2024-05-27T06:51:21.708 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-27T06:51:21.708 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} --name vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi --owner scheduled_vshankar@teuthology --verbose --archive /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636 -- /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/orig.config.yaml 2024-05-27T06:51:21.767 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T06:51:21.860 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636 branch: wip-vshankar-testing-20240521.063558-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} email: vshankar@redhat.com first_in_suite: false job_id: '7727636' kernel: flavor: default kdb: 1 sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240521.063558-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240521.063558-debug sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 owner: scheduled_vshankar@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2499 sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 sleep_before_teardown: 0 subset: 573/1024 suite: fs suite_branch: wip-vshankar-testing-20240521.063558-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL5tgIT7iRwUbK5jRLR5sCvZWnP5x8lLrYBgIEuSUeudVDX0sE0QWJmAs/9U19NmNdXDMqBn81eNl/J8hMe1p4c= smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNTGoIhgZE9x694KKZKAPAwsLlyA3k4xBxB8+AaRe5STwR29oeP33iMqmu6HX8S/CH3uRsgH/UriDw+1HEk/pA= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiWERPb/dnNOVjWO2WV1vsLBQbvV2AIoH/TZjDGB0vRX3s6z2fU3AMFkknTYiYBoP1ULxcRAVzV+ekjJK1YEyQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-27_05:52:49 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-27T06:51:21.860 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa; will attempt to use it 2024-05-27T06:51:21.861 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks 2024-05-27T06:51:21.861 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-27T06:51:21.863 INFO:teuthology.task.internal:Checking packages... 2024-05-27T06:51:21.887 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '38f9d900c83c63799fbdbe61acc9a11b0d3554a6' 2024-05-27T06:51:21.887 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-27T06:51:21.888 INFO:teuthology.packaging:ref: None 2024-05-27T06:51:21.888 INFO:teuthology.packaging:tag: None 2024-05-27T06:51:21.888 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240521.063558-debug 2024-05-27T06:51:21.888 INFO:teuthology.packaging:sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:51:21.888 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240521.063558-debug 2024-05-27T06:51:22.048 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3872.g38f9d900 2024-05-27T06:51:22.051 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-27T06:51:22.058 INFO:teuthology.task.internal:no buildpackages task found 2024-05-27T06:51:22.058 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-27T06:51:22.095 INFO:teuthology.task.internal:Saving configuration 2024-05-27T06:51:22.118 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-27T06:51:22.125 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-27T06:51:22.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:46:04.395691', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL5tgIT7iRwUbK5jRLR5sCvZWnP5x8lLrYBgIEuSUeudVDX0sE0QWJmAs/9U19NmNdXDMqBn81eNl/J8hMe1p4c='} 2024-05-27T06:51:22.172 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:46:04.396762', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNTGoIhgZE9x694KKZKAPAwsLlyA3k4xBxB8+AaRe5STwR29oeP33iMqmu6HX8S/CH3uRsgH/UriDw+1HEk/pA='} 2024-05-27T06:51:22.193 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:46:04.397819', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiWERPb/dnNOVjWO2WV1vsLBQbvV2AIoH/TZjDGB0vRX3s6z2fU3AMFkknTYiYBoP1ULxcRAVzV+ekjJK1YEyQ='} 2024-05-27T06:51:22.193 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-27T06:51:22.199 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-27T06:51:22.199 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-27T06:51:22.199 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-27T06:51:22.199 INFO:teuthology.run_tasks:Running task console_log... 2024-05-27T06:51:22.306 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2f25e17ac0>, signals=[15]) 2024-05-27T06:51:22.306 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-27T06:51:22.312 INFO:teuthology.task.internal:Opening connections... 2024-05-27T06:51:22.312 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T06:51:22.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:51:22.392 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-05-27T06:51:22.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:51:22.469 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-05-27T06:51:22.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:51:22.547 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-27T06:51:22.554 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-05-27T06:51:22.573 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-05-27T06:51:22.573 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-05-27T06:51:22.631 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-05-27T06:51:22.631 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-05-27T06:51:22.631 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-05-27T06:51:22.631 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-05-27T06:51:22.631 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T06:51:22.632 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T06:51:22.633 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-05-27T06:51:22.651 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-05-27T06:51:22.672 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-05-27T06:51:22.672 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-05-27T06:51:22.730 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-05-27T06:51:22.730 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-05-27T06:51:22.730 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-05-27T06:51:22.730 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-05-27T06:51:22.730 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-05-27T06:51:22.730 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-05-27T06:51:22.730 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T06:51:22.731 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-05-27T06:51:22.731 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-05-27T06:51:22.731 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T06:51:22.731 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-05-27T06:51:22.731 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T06:51:22.731 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T06:51:22.731 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T06:51:22.731 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-05-27T06:51:22.751 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-05-27T06:51:22.768 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-05-27T06:51:22.769 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-05-27T06:51:22.826 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-05-27T06:51:22.826 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-05-27T06:51:22.826 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-05-27T06:51:22.826 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T06:51:22.827 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T06:51:22.827 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-05-27T06:51:22.851 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-27T06:51:22.860 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-27T06:51:22.909 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-27T06:51:22.909 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:51:22.912 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:51:22.914 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:51:22.931 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-27T06:51:22.936 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-27T06:51:22.936 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:51:22.970 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:51:22.973 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:51:23.071 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-27T06:51:23.148 INFO:teuthology.run_tasks:Running task kernel... 2024-05-27T06:51:23.163 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'} 2024-05-27T06:51:23.163 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'mon.b': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'mon.c': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}}, timeout 300 2024-05-27T06:51:23.163 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:51:23.164 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:51:23.165 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:51:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:51:23.181 DEBUG:teuthology.task.kernel:sha1 for mon.c is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.182 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:51:23.182 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-05-27T06:51:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:51:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:51:23.183 DEBUG:teuthology.task.kernel:sha1 for mon.a is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.184 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:51:23.184 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-05-27T06:51:23.184 DEBUG:teuthology.task.kernel:sha1 for mon.b is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.184 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:51:23.184 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-05-27T06:51:23.239 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-437.el9.x86_64 2024-05-27T06:51:23.239 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.239 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:51:23.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.244 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-437.el9.x86_64 2024-05-27T06:51:23.244 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-437.el9.x86_64 2024-05-27T06:51:23.244 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.244 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:51:23.244 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.245 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.245 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:51:23.246 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:23.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum upgrade -y linux-firmware 2024-05-27T06:51:23.404 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum upgrade -y linux-firmware 2024-05-27T06:51:23.467 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum upgrade -y linux-firmware 2024-05-27T06:51:25.472 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.1 MB 00:00 2024-05-27T06:51:25.638 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-05-27T06:51:25.853 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.1 MB 00:00 2024-05-27T06:51:27.464 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-05-27T06:51:27.912 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-05-27T06:51:28.640 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 19 MB 00:02 2024-05-27T06:51:31.664 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-05-27T06:51:32.781 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-05-27T06:51:33.090 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 17 kB 00:00 2024-05-27T06:51:34.013 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.2 MB 00:02 2024-05-27T06:51:34.199 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-05-27T06:51:35.273 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 17 kB 00:01 2024-05-27T06:51:35.889 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 17 kB 00:00 2024-05-27T06:51:36.177 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-05-27T06:51:37.609 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-27T06:51:39.616 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-27T06:51:41.199 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-27T06:51:41.730 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-05-27T06:51:41.731 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-05-27T06:51:41.732 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-05-27T06:51:41.851 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-27T06:51:41.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:42.032 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-27T06:51:42.033 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-27T06:51:42.037 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi012.front.sepia.ceph.com to complete... 2024-05-27T06:51:42.990 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-27T06:51:43.182 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:01 ago on Mon 27 May 2024 06:51:42 AM UTC. 2024-05-27T06:51:43.225 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-05-27T06:51:43.226 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-05-27T06:51:43.226 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-05-27T06:51:43.360 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.c... 2024-05-27T06:51:43.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:43.593 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-27T06:51:43.593 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-27T06:51:43.595 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi162.front.sepia.ceph.com to complete... 2024-05-27T06:51:45.090 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-05-27T06:51:45.091 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-05-27T06:51:45.091 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-05-27T06:51:45.201 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-27T06:51:45.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:51:45.346 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-27T06:51:45.346 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-27T06:51:45.349 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi062.front.sepia.ceph.com to complete... 2024-05-27T06:52:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27 06:52:05 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-27T06:52:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:FINISHED --2024-05-27 06:52:05-- 2024-05-27T06:52:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:Total wall clock time: 23s 2024-05-27T06:52:05.543 INFO:teuthology.orchestra.run.smithi012.stderr:Downloaded: 1 files, 340M in 23s (14.7 MB/s) 2024-05-27T06:52:05.546 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-27T06:52:05.546 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-27T06:52:07.742 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying... ######################################## 2024-05-27T06:52:07.897 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing... ######################################## 2024-05-27T06:52:08.015 INFO:teuthology.orchestra.run.smithi012.stdout:Updating / installing... 2024-05-27T06:52:14.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27 06:52:14 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-27T06:52:14.195 INFO:teuthology.orchestra.run.smithi162.stderr:FINISHED --2024-05-27 06:52:14-- 2024-05-27T06:52:14.195 INFO:teuthology.orchestra.run.smithi162.stderr:Total wall clock time: 31s 2024-05-27T06:52:14.195 INFO:teuthology.orchestra.run.smithi162.stderr:Downloaded: 1 files, 340M in 30s (11.3 MB/s) 2024-05-27T06:52:14.198 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.c... 2024-05-27T06:52:14.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-27T06:52:16.289 INFO:teuthology.orchestra.run.smithi162.stdout:Verifying... ######################################## 2024-05-27T06:52:16.321 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-27T06:52:17.177 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing... ######################################## 2024-05-27T06:52:17.511 INFO:teuthology.orchestra.run.smithi162.stdout:Updating / installing... 2024-05-27T06:52:17.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27 06:52:17 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-27T06:52:17.690 INFO:teuthology.orchestra.run.smithi062.stderr:FINISHED --2024-05-27 06:52:17-- 2024-05-27T06:52:17.691 INFO:teuthology.orchestra.run.smithi062.stderr:Total wall clock time: 32s 2024-05-27T06:52:17.691 INFO:teuthology.orchestra.run.smithi062.stderr:Downloaded: 1 files, 340M in 32s (10.7 MB/s) 2024-05-27T06:52:17.693 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-27T06:52:17.693 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-27T06:52:19.867 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying... ######################################## 2024-05-27T06:52:20.030 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing... ######################################## 2024-05-27T06:52:21.348 INFO:teuthology.orchestra.run.smithi062.stdout:Updating / installing... 2024-05-27T06:52:25.460 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-27T06:52:29.634 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-27T06:52:54.189 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-27T06:52:54.190 DEBUG:teuthology.orchestra.run.smithi012:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-27T06:52:54.272 INFO:teuthology.orchestra.run.smithi012.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:52:54.272 INFO:teuthology.orchestra.run.smithi012.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-27T06:52:54.272 INFO:teuthology.orchestra.run.smithi012.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:52:54.273 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:52:54.274 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:52:54.275 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:52:54.276 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:52:54.277 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:52:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:52:54.279 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:52:54.280 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:52:54.281 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:52:54.282 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:52:54.283 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:52:54.284 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:52:54.285 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-27T06:52:54.286 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:52:54.287 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:52:54.288 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:52:54.289 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-27T06:52:54.290 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:52:54.291 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:52:54.292 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:52:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:52:54.294 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:52:54.295 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:52:54.296 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:52:54.297 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:52:54.298 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:52:54.299 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:52:54.300 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:52:54.301 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:52:54.302 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:52:54.303 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:52:54.304 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:52:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:52:54.306 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-27T06:52:54.307 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:52:54.308 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-27T06:52:54.309 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:52:54.310 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-27T06:52:54.311 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:52:54.312 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-27T06:52:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:52:54.314 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-27T06:52:54.315 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:52:54.316 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-27T06:52:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:52:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:52:54.319 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-27T06:52:54.320 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:52:54.321 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-27T06:52:54.322 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-27T06:52:54.323 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:52:54.324 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-27T06:52:54.325 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:52:54.326 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:52:54.327 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:52:54.328 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:52:54.329 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-27T06:52:54.330 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-27T06:52:54.331 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:52:54.332 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:52:54.333 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:52:54.334 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:52:54.335 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:52:54.336 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:52:54.337 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:52:54.338 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:52:54.339 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:52:54.340 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-27T06:52:54.341 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:52:54.342 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:52:54.343 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:52:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:52:54.345 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:52:54.346 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:52:54.347 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:52:54.348 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-27T06:52:54.349 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:52:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:52:54.351 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:52:54.352 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:52:54.353 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:52:54.354 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-27T06:52:54.355 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-27T06:52:54.356 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:52:54.357 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-27T06:52:54.358 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:52:54.359 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:52:54.360 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:52:54.361 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:52:54.362 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:52:54.363 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:52:54.364 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:52:54.365 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:52:54.366 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-27T06:52:54.367 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-27T06:52:54.368 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:52:54.369 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:52:54.370 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:52:54.371 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-27T06:52:54.372 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-27T06:52:54.373 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:52:54.374 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:52:54.375 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:52:54.376 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:52:54.377 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:52:54.378 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:52:54.379 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:52:54.380 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:52:54.381 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:52:54.382 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-27T06:52:54.383 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-27T06:52:54.384 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-27T06:52:54.385 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:52:54.386 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:52:54.387 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:52:54.388 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:52:54.389 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-27T06:52:54.390 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-27T06:52:54.391 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:52:54.392 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:52:54.393 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:52:54.394 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:52:54.395 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:52:54.396 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:52:54.397 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:52:54.398 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:52:54.399 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:52:54.400 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:52:54.401 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:52:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:52:54.403 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:52:54.404 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:52:54.405 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:52:54.406 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:52:54.407 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:52:54.408 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:52:54.409 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-27T06:52:54.410 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:52:54.411 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:52:54.412 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:52:54.413 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-27T06:52:54.414 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:52:54.415 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:52:54.416 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:52:54.417 INFO:teuthology.orchestra.run.smithi012.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:52:54.421 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:52:54.421 DEBUG:teuthology.orchestra.run.smithi012:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi012.stdout:postinstall scriptlet (using /bin/sh): 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi012.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi012.stdout:for file in vmlinuz System.map config; do 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi012.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi012.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout: fi 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout:done 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-05-27T06:52:54.446 INFO:teuthology.orchestra.run.smithi012.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-27T06:52:54.447 INFO:teuthology.orchestra.run.smithi012.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:52:54.447 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-27T06:52:54.447 INFO:teuthology.orchestra.run.smithi012.stdout:fi 2024-05-27T06:52:54.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2024-05-27T06:52:54.534 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T06:52:54.535 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T06:52:54.536 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T06:52:54.536 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-05-27T06:52:54.536 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T06:52:54.536 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T06:52:54.537 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T06:52:54.537 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 6.9.0-gb9459a26b088 2024-05-27T06:52:54.537 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T06:52:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-05-27T06:52:56.478 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T06:52:56.511 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-05-27T06:52:56.514 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-05-27T06:52:56.542 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.5A8I04lKsN 2024-05-27T06:52:56.543 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5A8I04lKsN 2024-05-27T06:52:56.608 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.5A8I04lKsN 2024-05-27T06:52:56.752 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.5A8I04lKsN is 6KB 2024-05-27T06:52:56.801 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.5A8I04lKsN 2024-05-27T06:52:56.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T06:52:56.878 INFO:teuthology.orchestra.run.smithi012.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T06:52:56.878 INFO:teuthology.orchestra.run.smithi012.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T06:52:56.878 INFO:teuthology.orchestra.run.smithi012.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T06:52:56.878 INFO:teuthology.orchestra.run.smithi012.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-27T06:52:56.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-27T06:52:57.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-05-27T06:53:02.944 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-27T06:53:02.944 DEBUG:teuthology.orchestra.run.smithi162:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-27T06:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:53:03.030 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:53:03.031 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:53:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:53:03.033 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:53:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:53:03.035 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:53:03.036 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:53:03.037 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:53:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:53:03.039 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:53:03.040 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-27T06:53:03.041 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:53:03.042 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:53:03.043 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:53:03.044 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-27T06:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-27T06:53:03.046 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:53:03.047 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-27T06:53:03.048 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:53:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:53:03.050 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:53:03.051 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:53:03.052 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-27T06:53:03.053 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:53:03.054 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:53:03.055 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:53:03.056 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:53:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:53:03.058 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:53:03.059 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-27T06:53:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:53:03.061 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:53:03.062 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:53:03.063 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:53:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:53:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:53:03.066 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-27T06:53:03.067 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:53:03.068 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:53:03.070 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-27T06:53:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:53:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:53:03.073 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:53:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:53:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-27T06:53:03.076 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-27T06:53:03.077 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:53:03.078 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:53:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:53:03.080 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:53:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-27T06:53:03.082 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:53:03.083 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-27T06:53:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-27T06:53:03.085 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-27T06:53:03.086 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:53:03.088 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:53:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:53:03.090 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:53:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-27T06:53:03.092 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:53:03.093 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:53:03.094 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:53:03.095 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-27T06:53:03.096 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-27T06:53:03.097 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:53:03.098 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:53:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:53:03.100 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:53:03.101 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:53:03.102 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:53:03.103 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-27T06:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:53:03.105 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-27T06:53:03.106 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:53:03.107 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-27T06:53:03.108 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-27T06:53:03.109 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:53:03.110 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:53:03.111 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:53:03.112 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:53:03.113 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-27T06:53:03.114 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:53:03.115 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:53:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:53:03.117 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:53:03.118 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:53:03.119 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:53:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:53:03.122 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:53:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-27T06:53:03.124 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:53:03.125 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:53:03.126 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:53:03.127 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:53:03.128 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:53:03.129 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:53:03.130 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:53:03.131 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-27T06:53:03.132 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-27T06:53:03.133 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:53:03.134 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:53:03.135 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:53:03.136 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-27T06:53:03.137 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:53:03.138 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-27T06:53:03.139 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-27T06:53:03.140 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-27T06:53:03.141 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:53:03.142 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:53:03.143 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:53:03.144 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:53:03.145 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:53:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:53:03.147 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:53:03.148 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:53:03.149 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:53:03.150 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:53:03.151 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-27T06:53:03.152 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:53:03.153 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:53:03.154 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-27T06:53:03.155 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:53:03.156 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:53:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:53:03.158 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:53:03.159 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:53:03.160 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:53:03.161 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:53:03.162 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:53:03.163 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:53:03.164 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:53:03.165 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:53:03.166 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:53:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:53:03.168 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:53:03.169 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:53:03.170 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:53:03.171 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:53:03.172 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-27T06:53:03.173 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:53:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:53:03.178 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:53:03.178 DEBUG:teuthology.orchestra.run.smithi162:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-27T06:53:03.201 INFO:teuthology.orchestra.run.smithi162.stdout:postinstall scriptlet (using /bin/sh): 2024-05-27T06:53:03.201 INFO:teuthology.orchestra.run.smithi162.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:53:03.201 INFO:teuthology.orchestra.run.smithi162.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:53:03.201 INFO:teuthology.orchestra.run.smithi162.stdout:fi 2024-05-27T06:53:03.201 INFO:teuthology.orchestra.run.smithi162.stdout:for file in vmlinuz System.map config; do 2024-05-27T06:53:03.201 INFO:teuthology.orchestra.run.smithi162.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout: fi 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout:done 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout:fi 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout:fi 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-27T06:53:03.202 INFO:teuthology.orchestra.run.smithi162.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:53:03.203 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-27T06:53:03.203 INFO:teuthology.orchestra.run.smithi162.stdout:fi 2024-05-27T06:53:03.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-05-27T06:53:03.276 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-05-27T06:53:03.276 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-05-27T06:53:03.276 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-05-27T06:53:03.277 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T06:53:03.278 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T06:53:03.279 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T06:53:03.279 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 6.9.0-gb9459a26b088 2024-05-27T06:53:03.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T06:53:03.851 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-05-27T06:53:05.243 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T06:53:05.271 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-05-27T06:53:05.273 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-05-27T06:53:05.294 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.9VvndR43FF 2024-05-27T06:53:05.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9VvndR43FF 2024-05-27T06:53:05.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.9VvndR43FF 2024-05-27T06:53:05.498 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.9VvndR43FF is 6KB 2024-05-27T06:53:05.546 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.9VvndR43FF 2024-05-27T06:53:05.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T06:53:05.622 INFO:teuthology.orchestra.run.smithi162.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T06:53:05.622 INFO:teuthology.orchestra.run.smithi162.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T06:53:05.622 INFO:teuthology.orchestra.run.smithi162.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T06:53:05.622 INFO:teuthology.orchestra.run.smithi162.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-27T06:53:05.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-27T06:53:05.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-05-27T06:53:07.446 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-27T06:53:07.446 DEBUG:teuthology.orchestra.run.smithi062:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-27T06:53:07.538 INFO:teuthology.orchestra.run.smithi062.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:53:07.538 INFO:teuthology.orchestra.run.smithi062.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-27T06:53:07.538 INFO:teuthology.orchestra.run.smithi062.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:53:07.538 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-27T06:53:07.538 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-27T06:53:07.538 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-27T06:53:07.539 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:53:07.540 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:53:07.541 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:53:07.542 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:53:07.543 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:53:07.544 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:53:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:53:07.546 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:53:07.547 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:53:07.548 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:53:07.549 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:53:07.550 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:53:07.551 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-27T06:53:07.552 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-27T06:53:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-27T06:53:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-27T06:53:07.557 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:53:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:53:07.559 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-27T06:53:07.560 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:53:07.561 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:53:07.562 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:53:07.563 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:53:07.564 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:53:07.565 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:53:07.566 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:53:07.567 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:53:07.568 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:53:07.570 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:53:07.571 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:53:07.572 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:53:07.573 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:53:07.574 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:53:07.575 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:53:07.576 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:53:07.577 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:53:07.578 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:53:07.579 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:53:07.580 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:53:07.581 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:53:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-27T06:53:07.583 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:53:07.584 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:53:07.585 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-27T06:53:07.586 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:53:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-27T06:53:07.588 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:53:07.589 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:53:07.590 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:53:07.591 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-27T06:53:07.592 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:53:07.593 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-27T06:53:07.594 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-27T06:53:07.595 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:53:07.596 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-27T06:53:07.597 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:53:07.598 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:53:07.599 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-27T06:53:07.600 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:53:07.601 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-27T06:53:07.602 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:53:07.603 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-27T06:53:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-27T06:53:07.605 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:53:07.606 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-27T06:53:07.607 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-27T06:53:07.608 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:53:07.609 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-27T06:53:07.610 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:53:07.611 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-27T06:53:07.612 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:53:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:53:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:53:07.615 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:53:07.616 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:53:07.617 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:53:07.618 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-27T06:53:07.619 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:53:07.620 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:53:07.621 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:53:07.622 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:53:07.623 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:53:07.624 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:53:07.625 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:53:07.626 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:53:07.627 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:53:07.628 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-27T06:53:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:53:07.630 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-27T06:53:07.631 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:53:07.632 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:53:07.633 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:53:07.634 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:53:07.635 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:53:07.636 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:53:07.637 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:53:07.638 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:53:07.639 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:53:07.640 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:53:07.641 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:53:07.642 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:53:07.643 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:53:07.644 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:53:07.645 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-27T06:53:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:53:07.647 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:53:07.648 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-27T06:53:07.649 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:53:07.650 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:53:07.651 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:53:07.652 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:53:07.653 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:53:07.654 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-27T06:53:07.655 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-27T06:53:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:53:07.657 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-27T06:53:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:53:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:53:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:53:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:53:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:53:07.663 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-27T06:53:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:53:07.665 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:53:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-27T06:53:07.667 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:53:07.668 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:53:07.669 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:53:07.670 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-27T06:53:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:53:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:53:07.673 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-27T06:53:07.674 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-27T06:53:07.675 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:53:07.676 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:53:07.677 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:53:07.678 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:53:07.679 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:53:07.680 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:53:07.681 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:53:07.682 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:53:07.683 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:53:07.684 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:53:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:53:07.686 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:53:07.687 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:53:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:53:07.689 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:53:07.690 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-27T06:53:07.691 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:53:07.692 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:53:07.693 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:53:07.694 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:53:07.695 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:53:07.696 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:53:07.697 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:53:07.698 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:53:07.699 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:53:07.700 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:53:07.701 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:53:07.702 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-27T06:53:07.703 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:53:07.704 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:53:07.705 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:53:07.706 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:53:07.710 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:53:07.710 DEBUG:teuthology.orchestra.run.smithi062:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-27T06:53:07.733 INFO:teuthology.orchestra.run.smithi062.stdout:postinstall scriptlet (using /bin/sh): 2024-05-27T06:53:07.733 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:53:07.733 INFO:teuthology.orchestra.run.smithi062.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout:for file in vmlinuz System.map config; do 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout: fi 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout:done 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:53:07.734 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-05-27T06:53:07.735 INFO:teuthology.orchestra.run.smithi062.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-27T06:53:07.735 INFO:teuthology.orchestra.run.smithi062.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-27T06:53:07.735 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-05-27T06:53:07.735 INFO:teuthology.orchestra.run.smithi062.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-27T06:53:07.735 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:53:07.735 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-27T06:53:07.735 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-05-27T06:53:07.736 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.06 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 68.el9 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Group : Unspecified 2024-05-27T06:53:07.809 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 8264254 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : builder@centos.org 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T06:53:07.810 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T06:53:07.811 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T06:53:07.811 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-05-27T06:53:07.811 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T06:53:07.811 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T06:53:07.812 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T06:53:07.812 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 6.9.0-gb9459a26b088 2024-05-27T06:53:07.812 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T06:53:08.382 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-05-27T06:53:09.773 INFO:teuthology.orchestra.run.smithi062.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T06:53:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-05-27T06:53:09.807 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-05-27T06:53:09.835 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.jIfM3OG0XC 2024-05-27T06:53:09.836 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jIfM3OG0XC 2024-05-27T06:53:09.904 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.jIfM3OG0XC 2024-05-27T06:53:10.047 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.jIfM3OG0XC is 6KB 2024-05-27T06:53:10.098 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.jIfM3OG0XC 2024-05-27T06:53:10.114 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T06:53:10.178 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T06:53:10.179 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T06:53:10.179 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T06:53:10.179 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-27T06:53:10.180 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-27T06:53:10.346 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-05-27T06:53:27.083 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-27T06:53:27.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-05-27T06:53:27.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:53:35.786 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-05-27T06:53:35.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T06:53:35.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:53:40.354 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-27T06:53:40.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-05-27T06:53:40.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:53:45.634 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-05-27T06:53:54.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-05-27T06:53:54.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-05-27T06:53:54.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:53:58.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-05-27T06:54:03.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T06:54:03.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:54:06.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-05-27T06:54:07.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-05-27T06:54:07.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:54:10.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-05-27T06:54:13.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-05-27T06:54:18.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T06:54:18.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:54:21.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-05-27T06:54:22.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-05-27T06:54:22.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:54:25.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-05-27T06:54:25.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:54:28.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-05-27T06:54:36.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T06:54:36.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:54:43.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-05-27T06:54:43.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:54:43.445 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-05-27T06:54:44.068 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-05-27T06:54:44.068 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.9.0-gb9459a26b088"... 2024-05-27T06:54:44.068 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-05-27T06:54:44.109 INFO:teuthology.orchestra.run.smithi012.stdout:6.9.0-gb9459a26b088 2024-05-27T06:54:44.109 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:54:44.110 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:54:44.110 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T06:54:45.110 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-27T06:54:45.110 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:54:45.229 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-05-27T06:54:45.260 DEBUG:teuthology.parallel:result is None 2024-05-27T06:54:54.987 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-05-27T06:54:55.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-05-27T06:54:55.614 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-05-27T06:54:55.614 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.9.0-gb9459a26b088"... 2024-05-27T06:54:55.614 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-05-27T06:54:55.654 INFO:teuthology.orchestra.run.smithi062.stdout:6.9.0-gb9459a26b088 2024-05-27T06:54:55.654 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:54:55.655 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:54:55.655 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T06:54:56.655 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-27T06:54:56.655 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:54:56.805 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-05-27T06:54:56.867 DEBUG:teuthology.parallel:result is None 2024-05-27T06:55:13.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T06:55:13.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:55:13.446 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-05-27T06:55:14.038 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T06:55:14.066 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "6.9.0-gb9459a26b088"... 2024-05-27T06:55:14.066 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-05-27T06:55:14.106 INFO:teuthology.orchestra.run.smithi162.stdout:6.9.0-gb9459a26b088 2024-05-27T06:55:14.106 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:55:14.107 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:55:14.107 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T06:55:15.107 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-05-27T06:55:15.108 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:55:15.234 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-05-27T06:55:15.270 DEBUG:teuthology.parallel:result is None 2024-05-27T06:55:15.270 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-27T06:55:15.277 INFO:teuthology.task.internal:Creating test directory... 2024-05-27T06:55:15.278 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:55:15.286 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:55:15.295 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:55:15.350 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-27T06:55:15.356 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-27T06:55:15.375 INFO:teuthology.task.internal:Creating archive directory... 2024-05-27T06:55:15.375 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:55:15.381 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:55:15.388 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:55:15.459 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-27T06:55:15.466 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-27T06:55:15.466 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:55:15.472 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:55:15.479 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:55:15.565 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:55:15.573 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:55:15.576 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:55:15.600 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:55:15.602 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:55:15.610 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:55:15.615 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-27T06:55:15.621 INFO:teuthology.task.internal:Configuring sudo... 2024-05-27T06:55:15.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:55:15.628 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:55:15.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:55:15.719 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-27T06:55:15.728 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-27T06:55:15.728 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:55:15.735 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:55:15.742 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:55:15.793 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:55:15.882 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:55:15.982 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:55:16.074 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:55:16.163 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T06:55:16.163 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:55:16.253 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:55:16.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:55:16.447 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:55:16.535 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:55:16.627 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T06:55:16.627 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:55:16.722 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:55:16.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:55:16.885 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:55:16.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:55:17.045 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T06:55:17.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:55:17.109 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-05-27T06:55:17.117 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-05-27T06:55:17.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-05-27T06:55:17.234 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T06:55:17.239 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T06:55:17.249 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T06:55:17.803 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-27T06:55:17.810 INFO:teuthology.task.internal:Starting timer... 2024-05-27T06:55:17.811 INFO:teuthology.run_tasks:Running task pcp... 2024-05-27T06:55:17.865 INFO:teuthology.run_tasks:Running task selinux... 2024-05-27T06:55:17.895 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-27T06:55:17.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-05-27T06:55:18.031 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-05-27T06:55:18.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-05-27T06:55:18.193 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-05-27T06:55:18.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-05-27T06:55:18.323 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-05-27T06:55:18.326 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-27T06:55:18.327 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-05-27T06:55:18.379 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-05-27T06:55:18.379 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-05-27T06:55:18.441 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-05-27T06:55:18.441 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-05-27T06:55:18.489 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-05-27T06:55:18.489 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-05-27T06:55:18.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T06:55:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:55:18.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T06:55:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:55:18.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T06:55:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:55:18.727 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-27T06:55:18.727 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-05-27T06:55:18.800 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-05-27T06:55:18.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-05-27T06:55:18.959 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-27T06:55:18.970 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-27T06:55:18.981 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-27T06:55:18.982 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi062.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-27T07:01:56.417 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-05-27T07:01:56.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-05-27T07:01:56.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:01:56.525 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-05-27T07:01:56.691 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-05-27T07:01:56.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-05-27T07:01:56.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:01:56.799 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-05-27T07:01:56.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-05-27T07:01:56.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T07:01:56.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:01:57.056 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-05-27T07:01:57.211 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-05-27T07:01:57.212 INFO:teuthology.run_tasks:Running task clock... 2024-05-27T07:01:57.223 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-27T07:01:57.223 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:01:57.223 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:01:57.231 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:01:57.231 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:01:57.238 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:01:57.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:01:57.323 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:01:57.332 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:01:57.333 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:01:57.367 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:01:57.374 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:01:57.379 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:01:57.447 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-05-27T07:01:57.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-05-27T07:01:57.457 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-05-27T07:01:57.481 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-05-27T07:01:57.482 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-05-27T07:01:57.491 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-05-27T07:01:57.522 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:01:57.526 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:01:57.537 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:01:57.565 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:01:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:01:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:01:57.706 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:01:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:01:57.708 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:01:57.714 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:01:57.714 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-05-27T07:01:57.714 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.715 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.715 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.715 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.717 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:01:57.717 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-05-27T07:01:57.717 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.717 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.717 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.717 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.718 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:01:57.718 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-05-27T07:01:57.718 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.718 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.718 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.718 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:01:57.720 INFO:teuthology.run_tasks:Running task install... 2024-05-27T07:01:57.729 DEBUG:teuthology.task.install:project ceph 2024-05-27T07:01:57.729 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}} 2024-05-27T07:01:57.729 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'} 2024-05-27T07:01:57.729 INFO:teuthology.task.install:Using flavor: default 2024-05-27T07:01:57.734 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-27T07:01:57.734 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-27T07:01:57.735 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:01:57.735 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:01:57.737 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:01:57.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:01:57.738 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:01:57.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:01:57.887 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:01:57.887 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:01:57.961 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:01:57.962 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:01:57.973 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:01:57.973 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:01:58.000 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:01:58.001 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:01:58.001 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:01:58.102 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:01:58.103 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:01:58.134 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:01:58.135 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:01:58.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:01:58.144 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:01:58.144 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:01:58.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:01:58.231 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:01:58.231 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:01:58.233 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:01:58.233 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:01:58.235 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:01:58.354 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:01:58.359 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:01:58.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:01:58.430 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2024-05-27T07:01:58.434 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2024-05-27T07:01:58.473 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:01:58.480 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:01:58.552 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-05-27T07:01:58.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-05-27T07:01:58.570 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-05-27T07:01:58.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-05-27T07:01:58.762 INFO:teuthology.orchestra.run.smithi062.stdout:45 files removed 2024-05-27T07:01:58.802 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:01:58.876 INFO:teuthology.orchestra.run.smithi162.stdout:45 files removed 2024-05-27T07:01:58.896 INFO:teuthology.orchestra.run.smithi012.stdout:45 files removed 2024-05-27T07:01:58.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:01:58.938 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:01:59.732 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 215 kB/s | 84 kB 00:00 2024-05-27T07:01:59.734 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 174 kB/s | 84 kB 00:00 2024-05-27T07:01:59.866 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 176 kB/s | 84 kB 00:00 2024-05-27T07:02:00.268 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 27 kB/s | 12 kB 00:00 2024-05-27T07:02:00.274 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 26 kB/s | 12 kB 00:00 2024-05-27T07:02:00.375 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-05-27T07:02:00.777 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-05-27T07:02:00.783 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-05-27T07:02:00.876 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-05-27T07:02:01.740 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.1 MB 00:00 2024-05-27T07:02:02.357 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.1 MB 00:01 2024-05-27T07:02:03.317 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.1 MB 00:02 2024-05-27T07:02:03.657 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-05-27T07:02:04.400 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-27T07:02:05.468 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-05-27T07:02:08.274 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.2 MB 00:00 2024-05-27T07:02:08.892 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-27T07:02:10.200 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 17 kB 00:00 2024-05-27T07:02:10.667 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 17 kB 00:00 2024-05-27T07:02:11.777 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.2 MB 00:02 2024-05-27T07:02:13.452 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 17 kB 00:00 2024-05-27T07:02:14.846 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-05-27T07:02:15.073 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 21 MB 00:04 2024-05-27T07:02:21.212 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-27T07:02:21.234 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-05-27T07:02:23.732 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:02:23.733 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:02:23.754 INFO:teuthology.orchestra.run.smithi062.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:02:23.756 INFO:teuthology.orchestra.run.smithi062.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:02:23.757 INFO:teuthology.orchestra.run.smithi062.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:02:23.758 INFO:teuthology.orchestra.run.smithi062.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:02:23.761 INFO:teuthology.orchestra.run.smithi062.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:02:23.764 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:02:23.765 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:02:23.766 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:02:23.767 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:02:23.785 INFO:teuthology.orchestra.run.smithi162.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:02:23.786 INFO:teuthology.orchestra.run.smithi162.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:02:23.787 INFO:teuthology.orchestra.run.smithi162.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:02:23.788 INFO:teuthology.orchestra.run.smithi162.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:02:23.791 INFO:teuthology.orchestra.run.smithi162.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:02:23.794 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:02:23.795 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:02:23.845 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-05-27T07:02:23.862 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-05-27T07:02:23.863 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-05-27T07:02:23.863 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-05-27T07:02:23.863 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-05-27T07:02:23.863 INFO:teuthology.orchestra.run.smithi062.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:02:23.863 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:02:23.863 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:02:23.864 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:02:23.865 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:02:23.865 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:02:23.865 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:02:23.865 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:02:23.865 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:02:23.865 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:02:23.866 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:02:23.866 INFO:teuthology.orchestra.run.smithi062.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:02:23.866 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:02:23.866 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:02:23.866 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:02:23.866 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:02:23.867 INFO:teuthology.orchestra.run.smithi062.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:02:23.867 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:02:23.867 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:02:23.867 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:02:23.867 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:02:23.868 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:02:23.869 INFO:teuthology.orchestra.run.smithi062.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:02:23.870 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:02:23.871 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:02:23.872 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:02:23.873 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:02:23.874 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:02:23.875 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:02:23.876 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:02:23.877 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout:Installing weak dependencies: 2024-05-27T07:02:23.878 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:02:23.879 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:02:23.879 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-05-27T07:02:23.879 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================= 2024-05-27T07:02:23.879 INFO:teuthology.orchestra.run.smithi062.stdout:Install 148 Packages 2024-05-27T07:02:23.879 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 11 Packages 2024-05-27T07:02:23.879 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:02:23.880 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-05-27T07:02:23.880 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 219 M 2024-05-27T07:02:23.880 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-05-27T07:02:23.884 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-05-27T07:02:23.884 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-05-27T07:02:23.884 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-05-27T07:02:23.884 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-05-27T07:02:23.884 INFO:teuthology.orchestra.run.smithi162.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:02:23.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:02:23.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:02:23.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:02:23.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:02:23.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:02:23.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:02:23.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:02:23.886 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:02:23.886 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:02:23.886 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:02:23.886 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:02:23.886 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:02:23.886 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:02:23.887 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:02:23.887 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:02:23.887 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:02:23.887 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:02:23.887 INFO:teuthology.orchestra.run.smithi162.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:02:23.887 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:02:23.888 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:02:23.888 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:02:23.888 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:02:23.888 INFO:teuthology.orchestra.run.smithi162.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:02:23.888 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:02:23.888 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:02:23.888 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:02:23.889 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:02:23.889 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:02:23.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:02:23.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:02:23.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:02:23.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:02:23.890 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:02:23.890 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:02:23.890 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:02:23.890 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:02:23.890 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-05-27T07:02:23.891 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:02:23.891 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:02:23.891 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:02:23.891 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:02:23.891 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:02:23.891 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:02:23.892 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:02:23.892 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:02:23.892 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:02:23.892 INFO:teuthology.orchestra.run.smithi162.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:02:23.892 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:02:23.893 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-05-27T07:02:23.893 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:02:23.893 INFO:teuthology.orchestra.run.smithi162.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:02:23.893 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:02:23.893 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:02:23.894 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:02:23.894 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:02:23.894 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:02:23.894 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:02:23.894 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:02:23.894 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:02:23.895 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:02:23.895 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:02:23.895 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:02:23.895 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:02:23.895 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:02:23.895 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:02:23.896 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:02:23.896 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:02:23.896 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:02:23.896 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:02:23.896 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:02:23.896 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:02:23.897 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:02:23.897 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:02:23.897 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:02:23.897 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:02:23.897 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:02:23.897 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:02:23.898 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:02:23.898 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:02:23.898 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:02:23.898 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:02:23.898 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:02:23.898 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:02:23.899 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:02:23.899 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:02:23.899 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:02:23.899 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:02:23.899 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:02:23.899 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:02:23.899 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:02:23.900 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:02:23.900 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:02:23.900 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:02:23.900 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:02:23.900 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:02:23.900 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:02:23.901 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:02:23.902 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:02:23.903 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:02:23.904 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:02:23.905 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout:Install 148 Packages 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 11 Packages 2024-05-27T07:02:23.906 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:02:23.907 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 219 M 2024-05-27T07:02:23.907 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-05-27T07:02:24.952 INFO:teuthology.orchestra.run.smithi162.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-05-27T07:02:25.272 INFO:teuthology.orchestra.run.smithi062.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-05-27T07:02:25.452 INFO:teuthology.orchestra.run.smithi162.stdout:(2/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 6.5 MB/s | 5.4 MB 00:00 2024-05-27T07:02:25.528 INFO:teuthology.orchestra.run.smithi162.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 1.9 MB/s | 147 kB 00:00 2024-05-27T07:02:25.886 INFO:teuthology.orchestra.run.smithi162.stdout:(4/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 6.3 MB/s | 2.3 MB 00:00 2024-05-27T07:02:26.119 INFO:teuthology.orchestra.run.smithi162.stdout:(5/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 6.5 MB/s | 1.5 MB 00:00 2024-05-27T07:02:26.156 INFO:teuthology.orchestra.run.smithi062.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 979 kB/s | 863 kB 00:00 2024-05-27T07:02:26.206 INFO:teuthology.orchestra.run.smithi062.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 2.9 MB/s | 147 kB 00:00 2024-05-27T07:02:26.369 INFO:teuthology.orchestra.run.smithi162.stdout:(6/159): ceph-common-19.0.0-3872.g38f9d900.el9. 11 MB/s | 20 MB 00:01 2024-05-27T07:02:26.405 INFO:teuthology.orchestra.run.smithi162.stdout:(7/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 596 kB/s | 863 kB 00:01 2024-05-27T07:02:26.448 INFO:teuthology.orchestra.run.smithi062.stdout:(4/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 9.4 MB/s | 2.3 MB 00:00 2024-05-27T07:02:26.890 INFO:teuthology.orchestra.run.smithi062.stdout:(5/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 2.7 MB/s | 5.4 MB 00:01 2024-05-27T07:02:26.998 INFO:teuthology.orchestra.run.smithi062.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 2.7 MB/s | 1.5 MB 00:00 2024-05-27T07:02:27.445 INFO:teuthology.orchestra.run.smithi162.stdout:(8/159): ceph-radosgw-19.0.0-3872.g38f9d900.el9 9.7 MB/s | 10 MB 00:01 2024-05-27T07:02:27.520 INFO:teuthology.orchestra.run.smithi162.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 339 kB/s | 25 kB 00:00 2024-05-27T07:02:27.741 INFO:teuthology.orchestra.run.smithi062.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 5.7 MB/s | 4.9 MB 00:00 2024-05-27T07:02:28.945 INFO:teuthology.orchestra.run.smithi162.stdout:(10/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 7.0 MB/s | 18 MB 00:02 2024-05-27T07:02:29.070 INFO:teuthology.orchestra.run.smithi162.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 1.8 MB/s | 224 kB 00:00 2024-05-27T07:02:29.145 INFO:teuthology.orchestra.run.smithi162.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 365 kB/s | 27 kB 00:00 2024-05-27T07:02:29.212 INFO:teuthology.orchestra.run.smithi162.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 482 kB/s | 32 kB 00:00 2024-05-27T07:02:29.337 INFO:teuthology.orchestra.run.smithi162.stdout:(14/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 6.0 MB/s | 755 kB 00:00 2024-05-27T07:02:29.366 INFO:teuthology.orchestra.run.smithi062.stdout:(8/159): ceph-radosgw-19.0.0-3872.g38f9d900.el9 6.2 MB/s | 10 MB 00:01 2024-05-27T07:02:29.396 INFO:teuthology.orchestra.run.smithi162.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 2.7 MB/s | 157 kB 00:00 2024-05-27T07:02:29.455 INFO:teuthology.orchestra.run.smithi162.stdout:(16/159): librados-devel-19.0.0-3872.g38f9d900. 2.2 MB/s | 127 kB 00:00 2024-05-27T07:02:29.457 INFO:teuthology.orchestra.run.smithi062.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 278 kB/s | 25 kB 00:00 2024-05-27T07:02:29.597 INFO:teuthology.orchestra.run.smithi162.stdout:(17/159): libradosstriper1-19.0.0-3872.g38f9d90 3.5 MB/s | 498 kB 00:00 2024-05-27T07:02:30.539 INFO:teuthology.orchestra.run.smithi162.stdout:(18/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 5.5 MB/s | 5.2 MB 00:00 2024-05-27T07:02:30.630 INFO:teuthology.orchestra.run.smithi162.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 499 kB/s | 45 kB 00:00 2024-05-27T07:02:30.714 INFO:teuthology.orchestra.run.smithi162.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.6 MB/s | 136 kB 00:00 2024-05-27T07:02:30.789 INFO:teuthology.orchestra.run.smithi162.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 2.1 MB/s | 161 kB 00:00 2024-05-27T07:02:30.865 INFO:teuthology.orchestra.run.smithi162.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 4.2 MB/s | 320 kB 00:00 2024-05-27T07:02:30.973 INFO:teuthology.orchestra.run.smithi162.stdout:(23/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x8 1.0 MB/s | 4.9 MB 00:04 2024-05-27T07:02:30.983 INFO:teuthology.orchestra.run.smithi062.stdout:(10/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 4.5 MB/s | 18 MB 00:03 2024-05-27T07:02:31.015 INFO:teuthology.orchestra.run.smithi162.stdout:(24/159): python3-rbd-19.0.0-3872.g38f9d900.el9 2.0 MB/s | 300 kB 00:00 2024-05-27T07:02:31.082 INFO:teuthology.orchestra.run.smithi162.stdout:(25/159): python3-rgw-19.0.0-3872.g38f9d900.el9 920 kB/s | 99 kB 00:00 2024-05-27T07:02:31.099 INFO:teuthology.orchestra.run.smithi062.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 1.9 MB/s | 224 kB 00:00 2024-05-27T07:02:31.116 INFO:teuthology.orchestra.run.smithi162.stdout:(26/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 865 kB/s | 86 kB 00:00 2024-05-27T07:02:31.175 INFO:teuthology.orchestra.run.smithi062.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 365 kB/s | 27 kB 00:00 2024-05-27T07:02:31.191 INFO:teuthology.orchestra.run.smithi162.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 2.2 MB/s | 169 kB 00:00 2024-05-27T07:02:31.242 INFO:teuthology.orchestra.run.smithi062.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 483 kB/s | 32 kB 00:00 2024-05-27T07:02:31.250 INFO:teuthology.orchestra.run.smithi162.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 593 kB/s | 34 kB 00:00 2024-05-27T07:02:31.333 INFO:teuthology.orchestra.run.smithi162.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 1.7 MB/s | 143 kB 00:00 2024-05-27T07:02:31.461 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-05-27T07:02:31.467 INFO:teuthology.orchestra.run.smithi062.stdout:(14/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 3.3 MB/s | 755 kB 00:00 2024-05-27T07:02:31.567 INFO:teuthology.orchestra.run.smithi062.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 1.5 MB/s | 157 kB 00:00 2024-05-27T07:02:31.659 INFO:teuthology.orchestra.run.smithi062.stdout:(16/159): librados-devel-19.0.0-3872.g38f9d900. 1.4 MB/s | 127 kB 00:00 2024-05-27T07:02:31.768 INFO:teuthology.orchestra.run.smithi062.stdout:(17/159): libradosstriper1-19.0.0-3872.g38f9d90 4.5 MB/s | 498 kB 00:00 2024-05-27T07:02:32.092 INFO:teuthology.orchestra.run.smithi162.stdout:(30/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 9.4 MB/s | 43 MB 00:04 2024-05-27T07:02:32.201 INFO:teuthology.orchestra.run.smithi162.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 4.1 MB/s | 3.5 MB 00:00 2024-05-27T07:02:32.326 INFO:teuthology.orchestra.run.smithi162.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 2.2 MB/s | 274 kB 00:00 2024-05-27T07:02:32.376 INFO:teuthology.orchestra.run.smithi162.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 1.0 MB/s | 49 kB 00:00 2024-05-27T07:02:32.393 INFO:teuthology.orchestra.run.smithi062.stdout:(18/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 8.3 MB/s | 5.2 MB 00:00 2024-05-27T07:02:32.418 INFO:teuthology.orchestra.run.smithi162.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 403 kB/s | 16 kB 00:00 2024-05-27T07:02:32.452 INFO:teuthology.orchestra.run.smithi062.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 787 kB/s | 45 kB 00:00 2024-05-27T07:02:32.510 INFO:teuthology.orchestra.run.smithi162.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 2.9 MB/s | 273 kB 00:00 2024-05-27T07:02:32.535 INFO:teuthology.orchestra.run.smithi062.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.6 MB/s | 136 kB 00:00 2024-05-27T07:02:32.611 INFO:teuthology.orchestra.run.smithi062.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 2.1 MB/s | 161 kB 00:00 2024-05-27T07:02:32.710 INFO:teuthology.orchestra.run.smithi162.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 3.7 MB/s | 764 kB 00:00 2024-05-27T07:02:32.777 INFO:teuthology.orchestra.run.smithi062.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 1.9 MB/s | 320 kB 00:00 2024-05-27T07:02:32.853 INFO:teuthology.orchestra.run.smithi062.stdout:(23/159): python3-rbd-19.0.0-3872.g38f9d900.el9 3.9 MB/s | 300 kB 00:00 2024-05-27T07:02:32.886 INFO:teuthology.orchestra.run.smithi162.stdout:(37/159): ceph-mgr-diskprediction-local-19.0.0- 9.3 MB/s | 7.4 MB 00:00 2024-05-27T07:02:32.902 INFO:teuthology.orchestra.run.smithi162.stdout:(38/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 95 kB/s | 18 kB 00:00 2024-05-27T07:02:32.920 INFO:teuthology.orchestra.run.smithi062.stdout:(24/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.5 MB/s | 99 kB 00:00 2024-05-27T07:02:32.970 INFO:teuthology.orchestra.run.smithi062.stdout:(25/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 1.7 MB/s | 86 kB 00:00 2024-05-27T07:02:33.203 INFO:teuthology.orchestra.run.smithi162.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 255 kB/s | 80 kB 00:00 2024-05-27T07:02:33.270 INFO:teuthology.orchestra.run.smithi062.stdout:(26/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 10 MB/s | 3.0 MB 00:00 2024-05-27T07:02:33.321 INFO:teuthology.orchestra.run.smithi062.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 3.4 MB/s | 169 kB 00:00 2024-05-27T07:02:33.336 INFO:teuthology.orchestra.run.smithi162.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-05-27T07:02:33.372 INFO:teuthology.orchestra.run.smithi062.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 696 kB/s | 34 kB 00:00 2024-05-27T07:02:33.421 INFO:teuthology.orchestra.run.smithi062.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 2.8 MB/s | 143 kB 00:00 2024-05-27T07:02:33.437 INFO:teuthology.orchestra.run.smithi162.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-05-27T07:02:33.512 INFO:teuthology.orchestra.run.smithi162.stdout:(42/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 1.3 MB/s | 3.0 MB 00:02 2024-05-27T07:02:33.687 INFO:teuthology.orchestra.run.smithi162.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 612 kB/s | 106 kB 00:00 2024-05-27T07:02:33.755 INFO:teuthology.orchestra.run.smithi062.stdout:(30/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 11 MB/s | 3.5 MB 00:00 2024-05-27T07:02:33.762 INFO:teuthology.orchestra.run.smithi162.stdout:(44/159): python3-cryptography-36.0.1-4.el9.x86 3.8 MB/s | 1.2 MB 00:00 2024-05-27T07:02:33.788 INFO:teuthology.orchestra.run.smithi162.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-05-27T07:02:33.813 INFO:teuthology.orchestra.run.smithi162.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 2.5 MB/s | 125 kB 00:00 2024-05-27T07:02:33.847 INFO:teuthology.orchestra.run.smithi162.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-05-27T07:02:34.031 INFO:teuthology.orchestra.run.smithi062.stdout:(31/159): ceph-common-19.0.0-3872.g38f9d900.el9 2.2 MB/s | 20 MB 00:09 2024-05-27T07:02:34.056 INFO:teuthology.orchestra.run.smithi162.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 689 kB 00:00 2024-05-27T07:02:34.106 INFO:teuthology.orchestra.run.smithi162.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 3.6 MB/s | 943 kB 00:00 2024-05-27T07:02:34.149 INFO:teuthology.orchestra.run.smithi162.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-05-27T07:02:34.164 INFO:teuthology.orchestra.run.smithi062.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 2.0 MB/s | 274 kB 00:00 2024-05-27T07:02:34.165 INFO:teuthology.orchestra.run.smithi162.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 373 kB/s | 22 kB 00:00 2024-05-27T07:02:34.198 INFO:teuthology.orchestra.run.smithi162.stdout:(52/159): flex-2.6.4-9.el9.x86_64.rpm 6.2 MB/s | 313 kB 00:00 2024-05-27T07:02:34.215 INFO:teuthology.orchestra.run.smithi162.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 670 kB/s | 33 kB 00:00 2024-05-27T07:02:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 600 kB/s | 49 kB 00:00 2024-05-27T07:02:34.266 INFO:teuthology.orchestra.run.smithi162.stdout:(54/159): libgfortran-11.4.1-3.el9.x86_64.rpm 589 kB/s | 803 kB 00:01 2024-05-27T07:02:34.283 INFO:teuthology.orchestra.run.smithi162.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 275 kB/s | 18 kB 00:00 2024-05-27T07:02:34.323 INFO:teuthology.orchestra.run.smithi062.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 222 kB/s | 16 kB 00:00 2024-05-27T07:02:34.341 INFO:teuthology.orchestra.run.smithi162.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 150 kB/s | 11 kB 00:00 2024-05-27T07:02:34.375 INFO:teuthology.orchestra.run.smithi162.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-05-27T07:02:34.392 INFO:teuthology.orchestra.run.smithi162.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 915 kB/s | 45 kB 00:00 2024-05-27T07:02:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 2.2 MB/s | 273 kB 00:00 2024-05-27T07:02:34.450 INFO:teuthology.orchestra.run.smithi162.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-05-27T07:02:34.534 INFO:teuthology.orchestra.run.smithi162.stdout:(60/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.9 MB/s | 3.0 MB 00:00 2024-05-27T07:02:34.551 INFO:teuthology.orchestra.run.smithi162.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 256 kB/s | 25 kB 00:00 2024-05-27T07:02:34.568 INFO:teuthology.orchestra.run.smithi162.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-05-27T07:02:34.618 INFO:teuthology.orchestra.run.smithi162.stdout:(63/159): libtool-2.4.6-45.el9.x86_64.rpm 2.5 MB/s | 582 kB 00:00 2024-05-27T07:02:34.652 INFO:teuthology.orchestra.run.smithi162.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-05-27T07:02:34.669 INFO:teuthology.orchestra.run.smithi162.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 713 kB/s | 35 kB 00:00 2024-05-27T07:02:34.673 INFO:teuthology.orchestra.run.smithi062.stdout:(36/159): ceph-mgr-diskprediction-local-19.0.0- 8.1 MB/s | 7.4 MB 00:00 2024-05-27T07:02:34.702 INFO:teuthology.orchestra.run.smithi162.stdout:(66/159): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-05-27T07:02:34.724 INFO:teuthology.orchestra.run.smithi062.stdout:(37/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 2.7 MB/s | 764 kB 00:00 2024-05-27T07:02:34.749 INFO:teuthology.orchestra.run.smithi062.stdout:(38/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 245 kB/s | 18 kB 00:00 2024-05-27T07:02:34.811 INFO:teuthology.orchestra.run.smithi162.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-05-27T07:02:34.858 INFO:teuthology.orchestra.run.smithi062.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 607 kB/s | 80 kB 00:00 2024-05-27T07:02:34.886 INFO:teuthology.orchestra.run.smithi162.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-05-27T07:02:34.941 INFO:teuthology.orchestra.run.smithi062.stdout:(40/159): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-05-27T07:02:34.975 INFO:teuthology.orchestra.run.smithi062.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-05-27T07:02:34.978 INFO:teuthology.orchestra.run.smithi162.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-27T07:02:35.008 INFO:teuthology.orchestra.run.smithi062.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.8 MB/s | 253 kB 00:00 2024-05-27T07:02:35.012 INFO:teuthology.orchestra.run.smithi162.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-05-27T07:02:35.112 INFO:teuthology.orchestra.run.smithi162.stdout:(71/159): openblas-openmp-0.3.21-2.el9.x86_64.r 10 MB/s | 4.7 MB 00:00 2024-05-27T07:02:35.154 INFO:teuthology.orchestra.run.smithi162.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-05-27T07:02:35.196 INFO:teuthology.orchestra.run.smithi162.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-05-27T07:02:35.246 INFO:teuthology.orchestra.run.smithi162.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-05-27T07:02:35.296 INFO:teuthology.orchestra.run.smithi162.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-05-27T07:02:35.347 INFO:teuthology.orchestra.run.smithi162.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 5.0 MB/s | 248 kB 00:00 2024-05-27T07:02:35.389 INFO:teuthology.orchestra.run.smithi162.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 624 kB/s | 26 kB 00:00 2024-05-27T07:02:35.439 INFO:teuthology.orchestra.run.smithi162.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 389 kB/s | 19 kB 00:00 2024-05-27T07:02:35.556 INFO:teuthology.orchestra.run.smithi162.stdout:(79/159): postgresql-server-13.14-1.el9.x86_64. 10 MB/s | 5.8 MB 00:00 2024-05-27T07:02:35.590 INFO:teuthology.orchestra.run.smithi162.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-05-27T07:02:35.607 INFO:teuthology.orchestra.run.smithi162.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 702 kB/s | 35 kB 00:00 2024-05-27T07:02:35.676 INFO:teuthology.orchestra.run.smithi062.stdout:(43/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 6.9 MB/s | 43 MB 00:06 2024-05-27T07:02:35.710 INFO:teuthology.orchestra.run.smithi062.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 152 kB/s | 106 kB 00:00 2024-05-27T07:02:35.716 INFO:teuthology.orchestra.run.smithi162.stdout:(82/159): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-05-27T07:02:35.749 INFO:teuthology.orchestra.run.smithi162.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-05-27T07:02:35.768 INFO:teuthology.orchestra.run.smithi062.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-05-27T07:02:35.783 INFO:teuthology.orchestra.run.smithi162.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-05-27T07:02:35.808 INFO:teuthology.orchestra.run.smithi162.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-27T07:02:35.809 INFO:teuthology.orchestra.run.smithi062.stdout:(46/159): python3-cryptography-36.0.1-4.el9.x86 1.5 MB/s | 1.2 MB 00:00 2024-05-27T07:02:35.833 INFO:teuthology.orchestra.run.smithi162.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 855 kB/s | 42 kB 00:00 2024-05-27T07:02:35.835 INFO:teuthology.orchestra.run.smithi062.stdout:(47/159): python3-pycparser-2.20-6.el9.noarch.r 855 kB/s | 135 kB 00:00 2024-05-27T07:02:35.867 INFO:teuthology.orchestra.run.smithi162.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-05-27T07:02:35.869 INFO:teuthology.orchestra.run.smithi062.stdout:(48/159): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-05-27T07:02:35.892 INFO:teuthology.orchestra.run.smithi162.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-27T07:02:35.926 INFO:teuthology.orchestra.run.smithi162.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-05-27T07:02:35.952 INFO:teuthology.orchestra.run.smithi162.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 883 kB/s | 51 kB 00:00 2024-05-27T07:02:35.976 INFO:teuthology.orchestra.run.smithi162.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-05-27T07:02:36.018 INFO:teuthology.orchestra.run.smithi162.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-05-27T07:02:36.060 INFO:teuthology.orchestra.run.smithi162.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-05-27T07:02:36.110 INFO:teuthology.orchestra.run.smithi162.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 221 kB 00:00 2024-05-27T07:02:36.211 INFO:teuthology.orchestra.run.smithi162.stdout:(95/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.2 MB/s | 5.1 MB 00:00 2024-05-27T07:02:36.236 INFO:teuthology.orchestra.run.smithi162.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 288 kB/s | 36 kB 00:00 2024-05-27T07:02:36.261 INFO:teuthology.orchestra.run.smithi162.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 960 kB/s | 47 kB 00:00 2024-05-27T07:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 305 kB 00:00 2024-05-27T07:02:36.320 INFO:teuthology.orchestra.run.smithi162.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-05-27T07:02:36.345 INFO:teuthology.orchestra.run.smithi162.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-05-27T07:02:36.371 INFO:teuthology.orchestra.run.smithi162.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-05-27T07:02:36.396 INFO:teuthology.orchestra.run.smithi162.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 919 kB/s | 45 kB 00:00 2024-05-27T07:02:36.680 INFO:teuthology.orchestra.run.smithi162.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-05-27T07:02:36.705 INFO:teuthology.orchestra.run.smithi162.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 67 kB/s | 22 kB 00:00 2024-05-27T07:02:36.736 INFO:teuthology.orchestra.run.smithi062.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 123 kB/s | 107 kB 00:00 2024-05-27T07:02:36.886 INFO:teuthology.orchestra.run.smithi062.stdout:(50/159): automake-1.16.2-8.el9.noarch.rpm 641 kB/s | 689 kB 00:01 2024-05-27T07:02:36.953 INFO:teuthology.orchestra.run.smithi062.stdout:(51/159): bison-3.7.4-5.el9.x86_64.rpm 844 kB/s | 943 kB 00:01 2024-05-27T07:02:37.355 INFO:teuthology.orchestra.run.smithi162.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 901 kB/s | 863 kB 00:00 2024-05-27T07:02:37.487 INFO:teuthology.orchestra.run.smithi062.stdout:(52/159): elfutils-libelf-devel-0.191-4.el9.x86 29 kB/s | 22 kB 00:00 2024-05-27T07:02:37.547 INFO:teuthology.orchestra.run.smithi162.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 367 kB/s | 309 kB 00:00 2024-05-27T07:02:37.605 INFO:teuthology.orchestra.run.smithi162.stdout:(107/159): abseil-cpp-20211102.0-3.el9.x86_64.r 593 kB/s | 548 kB 00:00 2024-05-27T07:02:37.662 INFO:teuthology.orchestra.run.smithi062.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 46 kB/s | 33 kB 00:00 2024-05-27T07:02:37.731 INFO:teuthology.orchestra.run.smithi162.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 213 kB/s | 26 kB 00:00 2024-05-27T07:02:37.823 INFO:teuthology.orchestra.run.smithi162.stdout:(109/159): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-05-27T07:02:37.857 INFO:teuthology.orchestra.run.smithi162.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-05-27T07:02:37.870 INFO:teuthology.orchestra.run.smithi062.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 319 kB/s | 313 kB 00:00 2024-05-27T07:02:37.881 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-05-27T07:02:37.998 INFO:teuthology.orchestra.run.smithi162.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-05-27T07:02:38.157 INFO:teuthology.orchestra.run.smithi162.stdout:(112/159): libarrow-9.0.0-11.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-05-27T07:02:38.174 INFO:teuthology.orchestra.run.smithi162.stdout:(113/159): libunwind-1.6.2-1.el9.x86_64.rpm 192 kB/s | 67 kB 00:00 2024-05-27T07:02:38.257 INFO:teuthology.orchestra.run.smithi162.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.2 MB/s | 839 kB 00:00 2024-05-27T07:02:38.316 INFO:teuthology.orchestra.run.smithi162.stdout:(115/159): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-05-27T07:02:38.333 INFO:teuthology.orchestra.run.smithi162.stdout:(116/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 275 kB/s | 43 kB 00:00 2024-05-27T07:02:38.367 INFO:teuthology.orchestra.run.smithi162.stdout:(117/159): python3-cachetools-4.2.4-1.el9.noarc 298 kB/s | 32 kB 00:00 2024-05-27T07:02:38.425 INFO:teuthology.orchestra.run.smithi162.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-05-27T07:02:38.470 INFO:teuthology.orchestra.run.smithi062.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 23 kB/s | 18 kB 00:00 2024-05-27T07:02:38.517 INFO:teuthology.orchestra.run.smithi162.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-05-27T07:02:38.521 INFO:teuthology.orchestra.run.smithi062.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 17 kB/s | 11 kB 00:00 2024-05-27T07:02:38.551 INFO:teuthology.orchestra.run.smithi162.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-05-27T07:02:38.609 INFO:teuthology.orchestra.run.smithi162.stdout:(121/159): python3-cheroot-8.6.0-4.el9.noarch.r 624 kB/s | 172 kB 00:00 2024-05-27T07:02:38.651 INFO:teuthology.orchestra.run.smithi162.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-05-27T07:02:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:(123/159): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-05-27T07:02:38.768 INFO:teuthology.orchestra.run.smithi162.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-05-27T07:02:38.802 INFO:teuthology.orchestra.run.smithi162.stdout:(125/159): python3-grpcio-tools-1.46.7-10.el9.x 752 kB/s | 144 kB 00:00 2024-05-27T07:02:38.861 INFO:teuthology.orchestra.run.smithi162.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-05-27T07:02:38.878 INFO:teuthology.orchestra.run.smithi162.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-05-27T07:02:38.920 INFO:teuthology.orchestra.run.smithi162.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 167 kB/s | 19 kB 00:00 2024-05-27T07:02:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-05-27T07:02:38.979 INFO:teuthology.orchestra.run.smithi062.stdout:(57/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.0 MB/s | 3.0 MB 00:01 2024-05-27T07:02:39.029 INFO:teuthology.orchestra.run.smithi162.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-05-27T07:02:39.045 INFO:teuthology.orchestra.run.smithi162.stdout:(131/159): python3-logutils-0.3.5-21.el9.noarch 370 kB/s | 46 kB 00:00 2024-05-27T07:02:39.087 INFO:teuthology.orchestra.run.smithi162.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 677 kB/s | 79 kB 00:00 2024-05-27T07:02:39.279 INFO:teuthology.orchestra.run.smithi162.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 231 kB/s | 58 kB 00:00 2024-05-27T07:02:39.297 INFO:teuthology.orchestra.run.smithi162.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 79 kB/s | 16 kB 00:00 2024-05-27T07:02:39.338 INFO:teuthology.orchestra.run.smithi162.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 932 kB/s | 272 kB 00:00 2024-05-27T07:02:39.339 INFO:teuthology.orchestra.run.smithi062.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 55 kB/s | 45 kB 00:00 2024-05-27T07:02:39.388 INFO:teuthology.orchestra.run.smithi162.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 833 kB/s | 90 kB 00:00 2024-05-27T07:02:39.397 INFO:teuthology.orchestra.run.smithi062.stdout:(59/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 173 kB/s | 160 kB 00:00 2024-05-27T07:02:39.405 INFO:teuthology.orchestra.run.smithi162.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 261 kB/s | 28 kB 00:00 2024-05-27T07:02:39.464 INFO:teuthology.orchestra.run.smithi162.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 246 kB/s | 31 kB 00:00 2024-05-27T07:02:39.514 INFO:teuthology.orchestra.run.smithi162.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-27T07:02:39.540 INFO:teuthology.orchestra.run.smithi162.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-05-27T07:02:39.590 INFO:teuthology.orchestra.run.smithi162.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2024-05-27T07:02:39.648 INFO:teuthology.orchestra.run.smithi162.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 383 kB/s | 51 kB 00:00 2024-05-27T07:02:39.682 INFO:teuthology.orchestra.run.smithi162.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-05-27T07:02:39.724 INFO:teuthology.orchestra.run.smithi162.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 673 kB/s | 90 kB 00:00 2024-05-27T07:02:39.791 INFO:teuthology.orchestra.run.smithi162.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-05-27T07:02:39.816 INFO:teuthology.orchestra.run.smithi162.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 149 kB/s | 20 kB 00:00 2024-05-27T07:02:39.892 INFO:teuthology.orchestra.run.smithi162.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-05-27T07:02:39.975 INFO:teuthology.orchestra.run.smithi162.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 8.7 MB/s | 1.6 MB 00:00 2024-05-27T07:02:40.164 INFO:teuthology.orchestra.run.smithi062.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 33 kB/s | 25 kB 00:00 2024-05-27T07:02:40.206 INFO:teuthology.orchestra.run.smithi062.stdout:(61/159): librdkafka-1.6.1-102.el9.x86_64.rpm 542 kB/s | 662 kB 00:01 2024-05-27T07:02:40.346 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:02:40.347 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:02:40.365 INFO:teuthology.orchestra.run.smithi012.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:02:40.366 INFO:teuthology.orchestra.run.smithi012.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:02:40.368 INFO:teuthology.orchestra.run.smithi012.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:02:40.369 INFO:teuthology.orchestra.run.smithi012.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:02:40.371 INFO:teuthology.orchestra.run.smithi012.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:02:40.374 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:02:40.375 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:02:40.450 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-05-27T07:02:40.468 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-05-27T07:02:40.468 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-05-27T07:02:40.468 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-05-27T07:02:40.468 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-05-27T07:02:40.468 INFO:teuthology.orchestra.run.smithi012.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:02:40.468 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:02:40.468 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:02:40.469 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:02:40.470 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:02:40.471 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-05-27T07:02:40.472 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:02:40.473 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:02:40.474 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:02:40.475 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:02:40.476 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:02:40.477 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:02:40.478 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:02:40.479 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:02:40.480 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-05-27T07:02:40.481 INFO:teuthology.orchestra.run.smithi012.stdout:Install 148 Packages 2024-05-27T07:02:40.482 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 11 Packages 2024-05-27T07:02:40.482 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:02:40.482 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 219 M 2024-05-27T07:02:40.482 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-05-27T07:02:40.581 INFO:teuthology.orchestra.run.smithi062.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 469 kB/s | 582 kB 00:01 2024-05-27T07:02:40.617 INFO:teuthology.orchestra.run.smithi162.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 4.0 MB/s | 2.3 MB 00:00 2024-05-27T07:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout:(150/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 3.6 MB/s | 3.1 MB 00:00 2024-05-27T07:02:40.892 INFO:teuthology.orchestra.run.smithi162.stdout:(151/159): NetworkManager-libnm-1.47.90-1.el9.x 6.6 MB/s | 1.8 MB 00:00 2024-05-27T07:02:40.909 INFO:teuthology.orchestra.run.smithi162.stdout:(152/159): NetworkManager-team-1.47.90-1.el9.x8 250 kB/s | 41 kB 00:00 2024-05-27T07:02:40.939 INFO:teuthology.orchestra.run.smithi062.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 61 kB/s | 47 kB 00:00 2024-05-27T07:02:40.951 INFO:teuthology.orchestra.run.smithi162.stdout:(153/159): NetworkManager-tui-1.47.90-1.el9.x86 4.2 MB/s | 248 kB 00:00 2024-05-27T07:02:41.001 INFO:teuthology.orchestra.run.smithi162.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 402 kB/s | 37 kB 00:00 2024-05-27T07:02:41.035 INFO:teuthology.orchestra.run.smithi162.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 2.4 MB/s | 207 kB 00:00 2024-05-27T07:02:41.119 INFO:teuthology.orchestra.run.smithi162.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.2 MB/s | 260 kB 00:00 2024-05-27T07:02:41.227 INFO:teuthology.orchestra.run.smithi162.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 7.1 MB/s | 1.3 MB 00:00 2024-05-27T07:02:41.298 INFO:teuthology.orchestra.run.smithi062.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 268 kB/s | 292 kB 00:01 2024-05-27T07:02:41.366 INFO:teuthology.orchestra.run.smithi012.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-05-27T07:02:41.369 INFO:teuthology.orchestra.run.smithi162.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.6 MB/s | 2.4 MB 00:00 2024-05-27T07:02:41.558 INFO:teuthology.orchestra.run.smithi012.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 4.4 MB/s | 863 kB 00:00 2024-05-27T07:02:41.617 INFO:teuthology.orchestra.run.smithi012.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 2.9 MB/s | 147 kB 00:00 2024-05-27T07:02:41.636 INFO:teuthology.orchestra.run.smithi162.stdout:(159/159): librados2-19.0.0-3872.g38f9d900.el9. 1.8 MB/s | 3.3 MB 00:01 2024-05-27T07:02:41.648 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:02:41.649 INFO:teuthology.orchestra.run.smithi062.stdout:(65/159): lua-5.4.4-4.el9.x86_64.rpm 177 kB/s | 188 kB 00:01 2024-05-27T07:02:41.649 INFO:teuthology.orchestra.run.smithi162.stdout:Total 12 MB/s | 219 MB 00:17 2024-05-27T07:02:41.674 INFO:teuthology.orchestra.run.smithi062.stdout:(66/159): openblas-0.3.21-2.el9.x86_64.rpm 48 kB/s | 35 kB 00:00 2024-05-27T07:02:41.833 INFO:teuthology.orchestra.run.smithi012.stdout:(4/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 10 MB/s | 2.3 MB 00:00 2024-05-27T07:02:42.042 INFO:teuthology.orchestra.run.smithi012.stdout:(5/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 7.3 MB/s | 1.5 MB 00:00 2024-05-27T07:02:42.192 INFO:teuthology.orchestra.run.smithi012.stdout:(6/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 4.9 MB/s | 5.4 MB 00:01 2024-05-27T07:02:42.501 INFO:teuthology.orchestra.run.smithi012.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 11 MB/s | 4.9 MB 00:00 2024-05-27T07:02:42.866 INFO:teuthology.orchestra.run.smithi062.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.0 MB/s | 4.7 MB 00:01 2024-05-27T07:02:43.041 INFO:teuthology.orchestra.run.smithi062.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 1.2 MB/s | 1.6 MB 00:01 2024-05-27T07:02:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-05-27T07:02:43.584 INFO:teuthology.orchestra.run.smithi012.stdout:(8/159): ceph-radosgw-19.0.0-3872.g38f9d900.el9 9.4 MB/s | 10 MB 00:01 2024-05-27T07:02:43.643 INFO:teuthology.orchestra.run.smithi012.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 441 kB/s | 25 kB 00:00 2024-05-27T07:02:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 725 kB/s | 911 kB 00:01 2024-05-27T07:02:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:(71/159): postgresql-private-libs-13.14-1.el9.x 119 kB/s | 133 kB 00:01 2024-05-27T07:02:44.168 INFO:teuthology.orchestra.run.smithi012.stdout:(10/159): ceph-common-19.0.0-3872.g38f9d900.el9 6.4 MB/s | 20 MB 00:03 2024-05-27T07:02:44.435 INFO:teuthology.orchestra.run.smithi012.stdout:(11/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 7.9 MB/s | 18 MB 00:02 2024-05-27T07:02:44.469 INFO:teuthology.orchestra.run.smithi012.stdout:(12/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 749 kB/s | 224 kB 00:00 2024-05-27T07:02:44.511 INFO:teuthology.orchestra.run.smithi012.stdout:(13/159): cephfs-shell-19.0.0-3872.g38f9d900.el 365 kB/s | 27 kB 00:00 2024-05-27T07:02:44.544 INFO:teuthology.orchestra.run.smithi012.stdout:(14/159): libcephfs-devel-19.0.0-3872.g38f9d900 426 kB/s | 32 kB 00:00 2024-05-27T07:02:44.628 INFO:teuthology.orchestra.run.smithi012.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 1.9 MB/s | 157 kB 00:00 2024-05-27T07:02:44.678 INFO:teuthology.orchestra.run.smithi012.stdout:(16/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 4.4 MB/s | 755 kB 00:00 2024-05-27T07:02:44.754 INFO:teuthology.orchestra.run.smithi012.stdout:(17/159): librados-devel-19.0.0-3872.g38f9d900. 1.0 MB/s | 127 kB 00:00 2024-05-27T07:02:44.795 INFO:teuthology.orchestra.run.smithi012.stdout:(18/159): libradosstriper1-19.0.0-3872.g38f9d90 4.2 MB/s | 498 kB 00:00 2024-05-27T07:02:44.800 INFO:teuthology.orchestra.run.smithi062.stdout:(72/159): postgresql-server-13.14-1.el9.x86_64. 3.7 MB/s | 5.8 MB 00:01 2024-05-27T07:02:44.898 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-05-27T07:02:44.904 INFO:teuthology.orchestra.run.smithi012.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 420 kB/s | 45 kB 00:00 2024-05-27T07:02:45.013 INFO:teuthology.orchestra.run.smithi012.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.2 MB/s | 136 kB 00:00 2024-05-27T07:02:45.050 INFO:teuthology.orchestra.run.smithi062.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 98 kB/s | 87 kB 00:00 2024-05-27T07:02:45.113 INFO:teuthology.orchestra.run.smithi012.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 1.6 MB/s | 161 kB 00:00 2024-05-27T07:02:45.152 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-05-27T07:02:45.152 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-05-27T07:02:45.213 INFO:teuthology.orchestra.run.smithi012.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 3.2 MB/s | 320 kB 00:00 2024-05-27T07:02:45.305 INFO:teuthology.orchestra.run.smithi012.stdout:(23/159): python3-rbd-19.0.0-3872.g38f9d900.el9 3.2 MB/s | 300 kB 00:00 2024-05-27T07:02:45.409 INFO:teuthology.orchestra.run.smithi062.stdout:(74/159): protobuf-3.14.0-13.el9.x86_64.rpm 806 kB/s | 1.0 MB 00:01 2024-05-27T07:02:45.431 INFO:teuthology.orchestra.run.smithi012.stdout:(24/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 7.6 MB/s | 5.2 MB 00:00 2024-05-27T07:02:45.464 INFO:teuthology.orchestra.run.smithi012.stdout:(25/159): python3-rgw-19.0.0-3872.g38f9d900.el9 631 kB/s | 99 kB 00:00 2024-05-27T07:02:45.523 INFO:teuthology.orchestra.run.smithi012.stdout:(26/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 949 kB/s | 86 kB 00:00 2024-05-27T07:02:45.623 INFO:teuthology.orchestra.run.smithi012.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 1.7 MB/s | 169 kB 00:00 2024-05-27T07:02:45.732 INFO:teuthology.orchestra.run.smithi012.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 318 kB/s | 34 kB 00:00 2024-05-27T07:02:45.893 INFO:teuthology.orchestra.run.smithi062.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 73 kB/s | 61 kB 00:00 2024-05-27T07:02:45.899 INFO:teuthology.orchestra.run.smithi012.stdout:(29/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 7.0 MB/s | 3.0 MB 00:00 2024-05-27T07:02:45.933 INFO:teuthology.orchestra.run.smithi012.stdout:(30/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 717 kB/s | 143 kB 00:00 2024-05-27T07:02:46.450 INFO:teuthology.orchestra.run.smithi012.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 6.5 MB/s | 3.5 MB 00:00 2024-05-27T07:02:46.550 INFO:teuthology.orchestra.run.smithi012.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 2.7 MB/s | 274 kB 00:00 2024-05-27T07:02:46.559 INFO:teuthology.orchestra.run.smithi062.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-05-27T07:02:46.635 INFO:teuthology.orchestra.run.smithi062.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 199 kB/s | 244 kB 00:01 2024-05-27T07:02:47.060 INFO:teuthology.orchestra.run.smithi062.stdout:(78/159): python3-jinja2-2.11.3-6.el9.noarch.rp 213 kB/s | 248 kB 00:01 2024-05-27T07:02:47.067 INFO:teuthology.orchestra.run.smithi012.stdout:(33/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 13 MB/s | 43 MB 00:03 2024-05-27T07:02:47.109 INFO:teuthology.orchestra.run.smithi012.stdout:(34/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 88 kB/s | 49 kB 00:00 2024-05-27T07:02:47.150 INFO:teuthology.orchestra.run.smithi012.stdout:(35/159): ceph-prometheus-alerts-19.0.0-3872.g3 200 kB/s | 16 kB 00:00 2024-05-27T07:02:47.210 INFO:teuthology.orchestra.run.smithi062.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 33 kB/s | 19 kB 00:00 2024-05-27T07:02:47.217 INFO:teuthology.orchestra.run.smithi012.stdout:(36/159): ceph-volume-19.0.0-3872.g38f9d900.el9 2.5 MB/s | 273 kB 00:00 2024-05-27T07:02:47.273 INFO:teuthology.orchestra.run.smithi062.stdout:(80/159): python3-jsonpatch-1.21-16.el9.noarch. 36 kB/s | 26 kB 00:00 2024-05-27T07:02:47.284 INFO:teuthology.orchestra.run.smithi012.stdout:(37/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 5.6 MB/s | 764 kB 00:00 2024-05-27T07:02:47.326 INFO:teuthology.orchestra.run.smithi012.stdout:(38/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 169 kB/s | 18 kB 00:00 2024-05-27T07:02:47.460 INFO:teuthology.orchestra.run.smithi012.stdout:(39/159): ceph-mgr-diskprediction-local-19.0.0- 4.8 MB/s | 7.4 MB 00:01 2024-05-27T07:02:47.569 INFO:teuthology.orchestra.run.smithi012.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 284 kB/s | 80 kB 00:00 2024-05-27T07:02:47.619 INFO:teuthology.orchestra.run.smithi012.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-05-27T07:02:47.661 INFO:teuthology.orchestra.run.smithi012.stdout:(42/159): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-05-27T07:02:47.694 INFO:teuthology.orchestra.run.smithi012.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-05-27T07:02:47.720 INFO:teuthology.orchestra.run.smithi012.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-27T07:02:47.747 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-05-27T07:02:47.747 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-05-27T07:02:47.753 INFO:teuthology.orchestra.run.smithi012.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-27T07:02:47.787 INFO:teuthology.orchestra.run.smithi012.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-27T07:02:47.821 INFO:teuthology.orchestra.run.smithi012.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-05-27T07:02:47.871 INFO:teuthology.orchestra.run.smithi012.stdout:(48/159): python3-cryptography-36.0.1-4.el9.x86 4.9 MB/s | 1.2 MB 00:00 2024-05-27T07:02:47.969 INFO:teuthology.orchestra.run.smithi062.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 46 kB/s | 35 kB 00:00 2024-05-27T07:02:48.130 INFO:teuthology.orchestra.run.smithi012.stdout:(49/159): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-05-27T07:02:48.155 INFO:teuthology.orchestra.run.smithi012.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 377 kB/s | 107 kB 00:00 2024-05-27T07:02:48.177 INFO:teuthology.orchestra.run.smithi062.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 154 kB/s | 172 kB 00:01 2024-05-27T07:02:48.214 INFO:teuthology.orchestra.run.smithi012.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 263 kB/s | 22 kB 00:00 2024-05-27T07:02:48.257 INFO:teuthology.orchestra.run.smithi012.stdout:(52/159): bison-3.7.4-5.el9.x86_64.rpm 2.1 MB/s | 943 kB 00:00 2024-05-27T07:02:48.306 INFO:teuthology.orchestra.run.smithi012.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 359 kB/s | 33 kB 00:00 2024-05-27T07:02:48.340 INFO:teuthology.orchestra.run.smithi012.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 1.7 MB/s | 313 kB 00:00 2024-05-27T07:02:48.390 INFO:teuthology.orchestra.run.smithi012.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 219 kB/s | 18 kB 00:00 2024-05-27T07:02:48.440 INFO:teuthology.orchestra.run.smithi012.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 112 kB/s | 11 kB 00:00 2024-05-27T07:02:48.474 INFO:teuthology.orchestra.run.smithi012.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-05-27T07:02:48.524 INFO:teuthology.orchestra.run.smithi012.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 547 kB/s | 45 kB 00:00 2024-05-27T07:02:48.575 INFO:teuthology.orchestra.run.smithi012.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-05-27T07:02:48.666 INFO:teuthology.orchestra.run.smithi012.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 280 kB/s | 25 kB 00:00 2024-05-27T07:02:48.708 INFO:teuthology.orchestra.run.smithi012.stdout:(61/159): libtool-2.4.6-45.el9.x86_64.rpm 3.1 MB/s | 582 kB 00:00 2024-05-27T07:02:48.759 INFO:teuthology.orchestra.run.smithi012.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 520 kB/s | 47 kB 00:00 2024-05-27T07:02:48.834 INFO:teuthology.orchestra.run.smithi012.stdout:(63/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.2 MB/s | 3.0 MB 00:00 2024-05-27T07:02:48.866 INFO:teuthology.orchestra.run.smithi062.stdout:(83/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.2 MB/s | 5.1 MB 00:01 2024-05-27T07:02:48.893 INFO:teuthology.orchestra.run.smithi012.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-05-27T07:02:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 354 kB/s | 35 kB 00:00 2024-05-27T07:02:48.960 INFO:teuthology.orchestra.run.smithi012.stdout:(66/159): lua-5.4.4-4.el9.x86_64.rpm 943 kB/s | 188 kB 00:00 2024-05-27T07:02:49.028 INFO:teuthology.orchestra.run.smithi062.stdout:(84/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 237 kB/s | 250 kB 00:01 2024-05-27T07:02:49.077 INFO:teuthology.orchestra.run.smithi012.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-05-27T07:02:49.152 INFO:teuthology.orchestra.run.smithi012.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-05-27T07:02:49.202 INFO:teuthology.orchestra.run.smithi012.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 2.7 MB/s | 133 kB 00:00 2024-05-27T07:02:49.211 INFO:teuthology.orchestra.run.smithi062.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 215 kB/s | 222 kB 00:01 2024-05-27T07:02:49.303 INFO:teuthology.orchestra.run.smithi012.stdout:(70/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-05-27T07:02:49.395 INFO:teuthology.orchestra.run.smithi012.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-05-27T07:02:49.445 INFO:teuthology.orchestra.run.smithi012.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-05-27T07:02:49.545 INFO:teuthology.orchestra.run.smithi012.stdout:(73/159): openblas-openmp-0.3.21-2.el9.x86_64.r 7.3 MB/s | 4.7 MB 00:00 2024-05-27T07:02:49.679 INFO:teuthology.orchestra.run.smithi062.stdout:(86/159): python3-packaging-20.9-5.el9.noarch.r 95 kB/s | 77 kB 00:00 2024-05-27T07:02:49.679 INFO:teuthology.orchestra.run.smithi012.stdout:(74/159): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-05-27T07:02:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 351 kB/s | 61 kB 00:00 2024-05-27T07:02:49.779 INFO:teuthology.orchestra.run.smithi012.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 4.2 MB/s | 244 kB 00:00 2024-05-27T07:02:49.834 INFO:teuthology.orchestra.run.smithi012.stdout:(77/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-05-27T07:02:49.880 INFO:teuthology.orchestra.run.smithi012.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 258 kB/s | 26 kB 00:00 2024-05-27T07:02:49.922 INFO:teuthology.orchestra.run.smithi012.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 222 kB/s | 19 kB 00:00 2024-05-27T07:02:49.929 INFO:teuthology.orchestra.run.smithi062.stdout:(87/159): python3-prettytable-0.7.2-27.el9.noar 47 kB/s | 42 kB 00:00 2024-05-27T07:02:49.955 INFO:teuthology.orchestra.run.smithi012.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.3 MB/s | 172 kB 00:00 2024-05-27T07:02:49.998 INFO:teuthology.orchestra.run.smithi012.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 467 kB/s | 35 kB 00:00 2024-05-27T07:02:50.065 INFO:teuthology.orchestra.run.smithi012.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-05-27T07:02:50.123 INFO:teuthology.orchestra.run.smithi012.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.8 MB/s | 222 kB 00:00 2024-05-27T07:02:50.212 INFO:teuthology.orchestra.run.smithi062.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 269 kB/s | 269 kB 00:00 2024-05-27T07:02:50.249 INFO:teuthology.orchestra.run.smithi012.stdout:(84/159): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 6.0 MB 00:00 2024-05-27T07:02:50.274 INFO:teuthology.orchestra.run.smithi012.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 516 kB/s | 77 kB 00:00 2024-05-27T07:02:50.316 INFO:teuthology.orchestra.run.smithi012.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 640 kB/s | 42 kB 00:00 2024-05-27T07:02:50.349 INFO:teuthology.orchestra.run.smithi012.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-05-27T07:02:50.383 INFO:teuthology.orchestra.run.smithi012.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-27T07:02:50.417 INFO:teuthology.orchestra.run.smithi012.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-05-27T07:02:50.467 INFO:teuthology.orchestra.run.smithi012.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 616 kB/s | 51 kB 00:00 2024-05-27T07:02:50.517 INFO:teuthology.orchestra.run.smithi012.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 539 kB/s | 54 kB 00:00 2024-05-27T07:02:50.563 INFO:teuthology.orchestra.run.smithi062.stdout:(89/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 180 kB/s | 159 kB 00:00 2024-05-27T07:02:50.643 INFO:teuthology.orchestra.run.smithi012.stdout:(92/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.4 MB/s | 5.1 MB 00:00 2024-05-27T07:02:50.693 INFO:teuthology.orchestra.run.smithi012.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 239 kB/s | 42 kB 00:00 2024-05-27T07:02:50.743 INFO:teuthology.orchestra.run.smithi012.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 445 kB/s | 44 kB 00:00 2024-05-27T07:02:50.802 INFO:teuthology.orchestra.run.smithi012.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-05-27T07:02:50.852 INFO:teuthology.orchestra.run.smithi012.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 333 kB/s | 36 kB 00:00 2024-05-27T07:02:50.903 INFO:teuthology.orchestra.run.smithi012.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 477 kB/s | 47 kB 00:00 2024-05-27T07:02:50.930 INFO:teuthology.orchestra.run.smithi062.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 71 kB/s | 51 kB 00:00 2024-05-27T07:02:50.936 INFO:teuthology.orchestra.run.smithi012.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-05-27T07:02:50.974 INFO:teuthology.orchestra.run.smithi062.stdout:(91/159): python3-pyasn1-modules-0.4.8-6.el9.no 288 kB/s | 279 kB 00:00 2024-05-27T07:02:50.986 INFO:teuthology.orchestra.run.smithi012.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 700 kB/s | 58 kB 00:00 2024-05-27T07:02:51.012 INFO:teuthology.orchestra.run.smithi012.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-05-27T07:02:51.062 INFO:teuthology.orchestra.run.smithi012.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 858 kB/s | 64 kB 00:00 2024-05-27T07:02:51.112 INFO:teuthology.orchestra.run.smithi012.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-05-27T07:02:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-05-27T07:02:51.221 INFO:teuthology.orchestra.run.smithi012.stdout:(104/159): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-05-27T07:02:51.322 INFO:teuthology.orchestra.run.smithi062.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 71 kB/s | 54 kB 00:00 2024-05-27T07:02:51.689 INFO:teuthology.orchestra.run.smithi062.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 58 kB/s | 42 kB 00:00 2024-05-27T07:02:51.713 INFO:teuthology.orchestra.run.smithi012.stdout:(105/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-05-27T07:02:51.888 INFO:teuthology.orchestra.run.smithi012.stdout:(106/159): abseil-cpp-20211102.0-3.el9.x86_64.r 757 kB/s | 548 kB 00:00 2024-05-27T07:02:51.922 INFO:teuthology.orchestra.run.smithi012.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 489 kB/s | 309 kB 00:00 2024-05-27T07:02:52.022 INFO:teuthology.orchestra.run.smithi012.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 261 kB/s | 26 kB 00:00 2024-05-27T07:02:52.023 INFO:teuthology.orchestra.run.smithi062.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 63 kB/s | 44 kB 00:00 2024-05-27T07:02:52.073 INFO:teuthology.orchestra.run.smithi012.stdout:(109/159): grpc-data-1.46.7-10.el9.noarch.rpm 54 kB/s | 19 kB 00:00 2024-05-27T07:02:52.124 INFO:teuthology.orchestra.run.smithi012.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-05-27T07:02:52.215 INFO:teuthology.orchestra.run.smithi012.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-05-27T07:02:52.240 INFO:teuthology.orchestra.run.smithi012.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 405 kB/s | 67 kB 00:00 2024-05-27T07:02:52.491 INFO:teuthology.orchestra.run.smithi012.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.0 MB/s | 839 kB 00:00 2024-05-27T07:02:52.532 INFO:teuthology.orchestra.run.smithi012.stdout:(114/159): python3-asyncssh-2.13.2-5.el9.noarch 1.8 MB/s | 548 kB 00:00 2024-05-27T07:02:52.556 INFO:teuthology.orchestra.run.smithi062.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 255 kB/s | 221 kB 00:00 2024-05-27T07:02:52.566 INFO:teuthology.orchestra.run.smithi012.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 586 kB/s | 43 kB 00:00 2024-05-27T07:02:52.608 INFO:teuthology.orchestra.run.smithi012.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 432 kB/s | 32 kB 00:00 2024-05-27T07:02:52.642 INFO:teuthology.orchestra.run.smithi012.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 201 kB/s | 15 kB 00:00 2024-05-27T07:02:52.750 INFO:teuthology.orchestra.run.smithi012.stdout:(118/159): libarrow-9.0.0-11.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-05-27T07:02:52.781 INFO:teuthology.orchestra.run.smithi062.stdout:(96/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.8 MB/s | 16 MB 00:01 2024-05-27T07:02:52.784 INFO:teuthology.orchestra.run.smithi012.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 983 kB/s | 172 kB 00:00 2024-05-27T07:02:52.815 INFO:teuthology.orchestra.run.smithi062.stdout:(97/159): qatlib-service-24.02.0-1.el9.x86_64.r 45 kB/s | 36 kB 00:00 2024-05-27T07:02:52.817 INFO:teuthology.orchestra.run.smithi012.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 2.0 MB/s | 358 kB 00:00 2024-05-27T07:02:52.851 INFO:teuthology.orchestra.run.smithi012.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-05-27T07:02:52.885 INFO:teuthology.orchestra.run.smithi012.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-05-27T07:02:52.937 INFO:teuthology.orchestra.run.smithi012.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-05-27T07:02:52.960 INFO:teuthology.orchestra.run.smithi012.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 144 kB/s | 11 kB 00:00 2024-05-27T07:02:53.036 INFO:teuthology.orchestra.run.smithi012.stdout:(125/159): python3-grpcio-1.46.7-10.el9.x86_64. 9.4 MB/s | 2.0 MB 00:00 2024-05-27T07:02:53.061 INFO:teuthology.orchestra.run.smithi012.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 141 kB/s | 18 kB 00:00 2024-05-27T07:02:53.086 INFO:teuthology.orchestra.run.smithi012.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 186 kB/s | 23 kB 00:00 2024-05-27T07:02:53.111 INFO:teuthology.orchestra.run.smithi012.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 261 kB/s | 19 kB 00:00 2024-05-27T07:02:53.153 INFO:teuthology.orchestra.run.smithi012.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 299 kB/s | 20 kB 00:00 2024-05-27T07:02:53.187 INFO:teuthology.orchestra.run.smithi012.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 620 kB/s | 46 kB 00:00 2024-05-27T07:02:53.246 INFO:teuthology.orchestra.run.smithi012.stdout:(131/159): python3-kubernetes-26.1.0-2.el9.noar 6.5 MB/s | 1.0 MB 00:00 2024-05-27T07:02:53.279 INFO:teuthology.orchestra.run.smithi012.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 632 kB/s | 79 kB 00:00 2024-05-27T07:02:53.304 INFO:teuthology.orchestra.run.smithi012.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 496 kB/s | 58 kB 00:00 2024-05-27T07:02:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-05-27T07:02:53.363 INFO:teuthology.orchestra.run.smithi012.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-05-27T07:02:53.389 INFO:teuthology.orchestra.run.smithi012.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-05-27T07:02:53.422 INFO:teuthology.orchestra.run.smithi012.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 340 kB/s | 28 kB 00:00 2024-05-27T07:02:53.448 INFO:teuthology.orchestra.run.smithi012.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-05-27T07:02:53.482 INFO:teuthology.orchestra.run.smithi012.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-05-27T07:02:53.507 INFO:teuthology.orchestra.run.smithi012.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-05-27T07:02:53.532 INFO:teuthology.orchestra.run.smithi012.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 430 kB/s | 36 kB 00:00 2024-05-27T07:02:53.540 INFO:teuthology.orchestra.run.smithi062.stdout:(98/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 48 kB/s | 47 kB 00:00 2024-05-27T07:02:53.557 INFO:teuthology.orchestra.run.smithi012.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-05-27T07:02:53.591 INFO:teuthology.orchestra.run.smithi012.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-05-27T07:02:53.616 INFO:teuthology.orchestra.run.smithi012.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-05-27T07:02:53.648 INFO:teuthology.orchestra.run.smithi062.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 69 kB/s | 58 kB 00:00 2024-05-27T07:02:53.680 INFO:teuthology.orchestra.run.smithi012.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.4 MB/s | 427 kB 00:00 2024-05-27T07:02:53.700 INFO:teuthology.orchestra.run.smithi012.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 185 kB/s | 20 kB 00:00 2024-05-27T07:02:53.733 INFO:teuthology.orchestra.run.smithi012.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-05-27T07:02:53.757 INFO:teuthology.orchestra.run.smithi062.stdout:(100/159): socat-1.7.4.1-5.el9.x86_64.rpm 313 kB/s | 305 kB 00:00 2024-05-27T07:02:53.818 INFO:teuthology.orchestra.run.smithi012.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-27T07:02:54.001 INFO:teuthology.orchestra.run.smithi012.stdout:(149/159): librados2-19.0.0-3872.g38f9d900.el9. 11 MB/s | 3.3 MB 00:00 2024-05-27T07:02:54.109 INFO:teuthology.orchestra.run.smithi012.stdout:(150/159): NetworkManager-1.47.90-1.el9.x86_64. 7.9 MB/s | 2.3 MB 00:00 2024-05-27T07:02:54.160 INFO:teuthology.orchestra.run.smithi012.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 847 kB/s | 41 kB 00:00 2024-05-27T07:02:54.218 INFO:teuthology.orchestra.run.smithi012.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 4.2 MB/s | 248 kB 00:00 2024-05-27T07:02:54.269 INFO:teuthology.orchestra.run.smithi012.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 748 kB/s | 37 kB 00:00 2024-05-27T07:02:54.344 INFO:teuthology.orchestra.run.smithi012.stdout:(154/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 5.1 MB/s | 3.1 MB 00:00 2024-05-27T07:02:54.402 INFO:teuthology.orchestra.run.smithi012.stdout:(155/159): NetworkManager-libnm-1.47.90-1.el9.x 4.5 MB/s | 1.8 MB 00:00 2024-05-27T07:02:54.428 INFO:teuthology.orchestra.run.smithi012.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.3 MB/s | 207 kB 00:00 2024-05-27T07:02:54.457 INFO:teuthology.orchestra.run.smithi062.stdout:(101/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 102 kB/s | 94 kB 00:00 2024-05-27T07:02:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-05-27T07:02:54.470 INFO:teuthology.orchestra.run.smithi012.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.0 MB/s | 260 kB 00:00 2024-05-27T07:02:54.603 INFO:teuthology.orchestra.run.smithi012.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-05-27T07:02:54.624 INFO:teuthology.orchestra.run.smithi062.stdout:(102/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 65 kB/s | 64 kB 00:00 2024-05-27T07:02:54.650 INFO:teuthology.orchestra.run.smithi062.stdout:(103/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 50 kB/s | 45 kB 00:00 2024-05-27T07:02:54.661 INFO:teuthology.orchestra.run.smithi012.stdout:(159/159): openssl-3.2.1-1.el9.x86_64.rpm 5.2 MB/s | 1.3 MB 00:00 2024-05-27T07:02:54.673 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:02:54.674 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:02:54.674 INFO:teuthology.orchestra.run.smithi012.stdout:Total 15 MB/s | 219 MB 00:14 2024-05-27T07:02:54.758 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:02:54.869 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:02:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:02:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:02:55.025 INFO:teuthology.orchestra.run.smithi062.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-05-27T07:02:55.100 INFO:teuthology.orchestra.run.smithi062.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 4.1 MB/s | 309 kB 00:00 2024-05-27T07:02:55.101 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:02:55.158 INFO:teuthology.orchestra.run.smithi062.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 355 kB/s | 19 kB 00:00 2024-05-27T07:02:55.201 INFO:teuthology.orchestra.run.smithi062.stdout:(107/159): lua-devel-5.4.4-4.el9.x86_64.rpm 30 kB/s | 22 kB 00:00 2024-05-27T07:02:55.376 INFO:teuthology.orchestra.run.smithi062.stdout:(108/159): libarrow-9.0.0-11.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-05-27T07:02:55.401 INFO:teuthology.orchestra.run.smithi062.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-05-27T07:02:55.435 INFO:teuthology.orchestra.run.smithi062.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 843 kB/s | 49 kB 00:00 2024-05-27T07:02:55.493 INFO:teuthology.orchestra.run.smithi062.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2024-05-27T07:02:55.518 INFO:teuthology.orchestra.run.smithi062.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2024-05-27T07:02:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:02:55.577 INFO:teuthology.orchestra.run.smithi062.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 839 kB 00:00 2024-05-27T07:02:55.636 INFO:teuthology.orchestra.run.smithi062.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 760 kB/s | 43 kB 00:00 2024-05-27T07:02:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:02:55.706 INFO:teuthology.orchestra.run.smithi062.stdout:(115/159): python3-asyncssh-2.13.2-5.el9.noarch 2.9 MB/s | 548 kB 00:00 2024-05-27T07:02:55.737 INFO:teuthology.orchestra.run.smithi062.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 323 kB/s | 32 kB 00:00 2024-05-27T07:02:55.762 INFO:teuthology.orchestra.run.smithi062.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 272 kB/s | 15 kB 00:00 2024-05-27T07:02:55.804 INFO:teuthology.orchestra.run.smithi062.stdout:(118/159): protobuf-compiler-3.14.0-13.el9.x86_ 734 kB/s | 863 kB 00:01 2024-05-27T07:02:55.838 INFO:teuthology.orchestra.run.smithi062.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-05-27T07:02:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:02:55.871 INFO:teuthology.orchestra.run.smithi062.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-05-27T07:02:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:02:55.905 INFO:teuthology.orchestra.run.smithi062.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-05-27T07:02:55.971 INFO:teuthology.orchestra.run.smithi062.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 2.2 MB/s | 144 kB 00:00 2024-05-27T07:02:55.993 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:02:56.055 INFO:teuthology.orchestra.run.smithi062.stdout:(123/159): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-05-27T07:02:56.080 INFO:teuthology.orchestra.run.smithi062.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-05-27T07:02:56.106 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:02:56.139 INFO:teuthology.orchestra.run.smithi062.stdout:(125/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 910 kB/s | 303 kB 00:00 2024-05-27T07:02:56.164 INFO:teuthology.orchestra.run.smithi062.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-05-27T07:02:56.189 INFO:teuthology.orchestra.run.smithi062.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-05-27T07:02:56.201 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:02:56.215 INFO:teuthology.orchestra.run.smithi062.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 262 kB/s | 19 kB 00:00 2024-05-27T07:02:56.248 INFO:teuthology.orchestra.run.smithi062.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 342 kB/s | 20 kB 00:00 2024-05-27T07:02:56.274 INFO:teuthology.orchestra.run.smithi062.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 798 kB/s | 46 kB 00:00 2024-05-27T07:02:56.310 INFO:teuthology.orchestra.run.smithi062.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-05-27T07:02:56.324 INFO:teuthology.orchestra.run.smithi062.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-05-27T07:02:56.375 INFO:teuthology.orchestra.run.smithi062.stdout:(133/159): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-05-27T07:02:56.391 INFO:teuthology.orchestra.run.smithi062.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 248 kB/s | 16 kB 00:00 2024-05-27T07:02:56.425 INFO:teuthology.orchestra.run.smithi062.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-05-27T07:02:56.450 INFO:teuthology.orchestra.run.smithi062.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 488 kB/s | 28 kB 00:00 2024-05-27T07:02:56.467 INFO:teuthology.orchestra.run.smithi062.stdout:(137/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 983 kB/s | 90 kB 00:00 2024-05-27T07:02:56.524 INFO:teuthology.orchestra.run.smithi062.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 316 kB/s | 31 kB 00:00 2024-05-27T07:02:56.551 INFO:teuthology.orchestra.run.smithi062.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 712 kB/s | 59 kB 00:00 2024-05-27T07:02:56.576 INFO:teuthology.orchestra.run.smithi062.stdout:(140/159): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-27T07:02:56.585 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:02:56.602 INFO:teuthology.orchestra.run.smithi062.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 465 kB/s | 36 kB 00:00 2024-05-27T07:02:56.627 INFO:teuthology.orchestra.run.smithi062.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 681 kB/s | 51 kB 00:00 2024-05-27T07:02:56.660 INFO:teuthology.orchestra.run.smithi062.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-05-27T07:02:56.667 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:02:56.686 INFO:teuthology.orchestra.run.smithi062.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-05-27T07:02:56.699 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:02:56.722 INFO:teuthology.orchestra.run.smithi062.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-05-27T07:02:56.745 INFO:teuthology.orchestra.run.smithi062.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 241 kB/s | 20 kB 00:00 2024-05-27T07:02:56.787 INFO:teuthology.orchestra.run.smithi062.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-05-27T07:02:56.852 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:02:56.862 INFO:teuthology.orchestra.run.smithi062.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-05-27T07:02:56.883 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:02:56.969 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:02:57.009 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:02:57.069 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:02:57.071 INFO:teuthology.orchestra.run.smithi062.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 11 MB/s | 2.3 MB 00:00 2024-05-27T07:02:57.163 INFO:teuthology.orchestra.run.smithi062.stdout:(150/159): librados2-19.0.0-3872.g38f9d900.el9. 8.0 MB/s | 3.3 MB 00:00 2024-05-27T07:02:57.221 INFO:teuthology.orchestra.run.smithi062.stdout:(151/159): NetworkManager-libnm-1.47.90-1.el9.x 12 MB/s | 1.8 MB 00:00 2024-05-27T07:02:57.247 INFO:teuthology.orchestra.run.smithi062.stdout:(152/159): NetworkManager-team-1.47.90-1.el9.x8 500 kB/s | 41 kB 00:00 2024-05-27T07:02:57.280 INFO:teuthology.orchestra.run.smithi062.stdout:(153/159): NetworkManager-tui-1.47.90-1.el9.x86 4.2 MB/s | 248 kB 00:00 2024-05-27T07:02:57.339 INFO:teuthology.orchestra.run.smithi062.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 403 kB/s | 37 kB 00:00 2024-05-27T07:02:57.372 INFO:teuthology.orchestra.run.smithi062.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 2.2 MB/s | 207 kB 00:00 2024-05-27T07:02:57.385 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:02:57.414 INFO:teuthology.orchestra.run.smithi062.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 3.4 MB/s | 260 kB 00:00 2024-05-27T07:02:57.490 INFO:teuthology.orchestra.run.smithi062.stdout:(157/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 4.4 MB/s | 3.1 MB 00:00 2024-05-27T07:02:57.565 INFO:teuthology.orchestra.run.smithi062.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 7.1 MB/s | 1.3 MB 00:00 2024-05-27T07:02:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:02:57.640 INFO:teuthology.orchestra.run.smithi062.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-05-27T07:02:57.647 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:02:57.647 INFO:teuthology.orchestra.run.smithi062.stdout:Total 6.5 MB/s | 219 MB 00:33 2024-05-27T07:02:57.799 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:02:57.886 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:02:57.999 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-05-27T07:02:58.031 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:02:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:02:58.249 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-05-27T07:02:58.250 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-05-27T07:02:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:02:58.643 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:02:58.682 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:02:58.729 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:02:58.840 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:02:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:02:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:02:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:02:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:02:59.163 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:02:59.229 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:02:59.265 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:02:59.320 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:02:59.457 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:02:59.494 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:02:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:02:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:02:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:02:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:02:59.883 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:03:00.063 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:03:00.130 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:03:00.814 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:03:00.888 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-05-27T07:03:00.889 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-05-27T07:03:00.929 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:03:00.951 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-05-27T07:03:00.973 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:03:01.023 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:03:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:03:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:03:01.219 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-05-27T07:03:01.219 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-05-27T07:03:01.230 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:03:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:03:01.502 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:03:01.647 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:03:01.724 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:03:01.812 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:03:01.852 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:03:01.889 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:03:01.955 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:03:02.007 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:03:02.068 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:03:02.135 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:03:02.333 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:03:02.384 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:03:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:03:02.566 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:03:02.631 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:03:02.680 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:03:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:03:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:03:02.953 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:03:03.124 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:03:03.170 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:03:03.217 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:03:03.268 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:03:03.330 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:03:03.383 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:03:03.424 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:03:03.470 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:03:03.520 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:03:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:03:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:03:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:03:03.859 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:03:03.901 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:03:03.971 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:03:04.077 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-05-27T07:03:04.078 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-05-27T07:03:04.120 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:03:04.188 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:03:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:03:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:03:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:03:05.372 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:03:05.440 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:03:05.496 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:03:05.605 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:03:05.650 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:03:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:03:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:03:06.030 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:03:06.860 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:03:07.007 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:03:07.600 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-05-27T07:03:07.707 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:03:07.779 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:03:07.894 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:03:07.953 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:03:07.989 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:03:08.098 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:03:08.550 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:03:08.640 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:03:08.662 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:03:08.745 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:03:08.782 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:03:08.817 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:03:08.836 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:03:08.862 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:03:08.999 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:03:09.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:03:09.143 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:03:09.207 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:03:09.250 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:03:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:03:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:03:09.465 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:03:09.530 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:03:09.614 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:03:09.645 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:03:09.818 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:03:09.844 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:03:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:03:09.994 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:03:10.036 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:03:10.066 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:03:10.069 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:03:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:03:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:03:10.380 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:03:10.738 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:03:10.901 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:03:10.974 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-05-27T07:03:11.006 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:03:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:03:11.199 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:03:11.271 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:03:11.286 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:03:11.422 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:03:11.483 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:03:11.516 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:03:11.607 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:03:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:03:11.802 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:03:11.843 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:03:11.900 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:03:12.022 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:03:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:03:12.075 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:03:12.150 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:03:12.171 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:03:12.205 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:03:12.240 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:03:12.332 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:03:12.356 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:03:12.389 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:03:12.405 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:03:12.444 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:03:12.497 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:03:12.521 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:03:12.563 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:03:12.611 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:03:12.656 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:03:12.674 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:03:12.706 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:03:12.738 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:03:12.811 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:12.850 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:12.859 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:03:13.057 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:03:13.080 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:03:13.119 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:03:13.144 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:03:13.199 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:03:13.310 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:03:13.337 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:03:13.385 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:03:13.463 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:03:13.474 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:03:13.505 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:03:13.516 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:13.570 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:03:13.580 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:03:13.742 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:03:13.743 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:03:13.743 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:03:13.743 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:13.788 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:03:13.899 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:03:14.187 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:03:14.191 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:03:14.296 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:03:14.336 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:03:14.375 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:03:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:03:14.432 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:03:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:03:14.499 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:03:14.565 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:03:14.692 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:03:14.817 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:03:14.847 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:03:14.900 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:03:15.037 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:03:15.122 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:03:15.205 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:03:15.219 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:03:15.257 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:03:15.296 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:03:15.365 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:03:15.405 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:03:15.406 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:03:15.435 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:03:15.467 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:03:15.484 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:03:15.518 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:03:15.588 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:03:15.648 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:03:15.713 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:03:15.718 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:03:15.768 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:03:15.792 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:03:15.846 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:03:15.846 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:03:15.978 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:03:15.984 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:03:16.024 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:03:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:03:16.069 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:03:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:03:16.106 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:03:16.139 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:03:16.141 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:03:16.254 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:03:16.297 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:03:16.354 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:03:16.357 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:03:16.389 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:03:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:03:16.543 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:03:16.578 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:03:16.587 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:03:16.627 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:03:16.681 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:03:16.719 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:03:16.772 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:03:16.829 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:03:16.878 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:03:16.911 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:03:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:03:16.981 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:03:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:03:17.059 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:03:17.102 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:03:17.251 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:03:17.319 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:03:17.369 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:03:17.444 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:03:17.612 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:03:17.701 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:03:17.728 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:03:17.838 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:03:17.885 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:03:17.924 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:03:17.981 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:03:18.033 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:03:18.110 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:03:18.335 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:03:18.405 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:03:18.529 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:03:18.611 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:03:18.711 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:03:18.731 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:03:18.780 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:03:18.820 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:03:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:03:18.876 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:03:18.889 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:03:18.936 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:03:18.999 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:03:19.027 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:03:19.076 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:03:19.092 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:03:19.150 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:03:19.274 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:03:19.292 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:03:19.317 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:03:19.337 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:03:19.392 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:03:19.553 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:03:19.583 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:03:19.603 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:03:19.613 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:03:19.651 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:03:19.685 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:03:19.718 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:03:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:03:19.937 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:03:20.129 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:03:20.182 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:03:20.227 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:03:20.298 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:03:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:03:20.423 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:03:20.472 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:03:20.525 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:03:20.576 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:03:20.654 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:03:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:03:20.705 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:03:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:03:20.855 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:03:20.914 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:03:20.963 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:03:21.036 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:03:21.205 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:03:21.278 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:03:22.294 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:03:22.352 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:03:22.385 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:03:22.516 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:03:22.533 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:03:22.586 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:03:22.628 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:03:22.641 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:03:22.663 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:03:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:03:22.765 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:03:22.805 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:03:23.053 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:03:23.092 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:03:23.101 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:03:23.149 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:03:23.208 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:03:23.273 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:03:23.306 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:03:23.362 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:03:23.966 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:03:24.010 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:03:24.111 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:03:24.125 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:03:24.137 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:03:24.280 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:03:26.009 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:03:26.125 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:03:26.162 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:03:26.197 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:03:26.598 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:03:26.637 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:03:26.795 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:03:26.827 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:03:26.893 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:03:26.898 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-05-27T07:03:27.101 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-05-27T07:03:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:27.329 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:03:27.377 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:03:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:27.377 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:03:27.377 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:27.498 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:03:27.548 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:03:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:27.650 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:03:27.678 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:03:27.684 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:03:27.846 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:03:27.847 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:03:27.847 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:03:27.847 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:27.888 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:03:29.122 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:03:29.201 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:03:29.201 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:29.202 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:03:29.202 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:03:29.202 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:03:29.202 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:03:29.525 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:03:29.525 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:29.525 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:03:29.525 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:03:29.525 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:03:29.526 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:30.031 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:03:30.105 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:03:30.106 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:30.106 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:03:30.106 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:03:30.106 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:03:30.106 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:30.240 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:03:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:03:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:03:30.601 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:30.624 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:03:30.647 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:30.686 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:03:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:03:30.921 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:03:31.191 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:03:31.260 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:03:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:03:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:03:31.355 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:03:31.392 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:03:31.553 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:03:31.553 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:03:31.553 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:03:31.553 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:31.602 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:03:31.994 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:03:32.031 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:03:32.087 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:03:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:03:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:03:33.569 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:03:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:03:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:03:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:03:34.905 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:03:35.094 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:03:35.126 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:03:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:03:35.206 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:35.206 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:03:35.206 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:03:35.206 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:03:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:03:35.304 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:03:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:03:35.512 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:03:36.558 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:03:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:03:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:03:36.663 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:36.664 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:03:36.664 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:03:36.664 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:03:36.664 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:36.713 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:03:36.798 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:03:36.798 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:36.798 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:03:36.798 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:03:36.974 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:03:36.974 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:36.974 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:03:36.974 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:03:36.974 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:03:36.974 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:37.389 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:03:37.465 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:03:37.466 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:37.466 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:03:37.466 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:03:37.466 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:03:37.469 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:41.419 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:03:41.419 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-05-27T07:03:41.419 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-05-27T07:03:41.419 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-05-27T07:03:41.420 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-05-27T07:03:41.420 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-05-27T07:03:41.420 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-05-27T07:03:41.420 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-05-27T07:03:41.420 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:41.665 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:03:41.714 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:03:41.715 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:41.715 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:03:41.715 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:42.137 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:03:42.188 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:03:42.243 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:03:42.277 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:03:42.315 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:03:42.592 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:03:42.641 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:03:42.683 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:03:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:03:42.770 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:03:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:03:43.523 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:03:43.604 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:03:43.604 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:43.604 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:03:43.604 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:03:43.604 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:03:43.604 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:43.756 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:03:43.813 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:03:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:03:43.874 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:03:43.946 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:03:43.947 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:43.947 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:03:43.947 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:03:43.947 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:03:43.947 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:44.475 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:03:44.548 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:03:44.549 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:44.549 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:03:44.549 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:03:44.549 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:03:44.549 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:44.684 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:03:44.746 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:03:44.824 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:03:45.059 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:03:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2024-05-27T07:03:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2024-05-27T07:03:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2024-05-27T07:03:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2024-05-27T07:03:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2024-05-27T07:03:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /root 2024-05-27T07:03:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2024-05-27T07:03:45.060 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:45.078 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:03:45.138 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:03:45.227 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:03:45.264 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:03:45.295 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:03:45.337 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:03:45.345 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:03:45.345 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:45.346 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:03:45.346 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:03:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:03:45.514 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:03:45.533 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:03:45.571 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:03:45.581 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:03:45.614 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:03:45.619 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:03:45.648 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:03:45.661 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:03:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:03:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:03:45.815 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:03:45.861 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:03:45.862 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:03:45.997 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:03:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:03:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:03:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:03:46.353 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:03:46.391 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:03:46.392 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:03:46.396 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:03:46.453 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:03:46.537 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:03:46.567 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:03:46.582 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:03:46.597 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:03:46.701 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:03:46.957 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:03:46.957 INFO:teuthology.orchestra.run.smithi162.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-27T07:03:46.957 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:47.147 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:03:47.230 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:03:47.230 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:47.231 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:03:47.231 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:03:47.231 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:03:47.231 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:47.505 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:03:47.576 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:03:47.577 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:47.577 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:03:47.577 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:03:47.577 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:03:47.577 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:47.932 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:03:47.965 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:03:48.114 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:03:48.191 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:03:48.191 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:48.192 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:03:48.192 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:03:48.192 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:03:48.192 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:48.332 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:03:48.403 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:03:48.479 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:03:48.727 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:03:48.778 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:03:48.868 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:03:49.010 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:03:49.133 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:03:49.157 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:03:49.188 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:03:49.244 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:03:49.297 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:03:49.323 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:03:49.519 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:03:49.559 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:03:49.643 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:03:49.644 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:49.644 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:03:49.644 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:03:49.644 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:03:49.644 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:49.710 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:03:49.751 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:03:49.867 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:03:49.930 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:03:50.014 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:03:50.054 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:03:50.112 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:03:50.225 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:03:50.268 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:03:51.024 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:03:51.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:03:51.136 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:03:51.136 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:51.136 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:03:51.136 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:03:51.137 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:03:51.137 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:51.196 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:03:51.296 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:03:51.296 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:51.296 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:03:51.296 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:51.372 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:03:51.478 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:03:51.478 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:51.478 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:03:51.479 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:03:51.479 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:03:51.479 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:51.633 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:03:51.673 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:03:51.889 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:03:51.992 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:03:51.992 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:51.993 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:03:51.993 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:03:51.993 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:03:51.993 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:03:52.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:03:52.697 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:03:52.698 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:03:52.698 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:03:52.698 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:03:52.698 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:03:52.710 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:03:52.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:03:52.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:03:52.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:03:52.714 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:03:52.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:03:52.716 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:03:52.717 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:03:52.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:03:52.719 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:03:52.720 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:03:52.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:03:52.835 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:03:52.864 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:03:53.029 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:03:53.227 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:03:53.258 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:03:53.345 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:03:53.345 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:53.345 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:03:53.345 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:03:53.345 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:03:53.345 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:53.395 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:03:53.432 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:03:53.550 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:03:53.631 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:03:54.723 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:03:54.753 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:03:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:03:54.826 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:54.826 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:03:54.826 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:03:54.826 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:03:54.826 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:54.886 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:03:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:03:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:54.944 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-05-27T07:03:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:03:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:03:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:03:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:03:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:03:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:03:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:03:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:03:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:03:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:03:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:03:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:03:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:03:54.956 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-05-27T07:03:54.962 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:03:54.962 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:54.963 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:03:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:55.050 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:03:55.130 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:03:55.131 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:55.131 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:03:55.131 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:03:55.131 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:03:55.131 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:55.522 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:03:55.596 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:03:55.596 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:03:55.596 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:03:55.596 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:03:55.596 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:03:55.596 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:03:55.621 DEBUG:teuthology.parallel:result is None 2024-05-27T07:03:57.047 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:03:57.117 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:03:57.184 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:03:57.223 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:03:57.262 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:03:57.606 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:03:57.660 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:03:57.693 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:03:57.724 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:03:57.771 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:03:57.817 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:03:58.775 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:03:58.821 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:03:58.875 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:04:00.330 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:04:00.444 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:04:00.501 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:04:00.528 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:04:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:04:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:04:00.651 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:04:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:04:00.688 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:04:00.721 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:04:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:04:00.755 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:04:00.796 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:04:00.796 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:04:00.930 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:04:00.976 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:04:00.976 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:04:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:04:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:04:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:04:01.131 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:04:01.168 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:04:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:04:01.202 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:04:01.203 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:04:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:04:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:04:01.521 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:04:01.521 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:04:01.668 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:04:01.713 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:04:01.846 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:04:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:04:02.117 INFO:teuthology.orchestra.run.smithi012.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-27T07:04:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:04:02.262 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:04:02.310 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:04:02.369 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:04:03.858 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:04:03.974 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:04:04.034 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:04:04.168 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:04:04.288 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:04:04.326 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:04:04.361 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:04:04.404 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:04:04.404 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:04:04.560 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:04:04.614 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:04:04.614 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:04:04.792 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:04:04.832 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:04:04.870 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:04:04.870 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:04:05.311 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:04:05.312 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:04:05.457 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:04:05.526 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:04:05.653 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:04:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:04:05.923 INFO:teuthology.orchestra.run.smithi062.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-27T07:04:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:04:08.117 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:04:08.118 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:04:08.119 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:04:08.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:04:08.121 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:04:08.122 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:04:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:04:08.124 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:04:08.125 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:04:08.126 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:04:08.127 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:04:08.128 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:04:08.129 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:04:10.505 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:04:10.505 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:04:10.505 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:04:10.506 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.507 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.508 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:04:10.509 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.510 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:04:10.511 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:04:10.512 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:04:10.513 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:04:10.514 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:04:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:04:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:04:10.517 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:04:10.518 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:04:10.519 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:04:10.520 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:04:10.521 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:04:10.521 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:04:10.521 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-05-27T07:04:11.216 DEBUG:teuthology.parallel:result is None 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:04:11.951 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:04:11.952 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:04:11.953 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:04:11.953 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:04:11.953 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:04:11.953 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:04:11.953 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:04:11.953 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:04:11.953 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:04:11.965 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:04:11.966 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:04:11.967 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:04:11.968 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:04:11.969 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:04:11.970 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:04:11.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:04:11.972 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:04:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:04:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:04:11.975 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:04:11.975 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:04:11.975 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:04:11.975 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:04:11.975 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:04:14.391 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.392 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.393 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:04:14.394 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:04:14.395 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:04:14.396 INFO:teuthology.orchestra.run.smithi062.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:04:14.397 INFO:teuthology.orchestra.run.smithi062.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:04:14.398 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:04:14.399 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:04:14.400 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.401 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:04:14.402 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:04:14.403 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:04:14.404 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:04:14.404 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:04:14.404 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-05-27T07:04:14.404 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-05-27T07:04:15.020 DEBUG:teuthology.parallel:result is None 2024-05-27T07:04:15.021 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:04:15.171 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:04:15.233 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:04:15.234 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:04:15.234 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:04:15.236 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:04:15.507 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:04:15.569 INFO:teuthology.orchestra.run.smithi062.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:04:15.569 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:04:15.569 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:04:15.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:04:15.868 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:04:15.941 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:04:15.941 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:04:15.941 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:04:15.943 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-27T07:04:15.943 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:04:15.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:04:16.041 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:04:16.041 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:04:16.111 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:04:16.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:04:16.210 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-27T07:04:16.211 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:04:16.211 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:04:16.304 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:04:16.464 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:04:16.464 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:04:16.537 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:04:16.645 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:04:16.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:04:16.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:04:16.862 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-27T07:04:16.862 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:04:16.862 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:04:16.957 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:04:17.090 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:04:17.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:04:17.164 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:04:17.271 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:04:17.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:04:17.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:04:17.488 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-27T07:04:17.489 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:04:17.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:04:17.583 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:04:17.672 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:04:17.672 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:04:17.740 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:04:17.849 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:04:17.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:04:17.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:04:18.006 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-27T07:04:18.143 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-05-27T07:04:18.144 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:04:18.144 INFO:tasks.cephadm:Cluster fsid is 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:04:18.144 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-27T07:04:18.144 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.12', 'mon.b': '172.21.15.62', 'mon.c': '172.21.15.162'} 2024-05-27T07:04:18.145 INFO:tasks.cephadm:First mon is mon.a on smithi012 2024-05-27T07:04:18.145 INFO:tasks.cephadm:First mgr is x 2024-05-27T07:04:18.145 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-27T07:04:18.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo hostname $(hostname -s) 2024-05-27T07:04:18.241 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname $(hostname -s) 2024-05-27T07:04:18.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-05-27T07:04:18.399 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-27T07:04:18.400 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:04:18.545 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79587/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 08:08:15.155937', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79583/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 07:54:08.730060', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-27T07:04:18.690 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240521.063558-debug, sha1 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:04:18.692 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T07:04:18.692 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T07:04:18.692 DEBUG:teuthology.orchestra.run.smithi012:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:19.733 INFO:teuthology.orchestra.run.smithi012.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:04 /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:19.733 DEBUG:teuthology.orchestra.run.smithi062:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:20.389 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:04 /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:20.389 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:21.495 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:04 /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:21.496 DEBUG:teuthology.orchestra.run.smithi012:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:21.559 DEBUG:teuthology.orchestra.run.smithi062:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:21.623 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:04:21.696 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 on all hosts... 2024-05-27T07:04:21.696 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:04:21.703 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:04:21.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:04:22.796 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:04:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:04:22.972 INFO:teuthology.orchestra.run.smithi012.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:05:08.668 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-05-27T07:05:08.668 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:05:08.668 INFO:teuthology.orchestra.run.smithi062.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:05:08.669 INFO:teuthology.orchestra.run.smithi062.stdout: "repo_digests": [ 2024-05-27T07:05:08.669 INFO:teuthology.orchestra.run.smithi062.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:05:08.669 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-05-27T07:05:08.669 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-05-27T07:05:09.558 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-05-27T07:05:09.558 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:05:09.558 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:05:09.558 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-05-27T07:05:09.558 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:05:09.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-05-27T07:05:09.558 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-05-27T07:05:11.270 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:05:11.270 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:05:11.270 INFO:teuthology.orchestra.run.smithi012.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:05:11.270 INFO:teuthology.orchestra.run.smithi012.stdout: "repo_digests": [ 2024-05-27T07:05:11.270 INFO:teuthology.orchestra.run.smithi012.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:05:11.270 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:05:11.271 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:05:11.338 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph 2024-05-27T07:05:11.429 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph 2024-05-27T07:05:11.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-05-27T07:05:11.627 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /etc/ceph 2024-05-27T07:05:11.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /etc/ceph 2024-05-27T07:05:11.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-05-27T07:05:11.903 INFO:tasks.cephadm:Writing seed config... 2024-05-27T07:05:11.904 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-27T07:05:11.904 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-27T07:05:11.904 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-27T07:05:11.904 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-27T07:05:11.905 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-27T07:05:11.906 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-27T07:05:11.910 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-27T07:05:11.911 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-27T07:05:11.912 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:05:11.912 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-27T07:05:11.969 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 55c251ba-1bf7-11ef-bc9b-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-27T07:05:11.970 DEBUG:teuthology.orchestra.run.smithi012:mon.a> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a.service 2024-05-27T07:05:12.018 DEBUG:teuthology.orchestra.run.smithi012:mgr.x> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.x.service 2024-05-27T07:05:12.064 INFO:tasks.cephadm:Bootstrapping... 2024-05-27T07:05:12.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 -v bootstrap --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.12 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:05:12.341 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:05:12.341 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6', '-v', 'bootstrap', '--fsid', '55c251ba-1bf7-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.12', '--skip-admin-label'] 2024-05-27T07:05:12.341 INFO:teuthology.orchestra.run.smithi012.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-27T07:05:12.342 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying podman|docker is present... 2024-05-27T07:05:12.404 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 5.0.2 2024-05-27T07:05:12.404 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying lvm2 is present... 2024-05-27T07:05:12.405 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying time synchronization is in place... 2024-05-27T07:05:12.423 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T07:05:12.423 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T07:05:12.440 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T07:05:12.441 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-05-27T07:05:12.464 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-05-27T07:05:12.487 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-05-27T07:05:12.488 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-05-27T07:05:12.488 INFO:teuthology.orchestra.run.smithi012.stdout:Repeating the final host check... 2024-05-27T07:05:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 5.0.2 2024-05-27T07:05:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-27T07:05:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl is present 2024-05-27T07:05:12.555 INFO:teuthology.orchestra.run.smithi012.stdout:lvcreate is present 2024-05-27T07:05:12.573 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T07:05:12.573 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T07:05:12.594 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T07:05:12.594 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-05-27T07:05:12.620 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-05-27T07:05:12.642 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-05-27T07:05:12.642 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-05-27T07:05:12.643 INFO:teuthology.orchestra.run.smithi012.stdout:Host looks OK 2024-05-27T07:05:12.643 INFO:teuthology.orchestra.run.smithi012.stdout:Cluster fsid: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:12.643 INFO:teuthology.orchestra.run.smithi012.stdout:Acquiring lock 139830777054544 on /run/cephadm/55c251ba-1bf7-11ef-bc9b-c7b262605968.lock 2024-05-27T07:05:12.644 INFO:teuthology.orchestra.run.smithi012.stdout:Lock 139830777054544 acquired on /run/cephadm/55c251ba-1bf7-11ef-bc9b-c7b262605968.lock 2024-05-27T07:05:12.645 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 3300 ... 2024-05-27T07:05:12.646 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 6789 ... 2024-05-27T07:05:12.646 INFO:teuthology.orchestra.run.smithi012.stdout:Base mon IP(s) is [172.21.15.12:3300, 172.21.15.12:6789], mon addrv is [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-05-27T07:05:12.661 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.12 metric 100 2024-05-27T07:05:12.661 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.12 metric 100 2024-05-27T07:05:12.674 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-27T07:05:12.674 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-27T07:05:12.674 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-27T07:05:12.686 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-27T07:05:12.686 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-27T07:05:12.686 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T07:05:12.687 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-27T07:05:12.687 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-05-27T07:05:12.687 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T07:05:12.689 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-05-27T07:05:12.689 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-05-27T07:05:12.690 INFO:teuthology.orchestra.run.smithi012.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-27T07:05:12.693 INFO:teuthology.orchestra.run.smithi012.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-27T07:05:12.695 INFO:teuthology.orchestra.run.smithi012.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:05:13.347 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T07:05:13.348 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:05:13.348 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Getting image source signatures 2024-05-27T07:05:13.348 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:cf46015854c6cdc81d3273f73f8f001fe4a1edc6ed77e464da8c9d5e01e9b108 2024-05-27T07:05:13.348 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-05-27T07:05:13.348 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying config sha256:e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T07:05:13.348 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-27T07:05:14.579 INFO:teuthology.orchestra.run.smithi012.stdout:ceph: stdout ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:05:14.579 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph version: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:05:14.579 INFO:teuthology.orchestra.run.smithi012.stdout:Extracting ceph user uid/gid from container image... 2024-05-27T07:05:15.772 INFO:teuthology.orchestra.run.smithi012.stdout:stat: stdout 167 167 2024-05-27T07:05:15.772 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial keys... 2024-05-27T07:05:16.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCsMFRmLDbNEhAAm0KSfOI8XiP312GbDYa0Xg== 2024-05-27T07:05:18.076 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCtMFRmP78KHBAA3q2+dHYWTIUZhxjxMl0FDg== 2024-05-27T07:05:19.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCuMFRme0m9JRAAlzg1vnNt2Uq6dmlMMCRQYQ== 2024-05-27T07:05:19.262 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial monmap... 2024-05-27T07:05:20.389 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T07:05:20.390 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-27T07:05:20.390 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:20.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T07:05:20.391 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool for a [v2:172.21.15.12:3300,v1:172.21.15.12:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T07:05:20.391 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-05-27T07:05:20.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: set fsid to 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:20.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T07:05:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:05:20.391 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mon... 2024-05-27T07:05:21.567 INFO:teuthology.orchestra.run.smithi012.stdout:create mon.a on 2024-05-27T07:05:21.895 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-27T07:05:22.151 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-27T07:05:22.473 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968.target. 2024-05-27T07:05:22.473 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968.target. 2024-05-27T07:05:22.895 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a 2024-05-27T07:05:22.895 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-05-27T07:05:23.198 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968.target.wants/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@.service. 2024-05-27T07:05:23.917 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:23 smithi012 podman[31089]: 2024-05-27 07:05:23.854426914 +0000 UTC m=+0.373647684 container start 333807efc21514326aa0b7095883f100b2a084a5d08e6ee0d27096f13caa5857 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:05:23.918 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:23 smithi012 bash[31089]: 333807efc21514326aa0b7095883f100b2a084a5d08e6ee0d27096f13caa5857 2024-05-27T07:05:23.965 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:05:23.966 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-05-27T07:05:23.986 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:05:23.986 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-05-27T07:05:23.986 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-05-27T07:05:23.987 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T07:05:23.987 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon to start... 2024-05-27T07:05:23.987 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon... 2024-05-27T07:05:24.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:23 smithi012 systemd[1]: Started Ceph mon.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:05:24.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:24 smithi012 ceph-mon[31107]: mkfs 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:24.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:24 smithi012 ceph-mon[31107]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T07:05:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T07:05:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: monmap epoch 1 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: last_changed 2024-05-27T07:05:19.787338+0000 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: created 2024-05-27T07:05:19.787338+0000 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: min_mon_release 19 (squid) 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: election_strategy: 1 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: fsmap 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: mgrmap e1: no daemons active 2024-05-27T07:05:25.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:25 smithi012 ceph-mon[31107]: from='client.? 172.21.15.12:0/637952747' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout cluster: 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout id: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout services: 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.743804s) 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-27T07:05:35.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout data: 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pgs: 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.660+0000 7f39f2a00700 1 Processor -- start 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.660+0000 7f39f2a00700 1 -- start start 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.661+0000 7f39f2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec07d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:35.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.661+0000 7f39f2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec07d750 con 0x7f39ec07ed90 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.662+0000 7f39ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec07d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.662+0000 7f39ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec07d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47138/0 (socket says 172.21.15.12:47138) 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.662+0000 7f39ebe00700 1 -- 172.21.15.12:0/2992033059 learned_addr learned my addr 172.21.15.12:0/2992033059 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.662+0000 7f39ebe00700 1 -- 172.21.15.12:0/2992033059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec07d8d0 con 0x7f39ec07ed90 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.663+0000 7f39ebe00700 1 --2- 172.21.15.12:0/2992033059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec07d210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098a0 tx=0x7f39dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a1f901e5417989 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.663+0000 7f39eaa00700 1 -- 172.21.15.12:0/2992033059 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc004160 con 0x7f39ec07ed90 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.663+0000 7f39eaa00700 1 -- 172.21.15.12:0/2992033059 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39dc004300 con 0x7f39ec07ed90 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.663+0000 7f39eaa00700 1 -- 172.21.15.12:0/2992033059 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc0036a0 con 0x7f39ec07ed90 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.664+0000 7f39f2a00700 1 -- 172.21.15.12:0/2992033059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 msgr2=0x7f39ec07d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.664+0000 7f39f2a00700 1 --2- 172.21.15.12:0/2992033059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec07d210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098a0 tx=0x7f39dc009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.664+0000 7f39f2a00700 1 -- 172.21.15.12:0/2992033059 shutdown_connections 2024-05-27T07:05:35.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.664+0000 7f39f2a00700 1 --2- 172.21.15.12:0/2992033059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec07d210 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.664+0000 7f39f2a00700 1 -- 172.21.15.12:0/2992033059 >> 172.21.15.12:0/2992033059 conn(0x7f39ec105020 msgr2=0x7f39ec1074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.664+0000 7f39f2a00700 1 -- 172.21.15.12:0/2992033059 shutdown_connections 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.664+0000 7f39f2a00700 1 -- 172.21.15.12:0/2992033059 wait complete. 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.666+0000 7f39f2a00700 1 Processor -- start 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.666+0000 7f39f2a00700 1 -- start start 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.666+0000 7f39f2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.666+0000 7f39f2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec07de00 con 0x7f39ec07ed90 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.667+0000 7f39ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.667+0000 7f39ebe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec19f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47146/0 (socket says 172.21.15.12:47146) 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.667+0000 7f39ebe00700 1 -- 172.21.15.12:0/637952747 learned_addr learned my addr 172.21.15.12:0/637952747 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.667+0000 7f39ebe00700 1 -- 172.21.15.12:0/637952747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec19fbc0 con 0x7f39ec07ed90 2024-05-27T07:05:35.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.668+0000 7f39ebe00700 1 --2- 172.21.15.12:0/637952747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec19f680 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39dc000c00 tx=0x7f39dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.668+0000 7f39e8c00700 1 -- 172.21.15.12:0/637952747 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc003db0 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.668+0000 7f39e8c00700 1 -- 172.21.15.12:0/637952747 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39dc011630 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.668+0000 7f39e8c00700 1 -- 172.21.15.12:0/637952747 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc011950 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.668+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ec19fe50 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.668+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ec1a2a60 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.668+0000 7f39e8c00700 1 -- 172.21.15.12:0/637952747 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f39dc01a460 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.669+0000 7f39e8c00700 1 -- 172.21.15.12:0/637952747 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f39dc01b780 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.669+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ec064e10 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.673+0000 7f39e8c00700 1 -- 172.21.15.12:0/637952747 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f39dc01b980 con 0x7f39ec07ed90 2024-05-27T07:05:35.600 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.755+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f39ec02e8b0 con 0x7f39ec07ed90 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.755+0000 7f39e8c00700 1 -- 172.21.15.12:0/637952747 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f39dc01bb60 con 0x7f39ec07ed90 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.757+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 msgr2=0x7f39ec19f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.757+0000 7f39f2a00700 1 --2- 172.21.15.12:0/637952747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec19f680 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39dc000c00 tx=0x7f39dc004750 comp rx=0 tx=0).stop 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.757+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 shutdown_connections 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.757+0000 7f39f2a00700 1 --2- 172.21.15.12:0/637952747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39ec07ed90 0x7f39ec19f680 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.757+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 >> 172.21.15.12:0/637952747 conn(0x7f39ec105020 msgr2=0x7f39ec105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.757+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 shutdown_connections 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:24.757+0000 7f39f2a00700 1 -- 172.21.15.12:0/637952747 wait complete. 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:mon is available 2024-05-27T07:05:35.601 INFO:teuthology.orchestra.run.smithi012.stdout:Assimilating anything we can from ceph.conf... 2024-05-27T07:05:36.823 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:36 smithi012 ceph-mon[31107]: from='client.? 172.21.15.12:0/4269160931' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T07:05:36.823 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:36 smithi012 ceph-mon[31107]: from='client.? 172.21.15.12:0/4269160931' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-27T07:05:37.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:37.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T07:05:37.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:37.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T07:05:37.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-05-27T07:05:37.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T07:05:37.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T07:05:37.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T07:05:37.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T07:05:37.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:37.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T07:05:37.129 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.413+0000 7f34e6a00700 1 Processor -- start 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.413+0000 7f34e6a00700 1 -- start start 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.413+0000 7f34e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e010bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.414+0000 7f34e6a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f34e010c3d0 con 0x7f34e010b990 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.414+0000 7f34e5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e010bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.414+0000 7f34e5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e010bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47938/0 (socket says 172.21.15.12:47938) 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.414+0000 7f34e5600700 1 -- 172.21.15.12:0/2594742420 learned_addr learned my addr 172.21.15.12:0/2594742420 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.415+0000 7f34e5600700 1 -- 172.21.15.12:0/2594742420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e010cb80 con 0x7f34e010b990 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.415+0000 7f34e5600700 1 --2- 172.21.15.12:0/2594742420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e010bdc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f34d40098a0 tx=0x7f34d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4624c344968dcdd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.415+0000 7f34dfe00700 1 -- 172.21.15.12:0/2594742420 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d4004160 con 0x7f34e010b990 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.415+0000 7f34dfe00700 1 -- 172.21.15.12:0/2594742420 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d4004300 con 0x7f34e010b990 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.415+0000 7f34dfe00700 1 -- 172.21.15.12:0/2594742420 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d40036a0 con 0x7f34e010b990 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.416+0000 7f34e6a00700 1 -- 172.21.15.12:0/2594742420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 msgr2=0x7f34e010bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:37.130 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.416+0000 7f34e6a00700 1 --2- 172.21.15.12:0/2594742420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e010bdc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f34d40098a0 tx=0x7f34d4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:37.131 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.416+0000 7f34e6a00700 1 -- 172.21.15.12:0/2594742420 shutdown_connections 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.416+0000 7f34e6a00700 1 --2- 172.21.15.12:0/2594742420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e010bdc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.416+0000 7f34e6a00700 1 -- 172.21.15.12:0/2594742420 >> 172.21.15.12:0/2594742420 conn(0x7f34e007efb0 msgr2=0x7f34e007f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.416+0000 7f34e6a00700 1 -- 172.21.15.12:0/2594742420 shutdown_connections 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.417+0000 7f34e6a00700 1 -- 172.21.15.12:0/2594742420 wait complete. 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.418+0000 7f34e6a00700 1 Processor -- start 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.418+0000 7f34e6a00700 1 -- start start 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.419+0000 7f34e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e019f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.419+0000 7f34e6a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f34e010d0b0 con 0x7f34e010b990 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.419+0000 7f34e5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e019f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:37.134 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.419+0000 7f34e5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e019f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47940/0 (socket says 172.21.15.12:47940) 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.419+0000 7f34e5600700 1 -- 172.21.15.12:0/4269160931 learned_addr learned my addr 172.21.15.12:0/4269160931 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.419+0000 7f34e5600700 1 -- 172.21.15.12:0/4269160931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e019f930 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.420+0000 7f34e5600700 1 --2- 172.21.15.12:0/4269160931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e019f3f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f34d4000c00 tx=0x7f34d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.420+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d4003db0 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.420+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d4011630 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.420+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d4011950 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.420+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e019fbc0 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.421+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e019ffe0 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.421+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f34d401a460 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.421+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f34d401aa90 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.422+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e0064e10 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.425+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f34d4042ec0 con 0x7f34e010b990 2024-05-27T07:05:37.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.504+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f34e002dc30 con 0x7f34e010b990 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.511+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f34d4031030 con 0x7f34e010b990 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.512+0000 7f34de000700 1 -- 172.21.15.12:0/4269160931 <== mon.0 v2:172.21.15.12:3300/0 8 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f34d4004260 con 0x7f34e010b990 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.512+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 msgr2=0x7f34e019f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.512+0000 7f34e6a00700 1 --2- 172.21.15.12:0/4269160931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e019f3f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f34d4000c00 tx=0x7f34d4004750 comp rx=0 tx=0).stop 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.513+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 shutdown_connections 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.513+0000 7f34e6a00700 1 --2- 172.21.15.12:0/4269160931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34e010b990 0x7f34e019f3f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.513+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 >> 172.21.15.12:0/4269160931 conn(0x7f34e007efb0 msgr2=0x7f34e0193420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.513+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 shutdown_connections 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:36.513+0000 7f34e6a00700 1 -- 172.21.15.12:0/4269160931 wait complete. 2024-05-27T07:05:37.136 INFO:teuthology.orchestra.run.smithi012.stdout:Generating new minimal ceph.conf... 2024-05-27T07:05:38.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:38 smithi012 ceph-mon[31107]: from='client.? 172.21.15.12:0/1161686717' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:38.551 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.861+0000 7fdb40e00700 1 Processor -- start 2024-05-27T07:05:38.551 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.861+0000 7fdb40e00700 1 -- start start 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.862+0000 7fdb40e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c10bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.862+0000 7fdb40e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c10c550 con 0x7fdb3c10bb50 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.862+0000 7fdb39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c10bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.862+0000 7fdb39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c10bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47956/0 (socket says 172.21.15.12:47956) 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.862+0000 7fdb39e00700 1 -- 172.21.15.12:0/3623234119 learned_addr learned my addr 172.21.15.12:0/3623234119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.863+0000 7fdb39e00700 1 -- 172.21.15.12:0/3623234119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c10cd20 con 0x7fdb3c10bb50 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.863+0000 7fdb39e00700 1 --2- 172.21.15.12:0/3623234119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c10bf60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009e80 tx=0x7fdb2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d02a9e13fef9f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.863+0000 7fdb39400700 1 -- 172.21.15.12:0/3623234119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb2c017070 con 0x7fdb3c10bb50 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.863+0000 7fdb39400700 1 -- 172.21.15.12:0/3623234119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fdb2c00b8c0 con 0x7fdb3c10bb50 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.863+0000 7fdb39400700 1 -- 172.21.15.12:0/3623234119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb2c004420 con 0x7fdb3c10bb50 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.864+0000 7fdb40e00700 1 -- 172.21.15.12:0/3623234119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 msgr2=0x7fdb3c10bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.864+0000 7fdb40e00700 1 --2- 172.21.15.12:0/3623234119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c10bf60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009e80 tx=0x7fdb2c00b330 comp rx=0 tx=0).stop 2024-05-27T07:05:38.552 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.864+0000 7fdb40e00700 1 -- 172.21.15.12:0/3623234119 shutdown_connections 2024-05-27T07:05:38.555 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.864+0000 7fdb40e00700 1 --2- 172.21.15.12:0/3623234119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c10bf60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:38.555 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.864+0000 7fdb40e00700 1 -- 172.21.15.12:0/3623234119 >> 172.21.15.12:0/3623234119 conn(0x7fdb3c107470 msgr2=0x7fdb3c109890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.864+0000 7fdb40e00700 1 -- 172.21.15.12:0/3623234119 shutdown_connections 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.865+0000 7fdb40e00700 1 -- 172.21.15.12:0/3623234119 wait complete. 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.866+0000 7fdb40e00700 1 Processor -- start 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.866+0000 7fdb40e00700 1 -- start start 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.866+0000 7fdb40e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c19fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.866+0000 7fdb40e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c1100d0 con 0x7fdb3c10bb50 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.867+0000 7fdb39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c19fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.867+0000 7fdb39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c19fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47966/0 (socket says 172.21.15.12:47966) 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.867+0000 7fdb39e00700 1 -- 172.21.15.12:0/1161686717 learned_addr learned my addr 172.21.15.12:0/1161686717 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.867+0000 7fdb39e00700 1 -- 172.21.15.12:0/1161686717 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c1a0410 con 0x7fdb3c10bb50 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.868+0000 7fdb39e00700 1 --2- 172.21.15.12:0/1161686717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c19fed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c004110 tx=0x7fdb2c009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:38.556 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.868+0000 7fdb33400700 1 -- 172.21.15.12:0/1161686717 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb2c020020 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.868+0000 7fdb33400700 1 -- 172.21.15.12:0/1161686717 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fdb2c003df0 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.868+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3c1a06a0 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.868+0000 7fdb33400700 1 -- 172.21.15.12:0/1161686717 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb2c017040 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.869+0000 7fdb33400700 1 -- 172.21.15.12:0/1161686717 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdb2c023020 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.869+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb3c1a32b0 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.869+0000 7fdb33400700 1 -- 172.21.15.12:0/1161686717 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdb2c00f050 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.870+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb3c064e10 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.873+0000 7fdb33400700 1 -- 172.21.15.12:0/1161686717 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fdb2c02a020 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.951+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fdb3c1a3480 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.951+0000 7fdb33400700 1 -- 172.21.15.12:0/1161686717 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fdb2c012b30 con 0x7fdb3c10bb50 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.954+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 msgr2=0x7fdb3c19fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.954+0000 7fdb40e00700 1 --2- 172.21.15.12:0/1161686717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c19fed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c004110 tx=0x7fdb2c009390 comp rx=0 tx=0).stop 2024-05-27T07:05:38.557 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.954+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 shutdown_connections 2024-05-27T07:05:38.558 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.954+0000 7fdb40e00700 1 --2- 172.21.15.12:0/1161686717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb3c10bb50 0x7fdb3c19fed0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:38.558 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.954+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 >> 172.21.15.12:0/1161686717 conn(0x7fdb3c107470 msgr2=0x7fdb3c108cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:38.558 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.954+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 shutdown_connections 2024-05-27T07:05:38.558 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:37.954+0000 7fdb40e00700 1 -- 172.21.15.12:0/1161686717 wait complete. 2024-05-27T07:05:38.558 INFO:teuthology.orchestra.run.smithi012.stdout:Restarting the monitor... 2024-05-27T07:05:38.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:38 smithi012 systemd[1]: Stopping Ceph mon.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:05:39.434 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a[31103]: 2024-05-27T07:05:39.153+0000 7f3abda00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:05:39.434 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a[31103]: 2024-05-27T07:05:39.153+0000 7f3abda00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-27T07:05:39.717 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:39 smithi012 podman[31350]: 2024-05-27 07:05:39.44327384 +0000 UTC m=+0.424945098 container died 333807efc21514326aa0b7095883f100b2a084a5d08e6ee0d27096f13caa5857 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:05:40.332 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:40 smithi012 podman[31350]: 2024-05-27 07:05:40.013793934 +0000 UTC m=+0.995465230 container cleanup 333807efc21514326aa0b7095883f100b2a084a5d08e6ee0d27096f13caa5857 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:05:40.332 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:40 smithi012 bash[31350]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a 2024-05-27T07:05:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:40 smithi012 podman[31363]: 2024-05-27 07:05:40.340516506 +0000 UTC m=+0.873465814 container remove 333807efc21514326aa0b7095883f100b2a084a5d08e6ee0d27096f13caa5857 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:05:41.321 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-27T07:05:41.321 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:40 smithi012 systemd[1]: Stopped Ceph mon.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:05:41.322 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a.service: Consumed 2.211s CPU time. 2024-05-27T07:05:41.322 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:40 smithi012 systemd[1]: Starting Ceph mon.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:05:41.580 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 podman[31454]: 2024-05-27 07:05:41.23508876 +0000 UTC m=+0.036237212 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:05:41.580 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 podman[31454]: 2024-05-27 07:05:41.341208785 +0000 UTC m=+0.142391572 container create f4a74345a656417d7e759b6600c0c5dcfb2fff6f335e12494f43eab188220ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:05:41.580 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 podman[31454]: 2024-05-27 07:05:41.577987593 +0000 UTC m=+0.379136151 container init f4a74345a656417d7e759b6600c0c5dcfb2fff6f335e12494f43eab188220ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:05:41.580 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 podman[31454]: 2024-05-27 07:05:41.588756871 +0000 UTC m=+0.389905366 container start f4a74345a656417d7e759b6600c0c5dcfb2fff6f335e12494f43eab188220ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T07:05:41.720 INFO:teuthology.orchestra.run.smithi012.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-27T07:05:41.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: pidfile_write: ignore empty --pid-file 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: load: jerasure load: lrc 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Git sha 0 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: DB SUMMARY 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: DB Session ID: 5ITLX6VKEABBTIEVOLJL 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-27T07:05:41.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103978 ; 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 bash[31454]: f4a74345a656417d7e759b6600c0c5dcfb2fff6f335e12494f43eab188220ae2 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.env: 0x5578a6181500 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.info_log: 0x5578d6850040 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.statistics: (nil) 2024-05-27T07:05:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.use_fsync: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.db_log_dir: 2024-05-27T07:05:41.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.wal_dir: 2024-05-27T07:05:41.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:05:41.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:05:41.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:05:41.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:05:41.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:05:41.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.write_buffer_manager: 0x5578d671a8c0 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.unordered_write: 0 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:05:41.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.row_cache: None 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.wal_filter: None 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.wal_compression: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:05:41.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:05:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_open_files: -1 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Compression algorithms supported: 2024-05-27T07:05:41.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kZSTD supported: 0 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.merge_operator: 2024-05-27T07:05:41.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_filter: None 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5578d6850200) 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_type: 0 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_index_type: 0 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_shortening: 1 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:05:41.852 INFO:journalctl@ceph.mon.a.smithi012.stdout: checksum: 4 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: no_block_cache: 0 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache: 0x5578d674d1f0 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_options: 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: capacity : 536870912 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_shard_bits : 4 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: strict_capacity_limit : 0 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_compressed: (nil) 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: persistent_cache: (nil) 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size: 4096 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size_deviation: 10 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_restart_interval: 16 2024-05-27T07:05:41.853 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_block_restart_interval: 1 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: metadata_block_size: 4096 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: partition_filters: 0 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: use_delta_encoding: 1 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: filter_policy: bloomfilter 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: whole_key_filtering: 1 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: verify_compression: 0 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: format_version: 5 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_index_compression: 1 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_align: 0 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: max_auto_readahead_size: 262144 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: prepopulate_block_cache: 0 2024-05-27T07:05:41.854 INFO:journalctl@ceph.mon.a.smithi012.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression: NoCompression 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.num_levels: 7 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:05:41.855 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:05:41.857 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:05:41.858 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:05:41.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:05:41.860 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.ttl: 2592000 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:05:41.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b6a2b57c-da79-4170-b4a6-bee14e31aab3 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793541720666, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793541722557, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99213, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97159, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13895, "raw_average_key_size": 49, "raw_value_size": 89568, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716793541, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b6a2b57c-da79-4170-b4a6-bee14e31aab3", "db_session_id": "5ITLX6VKEABBTIEVOLJL", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793541722801, "job": 1, "event": "recovery_finished"} 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-27T07:05:41.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5578d6731c00 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: DB pointer 0x5578db524000 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** DB Stats ** 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-05-27T07:05:41.863 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: L0 2/0 98.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 88.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: Sum 2/0 98.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 88.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 88.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 88.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-05-27T07:05:41.864 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative compaction: 0.00 GB write, 13.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval compaction: 0.00 GB write, 13.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache BinnedLRUCache@0x5578d674d1f0#2 capacity: 512.00 MB usage: 1.41 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.91 KB,0.000172853%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 systemd[1]: Started Ceph mon.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:05:41.865 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: starting mon.a rank 0 at public addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] at bind addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???) e1 preinit fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???).mds e1 new map 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???).mds e1 print_map 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout: e1 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout: legacy client fscid: -1 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout: No filesystems configured 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:05:41.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: monmap epoch 1 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: last_changed 2024-05-27T07:05:19.787338+0000 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: created 2024-05-27T07:05:19.787338+0000 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: min_mon_release 19 (squid) 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: election_strategy: 1 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: fsmap 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T07:05:41.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:41 smithi012 ceph-mon[31473]: mgrmap e1: no daemons active 2024-05-27T07:05:43.226 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.493+0000 7f8d91c00700 1 Processor -- start 2024-05-27T07:05:43.226 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.493+0000 7f8d91c00700 1 -- start start 2024-05-27T07:05:43.226 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.493+0000 7f8d91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c109d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:43.226 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.493+0000 7f8d91c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c10a350 con 0x7f8d8c109950 2024-05-27T07:05:43.226 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.494+0000 7f8d8aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.494+0000 7f8d8aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c109d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50206/0 (socket says 172.21.15.12:50206) 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.494+0000 7f8d8aa00700 1 -- 172.21.15.12:0/144329982 learned_addr learned my addr 172.21.15.12:0/144329982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.495+0000 7f8d8aa00700 1 -- 172.21.15.12:0/144329982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c10ab20 con 0x7f8d8c109950 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.495+0000 7f8d8aa00700 1 --2- 172.21.15.12:0/144329982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c109d60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0098a0 tx=0x7f8d7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9659703d3e4246 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.496+0000 7f8d89600700 1 -- 172.21.15.12:0/144329982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d7c004160 con 0x7f8d8c109950 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.496+0000 7f8d89600700 1 -- 172.21.15.12:0/144329982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f8d7c004300 con 0x7f8d8c109950 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.496+0000 7f8d89600700 1 -- 172.21.15.12:0/144329982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d7c0119e0 con 0x7f8d8c109950 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.497+0000 7f8d91c00700 1 -- 172.21.15.12:0/144329982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 msgr2=0x7f8d8c109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.497+0000 7f8d91c00700 1 --2- 172.21.15.12:0/144329982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c109d60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0098a0 tx=0x7f8d7c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.497+0000 7f8d91c00700 1 -- 172.21.15.12:0/144329982 shutdown_connections 2024-05-27T07:05:43.227 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.497+0000 7f8d91c00700 1 --2- 172.21.15.12:0/144329982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c109d60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.497+0000 7f8d91c00700 1 -- 172.21.15.12:0/144329982 >> 172.21.15.12:0/144329982 conn(0x7f8d8c105050 msgr2=0x7f8d8c1074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.497+0000 7f8d91c00700 1 -- 172.21.15.12:0/144329982 shutdown_connections 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.497+0000 7f8d91c00700 1 -- 172.21.15.12:0/144329982 wait complete. 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.498+0000 7f8d91c00700 1 Processor -- start 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.498+0000 7f8d91c00700 1 -- start start 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.499+0000 7f8d91c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.499+0000 7f8d91c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c10ded0 con 0x7f8d8c109950 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.499+0000 7f8d8aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.499+0000 7f8d8aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c19dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50214/0 (socket says 172.21.15.12:50214) 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.499+0000 7f8d8aa00700 1 -- 172.21.15.12:0/38400055 learned_addr learned my addr 172.21.15.12:0/38400055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.500+0000 7f8d8aa00700 1 -- 172.21.15.12:0/38400055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c19e210 con 0x7f8d8c109950 2024-05-27T07:05:43.228 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.500+0000 7f8d8aa00700 1 --2- 172.21.15.12:0/38400055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c19dcd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009fb0 tx=0x7f8d7c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.500+0000 7f8d83400700 1 -- 172.21.15.12:0/38400055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d7c0038b0 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.500+0000 7f8d83400700 1 -- 172.21.15.12:0/38400055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f8d7c003a50 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.500+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c19e4a0 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.500+0000 7f8d83400700 1 -- 172.21.15.12:0/38400055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d7c019550 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.501+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c1a10b0 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.501+0000 7f8d83400700 1 -- 172.21.15.12:0/38400055 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d7c019be0 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.501+0000 7f8d83400700 1 -- 172.21.15.12:0/38400055 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8d7c024730 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.502+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8c064e10 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.505+0000 7f8d83400700 1 -- 172.21.15.12:0/38400055 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8d7c032070 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.584+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f8d8c1a1300 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.588+0000 7f8d83400700 1 -- 172.21.15.12:0/38400055 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f8d7c024930 con 0x7f8d8c109950 2024-05-27T07:05:43.229 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.589+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 msgr2=0x7f8d8c19dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:43.230 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.589+0000 7f8d91c00700 1 --2- 172.21.15.12:0/38400055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c19dcd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009fb0 tx=0x7f8d7c011a60 comp rx=0 tx=0).stop 2024-05-27T07:05:43.230 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.589+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 shutdown_connections 2024-05-27T07:05:43.230 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.589+0000 7f8d91c00700 1 --2- 172.21.15.12:0/38400055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8d8c109950 0x7f8d8c19dcd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.230 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.589+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 >> 172.21.15.12:0/38400055 conn(0x7f8d8c105050 msgr2=0x7f8d8c106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:43.230 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.590+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 shutdown_connections 2024-05-27T07:05:43.230 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:42.590+0000 7f8d91c00700 1 -- 172.21.15.12:0/38400055 wait complete. 2024-05-27T07:05:43.230 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-27T07:05:43.250 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:05:43.250 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mgr... 2024-05-27T07:05:43.250 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-27T07:05:43.251 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-27T07:05:43.678 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T07:05:43.679 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-05-27T07:05:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:43 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/38400055' entity='client.admin' 2024-05-27T07:05:43.955 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968.target.wants/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@.service. 2024-05-27T07:05:44.342 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:43 smithi012 systemd[1]: Starting Ceph mgr.x for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:05:44.620 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:44 smithi012 podman[31699]: 2024-05-27 07:05:44.283828608 +0000 UTC m=+0.041940716 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:05:44.620 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:44 smithi012 podman[31699]: 2024-05-27 07:05:44.378662287 +0000 UTC m=+0.136774411 container create 0587ea20bc322350ce3c311a3ab3e53a21e46e7aeac9155f19fda338b71452f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:05:44.620 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:44 smithi012 podman[31699]: 2024-05-27 07:05:44.615563797 +0000 UTC m=+0.373675964 container init 0587ea20bc322350ce3c311a3ab3e53a21e46e7aeac9155f19fda338b71452f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:05:44.620 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:44 smithi012 podman[31699]: 2024-05-27 07:05:44.628037123 +0000 UTC m=+0.386149250 container start 0587ea20bc322350ce3c311a3ab3e53a21e46e7aeac9155f19fda338b71452f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:05:44.730 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:05:44.730 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-05-27T07:05:44.749 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:05:44.749 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-05-27T07:05:44.749 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-05-27T07:05:44.749 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T07:05:44.768 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:05:44.769 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-05-27T07:05:44.785 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:05:44.786 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-05-27T07:05:44.786 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-05-27T07:05:44.786 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-27T07:05:44.786 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr to start... 2024-05-27T07:05:44.786 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr... 2024-05-27T07:05:44.940 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:44 smithi012 bash[31699]: 0587ea20bc322350ce3c311a3ab3e53a21e46e7aeac9155f19fda338b71452f9 2024-05-27T07:05:44.940 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:44 smithi012 systemd[1]: Started Ceph mgr.x for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:05:45.216 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:44.965+0000 7fd9e9f22200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:05:45.546 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:45.240+0000 7fd9e9f22200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:05:45.546 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:45.391+0000 7fd9e9f22200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:05:45.818 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:45 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2235907799' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:05:46.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:45.825+0000 7fd9e9f22200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:05:46.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:45.947+0000 7fd9e9f22200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:05:46.290 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:46.292 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:05:46.292 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:05:46.292 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:05:46.292 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:05:46.292 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:05:46.292 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:05:46.292 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:46.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:05:46.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:46.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:05:46.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:05:46.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:05:46.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:05:46.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:05:24.015823+0000", 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:46.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.462+0000 7f632e800700 1 Processor -- start 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.462+0000 7f632e800700 1 -- start start 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.463+0000 7f632e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f6328109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.463+0000 7f632e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f632810a3c0 con 0x7f63281099c0 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.463+0000 7f6327400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f6328109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.463+0000 7f6327400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f6328109dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50242/0 (socket says 172.21.15.12:50242) 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.463+0000 7f6327400700 1 -- 172.21.15.12:0/3128887464 learned_addr learned my addr 172.21.15.12:0/3128887464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.464+0000 7f6327400700 1 -- 172.21.15.12:0/3128887464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632810ab40 con 0x7f63281099c0 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.464+0000 7f6327400700 1 --2- 172.21.15.12:0/3128887464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f6328109dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f63180098a0 tx=0x7f6318009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4add29dad08408c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.465+0000 7f6326000700 1 -- 172.21.15.12:0/3128887464 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6318004160 con 0x7f63281099c0 2024-05-27T07:05:46.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.465+0000 7f6326000700 1 -- 172.21.15.12:0/3128887464 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6318004300 con 0x7f63281099c0 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.465+0000 7f6326000700 1 -- 172.21.15.12:0/3128887464 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63180119e0 con 0x7f63281099c0 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.466+0000 7f632e800700 1 -- 172.21.15.12:0/3128887464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 msgr2=0x7f6328109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.466+0000 7f632e800700 1 --2- 172.21.15.12:0/3128887464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f6328109dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f63180098a0 tx=0x7f6318009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.466+0000 7f632e800700 1 -- 172.21.15.12:0/3128887464 shutdown_connections 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.466+0000 7f632e800700 1 --2- 172.21.15.12:0/3128887464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f6328109dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.466+0000 7f632e800700 1 -- 172.21.15.12:0/3128887464 >> 172.21.15.12:0/3128887464 conn(0x7f6328105020 msgr2=0x7f63281074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.466+0000 7f632e800700 1 -- 172.21.15.12:0/3128887464 shutdown_connections 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.466+0000 7f632e800700 1 -- 172.21.15.12:0/3128887464 wait complete. 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.467+0000 7f632e800700 1 Processor -- start 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.467+0000 7f632e800700 1 -- start start 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.468+0000 7f632e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f632819fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.468+0000 7f632e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f632810def0 con 0x7f63281099c0 2024-05-27T07:05:46.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.468+0000 7f6327400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f632819fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.468+0000 7f6327400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f632819fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50254/0 (socket says 172.21.15.12:50254) 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.468+0000 7f6327400700 1 -- 172.21.15.12:0/2235907799 learned_addr learned my addr 172.21.15.12:0/2235907799 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.469+0000 7f6327400700 1 -- 172.21.15.12:0/2235907799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63281a0420 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.469+0000 7f6327400700 1 --2- 172.21.15.12:0/2235907799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f632819fee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63180119b0 tx=0x7f6318011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.469+0000 7f631fe00700 1 -- 172.21.15.12:0/2235907799 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6318003970 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.469+0000 7f631fe00700 1 -- 172.21.15.12:0/2235907799 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6318003b10 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.469+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63281a06b0 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.469+0000 7f631fe00700 1 -- 172.21.15.12:0/2235907799 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6318024610 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.469+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63281a32c0 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.470+0000 7f631fe00700 1 -- 172.21.15.12:0/2235907799 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6318024c10 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.470+0000 7f631fe00700 1 -- 172.21.15.12:0/2235907799 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6318019700 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.471+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6328064e10 con 0x7f63281099c0 2024-05-27T07:05:46.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.474+0000 7f631fe00700 1 -- 172.21.15.12:0/2235907799 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f63180094f0 con 0x7f63281099c0 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.554+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f63281a34e0 con 0x7f63281099c0 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.555+0000 7f631fe00700 1 -- 172.21.15.12:0/2235907799 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f631801aae0 con 0x7f63281099c0 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.556+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 msgr2=0x7f632819fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.556+0000 7f632e800700 1 --2- 172.21.15.12:0/2235907799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f632819fee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63180119b0 tx=0x7f6318011a90 comp rx=0 tx=0).stop 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.556+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 shutdown_connections 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.556+0000 7f632e800700 1 --2- 172.21.15.12:0/2235907799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f63281099c0 0x7f632819fee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.556+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 >> 172.21.15.12:0/2235907799 conn(0x7f6328105020 msgr2=0x7f63281058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.557+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 shutdown_connections 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:45.557+0000 7f632e800700 1 -- 172.21.15.12:0/2235907799 wait complete. 2024-05-27T07:05:46.307 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (1/15)... 2024-05-27T07:05:48.051 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:47.602+0000 7fd9e9f22200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:05:48.051 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:47.837+0000 7fd9e9f22200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:05:48.051 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:48.058+0000 7fd9e9f22200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:05:48.740 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:48.468+0000 7fd9e9f22200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:05:48.741 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:48.600+0000 7fd9e9f22200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:05:49.079 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:48.865+0000 7fd9e9f22200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:05:49.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:49.151+0000 7fd9e9f22200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:05:49.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:49 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/919038757' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:05:49.710 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:49.710 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:05:49.710 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:49.711 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:05:49.712 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:05:49.720 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:05:49.721 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:05:49.722 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:05:24.015823+0000", 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.980+0000 7f807be00700 1 Processor -- start 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.980+0000 7f807be00700 1 -- start start 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.981+0000 7f807be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c07d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:49.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.981+0000 7f807be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f807c07d730 con 0x7f807c07edc0 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.981+0000 7f807aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c07d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.981+0000 7f807aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c07d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50262/0 (socket says 172.21.15.12:50262) 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.981+0000 7f807aa00700 1 -- 172.21.15.12:0/1339877164 learned_addr learned my addr 172.21.15.12:0/1339877164 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.982+0000 7f807aa00700 1 -- 172.21.15.12:0/1339877164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c07d8b0 con 0x7f807c07edc0 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.982+0000 7f807aa00700 1 --2- 172.21.15.12:0/1339877164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c07d1f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f806c0098a0 tx=0x7f806c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49ed04d7a82a55d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.983+0000 7f8079600700 1 -- 172.21.15.12:0/1339877164 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f806c004160 con 0x7f807c07edc0 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.983+0000 7f8079600700 1 -- 172.21.15.12:0/1339877164 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f806c004300 con 0x7f807c07edc0 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.983+0000 7f8079600700 1 -- 172.21.15.12:0/1339877164 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f806c0119e0 con 0x7f807c07edc0 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.983+0000 7f807be00700 1 -- 172.21.15.12:0/1339877164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 msgr2=0x7f807c07d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.983+0000 7f807be00700 1 --2- 172.21.15.12:0/1339877164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c07d1f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f806c0098a0 tx=0x7f806c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.984+0000 7f807be00700 1 -- 172.21.15.12:0/1339877164 shutdown_connections 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.984+0000 7f807be00700 1 --2- 172.21.15.12:0/1339877164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c07d1f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:49.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.984+0000 7f807be00700 1 -- 172.21.15.12:0/1339877164 >> 172.21.15.12:0/1339877164 conn(0x7f807c104e50 msgr2=0x7f807c1072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.984+0000 7f807be00700 1 -- 172.21.15.12:0/1339877164 shutdown_connections 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.984+0000 7f807be00700 1 -- 172.21.15.12:0/1339877164 wait complete. 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.985+0000 7f807be00700 1 Processor -- start 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.985+0000 7f807be00700 1 -- start start 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.986+0000 7f807be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.986+0000 7f807be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f807c10fd10 con 0x7f807c07edc0 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.986+0000 7f807aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.986+0000 7f807aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c19fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50268/0 (socket says 172.21.15.12:50268) 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.986+0000 7f807aa00700 1 -- 172.21.15.12:0/919038757 learned_addr learned my addr 172.21.15.12:0/919038757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.986+0000 7f807aa00700 1 -- 172.21.15.12:0/919038757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c1a01f0 con 0x7f807c07edc0 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.987+0000 7f807aa00700 1 --2- 172.21.15.12:0/919038757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c19fcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f806c0119b0 tx=0x7f806c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:49.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.987+0000 7f8073400700 1 -- 172.21.15.12:0/919038757 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f806c003af0 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.987+0000 7f8073400700 1 -- 172.21.15.12:0/919038757 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f806c01a8a0 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.987+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807c1a0480 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.987+0000 7f8073400700 1 -- 172.21.15.12:0/919038757 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f806c024830 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.987+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807c1a3090 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.987+0000 7f8073400700 1 -- 172.21.15.12:0/919038757 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f806c022020 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.988+0000 7f8073400700 1 -- 172.21.15.12:0/919038757 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f806c0193c0 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.988+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807c064e10 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:48.993+0000 7f8073400700 1 -- 172.21.15.12:0/919038757 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f806c042ec0 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.087+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f807c1a32b0 con 0x7f807c07edc0 2024-05-27T07:05:49.726 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.088+0000 7f8073400700 1 -- 172.21.15.12:0/919038757 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f806c031020 con 0x7f807c07edc0 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.089+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 msgr2=0x7f807c19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.089+0000 7f807be00700 1 --2- 172.21.15.12:0/919038757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c19fcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f806c0119b0 tx=0x7f806c011a90 comp rx=0 tx=0).stop 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.089+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 shutdown_connections 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.089+0000 7f807be00700 1 --2- 172.21.15.12:0/919038757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f807c07edc0 0x7f807c19fcb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.089+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 >> 172.21.15.12:0/919038757 conn(0x7f807c104e50 msgr2=0x7f807c1066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.089+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 shutdown_connections 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:49.089+0000 7f807be00700 1 -- 172.21.15.12:0/919038757 wait complete. 2024-05-27T07:05:49.727 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (2/15)... 2024-05-27T07:05:50.281 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:49.987+0000 7fd9e9f22200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:05:50.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:50.284+0000 7fd9e9f22200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:05:51.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:50.864+0000 7fd9e9f22200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:05:51.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:50.996+0000 7fd9e9f22200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:05:51.840 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:51 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:51.475+0000 7fd9e9f22200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:05:52.171 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:51 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:51.953+0000 7fd9e9f22200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:05:52.171 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:52.086+0000 7fd9e9f22200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:05:52.422 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:52.430+0000 7fd9e9f22200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:05:52.768 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:52 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1675518147' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:05:52.768 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:52.638+0000 7fd9e9f22200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:05:52.768 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:52.775+0000 7fd9e9f22200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:05:53.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:53.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:05:53.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:05:53.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:05:53.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:05:53.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:05:24.015823+0000", 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:53.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.408+0000 7f8069400700 1 Processor -- start 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.408+0000 7f8069400700 1 -- start start 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.409+0000 7f8069400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806407d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.409+0000 7f8069400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f806407d800 con 0x7f806407ee00 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.409+0000 7f8063e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806407d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.409+0000 7f8063e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806407d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55510/0 (socket says 172.21.15.12:55510) 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.409+0000 7f8063e00700 1 -- 172.21.15.12:0/3675099486 learned_addr learned my addr 172.21.15.12:0/3675099486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.410+0000 7f8063e00700 1 -- 172.21.15.12:0/3675099486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806407d980 con 0x7f806407ee00 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.410+0000 7f8063e00700 1 --2- 172.21.15.12:0/3675099486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806407d230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f80540098a0 tx=0x7f8054009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7502896f9f1188ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.410+0000 7f8062a00700 1 -- 172.21.15.12:0/3675099486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8054004160 con 0x7f806407ee00 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.411+0000 7f8062a00700 1 -- 172.21.15.12:0/3675099486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f8054004300 con 0x7f806407ee00 2024-05-27T07:05:53.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.411+0000 7f8062a00700 1 -- 172.21.15.12:0/3675099486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80540119e0 con 0x7f806407ee00 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.411+0000 7f8069400700 1 -- 172.21.15.12:0/3675099486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 msgr2=0x7f806407d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.411+0000 7f8069400700 1 --2- 172.21.15.12:0/3675099486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806407d230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f80540098a0 tx=0x7f8054009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.412+0000 7f8069400700 1 -- 172.21.15.12:0/3675099486 shutdown_connections 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.412+0000 7f8069400700 1 --2- 172.21.15.12:0/3675099486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806407d230 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.412+0000 7f8069400700 1 -- 172.21.15.12:0/3675099486 >> 172.21.15.12:0/3675099486 conn(0x7f8064104e50 msgr2=0x7f80641072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.412+0000 7f8069400700 1 -- 172.21.15.12:0/3675099486 shutdown_connections 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.412+0000 7f8069400700 1 -- 172.21.15.12:0/3675099486 wait complete. 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.413+0000 7f8069400700 1 Processor -- start 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.413+0000 7f8069400700 1 -- start start 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.413+0000 7f8069400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.413+0000 7f8069400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f806410fd30 con 0x7f806407ee00 2024-05-27T07:05:53.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.414+0000 7f8063e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.414+0000 7f8063e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806419fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55520/0 (socket says 172.21.15.12:55520) 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.414+0000 7f8063e00700 1 -- 172.21.15.12:0/1675518147 learned_addr learned my addr 172.21.15.12:0/1675518147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.414+0000 7f8063e00700 1 -- 172.21.15.12:0/1675518147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80641a0210 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.414+0000 7f8063e00700 1 --2- 172.21.15.12:0/1675518147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806419fcd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f80540116a0 tx=0x7f8054011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.415+0000 7f8060c00700 1 -- 172.21.15.12:0/1675518147 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80540039f0 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.415+0000 7f8060c00700 1 -- 172.21.15.12:0/1675518147 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f8054003b90 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.415+0000 7f8060c00700 1 -- 172.21.15.12:0/1675518147 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f805401a600 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.415+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80641a04a0 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.415+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80641a30b0 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.415+0000 7f8060c00700 1 -- 172.21.15.12:0/1675518147 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f805401ac90 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.415+0000 7f8060c00700 1 -- 172.21.15.12:0/1675518147 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8054019700 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.416+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8064064e10 con 0x7f806407ee00 2024-05-27T07:05:53.341 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.419+0000 7f8060c00700 1 -- 172.21.15.12:0/1675518147 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8054024a50 con 0x7f806407ee00 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.499+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f80641a33a0 con 0x7f806407ee00 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.499+0000 7f8060c00700 1 -- 172.21.15.12:0/1675518147 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f8054024430 con 0x7f806407ee00 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.501+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 msgr2=0x7f806419fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.501+0000 7f8069400700 1 --2- 172.21.15.12:0/1675518147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806419fcd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f80540116a0 tx=0x7f8054011a60 comp rx=0 tx=0).stop 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.501+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 shutdown_connections 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.501+0000 7f8069400700 1 --2- 172.21.15.12:0/1675518147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f806407ee00 0x7f806419fcd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.501+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 >> 172.21.15.12:0/1675518147 conn(0x7f8064104e50 msgr2=0x7f80641066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.501+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 shutdown_connections 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:52.501+0000 7f8069400700 1 -- 172.21.15.12:0/1675518147 wait complete. 2024-05-27T07:05:53.342 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (3/15)... 2024-05-27T07:05:53.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:53.211+0000 7fd9e9f22200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:05:53.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:53.349+0000 7fd9e9f22200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:05:54.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:54.149+0000 7fd9e9f22200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:05:54.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:05:54.447+0000 7fd9e9f22200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:05:54.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: Activating manager daemon x 2024-05-27T07:05:54.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: mgrmap e2: x(active, starting, since 0.0137472s) 2024-05-27T07:05:54.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:05:54.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:05:54.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:05:54.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:05:54.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:05:54.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: Manager daemon x is now available 2024-05-27T07:05:54.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:54 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:05:55.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:55 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' 2024-05-27T07:05:55.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:55 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:05:55.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:55 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' 2024-05-27T07:05:55.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:55 smithi012 ceph-mon[31473]: from='mgr.14100 172.21.15.12:0/2009566161' entity='mgr.x' 2024-05-27T07:05:55.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:55 smithi012 ceph-mon[31473]: mgrmap e3: x(active, since 1.0184s) 2024-05-27T07:05:56.711 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:56 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2117721044' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:05:57.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:57.003 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:05:57.004 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:57.006 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:05:57.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:05:57.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:57.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:05:57.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:57.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:05:57.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:05:57.007 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:05:57.015 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:05:57.016 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:05:24.015823+0000", 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.027+0000 7f4c68c00700 1 Processor -- start 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.027+0000 7f4c68c00700 1 -- start start 2024-05-27T07:05:57.017 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.028+0000 7f4c68c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6410c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.028+0000 7f4c68c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6407ce30 con 0x7f4c6410a400 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.028+0000 7f4c61e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6410c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.028+0000 7f4c61e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6410c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55588/0 (socket says 172.21.15.12:55588) 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.028+0000 7f4c61e00700 1 -- 172.21.15.12:0/1587474145 learned_addr learned my addr 172.21.15.12:0/1587474145 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.029+0000 7f4c61e00700 1 -- 172.21.15.12:0/1587474145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6410cd60 con 0x7f4c6410a400 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.029+0000 7f4c61e00700 1 --2- 172.21.15.12:0/1587474145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6410c820 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4c540098a0 tx=0x7f4c54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d924e5bd263dd081 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.029+0000 7f4c60a00700 1 -- 172.21.15.12:0/1587474145 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c54004160 con 0x7f4c6410a400 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.030+0000 7f4c60a00700 1 -- 172.21.15.12:0/1587474145 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f4c54004300 con 0x7f4c6410a400 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.030+0000 7f4c60a00700 1 -- 172.21.15.12:0/1587474145 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c540119e0 con 0x7f4c6410a400 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.030+0000 7f4c68c00700 1 -- 172.21.15.12:0/1587474145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 msgr2=0x7f4c6410c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.030+0000 7f4c68c00700 1 --2- 172.21.15.12:0/1587474145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6410c820 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4c540098a0 tx=0x7f4c54009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:57.018 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.031+0000 7f4c68c00700 1 -- 172.21.15.12:0/1587474145 shutdown_connections 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.031+0000 7f4c68c00700 1 --2- 172.21.15.12:0/1587474145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6410c820 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.031+0000 7f4c68c00700 1 -- 172.21.15.12:0/1587474145 >> 172.21.15.12:0/1587474145 conn(0x7f4c64103ec0 msgr2=0x7f4c64106310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.031+0000 7f4c68c00700 1 -- 172.21.15.12:0/1587474145 shutdown_connections 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.031+0000 7f4c68c00700 1 -- 172.21.15.12:0/1587474145 wait complete. 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.032+0000 7f4c68c00700 1 Processor -- start 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.032+0000 7f4c68c00700 1 -- start start 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.033+0000 7f4c68c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6419dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.033+0000 7f4c68c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6410dcc0 con 0x7f4c6410a400 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.033+0000 7f4c61e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6419dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.033+0000 7f4c61e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6419dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55596/0 (socket says 172.21.15.12:55596) 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.033+0000 7f4c61e00700 1 -- 172.21.15.12:0/2117721044 learned_addr learned my addr 172.21.15.12:0/2117721044 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.033+0000 7f4c61e00700 1 -- 172.21.15.12:0/2117721044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6419e180 con 0x7f4c6410a400 2024-05-27T07:05:57.019 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.034+0000 7f4c61e00700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6419dc40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c540119b0 tx=0x7f4c54011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.034+0000 7f4c5aa00700 1 -- 172.21.15.12:0/2117721044 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c54003af0 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.034+0000 7f4c5aa00700 1 -- 172.21.15.12:0/2117721044 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f4c5401a8a0 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.034+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6419e410 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.034+0000 7f4c5aa00700 1 -- 172.21.15.12:0/2117721044 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c54024830 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.034+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c641a1020 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.035+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c64064e10 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.035+0000 7f4c5aa00700 1 -- 172.21.15.12:0/2117721044 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 3) ==== 50105+0+0 (secure 0 0 0) 0x7f4c54022020 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.036+0000 7f4c5aa00700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f4c4803e410 0x7f4c48040920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.036+0000 7f4c5aa00700 1 -- 172.21.15.12:0/2117721044 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4c540532c0 con 0x7f4c6410a400 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.037+0000 7f4c61400700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f4c4803e410 0x7f4c48040920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.038+0000 7f4c61400700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f4c4803e410 0x7f4c48040920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4c50006fd0 tx=0x7f4c50006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:57.020 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.043+0000 7f4c5aa00700 1 -- 172.21.15.12:0/2117721044 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4c5400f2e0 con 0x7f4c6410a400 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.361+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4c641a1240 con 0x7f4c6410a400 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.362+0000 7f4c5aa00700 1 -- 172.21.15.12:0/2117721044 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4c5401aa50 con 0x7f4c6410a400 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f4c4803e410 msgr2=0x7f4c48040920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f4c4803e410 0x7f4c48040920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4c50006fd0 tx=0x7f4c50006e40 comp rx=0 tx=0).stop 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 msgr2=0x7f4c6419dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6419dc40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c540119b0 tx=0x7f4c54011a90 comp rx=0 tx=0).stop 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 shutdown_connections 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f4c4803e410 0x7f4c48040920 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 --2- 172.21.15.12:0/2117721044 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c6410a400 0x7f4c6419dc40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.365+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 >> 172.21.15.12:0/2117721044 conn(0x7f4c64103ec0 msgr2=0x7f4c641046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.366+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 shutdown_connections 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:56.366+0000 7f4c68c00700 1 -- 172.21.15.12:0/2117721044 wait complete. 2024-05-27T07:05:57.021 INFO:teuthology.orchestra.run.smithi012.stdout:mgr is available 2024-05-27T07:05:57.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:57 smithi012 ceph-mon[31473]: mgrmap e4: x(active, since 2s) 2024-05-27T07:05:58.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:58 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1553553494' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T07:05:58.630 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:58.630 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T07:05:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:05:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T07:05:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T07:05:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T07:05:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T07:05:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T07:05:58.631 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.722+0000 7f26d1000700 1 Processor -- start 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.722+0000 7f26d1000700 1 -- start start 2024-05-27T07:05:58.636 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.722+0000 7f26d1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc112b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.722+0000 7f26d1000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc113040 con 0x7f26cc1106e0 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.723+0000 7f26cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc112b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.723+0000 7f26cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc112b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55606/0 (socket says 172.21.15.12:55606) 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.723+0000 7f26cb400700 1 -- 172.21.15.12:0/1416688677 learned_addr learned my addr 172.21.15.12:0/1416688677 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.724+0000 7f26cb400700 1 -- 172.21.15.12:0/1416688677 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26cc1131c0 con 0x7f26cc1106e0 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.724+0000 7f26cb400700 1 --2- 172.21.15.12:0/1416688677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc112b00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0098a0 tx=0x7f26bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1fc6108f8c7e60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.724+0000 7f26caa00700 1 -- 172.21.15.12:0/1416688677 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26bc004160 con 0x7f26cc1106e0 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.725+0000 7f26caa00700 1 -- 172.21.15.12:0/1416688677 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f26bc004300 con 0x7f26cc1106e0 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.725+0000 7f26caa00700 1 -- 172.21.15.12:0/1416688677 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26bc0119e0 con 0x7f26cc1106e0 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.725+0000 7f26d1000700 1 -- 172.21.15.12:0/1416688677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 msgr2=0x7f26cc112b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:58.637 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.725+0000 7f26d1000700 1 --2- 172.21.15.12:0/1416688677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc112b00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0098a0 tx=0x7f26bc009bb0 comp rx=0 tx=0).stop 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.725+0000 7f26d1000700 1 -- 172.21.15.12:0/1416688677 shutdown_connections 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.725+0000 7f26d1000700 1 --2- 172.21.15.12:0/1416688677 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc112b00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.726+0000 7f26d1000700 1 -- 172.21.15.12:0/1416688677 >> 172.21.15.12:0/1416688677 conn(0x7f26cc07ef80 msgr2=0x7f26cc0813a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.726+0000 7f26d1000700 1 -- 172.21.15.12:0/1416688677 shutdown_connections 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.726+0000 7f26d1000700 1 -- 172.21.15.12:0/1416688677 wait complete. 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.727+0000 7f26d1000700 1 Processor -- start 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.727+0000 7f26d1000700 1 -- start start 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.727+0000 7f26d1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc1a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.727+0000 7f26d1000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc114130 con 0x7f26cc1106e0 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.728+0000 7f26cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc1a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:58.638 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.728+0000 7f26cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc1a4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55618/0 (socket says 172.21.15.12:55618) 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.728+0000 7f26cb400700 1 -- 172.21.15.12:0/1553553494 learned_addr learned my addr 172.21.15.12:0/1553553494 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.728+0000 7f26cb400700 1 -- 172.21.15.12:0/1553553494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26cc1a45a0 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.729+0000 7f26cb400700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc1a4060 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0116a0 tx=0x7f26bc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.729+0000 7f26c8c00700 1 -- 172.21.15.12:0/1553553494 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26bc0039f0 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.729+0000 7f26c8c00700 1 -- 172.21.15.12:0/1553553494 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f26bc003b90 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.729+0000 7f26c8c00700 1 -- 172.21.15.12:0/1553553494 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26bc01a600 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.729+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26cc1a4830 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.729+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26cc1a7440 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.730+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26cc064e10 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.730+0000 7f26c8c00700 1 -- 172.21.15.12:0/1553553494 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f26bc01a8b0 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.731+0000 7f26c8c00700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f26b003e4e0 0x7f26b00409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.731+0000 7f26c8c00700 1 -- 172.21.15.12:0/1553553494 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f26bc053c60 con 0x7f26cc1106e0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.731+0000 7f26c3e00700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f26b003e4e0 0x7f26b00409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:58.639 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.732+0000 7f26c3e00700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f26b003e4e0 0x7f26b00409f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f26b4006fd0 tx=0x7f26b4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:58.640 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.737+0000 7f26c8c00700 1 -- 172.21.15.12:0/1553553494 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f26bc02c350 con 0x7f26cc1106e0 2024-05-27T07:05:58.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.979+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f26cc1a7610 con 0x7f26cc1106e0 2024-05-27T07:05:58.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.982+0000 7f26c8c00700 1 -- 172.21.15.12:0/1553553494 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f26bc018dc0 con 0x7f26cc1106e0 2024-05-27T07:05:58.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.985+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f26b003e4e0 msgr2=0x7f26b00409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:58.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.985+0000 7f26d1000700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f26b003e4e0 0x7f26b00409f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f26b4006fd0 tx=0x7f26b4006e40 comp rx=0 tx=0).stop 2024-05-27T07:05:58.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.985+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 msgr2=0x7f26cc1a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:58.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.985+0000 7f26d1000700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc1a4060 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0116a0 tx=0x7f26bc011a60 comp rx=0 tx=0).stop 2024-05-27T07:05:58.641 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.986+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 shutdown_connections 2024-05-27T07:05:58.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.986+0000 7f26d1000700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f26b003e4e0 0x7f26b00409f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:58.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.986+0000 7f26d1000700 1 --2- 172.21.15.12:0/1553553494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26cc1106e0 0x7f26cc1a4060 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:58.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.986+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 >> 172.21.15.12:0/1553553494 conn(0x7f26cc07ef80 msgr2=0x7f26cc0807c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:58.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.986+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 shutdown_connections 2024-05-27T07:05:58.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:57.986+0000 7f26d1000700 1 -- 172.21.15.12:0/1553553494 wait complete. 2024-05-27T07:05:58.642 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling cephadm module... 2024-05-27T07:05:59.833 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:05:59 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/389772750' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-27T07:06:00.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:59 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: ignoring --setuser ceph since I am not root 2024-05-27T07:06:00.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:05:59 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: ignoring --setgroup ceph since I am not root 2024-05-27T07:06:00.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:00.017+0000 7f370801d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:06:00.269 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.280+0000 7f32bbe00700 1 Processor -- start 2024-05-27T07:06:00.270 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.280+0000 7f32bbe00700 1 -- start start 2024-05-27T07:06:00.270 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.281+0000 7f32bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc10be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:00.270 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.281+0000 7f32bbe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10c450 con 0x7f32bc10ba70 2024-05-27T07:06:00.270 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.281+0000 7f32baa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc10be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:00.270 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.281+0000 7f32baa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc10be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55630/0 (socket says 172.21.15.12:55630) 2024-05-27T07:06:00.271 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.281+0000 7f32baa00700 1 -- 172.21.15.12:0/3843313635 learned_addr learned my addr 172.21.15.12:0/3843313635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:00.271 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.282+0000 7f32baa00700 1 -- 172.21.15.12:0/3843313635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32bc10cb50 con 0x7f32bc10ba70 2024-05-27T07:06:00.271 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.282+0000 7f32baa00700 1 --2- 172.21.15.12:0/3843313635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc10be80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0098a0 tx=0x7f32ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4d1d3d7842b613 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:00.272 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.283+0000 7f32b9600700 1 -- 172.21.15.12:0/3843313635 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ac004160 con 0x7f32bc10ba70 2024-05-27T07:06:00.272 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.283+0000 7f32b9600700 1 -- 172.21.15.12:0/3843313635 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f32ac004300 con 0x7f32bc10ba70 2024-05-27T07:06:00.272 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.283+0000 7f32b9600700 1 -- 172.21.15.12:0/3843313635 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ac0119e0 con 0x7f32bc10ba70 2024-05-27T07:06:00.272 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.284+0000 7f32bbe00700 1 -- 172.21.15.12:0/3843313635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 msgr2=0x7f32bc10be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:00.272 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.284+0000 7f32bbe00700 1 --2- 172.21.15.12:0/3843313635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc10be80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0098a0 tx=0x7f32ac009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:00.272 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.284+0000 7f32bbe00700 1 -- 172.21.15.12:0/3843313635 shutdown_connections 2024-05-27T07:06:00.273 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.284+0000 7f32bbe00700 1 --2- 172.21.15.12:0/3843313635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc10be80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:00.273 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.284+0000 7f32bbe00700 1 -- 172.21.15.12:0/3843313635 >> 172.21.15.12:0/3843313635 conn(0x7f32bc107470 msgr2=0x7f32bc109890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:00.274 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.284+0000 7f32bbe00700 1 -- 172.21.15.12:0/3843313635 shutdown_connections 2024-05-27T07:06:00.274 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.284+0000 7f32bbe00700 1 -- 172.21.15.12:0/3843313635 wait complete. 2024-05-27T07:06:00.275 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.285+0000 7f32bbe00700 1 Processor -- start 2024-05-27T07:06:00.275 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.285+0000 7f32bbe00700 1 -- start start 2024-05-27T07:06:00.275 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.286+0000 7f32bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:00.275 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.286+0000 7f32bbe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10fe10 con 0x7f32bc10ba70 2024-05-27T07:06:00.275 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.286+0000 7f32baa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:00.275 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.286+0000 7f32baa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc19fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55632/0 (socket says 172.21.15.12:55632) 2024-05-27T07:06:00.275 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.286+0000 7f32baa00700 1 -- 172.21.15.12:0/389772750 learned_addr learned my addr 172.21.15.12:0/389772750 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:00.277 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.286+0000 7f32baa00700 1 -- 172.21.15.12:0/389772750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32bc1a0110 con 0x7f32bc10ba70 2024-05-27T07:06:00.277 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.287+0000 7f32baa00700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc19fbd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0119b0 tx=0x7f32ac011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.287+0000 7f32b3400700 1 -- 172.21.15.12:0/389772750 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ac003af0 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.287+0000 7f32b3400700 1 -- 172.21.15.12:0/389772750 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f32ac01a8a0 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.287+0000 7f32b3400700 1 -- 172.21.15.12:0/389772750 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ac024780 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.287+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32bc1a03a0 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.287+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32bc1a2fb0 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.289+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32bc064e10 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.289+0000 7f32b3400700 1 -- 172.21.15.12:0/389772750 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f32ac022020 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.290+0000 7f32b3400700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f32a003e4e0 0x7f32a00409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.290+0000 7f32b3400700 1 -- 172.21.15.12:0/389772750 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f32ac053870 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.290+0000 7f32ba000700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f32a003e4e0 0x7f32a00409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.291+0000 7f32ba000700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f32a003e4e0 0x7f32a00409f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32a8006fd0 tx=0x7f32a8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.295+0000 7f32b3400700 1 -- 172.21.15.12:0/389772750 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f32ac017e90 con 0x7f32bc10ba70 2024-05-27T07:06:00.278 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.588+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f32bc1a3220 con 0x7f32bc10ba70 2024-05-27T07:06:00.279 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.642+0000 7f32b3400700 1 -- 172.21.15.12:0/389772750 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f32ac052670 con 0x7f32bc10ba70 2024-05-27T07:06:00.279 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.645+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f32a003e4e0 msgr2=0x7f32a00409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:00.279 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.645+0000 7f32bbe00700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f32a003e4e0 0x7f32a00409f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32a8006fd0 tx=0x7f32a8006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:00.279 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.645+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 msgr2=0x7f32bc19fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:00.279 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.645+0000 7f32bbe00700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc19fbd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0119b0 tx=0x7f32ac011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:00.280 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.646+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 shutdown_connections 2024-05-27T07:06:00.280 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.646+0000 7f32bbe00700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f32a003e4e0 0x7f32a00409f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:00.280 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.646+0000 7f32bbe00700 1 --2- 172.21.15.12:0/389772750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32bc10ba70 0x7f32bc19fbd0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0119b0 tx=0x7f32ac011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:00.280 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.646+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 >> 172.21.15.12:0/389772750 conn(0x7f32bc107470 msgr2=0x7f32bc108c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:00.280 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.647+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 shutdown_connections 2024-05-27T07:06:00.280 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:05:59.647+0000 7f32bbe00700 1 -- 172.21.15.12:0/389772750 wait complete. 2024-05-27T07:06:00.545 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:00.266+0000 7f370801d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:06:00.545 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:00.414+0000 7f370801d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:06:00.827 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:00 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/389772750' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-27T07:06:00.827 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:00 smithi012 ceph-mon[31473]: mgrmap e5: x(active, since 5s) 2024-05-27T07:06:01.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:00.835+0000 7f370801d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:06:01.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:00.958+0000 7f370801d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:06:01.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:01 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/249943138' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T07:06:01.855 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.918+0000 7f5bdac00700 1 Processor -- start 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.918+0000 7f5bdac00700 1 -- start start 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.919+0000 7f5bdac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd41084a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.919+0000 7f5bdac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd407cdd0 con 0x7f5bd4108090 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.919+0000 7f5bd9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd41084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.919+0000 7f5bd9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd41084a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55660/0 (socket says 172.21.15.12:55660) 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.919+0000 7f5bd9800700 1 -- 172.21.15.12:0/1004869609 learned_addr learned my addr 172.21.15.12:0/1004869609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.920+0000 7f5bd9800700 1 -- 172.21.15.12:0/1004869609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd41089e0 con 0x7f5bd4108090 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.920+0000 7f5bd9800700 1 --2- 172.21.15.12:0/1004869609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd41084a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80098a0 tx=0x7f5bc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cead178851195a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.920+0000 7f5bd3e00700 1 -- 172.21.15.12:0/1004869609 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc8004160 con 0x7f5bd4108090 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.921+0000 7f5bd3e00700 1 -- 172.21.15.12:0/1004869609 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f5bc8004300 con 0x7f5bd4108090 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.921+0000 7f5bd3e00700 1 -- 172.21.15.12:0/1004869609 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc80119e0 con 0x7f5bd4108090 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.922+0000 7f5bdac00700 1 -- 172.21.15.12:0/1004869609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 msgr2=0x7f5bd41084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.922+0000 7f5bdac00700 1 --2- 172.21.15.12:0/1004869609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd41084a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80098a0 tx=0x7f5bc8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.922+0000 7f5bdac00700 1 -- 172.21.15.12:0/1004869609 shutdown_connections 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.922+0000 7f5bdac00700 1 --2- 172.21.15.12:0/1004869609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd41084a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.922+0000 7f5bdac00700 1 -- 172.21.15.12:0/1004869609 >> 172.21.15.12:0/1004869609 conn(0x7f5bd4103d20 msgr2=0x7f5bd4106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:01.857 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.922+0000 7f5bdac00700 1 -- 172.21.15.12:0/1004869609 shutdown_connections 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.922+0000 7f5bdac00700 1 -- 172.21.15.12:0/1004869609 wait complete. 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.924+0000 7f5bdac00700 1 Processor -- start 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.924+0000 7f5bdac00700 1 -- start start 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.925+0000 7f5bdac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.925+0000 7f5bdac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4109a00 con 0x7f5bd4108090 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.925+0000 7f5bd9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.925+0000 7f5bd9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55668/0 (socket says 172.21.15.12:55668) 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.925+0000 7f5bd9800700 1 -- 172.21.15.12:0/249943138 learned_addr learned my addr 172.21.15.12:0/249943138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.926+0000 7f5bd9800700 1 -- 172.21.15.12:0/249943138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd41a01b0 con 0x7f5bd4108090 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.926+0000 7f5bd9800700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd419fc70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80119b0 tx=0x7f5bc8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.926+0000 7f5bd2000700 1 -- 172.21.15.12:0/249943138 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc8003af0 con 0x7f5bd4108090 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.926+0000 7f5bd2000700 1 -- 172.21.15.12:0/249943138 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f5bc801a8a0 con 0x7f5bd4108090 2024-05-27T07:06:01.858 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.926+0000 7f5bd2000700 1 -- 172.21.15.12:0/249943138 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bc8024830 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.926+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd41a0440 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.926+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd41a3050 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.928+0000 7f5bd2000700 1 -- 172.21.15.12:0/249943138 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f5bc8022020 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.928+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd4064e10 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.928+0000 7f5bd2000700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 0x7f5bbc040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.928+0000 7f5bd2000700 1 -- 172.21.15.12:0/249943138 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5bc8053510 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.929+0000 7f5bd8e00700 1 -- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 msgr2=0x7f5bbc040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.929+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 0x7f5bbc040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:00.934+0000 7f5bd2000700 1 -- 172.21.15.12:0/249943138 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5bc800b3e0 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.130+0000 7f5bd8e00700 1 -- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 msgr2=0x7f5bbc040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.130+0000 7f5bd8e00700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 0x7f5bbc040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.229+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f5bd402e8e0 con 0x7f5bd4108090 2024-05-27T07:06:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.230+0000 7f5bd2000700 1 -- 172.21.15.12:0/249943138 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f5bc8029df0 con 0x7f5bd4108090 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 msgr2=0x7f5bbc040a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 0x7f5bbc040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 msgr2=0x7f5bd419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd419fc70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80119b0 tx=0x7f5bc8011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 shutdown_connections 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f5bbc03e580 0x7f5bbc040a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 --2- 172.21.15.12:0/249943138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bd4108090 0x7f5bd419fc70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 >> 172.21.15.12:0/249943138 conn(0x7f5bd4103d20 msgr2=0x7f5bd4193c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.233+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 shutdown_connections 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:01.234+0000 7f5bdac00700 1 -- 172.21.15.12:0/249943138 wait complete. 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-05-27T07:06:01.860 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 5... 2024-05-27T07:06:02.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:02 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:02.615+0000 7f370801d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:06:03.108 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:02 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:02.851+0000 7f370801d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:06:03.108 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:03.071+0000 7f370801d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:06:03.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:03.468+0000 7f370801d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:06:03.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:03.595+0000 7f370801d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:06:04.113 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:03.856+0000 7f370801d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:06:04.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:04 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:04.121+0000 7f370801d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:06:05.251 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:04 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:04.958+0000 7f370801d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:06:05.511 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:05.259+0000 7f370801d200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:06:06.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:05.839+0000 7f370801d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:06:06.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:05.973+0000 7f370801d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:06:06.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:06.453+0000 7f370801d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:06:07.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:06.924+0000 7f370801d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:06:07.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:07.054+0000 7f370801d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:06:07.706 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:07.391+0000 7f370801d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:06:07.707 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:07.579+0000 7f370801d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:06:07.707 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:07.714+0000 7f370801d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:06:08.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:08.124+0000 7f370801d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:06:08.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:08.253+0000 7f370801d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:06:09.334 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:09.043+0000 7f370801d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:06:09.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:09.342+0000 7f370801d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:06:09.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: Active manager daemon x restarted 2024-05-27T07:06:09.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: Activating manager daemon x 2024-05-27T07:06:09.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: osdmap e2: 0 total, 0 up, 0 in 2024-05-27T07:06:09.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: mgrmap e6: x(active, starting, since 0.00510573s) 2024-05-27T07:06:09.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:06:09.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:06:09.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:06:09.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:06:09.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:06:09.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:09 smithi012 ceph-mon[31473]: Manager daemon x is now available 2024-05-27T07:06:10.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: Found migration_current of "None". Setting to last migration. 2024-05-27T07:06:10.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:10.770 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:10.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:10.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:10.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:06:10.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:06:10.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:10.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:10.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:10 smithi012 ceph-mon[31473]: mgrmap e7: x(active, since 1.01874s) 2024-05-27T07:06:11.064 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:06:11.064 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-27T07:06:11.064 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T07:06:11.064 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:06:11.064 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.504+0000 7f0b8ee00700 1 Processor -- start 2024-05-27T07:06:11.064 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.504+0000 7f0b8ee00700 1 -- start start 2024-05-27T07:06:11.065 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.505+0000 7f0b8ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b88109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.505+0000 7f0b8ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8810a190 con 0x7f0b88109790 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.505+0000 7f0b8da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b88109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.505+0000 7f0b8da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b88109ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36716/0 (socket says 172.21.15.12:36716) 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.505+0000 7f0b8da00700 1 -- 172.21.15.12:0/2712643469 learned_addr learned my addr 172.21.15.12:0/2712643469 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.506+0000 7f0b8da00700 1 -- 172.21.15.12:0/2712643469 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8810a960 con 0x7f0b88109790 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.506+0000 7f0b8da00700 1 --2- 172.21.15.12:0/2712643469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b88109ba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0098a0 tx=0x7f0b7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e575b287921f27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.507+0000 7f0b87e00700 1 -- 172.21.15.12:0/2712643469 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b7c004160 con 0x7f0b88109790 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.507+0000 7f0b87e00700 1 -- 172.21.15.12:0/2712643469 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f0b7c004300 con 0x7f0b88109790 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.507+0000 7f0b87e00700 1 -- 172.21.15.12:0/2712643469 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b7c0119e0 con 0x7f0b88109790 2024-05-27T07:06:11.067 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.507+0000 7f0b8ee00700 1 -- 172.21.15.12:0/2712643469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 msgr2=0x7f0b88109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.507+0000 7f0b8ee00700 1 --2- 172.21.15.12:0/2712643469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b88109ba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0098a0 tx=0x7f0b7c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.508+0000 7f0b8ee00700 1 -- 172.21.15.12:0/2712643469 shutdown_connections 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.508+0000 7f0b8ee00700 1 --2- 172.21.15.12:0/2712643469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b88109ba0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.508+0000 7f0b8ee00700 1 -- 172.21.15.12:0/2712643469 >> 172.21.15.12:0/2712643469 conn(0x7f0b88104e50 msgr2=0x7f0b881072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.508+0000 7f0b8ee00700 1 -- 172.21.15.12:0/2712643469 shutdown_connections 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.508+0000 7f0b8ee00700 1 -- 172.21.15.12:0/2712643469 wait complete. 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.509+0000 7f0b8ee00700 1 Processor -- start 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.509+0000 7f0b8ee00700 1 -- start start 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.509+0000 7f0b8ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b8807ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:11.068 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.509+0000 7f0b8ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8810dd10 con 0x7f0b88109790 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.510+0000 7f0b8da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b8807ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.510+0000 7f0b8da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b8807ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36732/0 (socket says 172.21.15.12:36732) 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.510+0000 7f0b8da00700 1 -- 172.21.15.12:0/701655424 learned_addr learned my addr 172.21.15.12:0/701655424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.510+0000 7f0b8da00700 1 -- 172.21.15.12:0/701655424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8807ef60 con 0x7f0b88109790 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.510+0000 7f0b8da00700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b8807ea20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c009fb0 tx=0x7f0b7c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.511+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b7c0038b0 con 0x7f0b88109790 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.511+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f0b7c003a50 con 0x7f0b88109790 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.511+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b7c019550 con 0x7f0b88109790 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.511+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8807d1b0 con 0x7f0b88109790 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.511+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b8807d690 con 0x7f0b88109790 2024-05-27T07:06:11.073 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.513+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f0b7c0196f0 con 0x7f0b88109790 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.513+0000 7f0b86000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.513+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 --> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0b70041150 con 0x7f0b7003e530 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.514+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0b7c053af0 con 0x7f0b88109790 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.514+0000 7f0b8d000700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 msgr2=0x7f0b70040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.514+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.715+0000 7f0b8d000700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 msgr2=0x7f0b70040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:02.715+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:03.116+0000 7f0b8d000700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 msgr2=0x7f0b70040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:03.116+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:03.917+0000 7f0b8d000700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 msgr2=0x7f0b70040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:03.917+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:06:11.074 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:05.519+0000 7f0b8d000700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 msgr2=0x7f0b70040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:11.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:05.519+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T07:06:11.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:08.723+0000 7f0b8d000700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 msgr2=0x7f0b70040a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/579288494 2024-05-27T07:06:11.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:08.723+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-27T07:06:11.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:09.350+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7f0b7c019a30 con 0x7f0b88109790 2024-05-27T07:06:11.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:09.351+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 msgr2=0x7f0b70040a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:06:11.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:09.351+0000 7f0b86000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:11.075 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.357+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f0b7c055690 con 0x7f0b88109790 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.357+0000 7f0b86000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f0b70041e70 0x7f0b70044290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.357+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0b70041150 con 0x7f0b70041e70 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.359+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f0b70041e70 0x7f0b70044290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.361+0000 7f0b8d000700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f0b70041e70 0x7f0b70044290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0b74003a10 tx=0x7f0b740092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.362+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f0b70041150 con 0x7f0b70041e70 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.373+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f0b88064e10 con 0x7f0b70041e70 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.373+0000 7f0b86000700 1 -- 172.21.15.12:0/701655424 <== mgr.14120 v2:172.21.15.12:6800/2115905110 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f0b88064e10 con 0x7f0b70041e70 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f0b70041e70 msgr2=0x7f0b70044290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:11.078 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f0b70041e70 0x7f0b70044290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0b74003a10 tx=0x7f0b740092b0 comp rx=0 tx=0).stop 2024-05-27T07:06:11.079 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 msgr2=0x7f0b8807ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:11.079 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b8807ea20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c009fb0 tx=0x7f0b7c011a60 comp rx=0 tx=0).stop 2024-05-27T07:06:11.079 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 shutdown_connections 2024-05-27T07:06:11.079 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f0b70041e70 0x7f0b70044290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:11.079 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b88109790 0x7f0b8807ea20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:11.079 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 --2- 172.21.15.12:0/701655424 >> [v2:172.21.15.12:6800/579288494,v1:172.21.15.12:6801/579288494] conn(0x7f0b7003e530 0x7f0b70040a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:11.079 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 >> 172.21.15.12:0/701655424 conn(0x7f0b88104e50 msgr2=0x7f0b88106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:11.080 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 shutdown_connections 2024-05-27T07:06:11.080 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:10.374+0000 7f0b8ee00700 1 -- 172.21.15.12:0/701655424 wait complete. 2024-05-27T07:06:11.080 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 5 is available 2024-05-27T07:06:11.080 INFO:teuthology.orchestra.run.smithi012.stdout:Setting orchestrator backend to cephadm... 2024-05-27T07:06:11.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:11 smithi012 ceph-mon[31473]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T07:06:11.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:11 smithi012 ceph-mon[31473]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T07:06:11.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:11 smithi012 ceph-mon[31473]: [27/May/2024:07:06:10] ENGINE Bus STARTING 2024-05-27T07:06:11.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:11 smithi012 ceph-mon[31473]: [27/May/2024:07:06:10] ENGINE Serving on http://172.21.15.12:8765 2024-05-27T07:06:11.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:11 smithi012 ceph-mon[31473]: [27/May/2024:07:06:10] ENGINE Serving on https://172.21.15.12:7150 2024-05-27T07:06:11.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:11 smithi012 ceph-mon[31473]: [27/May/2024:07:06:10] ENGINE Bus STARTED 2024-05-27T07:06:11.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:11 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:12 smithi012 ceph-mon[31473]: mgrmap e8: x(active, since 2s) 2024-05-27T07:06:12.738 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:12 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:12.738 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:12 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:12.747 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.773+0000 7fa06b200700 1 Processor -- start 2024-05-27T07:06:12.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.773+0000 7fa06b200700 1 -- start start 2024-05-27T07:06:12.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.774+0000 7fa06b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06407dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:12.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.774+0000 7fa06b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa06407e0b0 con 0x7fa06407f6b0 2024-05-27T07:06:12.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.774+0000 7fa069e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06407dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.775+0000 7fa069e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06407dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36826/0 (socket says 172.21.15.12:36826) 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.775+0000 7fa069e00700 1 -- 172.21.15.12:0/1437351006 learned_addr learned my addr 172.21.15.12:0/1437351006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.775+0000 7fa069e00700 1 -- 172.21.15.12:0/1437351006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06407e230 con 0x7fa06407f6b0 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.775+0000 7fa069e00700 1 --2- 172.21.15.12:0/1437351006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06407dae0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0580098a0 tx=0x7fa058009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b693d889ee8dfe5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.776+0000 7fa068a00700 1 -- 172.21.15.12:0/1437351006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa058004160 con 0x7fa06407f6b0 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.776+0000 7fa068a00700 1 -- 172.21.15.12:0/1437351006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fa058004300 con 0x7fa06407f6b0 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.776+0000 7fa068a00700 1 -- 172.21.15.12:0/1437351006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0580119e0 con 0x7fa06407f6b0 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.777+0000 7fa06b200700 1 -- 172.21.15.12:0/1437351006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 msgr2=0x7fa06407dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:12.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.777+0000 7fa06b200700 1 --2- 172.21.15.12:0/1437351006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06407dae0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0580098a0 tx=0x7fa058009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.777+0000 7fa06b200700 1 -- 172.21.15.12:0/1437351006 shutdown_connections 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.777+0000 7fa06b200700 1 --2- 172.21.15.12:0/1437351006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06407dae0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.777+0000 7fa06b200700 1 -- 172.21.15.12:0/1437351006 >> 172.21.15.12:0/1437351006 conn(0x7fa064104e10 msgr2=0x7fa0641072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.777+0000 7fa06b200700 1 -- 172.21.15.12:0/1437351006 shutdown_connections 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.777+0000 7fa06b200700 1 -- 172.21.15.12:0/1437351006 wait complete. 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.778+0000 7fa06b200700 1 Processor -- start 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.778+0000 7fa06b200700 1 -- start start 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.778+0000 7fa06b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:12.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.778+0000 7fa06b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa06410fd80 con 0x7fa06407f6b0 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.779+0000 7fa069e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.779+0000 7fa069e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06419fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36842/0 (socket says 172.21.15.12:36842) 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.779+0000 7fa069e00700 1 -- 172.21.15.12:0/944746599 learned_addr learned my addr 172.21.15.12:0/944746599 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.779+0000 7fa069e00700 1 -- 172.21.15.12:0/944746599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0641a0160 con 0x7fa06407f6b0 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.779+0000 7fa069e00700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06419fc20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa0580119b0 tx=0x7fa058011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.779+0000 7fa05ea00700 1 -- 172.21.15.12:0/944746599 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa058003af0 con 0x7fa06407f6b0 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.779+0000 7fa05ea00700 1 -- 172.21.15.12:0/944746599 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fa05801a8a0 con 0x7fa06407f6b0 2024-05-27T07:06:12.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.780+0000 7fa05ea00700 1 -- 172.21.15.12:0/944746599 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa058024830 con 0x7fa06407f6b0 2024-05-27T07:06:12.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.780+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0641a03f0 con 0x7fa06407f6b0 2024-05-27T07:06:12.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.780+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0641a3000 con 0x7fa06407f6b0 2024-05-27T07:06:12.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.781+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa064064e10 con 0x7fa06407f6b0 2024-05-27T07:06:12.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.781+0000 7fa05ea00700 1 -- 172.21.15.12:0/944746599 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fa058022020 con 0x7fa06407f6b0 2024-05-27T07:06:12.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.781+0000 7fa05ea00700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fa04c03e580 0x7fa04c040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:12.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.782+0000 7fa05ea00700 1 -- 172.21.15.12:0/944746599 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa058053890 con 0x7fa06407f6b0 2024-05-27T07:06:12.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.782+0000 7fa069400700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fa04c03e580 0x7fa04c040a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.782+0000 7fa069400700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fa04c03e580 0x7fa04c040a90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa054006fd0 tx=0x7fa054006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:11.788+0000 7fa05ea00700 1 -- 172.21.15.12:0/944746599 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa058018530 con 0x7fa06407f6b0 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.030+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fa064105d20 con 0x7fa04c03e580 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.043+0000 7fa05ea00700 1 -- 172.21.15.12:0/944746599 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa064105d20 con 0x7fa04c03e580 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fa04c03e580 msgr2=0x7fa04c040a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fa04c03e580 0x7fa04c040a90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa054006fd0 tx=0x7fa054006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 msgr2=0x7fa06419fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06419fc20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa0580119b0 tx=0x7fa058011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 shutdown_connections 2024-05-27T07:06:12.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fa04c03e580 0x7fa04c040a90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:12.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 --2- 172.21.15.12:0/944746599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa06407f6b0 0x7fa06419fc20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:12.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 >> 172.21.15.12:0/944746599 conn(0x7fa064104e10 msgr2=0x7fa064105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:12.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 shutdown_connections 2024-05-27T07:06:12.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:12.047+0000 7fa06b200700 1 -- 172.21.15.12:0/944746599 wait complete. 2024-05-27T07:06:13.646 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:13 smithi012 ceph-mon[31473]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:14.293 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-27T07:06:14.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.402+0000 7f9e97200700 1 Processor -- start 2024-05-27T07:06:14.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.402+0000 7f9e97200700 1 -- start start 2024-05-27T07:06:14.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.403+0000 7f9e97200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9007d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:14.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.403+0000 7f9e97200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9007d730 con 0x7f9e9007ed70 2024-05-27T07:06:14.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.403+0000 7f9e8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9007d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:14.294 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.403+0000 7f9e8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9007d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43810/0 (socket says 172.21.15.12:43810) 2024-05-27T07:06:14.296 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.403+0000 7f9e8fe00700 1 -- 172.21.15.12:0/2327780450 learned_addr learned my addr 172.21.15.12:0/2327780450 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:14.296 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.404+0000 7f9e8fe00700 1 -- 172.21.15.12:0/2327780450 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9007d8b0 con 0x7f9e9007ed70 2024-05-27T07:06:14.296 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.404+0000 7f9e8fe00700 1 --2- 172.21.15.12:0/2327780450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9007d1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e800098a0 tx=0x7f9e80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea672bba8d36d5fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.404+0000 7f9e8ea00700 1 -- 172.21.15.12:0/2327780450 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e80004160 con 0x7f9e9007ed70 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.404+0000 7f9e8ea00700 1 -- 172.21.15.12:0/2327780450 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f9e80004300 con 0x7f9e9007ed70 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.404+0000 7f9e8ea00700 1 -- 172.21.15.12:0/2327780450 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e800119e0 con 0x7f9e9007ed70 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.405+0000 7f9e97200700 1 -- 172.21.15.12:0/2327780450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 msgr2=0x7f9e9007d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.405+0000 7f9e97200700 1 --2- 172.21.15.12:0/2327780450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9007d1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e800098a0 tx=0x7f9e80009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.405+0000 7f9e97200700 1 -- 172.21.15.12:0/2327780450 shutdown_connections 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.405+0000 7f9e97200700 1 --2- 172.21.15.12:0/2327780450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9007d1f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.405+0000 7f9e97200700 1 -- 172.21.15.12:0/2327780450 >> 172.21.15.12:0/2327780450 conn(0x7f9e90105020 msgr2=0x7f9e901074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.405+0000 7f9e97200700 1 -- 172.21.15.12:0/2327780450 shutdown_connections 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.405+0000 7f9e97200700 1 -- 172.21.15.12:0/2327780450 wait complete. 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.406+0000 7f9e97200700 1 Processor -- start 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.406+0000 7f9e97200700 1 -- start start 2024-05-27T07:06:14.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.407+0000 7f9e97200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9019fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:14.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.407+0000 7f9e97200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9010ff40 con 0x7f9e9007ed70 2024-05-27T07:06:14.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.407+0000 7f9e8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9019fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.407+0000 7f9e8fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9019fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43818/0 (socket says 172.21.15.12:43818) 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.407+0000 7f9e8fe00700 1 -- 172.21.15.12:0/1973935715 learned_addr learned my addr 172.21.15.12:0/1973935715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.408+0000 7f9e8fe00700 1 -- 172.21.15.12:0/1973935715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e901a0370 con 0x7f9e9007ed70 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.408+0000 7f9e8fe00700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9019fe30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e800119b0 tx=0x7f9e80011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.408+0000 7f9e8cc00700 1 -- 172.21.15.12:0/1973935715 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e80003af0 con 0x7f9e9007ed70 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.408+0000 7f9e8cc00700 1 -- 172.21.15.12:0/1973935715 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f9e8001a8a0 con 0x7f9e9007ed70 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.408+0000 7f9e8cc00700 1 -- 172.21.15.12:0/1973935715 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e80024830 con 0x7f9e9007ed70 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.408+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e901a0600 con 0x7f9e9007ed70 2024-05-27T07:06:14.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.408+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e901a3210 con 0x7f9e9007ed70 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.409+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e90064e10 con 0x7f9e9007ed70 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.410+0000 7f9e8cc00700 1 -- 172.21.15.12:0/1973935715 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f9e80022020 con 0x7f9e9007ed70 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.411+0000 7f9e8cc00700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9e7403e530 0x7f9e74040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.411+0000 7f9e8cc00700 1 -- 172.21.15.12:0/1973935715 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9e80053a20 con 0x7f9e9007ed70 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.411+0000 7f9e8f400700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9e7403e530 0x7f9e74040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.411+0000 7f9e8f400700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9e7403e530 0x7f9e74040a40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c006fd0 tx=0x7f9e7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.416+0000 7f9e8cc00700 1 -- 172.21.15.12:0/1973935715 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9e800212c0 con 0x7f9e9007ed70 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.651+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f9e901a33e0 con 0x7f9e7403e530 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.653+0000 7f9e8cc00700 1 -- 172.21.15.12:0/1973935715 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f9e901a33e0 con 0x7f9e7403e530 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9e7403e530 msgr2=0x7f9e74040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9e7403e530 0x7f9e74040a40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c006fd0 tx=0x7f9e7c006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:14.302 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 msgr2=0x7f9e9019fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:14.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9019fe30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e800119b0 tx=0x7f9e80011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:14.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 shutdown_connections 2024-05-27T07:06:14.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9e7403e530 0x7f9e74040a40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:14.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 --2- 172.21.15.12:0/1973935715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e9007ed70 0x7f9e9019fe30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:14.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 >> 172.21.15.12:0/1973935715 conn(0x7f9e90105020 msgr2=0x7f9e90105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:14.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.657+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 shutdown_connections 2024-05-27T07:06:14.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:13.658+0000 7f9e97200700 1 -- 172.21.15.12:0/1973935715 wait complete. 2024-05-27T07:06:14.304 INFO:teuthology.orchestra.run.smithi012.stdout:Generating ssh key... 2024-05-27T07:06:14.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:14 smithi012 ceph-mon[31473]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: Generating public/private rsa key pair. 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: Your identification has been saved in /tmp/tmptjxblvk4/key. 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: Your public key has been saved in /tmp/tmptjxblvk4/key.pub. 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: The key fingerprint is: 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: SHA256:jHQHwteRjQIqKbE2LajmZcE8QoIltYDpDqFWjGMMJO0 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: The key's randomart image is: 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: +---[RSA 3072]----+ 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: |&B= .o....= | 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: |XX== ...o.+ . | 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: |BB**. ..... | 2024-05-27T07:06:15.581 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: |=+E.o. + . | 2024-05-27T07:06:15.582 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: |=. o . S | 2024-05-27T07:06:15.582 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: |o.o | 2024-05-27T07:06:15.582 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: | . | 2024-05-27T07:06:15.582 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: | | 2024-05-27T07:06:15.582 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: | | 2024-05-27T07:06:15.582 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: +----[SHA256]-----+ 2024-05-27T07:06:16.303 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.045+0000 7f09b3400700 1 Processor -- start 2024-05-27T07:06:16.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.045+0000 7f09b3400700 1 -- start start 2024-05-27T07:06:16.304 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.046+0000 7f09b3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac07dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:16.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.046+0000 7f09b3400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac07e1c0 con 0x7f09ac07f7b0 2024-05-27T07:06:16.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.046+0000 7f09b0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac07dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:16.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.046+0000 7f09b0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac07dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43824/0 (socket says 172.21.15.12:43824) 2024-05-27T07:06:16.305 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.046+0000 7f09b0a00700 1 -- 172.21.15.12:0/3040570534 learned_addr learned my addr 172.21.15.12:0/3040570534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:16.306 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.047+0000 7f09b0a00700 1 -- 172.21.15.12:0/3040570534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac07e340 con 0x7f09ac07f7b0 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.047+0000 7f09b0a00700 1 --2- 172.21.15.12:0/3040570534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac07dc80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09a00098a0 tx=0x7f09a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc051bd6b6bc8353 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.048+0000 7f09ab400700 1 -- 172.21.15.12:0/3040570534 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09a0004160 con 0x7f09ac07f7b0 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.048+0000 7f09ab400700 1 -- 172.21.15.12:0/3040570534 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f09a0004300 con 0x7f09ac07f7b0 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.048+0000 7f09ab400700 1 -- 172.21.15.12:0/3040570534 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09a00119e0 con 0x7f09ac07f7b0 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.049+0000 7f09b3400700 1 -- 172.21.15.12:0/3040570534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 msgr2=0x7f09ac07dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.049+0000 7f09b3400700 1 --2- 172.21.15.12:0/3040570534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac07dc80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09a00098a0 tx=0x7f09a0009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.049+0000 7f09b3400700 1 -- 172.21.15.12:0/3040570534 shutdown_connections 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.049+0000 7f09b3400700 1 --2- 172.21.15.12:0/3040570534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac07dc80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.049+0000 7f09b3400700 1 -- 172.21.15.12:0/3040570534 >> 172.21.15.12:0/3040570534 conn(0x7f09ac1050c0 msgr2=0x7f09ac107510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.049+0000 7f09b3400700 1 -- 172.21.15.12:0/3040570534 shutdown_connections 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.049+0000 7f09b3400700 1 -- 172.21.15.12:0/3040570534 wait complete. 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.050+0000 7f09b3400700 1 Processor -- start 2024-05-27T07:06:16.307 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.050+0000 7f09b3400700 1 -- start start 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.051+0000 7f09b3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac19fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.051+0000 7f09b3400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac10fee0 con 0x7f09ac07f7b0 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.051+0000 7f09b0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac19fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.051+0000 7f09b0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac19fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43830/0 (socket says 172.21.15.12:43830) 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.051+0000 7f09b0a00700 1 -- 172.21.15.12:0/3148559575 learned_addr learned my addr 172.21.15.12:0/3148559575 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.051+0000 7f09b0a00700 1 -- 172.21.15.12:0/3148559575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac1a03a0 con 0x7f09ac07f7b0 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.052+0000 7f09b0a00700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac19fe60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f09a00119b0 tx=0x7f09a0011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.052+0000 7f09a9600700 1 -- 172.21.15.12:0/3148559575 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09a0003af0 con 0x7f09ac07f7b0 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.052+0000 7f09a9600700 1 -- 172.21.15.12:0/3148559575 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f09a001a8a0 con 0x7f09ac07f7b0 2024-05-27T07:06:16.310 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.052+0000 7f09a9600700 1 -- 172.21.15.12:0/3148559575 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09a0024830 con 0x7f09ac07f7b0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.052+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac1a0630 con 0x7f09ac07f7b0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.052+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac1a3240 con 0x7f09ac07f7b0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.053+0000 7f09a9600700 1 -- 172.21.15.12:0/3148559575 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f09a0022020 con 0x7f09ac07f7b0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.053+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09ac064e10 con 0x7f09ac07f7b0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.054+0000 7f09a9600700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f099403e530 0x7f0994040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.054+0000 7f09a9600700 1 -- 172.21.15.12:0/3148559575 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f09a0053a30 con 0x7f09ac07f7b0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.054+0000 7f09abe00700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f099403e530 0x7f0994040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.055+0000 7f09abe00700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f099403e530 0x7f0994040a40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0998006fd0 tx=0x7f0998006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.060+0000 7f09a9600700 1 -- 172.21.15.12:0/3148559575 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f09a0053300 con 0x7f09ac07f7b0 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.298+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f09ac1070b0 con 0x7f099403e530 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.504+0000 7f09a9600700 1 -- 172.21.15.12:0/3148559575 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f09ac1070b0 con 0x7f099403e530 2024-05-27T07:06:16.311 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f099403e530 msgr2=0x7f0994040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f099403e530 0x7f0994040a40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0998006fd0 tx=0x7f0998006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 msgr2=0x7f09ac19fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac19fe60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f09a00119b0 tx=0x7f09a0011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 shutdown_connections 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f099403e530 0x7f0994040a40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 --2- 172.21.15.12:0/3148559575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09ac07f7b0 0x7f09ac19fe60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.507+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 >> 172.21.15.12:0/3148559575 conn(0x7f09ac1050c0 msgr2=0x7f09ac1069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.508+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 shutdown_connections 2024-05-27T07:06:16.313 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:15.508+0000 7f09b3400700 1 -- 172.21.15.12:0/3148559575 wait complete. 2024-05-27T07:06:16.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:16 smithi012 ceph-mon[31473]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:16.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:16 smithi012 ceph-mon[31473]: Generating ssh key... 2024-05-27T07:06:16.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:16 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:16.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:16 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:17.869 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNfXOIZ3wIXptZ5rxwG8OXj5zV6rcoORkVPZ7X6IQy4SK5ZluRcNOTwxkVCX8wS76ouPZwAZRdDrQXI5i5Le+1W9NJ16tgqzTl7+HL/UnumwIoqGvWLhoO0YADToKgQL5GNK6zngdNBSMLNah2p+WPCWKPgCskgG31q96WLfdjRwY2FC2pUcrpy4X9OxfXJXI+itv1r03DnQOxKP4mwC8P4S+jTpLkThViiLkrtJUfFFaY9ogkCRAS3ksZ0mgrjuRc9HgwY3yY05TNZZcw8qP3pdmnjC4RxkH3xcIFAmAnib1VaP5LXseiivyIbuldSGMH4T292i13c9sw3dIuvPoEx5vdk75YF3W0zCbuMYcxSurjpYGIs3GtvRm2xJpbE5BcLpvD/iFLGHOvGxoymMtXlWMKxbhDW+vVlXKucqZZnww6FKA3bJ+VtfpOXaJ9ns0yk3XfCkD1a7IWAyBaekpNLAPyxXNDkOuQNPsLNxRGYkfB3S16csLU9uhOuYdgIHc= ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:06:17.869 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.009+0000 7f6693e00700 1 Processor -- start 2024-05-27T07:06:17.869 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.010+0000 7f6693e00700 1 -- start start 2024-05-27T07:06:17.869 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.010+0000 7f6693e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f6694112b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:17.869 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.010+0000 7f6693e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6694113040 con 0x7f66941106e0 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.011+0000 7f6692a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f6694112b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.011+0000 7f6692a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f6694112b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43832/0 (socket says 172.21.15.12:43832) 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.011+0000 7f6692a00700 1 -- 172.21.15.12:0/2070770699 learned_addr learned my addr 172.21.15.12:0/2070770699 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.011+0000 7f6692a00700 1 -- 172.21.15.12:0/2070770699 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66941131c0 con 0x7f66941106e0 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.011+0000 7f6692a00700 1 --2- 172.21.15.12:0/2070770699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f6694112b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f66840098a0 tx=0x7f6684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a8c1d3bbf9474b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.012+0000 7f6691600700 1 -- 172.21.15.12:0/2070770699 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684004160 con 0x7f66941106e0 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.012+0000 7f6691600700 1 -- 172.21.15.12:0/2070770699 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6684004300 con 0x7f66941106e0 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.012+0000 7f6691600700 1 -- 172.21.15.12:0/2070770699 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66840119e0 con 0x7f66941106e0 2024-05-27T07:06:17.870 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.013+0000 7f6693e00700 1 -- 172.21.15.12:0/2070770699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 msgr2=0x7f6694112b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.013+0000 7f6693e00700 1 --2- 172.21.15.12:0/2070770699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f6694112b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f66840098a0 tx=0x7f6684009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.013+0000 7f6693e00700 1 -- 172.21.15.12:0/2070770699 shutdown_connections 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.013+0000 7f6693e00700 1 --2- 172.21.15.12:0/2070770699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f6694112b00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.013+0000 7f6693e00700 1 -- 172.21.15.12:0/2070770699 >> 172.21.15.12:0/2070770699 conn(0x7f669407ef80 msgr2=0x7f66940813a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.013+0000 7f6693e00700 1 -- 172.21.15.12:0/2070770699 shutdown_connections 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.013+0000 7f6693e00700 1 -- 172.21.15.12:0/2070770699 wait complete. 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.014+0000 7f6693e00700 1 Processor -- start 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.014+0000 7f6693e00700 1 -- start start 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.015+0000 7f6693e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f66941a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.015+0000 7f6693e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6694114130 con 0x7f66941106e0 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.015+0000 7f6692a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f66941a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.015+0000 7f6692a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f66941a4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43838/0 (socket says 172.21.15.12:43838) 2024-05-27T07:06:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.015+0000 7f6692a00700 1 -- 172.21.15.12:0/4076505138 learned_addr learned my addr 172.21.15.12:0/4076505138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.016+0000 7f6692a00700 1 -- 172.21.15.12:0/4076505138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66941a45a0 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.016+0000 7f6692a00700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f66941a4060 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66840119b0 tx=0x7f6684011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.016+0000 7f668b400700 1 -- 172.21.15.12:0/4076505138 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684003af0 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.016+0000 7f668b400700 1 -- 172.21.15.12:0/4076505138 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f668401a8a0 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.016+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66941a4830 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.016+0000 7f668b400700 1 -- 172.21.15.12:0/4076505138 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684024830 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.016+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66941a7440 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.018+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6694064e10 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.018+0000 7f668b400700 1 -- 172.21.15.12:0/4076505138 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f6684022020 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.018+0000 7f668b400700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f667803e530 0x7f6678040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.018+0000 7f668b400700 1 -- 172.21.15.12:0/4076505138 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f66840539b0 con 0x7f66941106e0 2024-05-27T07:06:17.872 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.019+0000 7f6692000700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f667803e530 0x7f6678040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.020+0000 7f6692000700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f667803e530 0x7f6678040a40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.025+0000 7f668b400700 1 -- 172.21.15.12:0/4076505138 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6684053280 con 0x7f66941106e0 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.264+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6694080ed0 con 0x7f667803e530 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.266+0000 7f668b400700 1 -- 172.21.15.12:0/4076505138 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6694080ed0 con 0x7f667803e530 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.269+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f667803e530 msgr2=0x7f6678040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.269+0000 7f6693e00700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f667803e530 0x7f6678040a40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.269+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 msgr2=0x7f66941a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.269+0000 7f6693e00700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f66941a4060 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66840119b0 tx=0x7f6684011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.270+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 shutdown_connections 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.270+0000 7f6693e00700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f667803e530 0x7f6678040a40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.270+0000 7f6693e00700 1 --2- 172.21.15.12:0/4076505138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66941106e0 0x7f66941a4060 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.270+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 >> 172.21.15.12:0/4076505138 conn(0x7f669407ef80 msgr2=0x7f66940807c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:17.873 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.270+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 shutdown_connections 2024-05-27T07:06:17.874 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:17.270+0000 7f6693e00700 1 -- 172.21.15.12:0/4076505138 wait complete. 2024-05-27T07:06:17.874 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:06:17.874 INFO:teuthology.orchestra.run.smithi012.stdout:Adding key to root@localhost authorized_keys... 2024-05-27T07:06:17.874 INFO:teuthology.orchestra.run.smithi012.stdout:Adding host smithi012... 2024-05-27T07:06:18.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:17 smithi012 ceph-mon[31473]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:19.052 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:18 smithi012 ceph-mon[31473]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:21.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:20 smithi012 ceph-mon[31473]: Deploying cephadm binary to smithi012 2024-05-27T07:06:25.750 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Added host 'smithi012' with addr '172.21.15.12' 2024-05-27T07:06:25.750 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.547+0000 7fd768e00700 1 Processor -- start 2024-05-27T07:06:25.750 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.548+0000 7fd768e00700 1 -- start start 2024-05-27T07:06:25.751 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.548+0000 7fd768e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd76410c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:25.751 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.548+0000 7fd768e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd76407d680 con 0x7fd76410a550 2024-05-27T07:06:25.751 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.549+0000 7fd761e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd76410c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:25.751 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.549+0000 7fd761e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd76410c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43854/0 (socket says 172.21.15.12:43854) 2024-05-27T07:06:25.751 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.549+0000 7fd761e00700 1 -- 172.21.15.12:0/3166618555 learned_addr learned my addr 172.21.15.12:0/3166618555 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:25.751 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.549+0000 7fd761e00700 1 -- 172.21.15.12:0/3166618555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76410ceb0 con 0x7fd76410a550 2024-05-27T07:06:25.751 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.550+0000 7fd761e00700 1 --2- 172.21.15.12:0/3166618555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd76410c970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7540098a0 tx=0x7fd754009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102cb40b8d104d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.550+0000 7fd760a00700 1 -- 172.21.15.12:0/3166618555 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd754004160 con 0x7fd76410a550 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.550+0000 7fd760a00700 1 -- 172.21.15.12:0/3166618555 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fd754004300 con 0x7fd76410a550 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.550+0000 7fd760a00700 1 -- 172.21.15.12:0/3166618555 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7540119e0 con 0x7fd76410a550 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.551+0000 7fd768e00700 1 -- 172.21.15.12:0/3166618555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 msgr2=0x7fd76410c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.551+0000 7fd768e00700 1 --2- 172.21.15.12:0/3166618555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd76410c970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7540098a0 tx=0x7fd754009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.551+0000 7fd768e00700 1 -- 172.21.15.12:0/3166618555 shutdown_connections 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.551+0000 7fd768e00700 1 --2- 172.21.15.12:0/3166618555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd76410c970 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.551+0000 7fd768e00700 1 -- 172.21.15.12:0/3166618555 >> 172.21.15.12:0/3166618555 conn(0x7fd764103ff0 msgr2=0x7fd764106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:25.752 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.551+0000 7fd768e00700 1 -- 172.21.15.12:0/3166618555 shutdown_connections 2024-05-27T07:06:25.753 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.551+0000 7fd768e00700 1 -- 172.21.15.12:0/3166618555 wait complete. 2024-05-27T07:06:25.753 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.552+0000 7fd768e00700 1 Processor -- start 2024-05-27T07:06:25.753 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.553+0000 7fd768e00700 1 -- start start 2024-05-27T07:06:25.753 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.553+0000 7fd768e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd764109cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:25.753 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.553+0000 7fd768e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd76410dd40 con 0x7fd76410a550 2024-05-27T07:06:25.753 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.553+0000 7fd761e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd764109cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:25.753 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.553+0000 7fd761e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd764109cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43868/0 (socket says 172.21.15.12:43868) 2024-05-27T07:06:25.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.553+0000 7fd761e00700 1 -- 172.21.15.12:0/3166900516 learned_addr learned my addr 172.21.15.12:0/3166900516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:25.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.554+0000 7fd761e00700 1 -- 172.21.15.12:0/3166900516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76410a210 con 0x7fd76410a550 2024-05-27T07:06:25.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.554+0000 7fd761e00700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd764109cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7540119b0 tx=0x7fd754011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:25.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.554+0000 7fd75aa00700 1 -- 172.21.15.12:0/3166900516 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd754003af0 con 0x7fd76410a550 2024-05-27T07:06:25.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.555+0000 7fd75aa00700 1 -- 172.21.15.12:0/3166900516 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fd75401a8a0 con 0x7fd76410a550 2024-05-27T07:06:25.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.555+0000 7fd75aa00700 1 -- 172.21.15.12:0/3166900516 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd754024830 con 0x7fd76410a550 2024-05-27T07:06:25.754 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.555+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd764108460 con 0x7fd76410a550 2024-05-27T07:06:25.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.555+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd764108940 con 0x7fd76410a550 2024-05-27T07:06:25.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.556+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd764064e10 con 0x7fd76410a550 2024-05-27T07:06:25.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.556+0000 7fd75aa00700 1 -- 172.21.15.12:0/3166900516 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fd754022020 con 0x7fd76410a550 2024-05-27T07:06:25.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.557+0000 7fd75aa00700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fd74803e580 0x7fd748040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:25.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.557+0000 7fd75aa00700 1 -- 172.21.15.12:0/3166900516 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd754053950 con 0x7fd76410a550 2024-05-27T07:06:25.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.557+0000 7fd761400700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fd74803e580 0x7fd748040a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:25.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.558+0000 7fd761400700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fd74803e580 0x7fd748040a90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd750006fd0 tx=0x7fd750006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.562+0000 7fd75aa00700 1 -- 172.21.15.12:0/3166900516 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd7540212c0 con 0x7fd76410a550 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:18.805+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}) -- 0x7fd7641091c0 con 0x7fd74803e580 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.068+0000 7fd75aa00700 1 -- 172.21.15.12:0/3166900516 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fd7641091c0 con 0x7fd74803e580 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fd74803e580 msgr2=0x7fd748040a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fd74803e580 0x7fd748040a90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd750006fd0 tx=0x7fd750006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 msgr2=0x7fd764109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd764109cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7540119b0 tx=0x7fd754011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 shutdown_connections 2024-05-27T07:06:25.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fd74803e580 0x7fd748040a90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:25.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 --2- 172.21.15.12:0/3166900516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd76410a550 0x7fd764109cd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:25.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.073+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 >> 172.21.15.12:0/3166900516 conn(0x7fd764103ff0 msgr2=0x7fd7641058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:25.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.074+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 shutdown_connections 2024-05-27T07:06:25.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:25.074+0000 7fd768e00700 1 -- 172.21.15.12:0/3166900516 wait complete. 2024-05-27T07:06:25.757 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mon service... 2024-05-27T07:06:26.313 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:26 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:26.313 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:26 smithi012 ceph-mon[31473]: Added host smithi012 2024-05-27T07:06:26.314 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:26 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.561+0000 7fc85e200700 1 Processor -- start 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.561+0000 7fc85e200700 1 -- start start 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.562+0000 7fc85e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85807d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.562+0000 7fc85e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc85807d730 con 0x7fc85807edc0 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.562+0000 7fc85ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85807d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.562+0000 7fc85ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85807d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35596/0 (socket says 172.21.15.12:35596) 2024-05-27T07:06:27.992 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.563+0000 7fc85ce00700 1 -- 172.21.15.12:0/1797216238 learned_addr learned my addr 172.21.15.12:0/1797216238 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.563+0000 7fc85ce00700 1 -- 172.21.15.12:0/1797216238 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85807d8b0 con 0x7fc85807edc0 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.563+0000 7fc85ce00700 1 --2- 172.21.15.12:0/1797216238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85807d1f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0098a0 tx=0x7fc84c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7debb2c0029bb4e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.564+0000 7fc857400700 1 -- 172.21.15.12:0/1797216238 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc84c004160 con 0x7fc85807edc0 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.564+0000 7fc857400700 1 -- 172.21.15.12:0/1797216238 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc84c004300 con 0x7fc85807edc0 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.564+0000 7fc857400700 1 -- 172.21.15.12:0/1797216238 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc84c0119e0 con 0x7fc85807edc0 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.565+0000 7fc85e200700 1 -- 172.21.15.12:0/1797216238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 msgr2=0x7fc85807d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.565+0000 7fc85e200700 1 --2- 172.21.15.12:0/1797216238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85807d1f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0098a0 tx=0x7fc84c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.565+0000 7fc85e200700 1 -- 172.21.15.12:0/1797216238 shutdown_connections 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.565+0000 7fc85e200700 1 --2- 172.21.15.12:0/1797216238 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85807d1f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.565+0000 7fc85e200700 1 -- 172.21.15.12:0/1797216238 >> 172.21.15.12:0/1797216238 conn(0x7fc858104e50 msgr2=0x7fc8581072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.565+0000 7fc85e200700 1 -- 172.21.15.12:0/1797216238 shutdown_connections 2024-05-27T07:06:27.993 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.565+0000 7fc85e200700 1 -- 172.21.15.12:0/1797216238 wait complete. 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.567+0000 7fc85e200700 1 Processor -- start 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.567+0000 7fc85e200700 1 -- start start 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.567+0000 7fc85e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.568+0000 7fc85e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc85810fd10 con 0x7fc85807edc0 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.568+0000 7fc85ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.568+0000 7fc85ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85819fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35612/0 (socket says 172.21.15.12:35612) 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.568+0000 7fc85ce00700 1 -- 172.21.15.12:0/2746413111 learned_addr learned my addr 172.21.15.12:0/2746413111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.568+0000 7fc85ce00700 1 -- 172.21.15.12:0/2746413111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8581a01f0 con 0x7fc85807edc0 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.569+0000 7fc85ce00700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85819fcb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0119b0 tx=0x7fc84c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.569+0000 7fc855600700 1 -- 172.21.15.12:0/2746413111 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc84c003af0 con 0x7fc85807edc0 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.569+0000 7fc855600700 1 -- 172.21.15.12:0/2746413111 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc84c01a8a0 con 0x7fc85807edc0 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.569+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8581a0480 con 0x7fc85807edc0 2024-05-27T07:06:27.994 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.569+0000 7fc855600700 1 -- 172.21.15.12:0/2746413111 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc84c024830 con 0x7fc85807edc0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.569+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8581a3090 con 0x7fc85807edc0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.570+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc858064e10 con 0x7fc85807edc0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.571+0000 7fc855600700 1 -- 172.21.15.12:0/2746413111 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fc84c022020 con 0x7fc85807edc0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.571+0000 7fc855600700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fc84003e580 0x7fc840040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.571+0000 7fc855600700 1 -- 172.21.15.12:0/2746413111 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc84c053950 con 0x7fc85807edc0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.572+0000 7fc857e00700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fc84003e580 0x7fc840040a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.573+0000 7fc857e00700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fc84003e580 0x7fc840040a90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc844006fd0 tx=0x7fc844006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.578+0000 7fc855600700 1 -- 172.21.15.12:0/2746413111 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc84c0532a0 con 0x7fc85807edc0 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.842+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc858106dc0 con 0x7fc84003e580 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.852+0000 7fc855600700 1 -- 172.21.15.12:0/2746413111 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc858106dc0 con 0x7fc84003e580 2024-05-27T07:06:27.995 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.855+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fc84003e580 msgr2=0x7fc840040a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.855+0000 7fc85e200700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fc84003e580 0x7fc840040a90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc844006fd0 tx=0x7fc844006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.855+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 msgr2=0x7fc85819fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.855+0000 7fc85e200700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85819fcb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0119b0 tx=0x7fc84c011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.856+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 shutdown_connections 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.856+0000 7fc85e200700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fc84003e580 0x7fc840040a90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.856+0000 7fc85e200700 1 --2- 172.21.15.12:0/2746413111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc85807edc0 0x7fc85819fcb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.856+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 >> 172.21.15.12:0/2746413111 conn(0x7fc858104e50 msgr2=0x7fc8581066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.856+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 shutdown_connections 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:26.856+0000 7fc85e200700 1 -- 172.21.15.12:0/2746413111 wait complete. 2024-05-27T07:06:27.996 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mgr service... 2024-05-27T07:06:28.113 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:27 smithi012 ceph-mon[31473]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:28.113 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:27 smithi012 ceph-mon[31473]: Saving service mon spec with placement count:5 2024-05-27T07:06:28.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:27 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:29 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:29 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:29 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:29.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-27T07:06:29.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.992+0000 7f50e1600700 1 Processor -- start 2024-05-27T07:06:29.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.992+0000 7f50e1600700 1 -- start start 2024-05-27T07:06:29.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.993+0000 7f50e1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:29.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.993+0000 7f50e1600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc10a3c0 con 0x7f50dc1099c0 2024-05-27T07:06:29.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.994+0000 7f50daa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:29.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.994+0000 7f50daa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc109dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35620/0 (socket says 172.21.15.12:35620) 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.994+0000 7f50daa00700 1 -- 172.21.15.12:0/944133208 learned_addr learned my addr 172.21.15.12:0/944133208 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.994+0000 7f50daa00700 1 -- 172.21.15.12:0/944133208 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc10aaf0 con 0x7f50dc1099c0 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.994+0000 7f50daa00700 1 --2- 172.21.15.12:0/944133208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc109dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bba242f64578ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.995+0000 7f50d9600700 1 -- 172.21.15.12:0/944133208 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc004160 con 0x7f50dc1099c0 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.995+0000 7f50d9600700 1 -- 172.21.15.12:0/944133208 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50cc004300 con 0x7f50dc1099c0 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.995+0000 7f50d9600700 1 -- 172.21.15.12:0/944133208 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc011960 con 0x7f50dc1099c0 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.996+0000 7f50e1600700 1 -- 172.21.15.12:0/944133208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 msgr2=0x7f50dc109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.996+0000 7f50e1600700 1 --2- 172.21.15.12:0/944133208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc109dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.996+0000 7f50e1600700 1 -- 172.21.15.12:0/944133208 shutdown_connections 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.996+0000 7f50e1600700 1 --2- 172.21.15.12:0/944133208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc109dd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.996+0000 7f50e1600700 1 -- 172.21.15.12:0/944133208 >> 172.21.15.12:0/944133208 conn(0x7f50dc1050c0 msgr2=0x7f50dc107510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.996+0000 7f50e1600700 1 -- 172.21.15.12:0/944133208 shutdown_connections 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.996+0000 7f50e1600700 1 -- 172.21.15.12:0/944133208 wait complete. 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.997+0000 7f50e1600700 1 Processor -- start 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.997+0000 7f50e1600700 1 -- start start 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.998+0000 7f50e1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc19feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.998+0000 7f50e1600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc10dea0 con 0x7f50dc1099c0 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.998+0000 7f50daa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc19feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.998+0000 7f50daa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc19feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35636/0 (socket says 172.21.15.12:35636) 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.998+0000 7f50daa00700 1 -- 172.21.15.12:0/675726527 learned_addr learned my addr 172.21.15.12:0/675726527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.999+0000 7f50daa00700 1 -- 172.21.15.12:0/675726527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc1a03f0 con 0x7f50dc1099c0 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.999+0000 7f50daa00700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc19feb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006af0 tx=0x7f50cc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.999+0000 7f50d3400700 1 -- 172.21.15.12:0/675726527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc003af0 con 0x7f50dc1099c0 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.999+0000 7f50d3400700 1 -- 172.21.15.12:0/675726527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50cc01b8d0 con 0x7f50dc1099c0 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.999+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc1a0680 con 0x7f50dc1099c0 2024-05-27T07:06:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.999+0000 7f50d3400700 1 -- 172.21.15.12:0/675726527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc023820 con 0x7f50dc1099c0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:28.999+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc1a3290 con 0x7f50dc1099c0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.001+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50dc064e10 con 0x7f50dc1099c0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.001+0000 7f50d3400700 1 -- 172.21.15.12:0/675726527 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f50cc01b440 con 0x7f50dc1099c0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.002+0000 7f50d3400700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f50c003e530 0x7f50c0040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.002+0000 7f50d3400700 1 -- 172.21.15.12:0/675726527 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f50cc054010 con 0x7f50dc1099c0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.002+0000 7f50da000700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f50c003e530 0x7f50c0040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.003+0000 7f50da000700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f50c003e530 0x7f50c0040a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f50c8006fd0 tx=0x7f50c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.008+0000 7f50d3400700 1 -- 172.21.15.12:0/675726527 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f50cc01f030 con 0x7f50dc1099c0 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.266+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f50dc107050 con 0x7f50c003e530 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.273+0000 7f50d3400700 1 -- 172.21.15.12:0/675726527 <== mgr.14120 v2:172.21.15.12:6800/2115905110 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f50dc107050 con 0x7f50c003e530 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f50c003e530 msgr2=0x7f50c0040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f50c003e530 0x7f50c0040a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f50c8006fd0 tx=0x7f50c8006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 msgr2=0x7f50dc19feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc19feb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006af0 tx=0x7f50cc011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 shutdown_connections 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f50c003e530 0x7f50c0040a40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 --2- 172.21.15.12:0/675726527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50dc1099c0 0x7f50dc19feb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 >> 172.21.15.12:0/675726527 conn(0x7f50dc1050c0 msgr2=0x7f50dc106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 shutdown_connections 2024-05-27T07:06:29.941 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:29.276+0000 7f50e1600700 1 -- 172.21.15.12:0/675726527 wait complete. 2024-05-27T07:06:30.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:30 smithi012 ceph-mon[31473]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:30.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:30 smithi012 ceph-mon[31473]: Saving service mgr spec with placement count:2 2024-05-27T07:06:30.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:30 smithi012 ceph-mon[31473]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:06:31.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.583+0000 7f002ac00700 1 Processor -- start 2024-05-27T07:06:31.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.583+0000 7f002ac00700 1 -- start start 2024-05-27T07:06:31.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.583+0000 7f002ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002407d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:31.567 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.583+0000 7f002ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f002407d730 con 0x7f002407ed70 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.584+0000 7f0023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002407d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.584+0000 7f0023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002407d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35648/0 (socket says 172.21.15.12:35648) 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.584+0000 7f0023e00700 1 -- 172.21.15.12:0/2625571510 learned_addr learned my addr 172.21.15.12:0/2625571510 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.584+0000 7f0023e00700 1 -- 172.21.15.12:0/2625571510 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002407d8b0 con 0x7f002407ed70 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.585+0000 7f0023e00700 1 --2- 172.21.15.12:0/2625571510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002407d1f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f00140098a0 tx=0x7f0014009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2323143d1bb1fb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.585+0000 7f0022a00700 1 -- 172.21.15.12:0/2625571510 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0014004160 con 0x7f002407ed70 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.585+0000 7f0022a00700 1 -- 172.21.15.12:0/2625571510 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0014004300 con 0x7f002407ed70 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.585+0000 7f0022a00700 1 -- 172.21.15.12:0/2625571510 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0014011960 con 0x7f002407ed70 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.586+0000 7f002ac00700 1 -- 172.21.15.12:0/2625571510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 msgr2=0x7f002407d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.586+0000 7f002ac00700 1 --2- 172.21.15.12:0/2625571510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002407d1f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f00140098a0 tx=0x7f0014009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.586+0000 7f002ac00700 1 -- 172.21.15.12:0/2625571510 shutdown_connections 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.586+0000 7f002ac00700 1 --2- 172.21.15.12:0/2625571510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002407d1f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:31.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.586+0000 7f002ac00700 1 -- 172.21.15.12:0/2625571510 >> 172.21.15.12:0/2625571510 conn(0x7f0024105020 msgr2=0x7f00241074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.586+0000 7f002ac00700 1 -- 172.21.15.12:0/2625571510 shutdown_connections 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.586+0000 7f002ac00700 1 -- 172.21.15.12:0/2625571510 wait complete. 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.588+0000 7f002ac00700 1 Processor -- start 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.588+0000 7f002ac00700 1 -- start start 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.588+0000 7f002ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002419fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.588+0000 7f002ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f002410ff30 con 0x7f002407ed70 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.588+0000 7f0023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002419fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.588+0000 7f0023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002419fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35654/0 (socket says 172.21.15.12:35654) 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.588+0000 7f0023e00700 1 -- 172.21.15.12:0/1010203473 learned_addr learned my addr 172.21.15.12:0/1010203473 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.589+0000 7f0023e00700 1 -- 172.21.15.12:0/1010203473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00241a0390 con 0x7f002407ed70 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.589+0000 7f0023e00700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002419fe50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0014011670 tx=0x7f0014011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.589+0000 7f0020c00700 1 -- 172.21.15.12:0/1010203473 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0014003a60 con 0x7f002407ed70 2024-05-27T07:06:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.590+0000 7f0020c00700 1 -- 172.21.15.12:0/1010203473 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f001401b8d0 con 0x7f002407ed70 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.590+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00241a0620 con 0x7f002407ed70 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.590+0000 7f0020c00700 1 -- 172.21.15.12:0/1010203473 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0014023790 con 0x7f002407ed70 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.590+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00241a3230 con 0x7f002407ed70 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.591+0000 7f0020c00700 1 -- 172.21.15.12:0/1010203473 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f0014023a40 con 0x7f002407ed70 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.591+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0024064e10 con 0x7f002407ed70 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.592+0000 7f0020c00700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f000803e580 0x7f0008040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.592+0000 7f0020c00700 1 -- 172.21.15.12:0/1010203473 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0014054060 con 0x7f002407ed70 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.592+0000 7f0023400700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f000803e580 0x7f0008040a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.593+0000 7f0023400700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f000803e580 0x7f0008040a90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0010006fd0 tx=0x7f0010006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:31.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.597+0000 7f0020c00700 1 -- 172.21.15.12:0/1010203473 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f001401f030 con 0x7f002407ed70 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.886+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f00241a3830 con 0x7f002407ed70 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.895+0000 7f0020c00700 1 -- 172.21.15.12:0/1010203473 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f001400b3e0 con 0x7f002407ed70 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.898+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f000803e580 msgr2=0x7f0008040a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.898+0000 7f002ac00700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f000803e580 0x7f0008040a90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0010006fd0 tx=0x7f0010006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.898+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 msgr2=0x7f002419fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.898+0000 7f002ac00700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002419fe50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0014011670 tx=0x7f0014011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.899+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 shutdown_connections 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.899+0000 7f002ac00700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f000803e580 0x7f0008040a90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.899+0000 7f002ac00700 1 --2- 172.21.15.12:0/1010203473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f002407ed70 0x7f002419fe50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.899+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 >> 172.21.15.12:0/1010203473 conn(0x7f0024105020 msgr2=0x7f0024105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.899+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 shutdown_connections 2024-05-27T07:06:31.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:30.899+0000 7f002ac00700 1 -- 172.21.15.12:0/1010203473 wait complete. 2024-05-27T07:06:32.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:31 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1010203473' entity='client.admin' 2024-05-27T07:06:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:31 smithi012 ceph-mon[31473]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:06:33.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:33 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:33.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:33 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/692259705' entity='client.admin' 2024-05-27T07:06:33.674 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.693+0000 7f5dd3e00700 1 Processor -- start 2024-05-27T07:06:33.674 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.693+0000 7f5dd3e00700 1 -- start start 2024-05-27T07:06:33.674 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.694+0000 7f5dd3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd407d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:33.674 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.694+0000 7f5dd3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd407d730 con 0x7f5dd407edc0 2024-05-27T07:06:33.674 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.694+0000 7f5dd2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd407d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:33.674 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.694+0000 7f5dd2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd407d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53284/0 (socket says 172.21.15.12:53284) 2024-05-27T07:06:33.674 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.694+0000 7f5dd2a00700 1 -- 172.21.15.12:0/2519911407 learned_addr learned my addr 172.21.15.12:0/2519911407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.695+0000 7f5dd2a00700 1 -- 172.21.15.12:0/2519911407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd407d8b0 con 0x7f5dd407edc0 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.695+0000 7f5dd2a00700 1 --2- 172.21.15.12:0/2519911407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd407d1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40098a0 tx=0x7f5dc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d215ad97d4c809 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.696+0000 7f5dd1600700 1 -- 172.21.15.12:0/2519911407 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dc4004160 con 0x7f5dd407edc0 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.696+0000 7f5dd1600700 1 -- 172.21.15.12:0/2519911407 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc4004300 con 0x7f5dd407edc0 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.696+0000 7f5dd1600700 1 -- 172.21.15.12:0/2519911407 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dc4011960 con 0x7f5dd407edc0 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.697+0000 7f5dd3e00700 1 -- 172.21.15.12:0/2519911407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 msgr2=0x7f5dd407d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.697+0000 7f5dd3e00700 1 --2- 172.21.15.12:0/2519911407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd407d1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40098a0 tx=0x7f5dc4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.697+0000 7f5dd3e00700 1 -- 172.21.15.12:0/2519911407 shutdown_connections 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.697+0000 7f5dd3e00700 1 --2- 172.21.15.12:0/2519911407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd407d1f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:33.675 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.697+0000 7f5dd3e00700 1 -- 172.21.15.12:0/2519911407 >> 172.21.15.12:0/2519911407 conn(0x7f5dd4104e50 msgr2=0x7f5dd41072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.697+0000 7f5dd3e00700 1 -- 172.21.15.12:0/2519911407 shutdown_connections 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.697+0000 7f5dd3e00700 1 -- 172.21.15.12:0/2519911407 wait complete. 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.699+0000 7f5dd3e00700 1 Processor -- start 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.699+0000 7f5dd3e00700 1 -- start start 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.699+0000 7f5dd3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.699+0000 7f5dd3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd410fd50 con 0x7f5dd407edc0 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.700+0000 7f5dd2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.700+0000 7f5dd2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd419fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53294/0 (socket says 172.21.15.12:53294) 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.700+0000 7f5dd2a00700 1 -- 172.21.15.12:0/692259705 learned_addr learned my addr 172.21.15.12:0/692259705 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.700+0000 7f5dd2a00700 1 -- 172.21.15.12:0/692259705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd41a0210 con 0x7f5dd407edc0 2024-05-27T07:06:33.676 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.701+0000 7f5dd2a00700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd419fcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4011670 tx=0x7f5dc4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.701+0000 7f5dcb400700 1 -- 172.21.15.12:0/692259705 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dc4003a60 con 0x7f5dd407edc0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.701+0000 7f5dcb400700 1 -- 172.21.15.12:0/692259705 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dc401b8d0 con 0x7f5dd407edc0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.701+0000 7f5dcb400700 1 -- 172.21.15.12:0/692259705 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5dc4023790 con 0x7f5dd407edc0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.701+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd41a04a0 con 0x7f5dd407edc0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.701+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd41a30b0 con 0x7f5dd407edc0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.702+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd4064e10 con 0x7f5dd407edc0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.703+0000 7f5dcb400700 1 -- 172.21.15.12:0/692259705 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f5dc4023a40 con 0x7f5dd407edc0 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.703+0000 7f5dcb400700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f5db803e580 0x7f5db8040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:33.677 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.703+0000 7f5dcb400700 1 -- 172.21.15.12:0/692259705 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5dc4053fc0 con 0x7f5dd407edc0 2024-05-27T07:06:33.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.704+0000 7f5dd2000700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f5db803e580 0x7f5db8040a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:33.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.704+0000 7f5dd2000700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f5db803e580 0x7f5db8040a90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0006fd0 tx=0x7f5dc0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:33.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.710+0000 7f5dcb400700 1 -- 172.21.15.12:0/692259705 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5dc401f030 con 0x7f5dd407edc0 2024-05-27T07:06:33.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.945+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f5dd41a3710 con 0x7f5dd407edc0 2024-05-27T07:06:33.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.949+0000 7f5dcb400700 1 -- 172.21.15.12:0/692259705 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f5dc400b3e0 con 0x7f5dd407edc0 2024-05-27T07:06:33.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.952+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f5db803e580 msgr2=0x7f5db8040a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:33.678 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.952+0000 7f5dd3e00700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f5db803e580 0x7f5db8040a90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0006fd0 tx=0x7f5dc0006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.952+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 msgr2=0x7f5dd419fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.952+0000 7f5dd3e00700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd419fcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4011670 tx=0x7f5dc4011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.953+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 shutdown_connections 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.953+0000 7f5dd3e00700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f5db803e580 0x7f5db8040a90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.953+0000 7f5dd3e00700 1 --2- 172.21.15.12:0/692259705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd407edc0 0x7f5dd419fcd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.953+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 >> 172.21.15.12:0/692259705 conn(0x7f5dd4104e50 msgr2=0x7f5dd41066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.953+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 shutdown_connections 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:32.953+0000 7f5dd3e00700 1 -- 172.21.15.12:0/692259705 wait complete. 2024-05-27T07:06:33.679 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling the dashboard module... 2024-05-27T07:06:34.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:34 smithi012 ceph-mon[31473]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:06:35.888 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:35 smithi012 ceph-mon[31473]: from='mgr.14120 172.21.15.12:0/2875108797' entity='mgr.x' 2024-05-27T07:06:35.888 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:35 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2460189426' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-27T07:06:36.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:35 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: ignoring --setuser ceph since I am not root 2024-05-27T07:06:36.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:35 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: ignoring --setgroup ceph since I am not root 2024-05-27T07:06:36.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:36.088+0000 7f57f5672200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:06:36.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:36.348+0000 7f57f5672200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:06:36.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.336+0000 7fcedc400700 1 Processor -- start 2024-05-27T07:06:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.336+0000 7fcedc400700 1 -- start start 2024-05-27T07:06:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.336+0000 7fcedc400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced407dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.336+0000 7fcedc400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fced407e0b0 con 0x7fced407f6b0 2024-05-27T07:06:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.337+0000 7fced9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced407dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.337+0000 7fced9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced407dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53306/0 (socket says 172.21.15.12:53306) 2024-05-27T07:06:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.337+0000 7fced9a00700 1 -- 172.21.15.12:0/435992903 learned_addr learned my addr 172.21.15.12:0/435992903 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:36.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.338+0000 7fced9a00700 1 -- 172.21.15.12:0/435992903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced407e230 con 0x7fced407f6b0 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.338+0000 7fced9a00700 1 --2- 172.21.15.12:0/435992903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced407dae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcec80098a0 tx=0x7fcec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e6feb4911d5014 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.338+0000 7fced3e00700 1 -- 172.21.15.12:0/435992903 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcec8004160 con 0x7fced407f6b0 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.338+0000 7fced3e00700 1 -- 172.21.15.12:0/435992903 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcec8004300 con 0x7fced407f6b0 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.338+0000 7fced3e00700 1 -- 172.21.15.12:0/435992903 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcec8011960 con 0x7fced407f6b0 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.339+0000 7fcedc400700 1 -- 172.21.15.12:0/435992903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 msgr2=0x7fced407dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.339+0000 7fcedc400700 1 --2- 172.21.15.12:0/435992903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced407dae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcec80098a0 tx=0x7fcec8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.339+0000 7fcedc400700 1 -- 172.21.15.12:0/435992903 shutdown_connections 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.339+0000 7fcedc400700 1 --2- 172.21.15.12:0/435992903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced407dae0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.339+0000 7fcedc400700 1 -- 172.21.15.12:0/435992903 >> 172.21.15.12:0/435992903 conn(0x7fced4105050 msgr2=0x7fced41074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.340+0000 7fcedc400700 1 -- 172.21.15.12:0/435992903 shutdown_connections 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.340+0000 7fcedc400700 1 -- 172.21.15.12:0/435992903 wait complete. 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.341+0000 7fcedc400700 1 Processor -- start 2024-05-27T07:06:36.375 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.342+0000 7fcedc400700 1 -- start start 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.342+0000 7fcedc400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced419fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.342+0000 7fcedc400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fced410ff70 con 0x7fced407f6b0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.342+0000 7fced9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced419fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.343+0000 7fced9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced419fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53314/0 (socket says 172.21.15.12:53314) 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.343+0000 7fced9a00700 1 -- 172.21.15.12:0/2460189426 learned_addr learned my addr 172.21.15.12:0/2460189426 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.343+0000 7fced9a00700 1 -- 172.21.15.12:0/2460189426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced41a03d0 con 0x7fced407f6b0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.343+0000 7fced9a00700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced419fe90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcec8000c00 tx=0x7fcec8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.344+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcec8003b20 con 0x7fced407f6b0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.344+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcec801b8d0 con 0x7fced407f6b0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.344+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced41a0660 con 0x7fced407f6b0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.344+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcec8023820 con 0x7fced407f6b0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.344+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced41a3270 con 0x7fced407f6b0 2024-05-27T07:06:36.376 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.345+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb4005360 con 0x7fced407f6b0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.346+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fcec801b440 con 0x7fced407f6b0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.346+0000 7fced2000700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fcebc03e580 0x7fcebc040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.346+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcec8053de0 con 0x7fced407f6b0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.347+0000 7fced9000700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fcebc03e580 0x7fcebc040a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.348+0000 7fced9000700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fcebc03e580 0x7fcebc040a90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcec0006fd0 tx=0x7fcec0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.357+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcec801f030 con 0x7fced407f6b0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:34.666+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fceb4005650 con 0x7fced407f6b0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.670+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fcec801ba80 con 0x7fced407f6b0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.674+0000 7fced2000700 1 -- 172.21.15.12:0/2460189426 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fcec8059020 con 0x7fced407f6b0 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.675+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fcebc03e580 msgr2=0x7fcebc040a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.675+0000 7fcedc400700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fcebc03e580 0x7fcebc040a90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcec0006fd0 tx=0x7fcec0006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.675+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 msgr2=0x7fced419fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:36.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.675+0000 7fcedc400700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced419fe90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcec8000c00 tx=0x7fcec8011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:36.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.676+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 shutdown_connections 2024-05-27T07:06:36.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.676+0000 7fcedc400700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fcebc03e580 0x7fcebc040a90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:36.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.676+0000 7fcedc400700 1 --2- 172.21.15.12:0/2460189426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced407f6b0 0x7fced419fe90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:36.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.676+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 >> 172.21.15.12:0/2460189426 conn(0x7fced4105050 msgr2=0x7fced4106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:36.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.677+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 shutdown_connections 2024-05-27T07:06:36.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:35.677+0000 7fcedc400700 1 -- 172.21.15.12:0/2460189426 wait complete. 2024-05-27T07:06:36.661 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:36.492+0000 7f57f5672200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:06:36.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:36 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2460189426' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-27T07:06:36.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:36 smithi012 ceph-mon[31473]: mgrmap e9: x(active, since 26s) 2024-05-27T07:06:36.989 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:36.917+0000 7f57f5672200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:06:37.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:37.042+0000 7f57f5672200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:06:37.777 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:37 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3030111949' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T07:06:38.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:06:38.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-27T07:06:38.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:06:38.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T07:06:38.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T07:06:38.382 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:06:38.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.224+0000 7fbeb3e00700 1 Processor -- start 2024-05-27T07:06:38.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.224+0000 7fbeb3e00700 1 -- start start 2024-05-27T07:06:38.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.225+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb407d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:38.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.225+0000 7fbeb3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb407d7e0 con 0x7fbeb407ede0 2024-05-27T07:06:38.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.225+0000 7fbeb2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb407d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:38.383 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.225+0000 7fbeb2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb407d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53336/0 (socket says 172.21.15.12:53336) 2024-05-27T07:06:38.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.225+0000 7fbeb2a00700 1 -- 172.21.15.12:0/2359554992 learned_addr learned my addr 172.21.15.12:0/2359554992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:38.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.226+0000 7fbeb2a00700 1 -- 172.21.15.12:0/2359554992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb407d960 con 0x7fbeb407ede0 2024-05-27T07:06:38.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.226+0000 7fbeb2a00700 1 --2- 172.21.15.12:0/2359554992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb407d210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098a0 tx=0x7fbea4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d217573d2c66d69f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:38.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.226+0000 7fbeb1600700 1 -- 172.21.15.12:0/2359554992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea4004160 con 0x7fbeb407ede0 2024-05-27T07:06:38.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.227+0000 7fbeb1600700 1 -- 172.21.15.12:0/2359554992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbea4004300 con 0x7fbeb407ede0 2024-05-27T07:06:38.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.227+0000 7fbeb1600700 1 -- 172.21.15.12:0/2359554992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea4011960 con 0x7fbeb407ede0 2024-05-27T07:06:38.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.227+0000 7fbeb3e00700 1 -- 172.21.15.12:0/2359554992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 msgr2=0x7fbeb407d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:38.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.227+0000 7fbeb3e00700 1 --2- 172.21.15.12:0/2359554992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb407d210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098a0 tx=0x7fbea4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:38.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.228+0000 7fbeb3e00700 1 -- 172.21.15.12:0/2359554992 shutdown_connections 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.228+0000 7fbeb3e00700 1 --2- 172.21.15.12:0/2359554992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb407d210 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.228+0000 7fbeb3e00700 1 -- 172.21.15.12:0/2359554992 >> 172.21.15.12:0/2359554992 conn(0x7fbeb4104e50 msgr2=0x7fbeb41072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.228+0000 7fbeb3e00700 1 -- 172.21.15.12:0/2359554992 shutdown_connections 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.228+0000 7fbeb3e00700 1 -- 172.21.15.12:0/2359554992 wait complete. 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.229+0000 7fbeb3e00700 1 Processor -- start 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.229+0000 7fbeb3e00700 1 -- start start 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.229+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.229+0000 7fbeb3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb410fd90 con 0x7fbeb407ede0 2024-05-27T07:06:38.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.229+0000 7fbeb2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.229+0000 7fbeb2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb419fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53348/0 (socket says 172.21.15.12:53348) 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.229+0000 7fbeb2a00700 1 -- 172.21.15.12:0/3030111949 learned_addr learned my addr 172.21.15.12:0/3030111949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.230+0000 7fbeb2a00700 1 -- 172.21.15.12:0/3030111949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb41a0240 con 0x7fbeb407ede0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.230+0000 7fbeb2a00700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb419fd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbea4011670 tx=0x7fbea4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.230+0000 7fbeab400700 1 -- 172.21.15.12:0/3030111949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea4003a60 con 0x7fbeb407ede0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.230+0000 7fbeab400700 1 -- 172.21.15.12:0/3030111949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbea401b8d0 con 0x7fbeb407ede0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.230+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb41a04d0 con 0x7fbeb407ede0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.231+0000 7fbeab400700 1 -- 172.21.15.12:0/3030111949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea4023750 con 0x7fbeb407ede0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.231+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb41a30e0 con 0x7fbeb407ede0 2024-05-27T07:06:38.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.232+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb4064e10 con 0x7fbeb407ede0 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.233+0000 7fbeab400700 1 -- 172.21.15.12:0/3030111949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fbea401b440 con 0x7fbeb407ede0 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.233+0000 7fbeab400700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 0x7fbe98040aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.233+0000 7fbeab400700 1 -- 172.21.15.12:0/3030111949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbea40543e0 con 0x7fbeb407ede0 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.234+0000 7fbeb2000700 1 -- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 msgr2=0x7fbe98040aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.234+0000 7fbeb2000700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 0x7fbe98040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.239+0000 7fbeab400700 1 -- 172.21.15.12:0/3030111949 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbea401f030 con 0x7fbeb407ede0 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.435+0000 7fbeb2000700 1 -- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 msgr2=0x7fbe98040aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.435+0000 7fbeb2000700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 0x7fbe98040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.531+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fbeb41a32b0 con 0x7fbeb407ede0 2024-05-27T07:06:38.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.531+0000 7fbeab400700 1 -- 172.21.15.12:0/3030111949 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fbeb41a32b0 con 0x7fbeb407ede0 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 msgr2=0x7fbe98040aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 0x7fbe98040aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 msgr2=0x7fbeb419fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb419fd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbea4011670 tx=0x7fbea4011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 shutdown_connections 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7fbe9803e590 0x7fbe98040aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 --2- 172.21.15.12:0/3030111949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb407ede0 0x7fbeb419fd00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 >> 172.21.15.12:0/3030111949 conn(0x7fbeb4104e50 msgr2=0x7fbeb41066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 shutdown_connections 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:37.534+0000 7fbeb3e00700 1 -- 172.21.15.12:0/3030111949 wait complete. 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-05-27T07:06:38.396 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 9... 2024-05-27T07:06:38.941 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:38 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:38.687+0000 7f57f5672200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:06:38.941 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:38 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:38.937+0000 7f57f5672200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:06:39.297 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:39.162+0000 7f57f5672200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:06:39.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:39.564+0000 7f57f5672200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:06:39.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:39.693+0000 7f57f5672200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:06:40.218 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:39.957+0000 7f57f5672200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:06:40.218 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:40 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:40.226+0000 7f57f5672200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:06:41.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:41.057+0000 7f57f5672200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:06:41.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:41.368+0000 7f57f5672200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:06:42.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:41.959+0000 7f57f5672200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:06:42.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:42.097+0000 7f57f5672200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:06:42.834 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:42.586+0000 7f57f5672200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:06:43.091 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:43.066+0000 7f57f5672200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:06:43.530 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:43.198+0000 7f57f5672200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:06:43.531 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:43.538+0000 7f57f5672200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:06:43.841 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:43.727+0000 7f57f5672200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:06:44.266 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:43.863+0000 7f57f5672200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:06:44.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:44.273+0000 7f57f5672200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:06:44.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:44.403+0000 7f57f5672200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:06:45.496 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:45.202+0000 7f57f5672200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: Active manager daemon x restarted 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: Activating manager daemon x 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: osdmap e3: 0 total, 0 up, 0 in 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: mgrmap e10: x(active, starting, since 0.00781626s) 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:06:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:06:45.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:45 smithi012 ceph-mon[31473]: Manager daemon x is now available 2024-05-27T07:06:45.842 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:06:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:06:45.503+0000 7f57f5672200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:06:46.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:46.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:06:46.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:06:46.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:46 smithi012 ceph-mon[31473]: mgrmap e11: x(active, since 1.01141s) 2024-05-27T07:06:47.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.095+0000 7f9df1e00700 1 Processor -- start 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.096+0000 7f9df1e00700 1 -- start start 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.096+0000 7f9df1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec109950 0x7f9dec109d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.096+0000 7f9df1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec10a350 con 0x7f9dec109950 2024-05-27T07:06:47.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.096+0000 7f9deaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec109950 0x7f9dec109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:47.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.097+0000 7f9deaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec109950 0x7f9dec109d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53356/0 (socket says 172.21.15.12:53356) 2024-05-27T07:06:47.471 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.097+0000 7f9deaa00700 1 -- 172.21.15.12:0/4026029857 learned_addr learned my addr 172.21.15.12:0/4026029857 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.097+0000 7f9deaa00700 1 -- 172.21.15.12:0/4026029857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec10ab20 con 0x7f9dec109950 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.097+0000 7f9deaa00700 1 --2- 172.21.15.12:0/4026029857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec109950 0x7f9dec109d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0098a0 tx=0x7f9ddc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2cf7381d842d05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.098+0000 7f9de9600700 1 -- 172.21.15.12:0/4026029857 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ddc004160 con 0x7f9dec109950 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.098+0000 7f9de9600700 1 -- 172.21.15.12:0/4026029857 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9ddc004300 con 0x7f9dec109950 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.098+0000 7f9de9600700 1 -- 172.21.15.12:0/4026029857 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ddc011960 con 0x7f9dec109950 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.098+0000 7f9df1e00700 1 -- 172.21.15.12:0/4026029857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec109950 msgr2=0x7f9dec109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.098+0000 7f9df1e00700 1 --2- 172.21.15.12:0/4026029857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec109950 0x7f9dec109d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0098a0 tx=0x7f9ddc009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.099+0000 7f9df1e00700 1 -- 172.21.15.12:0/4026029857 shutdown_connections 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.099+0000 7f9df1e00700 1 --2- 172.21.15.12:0/4026029857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec109950 0x7f9dec109d60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.099+0000 7f9df1e00700 1 -- 172.21.15.12:0/4026029857 >> 172.21.15.12:0/4026029857 conn(0x7f9dec105050 msgr2=0x7f9dec1074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.099+0000 7f9df1e00700 1 -- 172.21.15.12:0/4026029857 shutdown_connections 2024-05-27T07:06:47.472 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.099+0000 7f9df1e00700 1 -- 172.21.15.12:0/4026029857 wait complete. 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.100+0000 7f9df1e00700 1 Processor -- start 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.100+0000 7f9df1e00700 1 -- start start 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.100+0000 7f9df1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec07f230 0x7f9dec07da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.100+0000 7f9df1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec10ded0 con 0x7f9dec07f230 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.101+0000 7f9deaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec07f230 0x7f9dec07da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.101+0000 7f9deaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec07f230 0x7f9dec07da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53368/0 (socket says 172.21.15.12:53368) 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.101+0000 7f9deaa00700 1 -- 172.21.15.12:0/1420114057 learned_addr learned my addr 172.21.15.12:0/1420114057 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.101+0000 7f9deaa00700 1 -- 172.21.15.12:0/1420114057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec07f6a0 con 0x7f9dec07f230 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.101+0000 7f9deaa00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec07f230 0x7f9dec07da40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc000c00 tx=0x7f9ddc012f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.102+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ddc011900 con 0x7f9dec07f230 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.102+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dec07f930 con 0x7f9dec07f230 2024-05-27T07:06:47.473 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.102+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9ddc003b30 con 0x7f9dec07f230 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.102+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ddc025c50 con 0x7f9dec07f230 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.102+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dec07e290 con 0x7f9dec07f230 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.103+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f9ddc01b450 con 0x7f9dec07f230 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.104+0000 7f9df0a00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.104+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 --> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9dd00411a0 con 0x7f9dd003e580 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.104+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9ddc053c30 con 0x7f9dec07f230 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.104+0000 7f9dea000700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 msgr2=0x7f9dd0040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.104+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.305+0000 7f9dea000700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 msgr2=0x7f9dd0040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.305+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.706+0000 7f9dea000700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 msgr2=0x7f9dd0040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:47.474 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:39.706+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:40.507+0000 7f9dea000700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 msgr2=0x7f9dd0040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:40.507+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:42.109+0000 7f9dea000700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 msgr2=0x7f9dd0040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:42.109+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:45.312+0000 7f9dea000700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 msgr2=0x7f9dd0040a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2115905110 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:45.312+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:45.514+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f9ddc01b790 con 0x7f9dec07f230 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:45.514+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 msgr2=0x7f9dd0040a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:45.514+0000 7f9df0a00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.514+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f9ddc056070 con 0x7f9dec07f230 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.514+0000 7f9df0a00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ddc025070 0x7f9dd0043fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.514+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9dd00411a0 con 0x7f9ddc025070 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.515+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ddc025070 0x7f9dd0043fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:47.475 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.517+0000 7f9dea000700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ddc025070 0x7f9dd0043fc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8003a10 tx=0x7f9dd80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.519+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f9dd00411a0 con 0x7f9ddc025070 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.534+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9dec064e10 con 0x7f9ddc025070 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.535+0000 7f9df0a00700 1 -- 172.21.15.12:0/1420114057 <== mgr.14152 v2:172.21.15.12:6800/2856894494 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9dec064e10 con 0x7f9ddc025070 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.535+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ddc025070 msgr2=0x7f9dd0043fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.535+0000 7f9df1e00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ddc025070 0x7f9dd0043fc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8003a10 tx=0x7f9dd80092b0 comp rx=0 tx=0).stop 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.535+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec07f230 msgr2=0x7f9dec07da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.535+0000 7f9df1e00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec07f230 0x7f9dec07da40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc000c00 tx=0x7f9ddc012f00 comp rx=0 tx=0).stop 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.536+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 shutdown_connections 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.536+0000 7f9df1e00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ddc025070 0x7f9dd0043fc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.536+0000 7f9df1e00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9dec07f230 0x7f9dec07da40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.536+0000 7f9df1e00700 1 --2- 172.21.15.12:0/1420114057 >> [v2:172.21.15.12:6800/2115905110,v1:172.21.15.12:6801/2115905110] conn(0x7f9dd003e580 0x7f9dd0040a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.536+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 >> 172.21.15.12:0/1420114057 conn(0x7f9dec105050 msgr2=0x7f9dec106a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.536+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 shutdown_connections 2024-05-27T07:06:47.476 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:46.536+0000 7f9df1e00700 1 -- 172.21.15.12:0/1420114057 wait complete. 2024-05-27T07:06:47.477 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 9 is available 2024-05-27T07:06:47.477 INFO:teuthology.orchestra.run.smithi012.stdout:Generating a dashboard self-signed certificate... 2024-05-27T07:06:47.621 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:47 smithi012 ceph-mon[31473]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T07:06:47.621 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:47 smithi012 ceph-mon[31473]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T07:06:48.837 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:48 smithi012 ceph-mon[31473]: [27/May/2024:07:06:47] ENGINE Bus STARTING 2024-05-27T07:06:48.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:48 smithi012 ceph-mon[31473]: [27/May/2024:07:06:47] ENGINE Serving on https://172.21.15.12:7150 2024-05-27T07:06:48.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:48 smithi012 ceph-mon[31473]: [27/May/2024:07:06:47] ENGINE Serving on http://172.21.15.12:8765 2024-05-27T07:06:48.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:48 smithi012 ceph-mon[31473]: [27/May/2024:07:06:47] ENGINE Bus STARTED 2024-05-27T07:06:48.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:48 smithi012 ceph-mon[31473]: mgrmap e12: x(active, since 2s) 2024-05-27T07:06:49.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-27T07:06:49.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.360+0000 7f28cf200700 1 Processor -- start 2024-05-27T07:06:49.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.360+0000 7f28cf200700 1 -- start start 2024-05-27T07:06:49.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.360+0000 7f28cf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d007d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:49.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.361+0000 7f28cf200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28d007d7e0 con 0x7f28d007ede0 2024-05-27T07:06:49.479 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.361+0000 7f28cde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d007d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.361+0000 7f28cde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d007d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56790/0 (socket says 172.21.15.12:56790) 2024-05-27T07:06:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.361+0000 7f28cde00700 1 -- 172.21.15.12:0/3360506524 learned_addr learned my addr 172.21.15.12:0/3360506524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.362+0000 7f28cde00700 1 -- 172.21.15.12:0/3360506524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d007d960 con 0x7f28d007ede0 2024-05-27T07:06:49.480 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.362+0000 7f28cde00700 1 --2- 172.21.15.12:0/3360506524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d007d210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009e80 tx=0x7f28c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef56791b6efbdc16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:49.486 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.362+0000 7f28cca00700 1 -- 172.21.15.12:0/3360506524 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c0017070 con 0x7f28d007ede0 2024-05-27T07:06:49.486 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.362+0000 7f28cca00700 1 -- 172.21.15.12:0/3360506524 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28c000b8c0 con 0x7f28d007ede0 2024-05-27T07:06:49.486 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.362+0000 7f28cca00700 1 -- 172.21.15.12:0/3360506524 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c0004420 con 0x7f28d007ede0 2024-05-27T07:06:49.486 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.363+0000 7f28cf200700 1 -- 172.21.15.12:0/3360506524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 msgr2=0x7f28d007d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:49.486 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.363+0000 7f28cf200700 1 --2- 172.21.15.12:0/3360506524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d007d210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009e80 tx=0x7f28c000b330 comp rx=0 tx=0).stop 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.363+0000 7f28cf200700 1 -- 172.21.15.12:0/3360506524 shutdown_connections 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.363+0000 7f28cf200700 1 --2- 172.21.15.12:0/3360506524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d007d210 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.363+0000 7f28cf200700 1 -- 172.21.15.12:0/3360506524 >> 172.21.15.12:0/3360506524 conn(0x7f28d0104e50 msgr2=0x7f28d01072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.363+0000 7f28cf200700 1 -- 172.21.15.12:0/3360506524 shutdown_connections 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.364+0000 7f28cf200700 1 -- 172.21.15.12:0/3360506524 wait complete. 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.365+0000 7f28cf200700 1 Processor -- start 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.365+0000 7f28cf200700 1 -- start start 2024-05-27T07:06:49.487 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.365+0000 7f28cf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.365+0000 7f28cf200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28d010fd90 con 0x7f28d007ede0 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.365+0000 7f28cde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.365+0000 7f28cde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d019fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56792/0 (socket says 172.21.15.12:56792) 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.365+0000 7f28cde00700 1 -- 172.21.15.12:0/1537267604 learned_addr learned my addr 172.21.15.12:0/1537267604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.366+0000 7f28cde00700 1 -- 172.21.15.12:0/1537267604 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d01a0240 con 0x7f28d007ede0 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.366+0000 7f28cde00700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d019fd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28c000bca0 tx=0x7f28c000bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.366+0000 7f28c6a00700 1 -- 172.21.15.12:0/1537267604 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c0020020 con 0x7f28d007ede0 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.366+0000 7f28c6a00700 1 -- 172.21.15.12:0/1537267604 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28c00128c0 con 0x7f28d007ede0 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.366+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d01a04d0 con 0x7f28d007ede0 2024-05-27T07:06:49.488 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.366+0000 7f28c6a00700 1 -- 172.21.15.12:0/1537267604 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c0017040 con 0x7f28d007ede0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.367+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d01a30e0 con 0x7f28d007ede0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.368+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d0064e10 con 0x7f28d007ede0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.368+0000 7f28c6a00700 1 -- 172.21.15.12:0/1537267604 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f28c0023020 con 0x7f28d007ede0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.369+0000 7f28c6a00700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f28b403e5e0 0x7f28b4040af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.369+0000 7f28c6a00700 1 -- 172.21.15.12:0/1537267604 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f28c000f050 con 0x7f28d007ede0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.370+0000 7f28cd400700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f28b403e5e0 0x7f28b4040af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.370+0000 7f28cd400700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f28b403e5e0 0x7f28b4040af0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28bc006fd0 tx=0x7f28bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.375+0000 7f28c6a00700 1 -- 172.21.15.12:0/1537267604 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f28c002a020 con 0x7f28d007ede0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.630+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f28d0106e00 con 0x7f28b403e5e0 2024-05-27T07:06:49.489 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.815+0000 7f28c6a00700 1 -- 172.21.15.12:0/1537267604 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f28d0106e00 con 0x7f28b403e5e0 2024-05-27T07:06:49.490 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f28b403e5e0 msgr2=0x7f28b4040af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:49.490 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f28b403e5e0 0x7f28b4040af0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28bc006fd0 tx=0x7f28bc006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:49.491 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 msgr2=0x7f28d019fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:49.491 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d019fd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28c000bca0 tx=0x7f28c000bd80 comp rx=0 tx=0).stop 2024-05-27T07:06:49.491 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 shutdown_connections 2024-05-27T07:06:49.491 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f28b403e5e0 0x7f28b4040af0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:49.491 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 --2- 172.21.15.12:0/1537267604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d007ede0 0x7f28d019fd00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:49.491 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 >> 172.21.15.12:0/1537267604 conn(0x7f28d0104e50 msgr2=0x7f28d01066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:49.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.818+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 shutdown_connections 2024-05-27T07:06:49.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:48.819+0000 7f28cf200700 1 -- 172.21.15.12:0/1537267604 wait complete. 2024-05-27T07:06:49.492 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial admin user... 2024-05-27T07:06:49.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:49 smithi012 ceph-mon[31473]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:49.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:49 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:06:49.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:49 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:06:49.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:49 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:06:51.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$fan3sEy5YROvMcy8wc/Ba.In6483Vvf6r6TFz9bhzUIeSYcWXB0x6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716793610, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-27T07:06:51.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.180+0000 7fcc83800700 1 Processor -- start 2024-05-27T07:06:51.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.180+0000 7fcc83800700 1 -- start start 2024-05-27T07:06:51.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.180+0000 7fcc83800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c108c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:51.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.180+0000 7fcc83800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c109220 con 0x7fcc7c108820 2024-05-27T07:06:51.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.181+0000 7fcc80e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c108c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.181+0000 7fcc80e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c108c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56796/0 (socket says 172.21.15.12:56796) 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.181+0000 7fcc80e00700 1 -- 172.21.15.12:0/980150421 learned_addr learned my addr 172.21.15.12:0/980150421 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.181+0000 7fcc80e00700 1 -- 172.21.15.12:0/980150421 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c1099f0 con 0x7fcc7c108820 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.182+0000 7fcc80e00700 1 --2- 172.21.15.12:0/980150421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c108c30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc700098a0 tx=0x7fcc70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4c4869d71dea91 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.182+0000 7fcc7b400700 1 -- 172.21.15.12:0/980150421 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70004160 con 0x7fcc7c108820 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.182+0000 7fcc7b400700 1 -- 172.21.15.12:0/980150421 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc70004300 con 0x7fcc7c108820 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.182+0000 7fcc7b400700 1 -- 172.21.15.12:0/980150421 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70011960 con 0x7fcc7c108820 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.183+0000 7fcc83800700 1 -- 172.21.15.12:0/980150421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 msgr2=0x7fcc7c108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.183+0000 7fcc83800700 1 --2- 172.21.15.12:0/980150421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c108c30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc700098a0 tx=0x7fcc70009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.183+0000 7fcc83800700 1 -- 172.21.15.12:0/980150421 shutdown_connections 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.183+0000 7fcc83800700 1 --2- 172.21.15.12:0/980150421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c108c30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.183+0000 7fcc83800700 1 -- 172.21.15.12:0/980150421 >> 172.21.15.12:0/980150421 conn(0x7fcc7c103ec0 msgr2=0x7fcc7c106310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.183+0000 7fcc83800700 1 -- 172.21.15.12:0/980150421 shutdown_connections 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.183+0000 7fcc83800700 1 -- 172.21.15.12:0/980150421 wait complete. 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.185+0000 7fcc83800700 1 Processor -- start 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.185+0000 7fcc83800700 1 -- start start 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.185+0000 7fcc83800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c19feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.185+0000 7fcc83800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c10cda0 con 0x7fcc7c108820 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.185+0000 7fcc80e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c19feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.185+0000 7fcc80e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c19feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56808/0 (socket says 172.21.15.12:56808) 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.185+0000 7fcc80e00700 1 -- 172.21.15.12:0/632053342 learned_addr learned my addr 172.21.15.12:0/632053342 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.186+0000 7fcc80e00700 1 -- 172.21.15.12:0/632053342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c1a03f0 con 0x7fcc7c108820 2024-05-27T07:06:51.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.186+0000 7fcc80e00700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c19feb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcc700119b0 tx=0x7fcc70011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.186+0000 7fcc79600700 1 -- 172.21.15.12:0/632053342 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70003aa0 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.187+0000 7fcc79600700 1 -- 172.21.15.12:0/632053342 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc7001b8d0 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.187+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7c1a0680 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.187+0000 7fcc79600700 1 -- 172.21.15.12:0/632053342 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70023860 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.187+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7c1a3290 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.188+0000 7fcc79600700 1 -- 172.21.15.12:0/632053342 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7fcc7001b440 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.188+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7c064e10 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.189+0000 7fcc79600700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcc6403e530 0x7fcc64040a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.189+0000 7fcc79600700 1 -- 172.21.15.12:0/632053342 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcc70054010 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.189+0000 7fcc7be00700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcc6403e530 0x7fcc64040a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.190+0000 7fcc7be00700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcc6403e530 0x7fcc64040a40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc68006fd0 tx=0x7fcc68006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.195+0000 7fcc79600700 1 -- 172.21.15.12:0/632053342 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcc7001f030 con 0x7fcc7c108820 2024-05-27T07:06:51.395 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.457+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fcc7c1a3520 con 0x7fcc6403e530 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.711+0000 7fcc79600700 1 -- 172.21.15.12:0/632053342 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fcc7c1a3520 con 0x7fcc6403e530 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcc6403e530 msgr2=0x7fcc64040a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcc6403e530 0x7fcc64040a40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc68006fd0 tx=0x7fcc68006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 msgr2=0x7fcc7c19feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c19feb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcc700119b0 tx=0x7fcc70011a90 comp rx=0 tx=0).stop 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 shutdown_connections 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcc6403e530 0x7fcc64040a40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 --2- 172.21.15.12:0/632053342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc7c108820 0x7fcc7c19feb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.715+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 >> 172.21.15.12:0/632053342 conn(0x7fcc7c103ec0 msgr2=0x7fcc7c104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.716+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 shutdown_connections 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:50.716+0000 7fcc83800700 1 -- 172.21.15.12:0/632053342 wait complete. 2024-05-27T07:06:51.396 INFO:teuthology.orchestra.run.smithi012.stdout:Fetching dashboard port number... 2024-05-27T07:06:51.954 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:51 smithi012 ceph-mon[31473]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:51.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:06:51.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:06:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:52 smithi012 ceph-mon[31473]: mgrmap e13: x(active, since 6s) 2024-05-27T07:06:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:52 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3165016784' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 8443 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.090+0000 7fc2dea00700 1 Processor -- start 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.091+0000 7fc2dea00700 1 -- start start 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.091+0000 7fc2dea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d8109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.091+0000 7fc2dea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d810a3c0 con 0x7fc2d81099c0 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.091+0000 7fc2d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d8109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.092+0000 7fc2d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d8109dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57560/0 (socket says 172.21.15.12:57560) 2024-05-27T07:06:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.092+0000 7fc2d7e00700 1 -- 172.21.15.12:0/2569736320 learned_addr learned my addr 172.21.15.12:0/2569736320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.092+0000 7fc2d7e00700 1 -- 172.21.15.12:0/2569736320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d810ab40 con 0x7fc2d81099c0 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.092+0000 7fc2d7e00700 1 --2- 172.21.15.12:0/2569736320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d8109dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80098a0 tx=0x7fc2c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e1ee00921b6fda server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.093+0000 7fc2d6a00700 1 -- 172.21.15.12:0/2569736320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2c8004160 con 0x7fc2d81099c0 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.093+0000 7fc2d6a00700 1 -- 172.21.15.12:0/2569736320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2c8004300 con 0x7fc2d81099c0 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.093+0000 7fc2d6a00700 1 -- 172.21.15.12:0/2569736320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2c8011960 con 0x7fc2d81099c0 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.094+0000 7fc2dea00700 1 -- 172.21.15.12:0/2569736320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 msgr2=0x7fc2d8109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.094+0000 7fc2dea00700 1 --2- 172.21.15.12:0/2569736320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d8109dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80098a0 tx=0x7fc2c8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.094+0000 7fc2dea00700 1 -- 172.21.15.12:0/2569736320 shutdown_connections 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.094+0000 7fc2dea00700 1 --2- 172.21.15.12:0/2569736320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d8109dd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.094+0000 7fc2dea00700 1 -- 172.21.15.12:0/2569736320 >> 172.21.15.12:0/2569736320 conn(0x7fc2d8105020 msgr2=0x7fc2d81074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.094+0000 7fc2dea00700 1 -- 172.21.15.12:0/2569736320 shutdown_connections 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.094+0000 7fc2dea00700 1 -- 172.21.15.12:0/2569736320 wait complete. 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.095+0000 7fc2dea00700 1 Processor -- start 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.095+0000 7fc2dea00700 1 -- start start 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.096+0000 7fc2dea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d807ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.096+0000 7fc2dea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d810def0 con 0x7fc2d81099c0 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.096+0000 7fc2d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d807ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.096+0000 7fc2d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d807ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57566/0 (socket says 172.21.15.12:57566) 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.096+0000 7fc2d7e00700 1 -- 172.21.15.12:0/3165016784 learned_addr learned my addr 172.21.15.12:0/3165016784 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.096+0000 7fc2d7e00700 1 -- 172.21.15.12:0/3165016784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d807ef60 con 0x7fc2d81099c0 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.097+0000 7fc2d7e00700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d807ea20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8006af0 tx=0x7fc2c8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.097+0000 7fc2d4c00700 1 -- 172.21.15.12:0/3165016784 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2c8003af0 con 0x7fc2d81099c0 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.097+0000 7fc2d4c00700 1 -- 172.21.15.12:0/3165016784 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2c801b8d0 con 0x7fc2d81099c0 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.097+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d807d1b0 con 0x7fc2d81099c0 2024-05-27T07:06:53.379 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.097+0000 7fc2d4c00700 1 -- 172.21.15.12:0/3165016784 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2c8023820 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.097+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d807d690 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.099+0000 7fc2d4c00700 1 -- 172.21.15.12:0/3165016784 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc2c801b440 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.099+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d8064e10 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.099+0000 7fc2d4c00700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc2bc03e620 0x7fc2bc040b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.099+0000 7fc2d4c00700 1 -- 172.21.15.12:0/3165016784 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc2c8054160 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.100+0000 7fc2d7400700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc2bc03e620 0x7fc2bc040b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.101+0000 7fc2d7400700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc2bc03e620 0x7fc2bc040b30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4006fd0 tx=0x7fc2c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.110+0000 7fc2d4c00700 1 -- 172.21.15.12:0/3165016784 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc2c801f030 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.351+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fc2d807dd80 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.352+0000 7fc2d4c00700 1 -- 172.21.15.12:0/3165016784 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fc2c8017e90 con 0x7fc2d81099c0 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc2bc03e620 msgr2=0x7fc2bc040b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:53.380 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc2bc03e620 0x7fc2bc040b30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4006fd0 tx=0x7fc2c4006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 msgr2=0x7fc2d807ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d807ea20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8006af0 tx=0x7fc2c8011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 shutdown_connections 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc2bc03e620 0x7fc2bc040b30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 --2- 172.21.15.12:0/3165016784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2d81099c0 0x7fc2d807ea20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.355+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 >> 172.21.15.12:0/3165016784 conn(0x7fc2d8105020 msgr2=0x7fc2d81058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.356+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 shutdown_connections 2024-05-27T07:06:53.381 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:52.356+0000 7fc2dea00700 1 -- 172.21.15.12:0/3165016784 wait complete. 2024-05-27T07:06:53.412 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:06:53.412 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-05-27T07:06:53.438 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:06:53.438 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-05-27T07:06:53.439 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-05-27T07:06:53.439 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-27T07:06:53.446 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph Dashboard is now available at: 2024-05-27T07:06:53.446 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:53.446 INFO:teuthology.orchestra.run.smithi012.stdout: URL: https://smithi012.front.sepia.ceph.com:8443/ 2024-05-27T07:06:53.447 INFO:teuthology.orchestra.run.smithi012.stdout: User: admin 2024-05-27T07:06:53.447 INFO:teuthology.orchestra.run.smithi012.stdout: Password: wyka01hv12 2024-05-27T07:06:53.447 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:53.448 INFO:teuthology.orchestra.run.smithi012.stdout:Saving cluster configuration to /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config directory 2024-05-27T07:06:53.450 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling autotune for osd_memory_target 2024-05-27T07:06:55.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.458+0000 7fc65dc00700 1 Processor -- start 2024-05-27T07:06:55.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.458+0000 7fc65dc00700 1 -- start start 2024-05-27T07:06:55.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.458+0000 7fc65dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc65807d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:55.385 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.458+0000 7fc65dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc65807d820 con 0x7fc65807edd0 2024-05-27T07:06:55.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.458+0000 7fc657e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc65807d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:55.386 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.458+0000 7fc657e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc65807d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57570/0 (socket says 172.21.15.12:57570) 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.458+0000 7fc657e00700 1 -- 172.21.15.12:0/3917762808 learned_addr learned my addr 172.21.15.12:0/3917762808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.459+0000 7fc657e00700 1 -- 172.21.15.12:0/3917762808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65807d9a0 con 0x7fc65807edd0 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.459+0000 7fc657e00700 1 --2- 172.21.15.12:0/3917762808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc65807d250 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6480098a0 tx=0x7fc648009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d48a2c7424c9add server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.459+0000 7fc656a00700 1 -- 172.21.15.12:0/3917762808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc648004160 con 0x7fc65807edd0 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.460+0000 7fc656a00700 1 -- 172.21.15.12:0/3917762808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc648004300 con 0x7fc65807edd0 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.460+0000 7fc656a00700 1 -- 172.21.15.12:0/3917762808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc648011960 con 0x7fc65807edd0 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.460+0000 7fc65dc00700 1 -- 172.21.15.12:0/3917762808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 msgr2=0x7fc65807d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.460+0000 7fc65dc00700 1 --2- 172.21.15.12:0/3917762808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc65807d250 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6480098a0 tx=0x7fc648009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.391 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.461+0000 7fc65dc00700 1 -- 172.21.15.12:0/3917762808 shutdown_connections 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.461+0000 7fc65dc00700 1 --2- 172.21.15.12:0/3917762808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc65807d250 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.461+0000 7fc65dc00700 1 -- 172.21.15.12:0/3917762808 >> 172.21.15.12:0/3917762808 conn(0x7fc658104e80 msgr2=0x7fc6581072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.461+0000 7fc65dc00700 1 -- 172.21.15.12:0/3917762808 shutdown_connections 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.461+0000 7fc65dc00700 1 -- 172.21.15.12:0/3917762808 wait complete. 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.462+0000 7fc65dc00700 1 Processor -- start 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.462+0000 7fc65dc00700 1 -- start start 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.462+0000 7fc65dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc6581a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.462+0000 7fc65dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc65810b9b0 con 0x7fc65807edd0 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.463+0000 7fc657e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc6581a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:55.392 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.463+0000 7fc657e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc6581a40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57582/0 (socket says 172.21.15.12:57582) 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.463+0000 7fc657e00700 1 -- 172.21.15.12:0/3340930801 learned_addr learned my addr 172.21.15.12:0/3340930801 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.463+0000 7fc657e00700 1 -- 172.21.15.12:0/3340930801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6581a4600 con 0x7fc65807edd0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.463+0000 7fc657e00700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc6581a40c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc648011670 tx=0x7fc648011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.464+0000 7fc654c00700 1 -- 172.21.15.12:0/3340930801 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc648003a60 con 0x7fc65807edd0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.464+0000 7fc654c00700 1 -- 172.21.15.12:0/3340930801 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc64801b8d0 con 0x7fc65807edd0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.464+0000 7fc654c00700 1 -- 172.21.15.12:0/3340930801 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc648023750 con 0x7fc65807edd0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.464+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6581a4890 con 0x7fc65807edd0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.464+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6581a74a0 con 0x7fc65807edd0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.465+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc658064e10 con 0x7fc65807edd0 2024-05-27T07:06:55.393 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.465+0000 7fc654c00700 1 -- 172.21.15.12:0/3340930801 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc64801b440 con 0x7fc65807edd0 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.466+0000 7fc654c00700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc63c03e630 0x7fc63c040b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.466+0000 7fc654c00700 1 -- 172.21.15.12:0/3340930801 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc6480542a0 con 0x7fc65807edd0 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.466+0000 7fc657400700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc63c03e630 0x7fc63c040b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.467+0000 7fc657400700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc63c03e630 0x7fc63c040b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc644006fd0 tx=0x7fc644006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.472+0000 7fc654c00700 1 -- 172.21.15.12:0/3340930801 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc64801f030 con 0x7fc65807edd0 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.710+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fc65802e880 con 0x7fc65807edd0 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.710+0000 7fc654c00700 1 -- 172.21.15.12:0/3340930801 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fc648018350 con 0x7fc65807edd0 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc63c03e630 msgr2=0x7fc63c040b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc63c03e630 0x7fc63c040b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc644006fd0 tx=0x7fc644006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:55.394 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 msgr2=0x7fc6581a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:55.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc6581a40c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc648011670 tx=0x7fc648011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:55.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 shutdown_connections 2024-05-27T07:06:55.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc63c03e630 0x7fc63c040b40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 --2- 172.21.15.12:0/3340930801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65807edd0 0x7fc6581a40c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.714+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 >> 172.21.15.12:0/3340930801 conn(0x7fc658104e80 msgr2=0x7fc65819ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:55.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.715+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 shutdown_connections 2024-05-27T07:06:55.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:54.715+0000 7fc65dc00700 1 -- 172.21.15.12:0/3340930801 wait complete. 2024-05-27T07:06:57.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.096+0000 7f5d96800700 1 Processor -- start 2024-05-27T07:06:57.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.096+0000 7f5d96800700 1 -- start start 2024-05-27T07:06:57.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.097+0000 7f5d96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9010bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:57.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.097+0000 7f5d96800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9010c550 con 0x7f5d9010bb50 2024-05-27T07:06:57.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.097+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9010bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.097+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9010bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57590/0 (socket says 172.21.15.12:57590) 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.097+0000 7f5d8f400700 1 -- 172.21.15.12:0/1379008217 learned_addr learned my addr 172.21.15.12:0/1379008217 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.098+0000 7f5d8f400700 1 -- 172.21.15.12:0/1379008217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9010cd20 con 0x7f5d9010bb50 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.098+0000 7f5d8f400700 1 --2- 172.21.15.12:0/1379008217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9010bf60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5d800098a0 tx=0x7f5d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70835b21bd9c37d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.098+0000 7f5d8ea00700 1 -- 172.21.15.12:0/1379008217 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d80004160 con 0x7f5d9010bb50 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.099+0000 7f5d8ea00700 1 -- 172.21.15.12:0/1379008217 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d80004300 con 0x7f5d9010bb50 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.099+0000 7f5d8ea00700 1 -- 172.21.15.12:0/1379008217 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d80011960 con 0x7f5d9010bb50 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.099+0000 7f5d96800700 1 -- 172.21.15.12:0/1379008217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 msgr2=0x7f5d9010bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.099+0000 7f5d96800700 1 --2- 172.21.15.12:0/1379008217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9010bf60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5d800098a0 tx=0x7f5d80009bb0 comp rx=0 tx=0).stop 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.099+0000 7f5d96800700 1 -- 172.21.15.12:0/1379008217 shutdown_connections 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.099+0000 7f5d96800700 1 --2- 172.21.15.12:0/1379008217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9010bf60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:57.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.099+0000 7f5d96800700 1 -- 172.21.15.12:0/1379008217 >> 172.21.15.12:0/1379008217 conn(0x7f5d90107470 msgr2=0x7f5d90109890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.100+0000 7f5d96800700 1 -- 172.21.15.12:0/1379008217 shutdown_connections 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.100+0000 7f5d96800700 1 -- 172.21.15.12:0/1379008217 wait complete. 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.101+0000 7f5d96800700 1 Processor -- start 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.101+0000 7f5d96800700 1 -- start start 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.101+0000 7f5d96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.101+0000 7f5d96800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d901100d0 con 0x7f5d9010bb50 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.102+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.102+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9019fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57600/0 (socket says 172.21.15.12:57600) 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.102+0000 7f5d8f400700 1 -- 172.21.15.12:0/1298524647 learned_addr learned my addr 172.21.15.12:0/1298524647 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.102+0000 7f5d8f400700 1 -- 172.21.15.12:0/1298524647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d901a03e0 con 0x7f5d9010bb50 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.103+0000 7f5d8f400700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9019fea0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d80006af0 tx=0x7f5d80011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.103+0000 7f5d8cc00700 1 -- 172.21.15.12:0/1298524647 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d80003af0 con 0x7f5d9010bb50 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.103+0000 7f5d8cc00700 1 -- 172.21.15.12:0/1298524647 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d8001b8d0 con 0x7f5d9010bb50 2024-05-27T07:06:57.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.103+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d901a0670 con 0x7f5d9010bb50 2024-05-27T07:06:57.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.103+0000 7f5d8cc00700 1 -- 172.21.15.12:0/1298524647 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d80023820 con 0x7f5d9010bb50 2024-05-27T07:06:57.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.103+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d901a3280 con 0x7f5d9010bb50 2024-05-27T07:06:57.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.104+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d90064e10 con 0x7f5d9010bb50 2024-05-27T07:06:57.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.104+0000 7f5d8cc00700 1 -- 172.21.15.12:0/1298524647 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5d8001b440 con 0x7f5d9010bb50 2024-05-27T07:06:57.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.105+0000 7f5d8cc00700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5d7403e620 0x7f5d74040b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:57.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.105+0000 7f5d8cc00700 1 -- 172.21.15.12:0/1298524647 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5d80054140 con 0x7f5d9010bb50 2024-05-27T07:06:57.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.105+0000 7f5d87e00700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5d7403e620 0x7f5d74040b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.106+0000 7f5d87e00700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5d7403e620 0x7f5d74040b30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006fd0 tx=0x7f5d78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.112+0000 7f5d8cc00700 1 -- 172.21.15.12:0/1298524647 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5d8001f030 con 0x7f5d9010bb50 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.474+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f5d901a3590 con 0x7f5d9010bb50 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.478+0000 7f5d8cc00700 1 -- 172.21.15.12:0/1298524647 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f5d80020da0 con 0x7f5d9010bb50 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5d7403e620 msgr2=0x7f5d74040b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5d7403e620 0x7f5d74040b30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006fd0 tx=0x7f5d78006e40 comp rx=0 tx=0).stop 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 msgr2=0x7f5d9019fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:57.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9019fea0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d80006af0 tx=0x7f5d80011a30 comp rx=0 tx=0).stop 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 shutdown_connections 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5d7403e620 0x7f5d74040b30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 --2- 172.21.15.12:0/1298524647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d9010bb50 0x7f5d9019fea0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.481+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 >> 172.21.15.12:0/1298524647 conn(0x7f5d90107470 msgr2=0x7f5d90108c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.483+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 shutdown_connections 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-05-27T07:06:56.483+0000 7f5d96800700 1 -- 172.21.15.12:0/1298524647 wait complete. 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-27T07:06:57.140 INFO:teuthology.orchestra.run.smithi012.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout:Or, if you are only running a single cluster on this host: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: ceph telemetry on 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout:For more information see: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.141 INFO:teuthology.orchestra.run.smithi012.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-27T07:06:57.142 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:06:57.142 INFO:teuthology.orchestra.run.smithi012.stdout:Bootstrap complete. 2024-05-27T07:06:57.240 INFO:tasks.cephadm:Fetching config... 2024-05-27T07:06:57.241 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:06:57.241 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-27T07:06:57.289 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-27T07:06:57.290 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:06:57.290 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-27T07:06:57.374 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-27T07:06:57.374 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:06:57.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-27T07:06:57.488 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-27T07:06:57.488 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:06:57.489 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-27T07:06:57.587 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-27T07:06:57.587 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNfXOIZ3wIXptZ5rxwG8OXj5zV6rcoORkVPZ7X6IQy4SK5ZluRcNOTwxkVCX8wS76ouPZwAZRdDrQXI5i5Le+1W9NJ16tgqzTl7+HL/UnumwIoqGvWLhoO0YADToKgQL5GNK6zngdNBSMLNah2p+WPCWKPgCskgG31q96WLfdjRwY2FC2pUcrpy4X9OxfXJXI+itv1r03DnQOxKP4mwC8P4S+jTpLkThViiLkrtJUfFFaY9ogkCRAS3ksZ0mgrjuRc9HgwY3yY05TNZZcw8qP3pdmnjC4RxkH3xcIFAmAnib1VaP5LXseiivyIbuldSGMH4T292i13c9sw3dIuvPoEx5vdk75YF3W0zCbuMYcxSurjpYGIs3GtvRm2xJpbE5BcLpvD/iFLGHOvGxoymMtXlWMKxbhDW+vVlXKucqZZnww6FKA3bJ+VtfpOXaJ9ns0yk3XfCkD1a7IWAyBaekpNLAPyxXNDkOuQNPsLNxRGYkfB3S16csLU9uhOuYdgIHc= ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:06:57.676 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:06:57 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1298524647' entity='client.admin' 2024-05-27T07:06:57.720 INFO:teuthology.orchestra.run.smithi012.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNfXOIZ3wIXptZ5rxwG8OXj5zV6rcoORkVPZ7X6IQy4SK5ZluRcNOTwxkVCX8wS76ouPZwAZRdDrQXI5i5Le+1W9NJ16tgqzTl7+HL/UnumwIoqGvWLhoO0YADToKgQL5GNK6zngdNBSMLNah2p+WPCWKPgCskgG31q96WLfdjRwY2FC2pUcrpy4X9OxfXJXI+itv1r03DnQOxKP4mwC8P4S+jTpLkThViiLkrtJUfFFaY9ogkCRAS3ksZ0mgrjuRc9HgwY3yY05TNZZcw8qP3pdmnjC4RxkH3xcIFAmAnib1VaP5LXseiivyIbuldSGMH4T292i13c9sw3dIuvPoEx5vdk75YF3W0zCbuMYcxSurjpYGIs3GtvRm2xJpbE5BcLpvD/iFLGHOvGxoymMtXlWMKxbhDW+vVlXKucqZZnww6FKA3bJ+VtfpOXaJ9ns0yk3XfCkD1a7IWAyBaekpNLAPyxXNDkOuQNPsLNxRGYkfB3S16csLU9uhOuYdgIHc= ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:06:57.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNfXOIZ3wIXptZ5rxwG8OXj5zV6rcoORkVPZ7X6IQy4SK5ZluRcNOTwxkVCX8wS76ouPZwAZRdDrQXI5i5Le+1W9NJ16tgqzTl7+HL/UnumwIoqGvWLhoO0YADToKgQL5GNK6zngdNBSMLNah2p+WPCWKPgCskgG31q96WLfdjRwY2FC2pUcrpy4X9OxfXJXI+itv1r03DnQOxKP4mwC8P4S+jTpLkThViiLkrtJUfFFaY9ogkCRAS3ksZ0mgrjuRc9HgwY3yY05TNZZcw8qP3pdmnjC4RxkH3xcIFAmAnib1VaP5LXseiivyIbuldSGMH4T292i13c9sw3dIuvPoEx5vdk75YF3W0zCbuMYcxSurjpYGIs3GtvRm2xJpbE5BcLpvD/iFLGHOvGxoymMtXlWMKxbhDW+vVlXKucqZZnww6FKA3bJ+VtfpOXaJ9ns0yk3XfCkD1a7IWAyBaekpNLAPyxXNDkOuQNPsLNxRGYkfB3S16csLU9uhOuYdgIHc= ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:06:57.881 INFO:teuthology.orchestra.run.smithi062.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNfXOIZ3wIXptZ5rxwG8OXj5zV6rcoORkVPZ7X6IQy4SK5ZluRcNOTwxkVCX8wS76ouPZwAZRdDrQXI5i5Le+1W9NJ16tgqzTl7+HL/UnumwIoqGvWLhoO0YADToKgQL5GNK6zngdNBSMLNah2p+WPCWKPgCskgG31q96WLfdjRwY2FC2pUcrpy4X9OxfXJXI+itv1r03DnQOxKP4mwC8P4S+jTpLkThViiLkrtJUfFFaY9ogkCRAS3ksZ0mgrjuRc9HgwY3yY05TNZZcw8qP3pdmnjC4RxkH3xcIFAmAnib1VaP5LXseiivyIbuldSGMH4T292i13c9sw3dIuvPoEx5vdk75YF3W0zCbuMYcxSurjpYGIs3GtvRm2xJpbE5BcLpvD/iFLGHOvGxoymMtXlWMKxbhDW+vVlXKucqZZnww6FKA3bJ+VtfpOXaJ9ns0yk3XfCkD1a7IWAyBaekpNLAPyxXNDkOuQNPsLNxRGYkfB3S16csLU9uhOuYdgIHc= ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:06:57.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNfXOIZ3wIXptZ5rxwG8OXj5zV6rcoORkVPZ7X6IQy4SK5ZluRcNOTwxkVCX8wS76ouPZwAZRdDrQXI5i5Le+1W9NJ16tgqzTl7+HL/UnumwIoqGvWLhoO0YADToKgQL5GNK6zngdNBSMLNah2p+WPCWKPgCskgG31q96WLfdjRwY2FC2pUcrpy4X9OxfXJXI+itv1r03DnQOxKP4mwC8P4S+jTpLkThViiLkrtJUfFFaY9ogkCRAS3ksZ0mgrjuRc9HgwY3yY05TNZZcw8qP3pdmnjC4RxkH3xcIFAmAnib1VaP5LXseiivyIbuldSGMH4T292i13c9sw3dIuvPoEx5vdk75YF3W0zCbuMYcxSurjpYGIs3GtvRm2xJpbE5BcLpvD/iFLGHOvGxoymMtXlWMKxbhDW+vVlXKucqZZnww6FKA3bJ+VtfpOXaJ9ns0yk3XfCkD1a7IWAyBaekpNLAPyxXNDkOuQNPsLNxRGYkfB3S16csLU9uhOuYdgIHc= ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:06:58.031 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNfXOIZ3wIXptZ5rxwG8OXj5zV6rcoORkVPZ7X6IQy4SK5ZluRcNOTwxkVCX8wS76ouPZwAZRdDrQXI5i5Le+1W9NJ16tgqzTl7+HL/UnumwIoqGvWLhoO0YADToKgQL5GNK6zngdNBSMLNah2p+WPCWKPgCskgG31q96WLfdjRwY2FC2pUcrpy4X9OxfXJXI+itv1r03DnQOxKP4mwC8P4S+jTpLkThViiLkrtJUfFFaY9ogkCRAS3ksZ0mgrjuRc9HgwY3yY05TNZZcw8qP3pdmnjC4RxkH3xcIFAmAnib1VaP5LXseiivyIbuldSGMH4T292i13c9sw3dIuvPoEx5vdk75YF3W0zCbuMYcxSurjpYGIs3GtvRm2xJpbE5BcLpvD/iFLGHOvGxoymMtXlWMKxbhDW+vVlXKucqZZnww6FKA3bJ+VtfpOXaJ9ns0yk3XfCkD1a7IWAyBaekpNLAPyxXNDkOuQNPsLNxRGYkfB3S16csLU9uhOuYdgIHc= ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:06:58.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-27T07:06:58.526 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.398+0000 7f3cfac00700 1 -- 172.21.15.12:0/1096641323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf4103bd0 msgr2=0x7f3cf4103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.398+0000 7f3cfac00700 1 --2- 172.21.15.12:0/1096641323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf4103bd0 0x7f3cf4103fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009910 tx=0x7f3ce4009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.399+0000 7f3cfac00700 1 -- 172.21.15.12:0/1096641323 shutdown_connections 2024-05-27T07:07:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.399+0000 7f3cfac00700 1 --2- 172.21.15.12:0/1096641323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf4103bd0 0x7f3cf4103fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.399+0000 7f3cfac00700 1 -- 172.21.15.12:0/1096641323 >> 172.21.15.12:0/1096641323 conn(0x7f3cf40ff210 msgr2=0x7f3cf41016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.399+0000 7f3cfac00700 1 -- 172.21.15.12:0/1096641323 shutdown_connections 2024-05-27T07:07:01.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.399+0000 7f3cfac00700 1 -- 172.21.15.12:0/1096641323 wait complete. 2024-05-27T07:07:01.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.400+0000 7f3cfac00700 1 Processor -- start 2024-05-27T07:07:01.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.400+0000 7f3cfac00700 1 -- start start 2024-05-27T07:07:01.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.401+0000 7f3cfac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf41996f0 0x7f3cf4199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:01.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.401+0000 7f3cfac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4109a40 con 0x7f3cf41996f0 2024-05-27T07:07:01.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.401+0000 7f3cf3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf41996f0 0x7f3cf4199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:01.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.401+0000 7f3cf3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf41996f0 0x7f3cf4199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57604/0 (socket says 172.21.15.12:57604) 2024-05-27T07:07:01.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.401+0000 7f3cf3e00700 1 -- 172.21.15.12:0/3573779094 learned_addr learned my addr 172.21.15.12:0/3573779094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:07:01.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.401+0000 7f3cf3e00700 1 -- 172.21.15.12:0/3573779094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf419a0a0 con 0x7f3cf41996f0 2024-05-27T07:07:01.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.402+0000 7f3cf3e00700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf41996f0 0x7f3cf4199b20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4005950 tx=0x7f3ce4005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:01.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.402+0000 7f3cf0c00700 1 -- 172.21.15.12:0/3573779094 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce4027070 con 0x7f3cf41996f0 2024-05-27T07:07:01.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.402+0000 7f3cf0c00700 1 -- 172.21.15.12:0/3573779094 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ce400bdf0 con 0x7f3cf41996f0 2024-05-27T07:07:01.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.402+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf419a330 con 0x7f3cf41996f0 2024-05-27T07:07:01.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.402+0000 7f3cf0c00700 1 -- 172.21.15.12:0/3573779094 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce400fcb0 con 0x7f3cf41996f0 2024-05-27T07:07:01.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.402+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf419ce60 con 0x7f3cf41996f0 2024-05-27T07:07:01.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.403+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf410a1c0 con 0x7f3cf41996f0 2024-05-27T07:07:01.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.404+0000 7f3cf0c00700 1 -- 172.21.15.12:0/3573779094 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f3ce402a070 con 0x7f3cf41996f0 2024-05-27T07:07:01.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.404+0000 7f3cf0c00700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3cd803e570 0x7f3cd8040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:01.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.404+0000 7f3cf0c00700 1 -- 172.21.15.12:0/3573779094 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3ce4053740 con 0x7f3cf41996f0 2024-05-27T07:07:01.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.405+0000 7f3cf3400700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3cd803e570 0x7f3cd8040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:01.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.405+0000 7f3cf3400700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3cd803e570 0x7f3cd8040a80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0006fd0 tx=0x7f3ce0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:01.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.410+0000 7f3cf0c00700 1 -- 172.21.15.12:0/3573779094 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3ce4013070 con 0x7f3cf41996f0 2024-05-27T07:07:01.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.650+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f3cf402e720 con 0x7f3cf41996f0 2024-05-27T07:07:01.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.654+0000 7f3cf0c00700 1 -- 172.21.15.12:0/3573779094 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f3ce400d340 con 0x7f3cf41996f0 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3cd803e570 msgr2=0x7f3cd8040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3cd803e570 0x7f3cd8040a80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0006fd0 tx=0x7f3ce0006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf41996f0 msgr2=0x7f3cf4199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf41996f0 0x7f3cf4199b20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4005950 tx=0x7f3ce4005810 comp rx=0 tx=0).stop 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 shutdown_connections 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3cd803e570 0x7f3cd8040a80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 --2- 172.21.15.12:0/3573779094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cf41996f0 0x7f3cf4199b20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.657+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 >> 172.21.15.12:0/3573779094 conn(0x7f3cf40ff210 msgr2=0x7f3cf40ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:01.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.658+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 shutdown_connections 2024-05-27T07:07:01.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:01.658+0000 7f3cfac00700 1 -- 172.21.15.12:0/3573779094 wait complete. 2024-05-27T07:07:02.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:01 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:02.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:01 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:02.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:01 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi012", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:07:02.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:01 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:02.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:01 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:02.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:01 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:02.442 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-27T07:07:02.442 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-27T07:07:02.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:02 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3573779094' entity='client.admin' 2024-05-27T07:07:02.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:02.986 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:03.514 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:05.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.488+0000 7f1fe3600700 1 -- 172.21.15.12:0/3810001679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 msgr2=0x7f1fdc10a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.488+0000 7f1fe3600700 1 --2- 172.21.15.12:0/3810001679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc10a5e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009910 tx=0x7f1fd0009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.488+0000 7f1fe3600700 1 -- 172.21.15.12:0/3810001679 shutdown_connections 2024-05-27T07:07:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.488+0000 7f1fe3600700 1 --2- 172.21.15.12:0/3810001679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc10a5e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.488+0000 7f1fe3600700 1 -- 172.21.15.12:0/3810001679 >> 172.21.15.12:0/3810001679 conn(0x7f1fdc079240 msgr2=0x7f1fdc07b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.488+0000 7f1fe3600700 1 -- 172.21.15.12:0/3810001679 shutdown_connections 2024-05-27T07:07:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.488+0000 7f1fe3600700 1 -- 172.21.15.12:0/3810001679 wait complete. 2024-05-27T07:07:05.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.490+0000 7f1fe3600700 1 Processor -- start 2024-05-27T07:07:05.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.490+0000 7f1fe3600700 1 -- start start 2024-05-27T07:07:05.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.491+0000 7f1fe3600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:05.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.491+0000 7f1fe3600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1fdc110040 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.491+0000 7f1fe0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.491+0000 7f1fe0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc19fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35442/0 (socket says 172.21.15.12:35442) 2024-05-27T07:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.491+0000 7f1fe0c00700 1 -- 172.21.15.12:0/4107607987 learned_addr learned my addr 172.21.15.12:0/4107607987 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:07:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.491+0000 7f1fe0c00700 1 -- 172.21.15.12:0/4107607987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc1a0170 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.492+0000 7f1fe0c00700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc19fc30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0005950 tx=0x7f1fd0005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.492+0000 7f1fd9600700 1 -- 172.21.15.12:0/4107607987 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fd0027070 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.492+0000 7f1fd9600700 1 -- 172.21.15.12:0/4107607987 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fd000bdf0 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.492+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fdc1a0400 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.492+0000 7f1fd9600700 1 -- 172.21.15.12:0/4107607987 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fd000fdb0 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.492+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fdc1a3010 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.493+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fdc1107c0 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.494+0000 7f1fd9600700 1 -- 172.21.15.12:0/4107607987 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f1fd002a070 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.494+0000 7f1fd9600700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1fc403e570 0x7f1fc4040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.494+0000 7f1fd9600700 1 -- 172.21.15.12:0/4107607987 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1fd0053840 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.494+0000 7f1fdbe00700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1fc403e570 0x7f1fc4040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.495+0000 7f1fdbe00700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1fc403e570 0x7f1fc4040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc006fd0 tx=0x7f1fcc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.500+0000 7f1fd9600700 1 -- 172.21.15.12:0/4107607987 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1fd0013070 con 0x7f1fdc10a1d0 2024-05-27T07:07:05.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.742+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f1fdc02e680 con 0x7f1fc403e570 2024-05-27T07:07:05.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.752+0000 7f1fd9600700 1 -- 172.21.15.12:0/4107607987 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1fdc02e680 con 0x7f1fc403e570 2024-05-27T07:07:05.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1fc403e570 msgr2=0x7f1fc4040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1fc403e570 0x7f1fc4040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc006fd0 tx=0x7f1fcc006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 msgr2=0x7f1fdc19fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc19fc30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0005950 tx=0x7f1fd0005810 comp rx=0 tx=0).stop 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 shutdown_connections 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1fc403e570 0x7f1fc4040a80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 --2- 172.21.15.12:0/4107607987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fdc10a1d0 0x7f1fdc19fc30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.755+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 >> 172.21.15.12:0/4107607987 conn(0x7f1fdc079240 msgr2=0x7f1fdc079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.756+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 shutdown_connections 2024-05-27T07:07:05.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:05.756+0000 7f1fe3600700 1 -- 172.21.15.12:0/4107607987 wait complete. 2024-05-27T07:07:06.508 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi062 2024-05-27T07:07:06.508 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:07:06.508 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/etc/ceph/ceph.conf 2024-05-27T07:07:06.573 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:07:06.574 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:07:06.674 INFO:tasks.cephadm:Adding host smithi062 to orchestrator... 2024-05-27T07:07:06.674 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch host add smithi062 2024-05-27T07:07:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:06 smithi012 ceph-mon[31473]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:06 smithi012 ceph-mon[31473]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:07:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:07:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:06 smithi012 ceph-mon[31473]: Updating smithi012:/etc/ceph/ceph.conf 2024-05-27T07:07:07.102 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:07 smithi012 ceph-mon[31473]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:07:08.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:08 smithi012 ceph-mon[31473]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:07:08.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:08 smithi012 ceph-mon[31473]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:08.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:08 smithi012 ceph-mon[31473]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:07:08.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:08.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:08.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:09.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.149+0000 7fab62c00700 1 -- 172.21.15.12:0/3895718457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 msgr2=0x7fab5c104f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.149+0000 7fab62c00700 1 --2- 172.21.15.12:0/3895718457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c104f40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fab50009910 tx=0x7fab50009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.149+0000 7fab62c00700 1 -- 172.21.15.12:0/3895718457 shutdown_connections 2024-05-27T07:07:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.149+0000 7fab62c00700 1 --2- 172.21.15.12:0/3895718457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c104f40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.149+0000 7fab62c00700 1 -- 172.21.15.12:0/3895718457 >> 172.21.15.12:0/3895718457 conn(0x7fab5c078440 msgr2=0x7fab5c078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.150+0000 7fab62c00700 1 -- 172.21.15.12:0/3895718457 shutdown_connections 2024-05-27T07:07:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.150+0000 7fab62c00700 1 -- 172.21.15.12:0/3895718457 wait complete. 2024-05-27T07:07:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.151+0000 7fab62c00700 1 Processor -- start 2024-05-27T07:07:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.151+0000 7fab62c00700 1 -- start start 2024-05-27T07:07:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.152+0000 7fab62c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c1a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.152+0000 7fab62c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c10dde0 con 0x7fab5c104b30 2024-05-27T07:07:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.152+0000 7fab61800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c1a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.152+0000 7fab61800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c1a1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35468/0 (socket says 172.21.15.12:35468) 2024-05-27T07:07:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.152+0000 7fab61800700 1 -- 172.21.15.12:0/1702820413 learned_addr learned my addr 172.21.15.12:0/1702820413 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:07:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.153+0000 7fab61800700 1 -- 172.21.15.12:0/1702820413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c1a1ad0 con 0x7fab5c104b30 2024-05-27T07:07:09.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.153+0000 7fab61800700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c1a1590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab50006010 tx=0x7fab50005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:09.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.153+0000 7fab5a000700 1 -- 172.21.15.12:0/1702820413 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab50026070 con 0x7fab5c104b30 2024-05-27T07:07:09.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.153+0000 7fab5a000700 1 -- 172.21.15.12:0/1702820413 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab5000b750 con 0x7fab5c104b30 2024-05-27T07:07:09.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.153+0000 7fab5a000700 1 -- 172.21.15.12:0/1702820413 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab50003810 con 0x7fab5c104b30 2024-05-27T07:07:09.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.153+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c1a1d60 con 0x7fab5c104b30 2024-05-27T07:07:09.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.154+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c19b720 con 0x7fab5c104b30 2024-05-27T07:07:09.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.155+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5c06b620 con 0x7fab5c104b30 2024-05-27T07:07:09.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.155+0000 7fab5a000700 1 -- 172.21.15.12:0/1702820413 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fab50029070 con 0x7fab5c104b30 2024-05-27T07:07:09.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.156+0000 7fab5a000700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fab4403e570 0x7fab44040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:09.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.156+0000 7fab5a000700 1 -- 172.21.15.12:0/1702820413 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fab5001e070 con 0x7fab5c104b30 2024-05-27T07:07:09.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.156+0000 7fab60e00700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fab4403e570 0x7fab44040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:09.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.157+0000 7fab60e00700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fab4403e570 0x7fab44040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fab48006fd0 tx=0x7fab48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:09.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.162+0000 7fab5a000700 1 -- 172.21.15.12:0/1702820413 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fab50013070 con 0x7fab5c104b30 2024-05-27T07:07:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:09.406+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi062", "target": ["mon-mgr", ""]}) -- 0x7fab5c064e10 con 0x7fab4403e570 2024-05-27T07:07:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:10 smithi012 ceph-mon[31473]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi062", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:07:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:10 smithi012 ceph-mon[31473]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:12.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:11 smithi012 ceph-mon[31473]: Deploying cephadm binary to smithi062 2024-05-27T07:07:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:12 smithi012 ceph-mon[31473]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:15.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:14 smithi012 ceph-mon[31473]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:15.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.952+0000 7fab5a000700 1 -- 172.21.15.12:0/1702820413 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fab5c064e10 con 0x7fab4403e570 2024-05-27T07:07:15.944 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi062' with addr '172.21.15.62' 2024-05-27T07:07:15.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fab4403e570 msgr2=0x7fab44040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:15.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fab4403e570 0x7fab44040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fab48006fd0 tx=0x7fab48006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:15.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 msgr2=0x7fab5c1a1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:15.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c1a1590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab50006010 tx=0x7fab50005710 comp rx=0 tx=0).stop 2024-05-27T07:07:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 shutdown_connections 2024-05-27T07:07:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fab4403e570 0x7fab44040a80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 --2- 172.21.15.12:0/1702820413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab5c104b30 0x7fab5c1a1590 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 >> 172.21.15.12:0/1702820413 conn(0x7fab5c078440 msgr2=0x7fab5c109d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 shutdown_connections 2024-05-27T07:07:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:15.956+0000 7fab62c00700 1 -- 172.21.15.12:0/1702820413 wait complete. 2024-05-27T07:07:16.678 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T07:07:17.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:16 smithi012 ceph-mon[31473]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:18.335 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:17 smithi012 ceph-mon[31473]: Added host smithi062 2024-05-27T07:07:18.335 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:17 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:18.335 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:17 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:18.966 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:18 smithi012 ceph-mon[31473]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.225+0000 7faad1e00700 1 -- 172.21.15.12:0/2860180160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 msgr2=0x7faacc10a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:19.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.225+0000 7faad1e00700 1 --2- 172.21.15.12:0/2860180160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc10a040 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faabc009910 tx=0x7faabc009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:19.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.225+0000 7faad1e00700 1 -- 172.21.15.12:0/2860180160 shutdown_connections 2024-05-27T07:07:19.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.225+0000 7faad1e00700 1 --2- 172.21.15.12:0/2860180160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc10a040 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:19.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.225+0000 7faad1e00700 1 -- 172.21.15.12:0/2860180160 >> 172.21.15.12:0/2860180160 conn(0x7faacc0ff650 msgr2=0x7faacc101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.225+0000 7faad1e00700 1 -- 172.21.15.12:0/2860180160 shutdown_connections 2024-05-27T07:07:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.226+0000 7faad1e00700 1 -- 172.21.15.12:0/2860180160 wait complete. 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.227+0000 7faad1e00700 1 Processor -- start 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.227+0000 7faad1e00700 1 -- start start 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.227+0000 7faad1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc1a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.228+0000 7faad1e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faacc10dfc0 con 0x7faacc109c30 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.228+0000 7faacaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc1a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.228+0000 7faacaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc1a1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40556/0 (socket says 172.21.15.12:40556) 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.228+0000 7faacaa00700 1 -- 172.21.15.12:0/1667711040 learned_addr learned my addr 172.21.15.12:0/1667711040 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.228+0000 7faacaa00700 1 -- 172.21.15.12:0/1667711040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faacc1a1b70 con 0x7faacc109c30 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.229+0000 7faacaa00700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc1a1630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faabc0092e0 tx=0x7faabc005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.229+0000 7faad0a00700 1 -- 172.21.15.12:0/1667711040 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faabc026070 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.229+0000 7faad0a00700 1 -- 172.21.15.12:0/1667711040 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faabc00b750 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.229+0000 7faad0a00700 1 -- 172.21.15.12:0/1667711040 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faabc003810 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.229+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faacc1a1e00 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.229+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faacc19b870 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.231+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faacc105200 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.231+0000 7faad0a00700 1 -- 172.21.15.12:0/1667711040 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7faabc029070 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.231+0000 7faad0a00700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7faab003e570 0x7faab0040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.232+0000 7faad0a00700 1 -- 172.21.15.12:0/1667711040 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faabc01e070 con 0x7faacc109c30 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.232+0000 7faaca000700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7faab003e570 0x7faab0040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.233+0000 7faaca000700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7faab003e570 0x7faab0040a80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faab8006fd0 tx=0x7faab8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.238+0000 7faad0a00700 1 -- 172.21.15.12:0/1667711040 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faabc013070 con 0x7faacc109c30 2024-05-27T07:07:19.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.487+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faacc19cd80 con 0x7faab003e570 2024-05-27T07:07:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.489+0000 7faad0a00700 1 -- 172.21.15.12:0/1667711040 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7faacc19cd80 con 0x7faab003e570 2024-05-27T07:07:19.481 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:07:19.481 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}] 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7faab003e570 msgr2=0x7faab0040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7faab003e570 0x7faab0040a80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faab8006fd0 tx=0x7faab8006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 msgr2=0x7faacc1a1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc1a1630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faabc0092e0 tx=0x7faabc005710 comp rx=0 tx=0).stop 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 shutdown_connections 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7faab003e570 0x7faab0040a80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 --2- 172.21.15.12:0/1667711040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faacc109c30 0x7faacc1a1630 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 >> 172.21.15.12:0/1667711040 conn(0x7faacc0ff650 msgr2=0x7faacc0ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 shutdown_connections 2024-05-27T07:07:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:19.493+0000 7faad1e00700 1 -- 172.21.15.12:0/1667711040 wait complete. 2024-05-27T07:07:20.256 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-05-27T07:07:20.256 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:07:20.256 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-05-27T07:07:20.315 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:07:20.316 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:07:20.411 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-05-27T07:07:20.411 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch host add smithi162 2024-05-27T07:07:20.884 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:21.184 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:20 smithi012 ceph-mon[31473]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:07:21.184 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:20 smithi012 ceph-mon[31473]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:21.185 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:22.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.914+0000 7f3f22200700 1 -- 172.21.15.12:0/3711532326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 msgr2=0x7f3f1c10a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:22.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.914+0000 7f3f22200700 1 --2- 172.21.15.12:0/3711532326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c10a7d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009910 tx=0x7f3f0c009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:22.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.915+0000 7f3f22200700 1 -- 172.21.15.12:0/3711532326 shutdown_connections 2024-05-27T07:07:22.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.915+0000 7f3f22200700 1 --2- 172.21.15.12:0/3711532326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c10a7d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:22.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.915+0000 7f3f22200700 1 -- 172.21.15.12:0/3711532326 >> 172.21.15.12:0/3711532326 conn(0x7f3f1c0799e0 msgr2=0x7f3f1c1085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:22.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.915+0000 7f3f22200700 1 -- 172.21.15.12:0/3711532326 shutdown_connections 2024-05-27T07:07:22.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.915+0000 7f3f22200700 1 -- 172.21.15.12:0/3711532326 wait complete. 2024-05-27T07:07:22.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:22 smithi012 ceph-mon[31473]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:22.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.916+0000 7f3f22200700 1 Processor -- start 2024-05-27T07:07:22.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.916+0000 7f3f22200700 1 -- start start 2024-05-27T07:07:22.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.917+0000 7f3f22200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c19fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:22.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.917+0000 7f3f22200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c114560 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.917+0000 7f3f1b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c19fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:22.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.917+0000 7f3f1b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c19fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45676/0 (socket says 172.21.15.12:45676) 2024-05-27T07:07:22.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.917+0000 7f3f1b400700 1 -- 172.21.15.12:0/4251305789 learned_addr learned my addr 172.21.15.12:0/4251305789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:07:22.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.917+0000 7f3f1b400700 1 -- 172.21.15.12:0/4251305789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1c1a03c0 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.918+0000 7f3f1b400700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c19fe80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006010 tx=0x7f3f0c005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:22.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.918+0000 7f3f13e00700 1 -- 172.21.15.12:0/4251305789 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f0c026070 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.918+0000 7f3f13e00700 1 -- 172.21.15.12:0/4251305789 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f0c00b6f0 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.918+0000 7f3f13e00700 1 -- 172.21.15.12:0/4251305789 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f0c003810 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.918+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1c1a06b0 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.918+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f1c1a4660 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.919+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f1c10b870 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.920+0000 7f3f13e00700 1 -- 172.21.15.12:0/4251305789 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f3f0c029070 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.920+0000 7f3f13e00700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3f0003e570 0x7f3f00040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:22.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.920+0000 7f3f13e00700 1 -- 172.21.15.12:0/4251305789 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3f0c01e070 con 0x7f3f1c10a3c0 2024-05-27T07:07:22.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.921+0000 7f3f1aa00700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3f0003e570 0x7f3f00040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:22.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.921+0000 7f3f1aa00700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3f0003e570 0x7f3f00040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3f08006fd0 tx=0x7f3f08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:22.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:22.926+0000 7f3f13e00700 1 -- 172.21.15.12:0/4251305789 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3f0c013070 con 0x7f3f1c10a3c0 2024-05-27T07:07:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:23.170+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}) -- 0x7f3f1c1a1190 con 0x7f3f0003e570 2024-05-27T07:07:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:24 smithi012 ceph-mon[31473]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:07:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:24 smithi012 ceph-mon[31473]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:26.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: Deploying cephadm binary to smithi162 2024-05-27T07:07:26.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:26.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:26.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:26.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:26.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:07:26.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:26.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:07:27.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:26 smithi012 ceph-mon[31473]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:07:27.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:26 smithi012 ceph-mon[31473]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:28 smithi012 ceph-mon[31473]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.505+0000 7f3f13e00700 1 -- 172.21.15.12:0/4251305789 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3f1c1a1190 con 0x7f3f0003e570 2024-05-27T07:07:29.497 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-05-27T07:07:29.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3f0003e570 msgr2=0x7f3f00040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:29.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3f0003e570 0x7f3f00040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3f08006fd0 tx=0x7f3f08006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:29.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 msgr2=0x7f3f1c19fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:29.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c19fe80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006010 tx=0x7f3f0c005710 comp rx=0 tx=0).stop 2024-05-27T07:07:29.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 shutdown_connections 2024-05-27T07:07:29.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f3f0003e570 0x7f3f00040a80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:29.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 --2- 172.21.15.12:0/4251305789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f1c10a3c0 0x7f3f1c19fe80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:29.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.509+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 >> 172.21.15.12:0/4251305789 conn(0x7f3f1c0799e0 msgr2=0x7f3f1c10bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:29.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.510+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 shutdown_connections 2024-05-27T07:07:29.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:29.510+0000 7f3f22200700 1 -- 172.21.15.12:0/4251305789 wait complete. 2024-05-27T07:07:30.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:29 smithi012 ceph-mon[31473]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:07:30.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:30.234 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T07:07:30.681 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:30 smithi012 ceph-mon[31473]: Added host smithi162 2024-05-27T07:07:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:30 smithi012 ceph-mon[31473]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:30 smithi012 ceph-mon[31473]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:07:31.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:31 smithi012 ceph-mon[31473]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:07:31.982 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:31 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:31.983 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:31 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:31.983 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:31 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:31.983 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:31 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.854+0000 7f264d800700 1 -- 172.21.15.12:0/676685497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 msgr2=0x7f26481041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.854+0000 7f264d800700 1 --2- 172.21.15.12:0/676685497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481041f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f263800b0a0 tx=0x7f263800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:07:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.855+0000 7f264d800700 1 -- 172.21.15.12:0/676685497 shutdown_connections 2024-05-27T07:07:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.855+0000 7f264d800700 1 --2- 172.21.15.12:0/676685497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481041f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.855+0000 7f264d800700 1 -- 172.21.15.12:0/676685497 >> 172.21.15.12:0/676685497 conn(0x7f26480ff690 msgr2=0x7f2648101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.855+0000 7f264d800700 1 -- 172.21.15.12:0/676685497 shutdown_connections 2024-05-27T07:07:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.855+0000 7f264d800700 1 -- 172.21.15.12:0/676685497 wait complete. 2024-05-27T07:07:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.856+0000 7f264d800700 1 Processor -- start 2024-05-27T07:07:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.856+0000 7f264d800700 1 -- start start 2024-05-27T07:07:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.857+0000 7f264d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.857+0000 7f264d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f264810df80 con 0x7f2648103de0 2024-05-27T07:07:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.857+0000 7f2646a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.857+0000 7f2646a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37822/0 (socket says 172.21.15.12:37822) 2024-05-27T07:07:32.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.857+0000 7f2646a00700 1 -- 172.21.15.12:0/349013354 learned_addr learned my addr 172.21.15.12:0/349013354 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:07:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.857+0000 7f2646a00700 1 -- 172.21.15.12:0/349013354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26481a1ba0 con 0x7f2648103de0 2024-05-27T07:07:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.858+0000 7f2646a00700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481a1660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2638007600 tx=0x7f263800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.858+0000 7f263fe00700 1 -- 172.21.15.12:0/349013354 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2638017040 con 0x7f2648103de0 2024-05-27T07:07:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.858+0000 7f263fe00700 1 -- 172.21.15.12:0/349013354 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2638022070 con 0x7f2648103de0 2024-05-27T07:07:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.858+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26481a1e30 con 0x7f2648103de0 2024-05-27T07:07:32.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.859+0000 7f263fe00700 1 -- 172.21.15.12:0/349013354 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2638004870 con 0x7f2648103de0 2024-05-27T07:07:32.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.859+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f264819b880 con 0x7f2648103de0 2024-05-27T07:07:32.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.860+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2648105290 con 0x7f2648103de0 2024-05-27T07:07:32.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.860+0000 7f263fe00700 1 -- 172.21.15.12:0/349013354 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f2638003a00 con 0x7f2648103de0 2024-05-27T07:07:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.861+0000 7f263fe00700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f262c03e560 0x7f262c040a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.861+0000 7f263fe00700 1 -- 172.21.15.12:0/349013354 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f263800f050 con 0x7f2648103de0 2024-05-27T07:07:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.861+0000 7f263e000700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f262c03e560 0x7f262c040a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.862+0000 7f263e000700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f262c03e560 0x7f262c040a70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2630006fd0 tx=0x7f2630006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:32.867+0000 7f263fe00700 1 -- 172.21.15.12:0/349013354 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f263802d250 con 0x7f2648103de0 2024-05-27T07:07:33.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.112+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f264819cdb0 con 0x7f262c03e560 2024-05-27T07:07:33.105 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:32 smithi012 ceph-mon[31473]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:33.105 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:33.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.114+0000 7f263fe00700 1 -- 172.21.15.12:0/349013354 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f264819cdb0 con 0x7f262c03e560 2024-05-27T07:07:33.107 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:07:33.107 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f262c03e560 msgr2=0x7f262c040a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f262c03e560 0x7f262c040a70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2630006fd0 tx=0x7f2630006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 msgr2=0x7f26481a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481a1660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2638007600 tx=0x7f263800bd40 comp rx=0 tx=0).stop 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 shutdown_connections 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f262c03e560 0x7f262c040a70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 --2- 172.21.15.12:0/349013354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2648103de0 0x7f26481a1660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 >> 172.21.15.12:0/349013354 conn(0x7f26480ff690 msgr2=0x7f2648075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 shutdown_connections 2024-05-27T07:07:33.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:33.118+0000 7f264d800700 1 -- 172.21.15.12:0/349013354 wait complete. 2024-05-27T07:07:33.949 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-27T07:07:33.950 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-05-27T07:07:34.411 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:34.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:34 smithi012 ceph-mon[31473]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:07:34.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:34.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:34 smithi012 ceph-mon[31473]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.501+0000 7fa7ac800700 1 -- 172.21.15.12:0/1557478365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 msgr2=0x7fa7a4108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.501+0000 7fa7ac800700 1 --2- 172.21.15.12:0/1557478365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a4108e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa798009910 tx=0x7fa798009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.501+0000 7fa7ac800700 1 -- 172.21.15.12:0/1557478365 shutdown_connections 2024-05-27T07:07:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.501+0000 7fa7ac800700 1 --2- 172.21.15.12:0/1557478365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a4108e90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:36.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.501+0000 7fa7ac800700 1 -- 172.21.15.12:0/1557478365 >> 172.21.15.12:0/1557478365 conn(0x7fa7a40785a0 msgr2=0x7fa7a4078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.501+0000 7fa7ac800700 1 -- 172.21.15.12:0/1557478365 shutdown_connections 2024-05-27T07:07:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.501+0000 7fa7ac800700 1 -- 172.21.15.12:0/1557478365 wait complete. 2024-05-27T07:07:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.503+0000 7fa7ac800700 1 Processor -- start 2024-05-27T07:07:36.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.503+0000 7fa7ac800700 1 -- start start 2024-05-27T07:07:36.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.503+0000 7fa7ac800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:36.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.503+0000 7fa7ac800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a406b680 con 0x7fa7a4108a80 2024-05-27T07:07:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.504+0000 7fa7a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.504+0000 7fa7a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a419f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37834/0 (socket says 172.21.15.12:37834) 2024-05-27T07:07:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.504+0000 7fa7a9e00700 1 -- 172.21.15.12:0/2407117628 learned_addr learned my addr 172.21.15.12:0/2407117628 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:07:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.504+0000 7fa7a9e00700 1 -- 172.21.15.12:0/2407117628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a419fa50 con 0x7fa7a4108a80 2024-05-27T07:07:36.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.504+0000 7fa7a9e00700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a419f510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa7980092e0 tx=0x7fa798005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.505+0000 7fa79ea00700 1 -- 172.21.15.12:0/2407117628 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa798026070 con 0x7fa7a4108a80 2024-05-27T07:07:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.505+0000 7fa79ea00700 1 -- 172.21.15.12:0/2407117628 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa79800b6f0 con 0x7fa7a4108a80 2024-05-27T07:07:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.505+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a419fd40 con 0x7fa7a4108a80 2024-05-27T07:07:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.505+0000 7fa79ea00700 1 -- 172.21.15.12:0/2407117628 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7980036c0 con 0x7fa7a4108a80 2024-05-27T07:07:36.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.505+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a4199870 con 0x7fa7a4108a80 2024-05-27T07:07:36.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.506+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a4104190 con 0x7fa7a4108a80 2024-05-27T07:07:36.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.507+0000 7fa79ea00700 1 -- 172.21.15.12:0/2407117628 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa798029070 con 0x7fa7a4108a80 2024-05-27T07:07:36.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.507+0000 7fa79ea00700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa78c03e510 0x7fa78c040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:36.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.507+0000 7fa79ea00700 1 -- 172.21.15.12:0/2407117628 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa79801e070 con 0x7fa7a4108a80 2024-05-27T07:07:36.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.508+0000 7fa7a9400700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa78c03e510 0x7fa78c040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:36.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.508+0000 7fa7a9400700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa78c03e510 0x7fa78c040a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa794006fd0 tx=0x7fa794006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:36.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.513+0000 7fa79ea00700 1 -- 172.21.15.12:0/2407117628 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa798013070 con 0x7fa7a4108a80 2024-05-27T07:07:36.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:36.763+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa7a419ff70 con 0x7fa7a4108a80 2024-05-27T07:07:36.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:36 smithi012 ceph-mon[31473]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:36.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.598+0000 7fa79ea00700 1 -- 172.21.15.12:0/2407117628 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fa79800d340 con 0x7fa7a4108a80 2024-05-27T07:07:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.602+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa78c03e510 msgr2=0x7fa78c040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.602+0000 7fa7ac800700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa78c03e510 0x7fa78c040a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa794006fd0 tx=0x7fa794006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.602+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 msgr2=0x7fa7a419f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:37.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.602+0000 7fa7ac800700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a419f510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa7980092e0 tx=0x7fa798005710 comp rx=0 tx=0).stop 2024-05-27T07:07:37.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.603+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 shutdown_connections 2024-05-27T07:07:37.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.603+0000 7fa7ac800700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa78c03e510 0x7fa78c040a20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:37.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.603+0000 7fa7ac800700 1 --2- 172.21.15.12:0/2407117628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7a4108a80 0x7fa7a419f510 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:37.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.603+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 >> 172.21.15.12:0/2407117628 conn(0x7fa7a40785a0 msgr2=0x7fa7a41011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:37.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.603+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 shutdown_connections 2024-05-27T07:07:37.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:07:37.603+0000 7fa7ac800700 1 -- 172.21.15.12:0/2407117628 wait complete. 2024-05-27T07:07:37.596 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-05-27T07:07:37.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:37 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2407117628' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:07:38.342 INFO:tasks.cephadm:Adding mon.a on smithi012 2024-05-27T07:07:38.342 INFO:tasks.cephadm:Adding mon.b on smithi062 2024-05-27T07:07:38.342 INFO:tasks.cephadm:Adding mon.c on smithi162 2024-05-27T07:07:38.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi012:172.21.15.12=a;smithi062:172.21.15.62=b;smithi162:172.21.15.162=c' 2024-05-27T07:07:38.754 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-27T07:07:38.862 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-27T07:07:39.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:38 smithi012 ceph-mon[31473]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:39.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:38 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2407117628' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:07:39.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:38 smithi012 ceph-mon[31473]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:07:41.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:40 smithi012 ceph-mon[31473]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:41.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.534+0000 7f8adf200700 1 -- 172.21.15.162:0/4088950279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 msgr2=0x7f8ad8078500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:41.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.534+0000 7f8adf200700 1 --2- 172.21.15.162:0/4088950279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad8078500 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009910 tx=0x7f8acc009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:41.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.534+0000 7f8adf200700 1 -- 172.21.15.162:0/4088950279 shutdown_connections 2024-05-27T07:07:41.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.534+0000 7f8adf200700 1 --2- 172.21.15.162:0/4088950279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad8078500 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:41.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.534+0000 7f8adf200700 1 -- 172.21.15.162:0/4088950279 >> 172.21.15.162:0/4088950279 conn(0x7f8ad8101250 msgr2=0x7f8ad8103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:41.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.534+0000 7f8adf200700 1 -- 172.21.15.162:0/4088950279 shutdown_connections 2024-05-27T07:07:41.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.534+0000 7f8adf200700 1 -- 172.21.15.162:0/4088950279 wait complete. 2024-05-27T07:07:41.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.535+0000 7f8adf200700 1 Processor -- start 2024-05-27T07:07:41.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.536+0000 7f8adf200700 1 -- start start 2024-05-27T07:07:41.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.536+0000 7f8adf200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:41.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.536+0000 7f8adf200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad810ff00 con 0x7f8ad80780f0 2024-05-27T07:07:41.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.537+0000 7f8adde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:41.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.537+0000 7f8adde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad819fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.162:49456/0 (socket says 172.21.15.162:49456) 2024-05-27T07:07:41.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.537+0000 7f8adde00700 1 -- 172.21.15.162:0/4205682294 learned_addr learned my addr 172.21.15.162:0/4205682294 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:07:41.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.537+0000 7f8adde00700 1 -- 172.21.15.162:0/4205682294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad819fff0 con 0x7f8ad80780f0 2024-05-27T07:07:41.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.538+0000 7f8adde00700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad819fab0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8acc005790 tx=0x7f8acc004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:41.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.538+0000 7f8ad2a00700 1 -- 172.21.15.162:0/4205682294 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8acc027070 con 0x7f8ad80780f0 2024-05-27T07:07:41.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.538+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad81a0280 con 0x7f8ad80780f0 2024-05-27T07:07:41.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.538+0000 7f8ad2a00700 1 -- 172.21.15.162:0/4205682294 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8acc00bdf0 con 0x7f8ad80780f0 2024-05-27T07:07:41.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.538+0000 7f8ad2a00700 1 -- 172.21.15.162:0/4205682294 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8acc00fd90 con 0x7f8ad80780f0 2024-05-27T07:07:41.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.538+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad81a2e90 con 0x7f8ad80780f0 2024-05-27T07:07:41.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.540+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad8198cf0 con 0x7f8ad80780f0 2024-05-27T07:07:41.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.541+0000 7f8ad2a00700 1 -- 172.21.15.162:0/4205682294 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f8acc02a070 con 0x7f8ad80780f0 2024-05-27T07:07:41.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.541+0000 7f8ad2a00700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8ac003e510 0x7f8ac0040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:41.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.541+0000 7f8ad2a00700 1 -- 172.21.15.162:0/4205682294 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8acc053b00 con 0x7f8ad80780f0 2024-05-27T07:07:41.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.542+0000 7f8add400700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8ac003e510 0x7f8ac0040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:41.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.543+0000 7f8add400700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8ac003e510 0x7f8ac0040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4006fd0 tx=0x7f8ac4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:41.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.558+0000 7f8ad2a00700 1 -- 172.21.15.162:0/4205682294 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8acc013070 con 0x7f8ad80780f0 2024-05-27T07:07:41.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.823+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi062:172.21.15.62=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}) -- 0x7f8ad802e7b0 con 0x7f8ac003e510 2024-05-27T07:07:41.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.840+0000 7f8ad2a00700 1 -- 172.21.15.162:0/4205682294 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8ad802e7b0 con 0x7f8ac003e510 2024-05-27T07:07:41.831 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-05-27T07:07:41.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8ac003e510 msgr2=0x7f8ac0040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8ac003e510 0x7f8ac0040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4006fd0 tx=0x7f8ac4006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 msgr2=0x7f8ad819fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad819fab0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8acc005790 tx=0x7f8acc004f60 comp rx=0 tx=0).stop 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 shutdown_connections 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8ac003e510 0x7f8ac0040a20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 --2- 172.21.15.162:0/4205682294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ad80780f0 0x7f8ad819fab0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.843+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 >> 172.21.15.162:0/4205682294 conn(0x7f8ad8101250 msgr2=0x7f8ad8101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.844+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 shutdown_connections 2024-05-27T07:07:41.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:41.844+0000 7f8adf200700 1 -- 172.21.15.162:0/4205682294 wait complete. 2024-05-27T07:07:42.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:42.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:42.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:42.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi062:172.21.15.62=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi062:172.21.15.62=b;smithi162:172.21.15.162=c;count:3 2024-05-27T07:07:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:42.588 DEBUG:teuthology.orchestra.run.smithi062:mon.b> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.b.service 2024-05-27T07:07:42.597 DEBUG:teuthology.orchestra.run.smithi162:mon.c> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.c.service 2024-05-27T07:07:42.601 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:07:42.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:07:43.018 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:07:43.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:42 smithi012 ceph-mon[31473]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:07:44.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.994+0000 7fc3c4400700 1 -- 172.21.15.162:0/3530658248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 msgr2=0x7fc3bc1041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:44.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.994+0000 7fc3c4400700 1 --2- 172.21.15.162:0/3530658248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc1041f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0009910 tx=0x7fc3b0009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:44.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.994+0000 7fc3c4400700 1 -- 172.21.15.162:0/3530658248 shutdown_connections 2024-05-27T07:07:44.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.994+0000 7fc3c4400700 1 --2- 172.21.15.162:0/3530658248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc1041f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:44.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.994+0000 7fc3c4400700 1 -- 172.21.15.162:0/3530658248 >> 172.21.15.162:0/3530658248 conn(0x7fc3bc0ff690 msgr2=0x7fc3bc101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:44.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.994+0000 7fc3c4400700 1 -- 172.21.15.162:0/3530658248 shutdown_connections 2024-05-27T07:07:44.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.994+0000 7fc3c4400700 1 -- 172.21.15.162:0/3530658248 wait complete. 2024-05-27T07:07:44.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.996+0000 7fc3c4400700 1 Processor -- start 2024-05-27T07:07:44.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.996+0000 7fc3c4400700 1 -- start start 2024-05-27T07:07:44.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.997+0000 7fc3c4400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:44.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.997+0000 7fc3c4400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc10df80 con 0x7fc3bc103de0 2024-05-27T07:07:44.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.997+0000 7fc3c1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:44.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.997+0000 7fc3c1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.162:42858/0 (socket says 172.21.15.162:42858) 2024-05-27T07:07:44.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.997+0000 7fc3c1a00700 1 -- 172.21.15.162:0/3809154028 learned_addr learned my addr 172.21.15.162:0/3809154028 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:07:44.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.998+0000 7fc3c1a00700 1 -- 172.21.15.162:0/3809154028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc077900 con 0x7fc3bc103de0 2024-05-27T07:07:44.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.999+0000 7fc3c1a00700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc075ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0005660 tx=0x7fc3b0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:44.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.999+0000 7fc3ba000700 1 -- 172.21.15.162:0/3809154028 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3b0026070 con 0x7fc3bc103de0 2024-05-27T07:07:44.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.999+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3bc075ff0 con 0x7fc3bc103de0 2024-05-27T07:07:44.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.999+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3bc0764b0 con 0x7fc3bc103de0 2024-05-27T07:07:44.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.999+0000 7fc3ba000700 1 -- 172.21.15.162:0/3809154028 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3b000b6f0 con 0x7fc3bc103de0 2024-05-27T07:07:44.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:44.999+0000 7fc3ba000700 1 -- 172.21.15.162:0/3809154028 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3b00037d0 con 0x7fc3bc103de0 2024-05-27T07:07:44.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.001+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3bc105290 con 0x7fc3bc103de0 2024-05-27T07:07:44.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.001+0000 7fc3ba000700 1 -- 172.21.15.162:0/3809154028 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc3b0029070 con 0x7fc3bc103de0 2024-05-27T07:07:44.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.002+0000 7fc3ba000700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc3a403e570 0x7fc3a4040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:44.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.002+0000 7fc3ba000700 1 -- 172.21.15.162:0/3809154028 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc3b001e070 con 0x7fc3bc103de0 2024-05-27T07:07:44.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.002+0000 7fc3c1000700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc3a403e570 0x7fc3a4040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:44.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.004+0000 7fc3c1000700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc3a403e570 0x7fc3a4040a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8006fd0 tx=0x7fc3a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:45.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.019+0000 7fc3ba000700 1 -- 172.21.15.162:0/3809154028 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc3b0013070 con 0x7fc3bc103de0 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:44 smithi012 ceph-mon[31473]: Deploying daemon mon.c on smithi162 2024-05-27T07:07:45.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.359+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc3bc077010 con 0x7fc3bc103de0 2024-05-27T07:07:45.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.361+0000 7fc3ba000700 1 -- 172.21.15.162:0/3809154028 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fc3b0030300 con 0x7fc3bc103de0 2024-05-27T07:07:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:07:45.352 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:05:19.787338Z","created":"2024-05-27T07:05:19.787338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc3a403e570 msgr2=0x7fc3a4040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc3a403e570 0x7fc3a4040a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8006fd0 tx=0x7fc3a8006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 msgr2=0x7fc3bc075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc075ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0005660 tx=0x7fc3b0005740 comp rx=0 tx=0).stop 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 shutdown_connections 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc3a403e570 0x7fc3a4040a80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 --2- 172.21.15.162:0/3809154028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3bc103de0 0x7fc3bc075ab0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 >> 172.21.15.162:0/3809154028 conn(0x7fc3bc0ff690 msgr2=0x7fc3bc101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 shutdown_connections 2024-05-27T07:07:45.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:45.365+0000 7fc3c4400700 1 -- 172.21.15.162:0/3809154028 wait complete. 2024-05-27T07:07:45.357 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-05-27T07:07:46.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:45 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/3809154028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:07:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:46 smithi012 ceph-mon[31473]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:47.769 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:07:47.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:07:48.238 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:07:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:48 smithi012 ceph-mon[31473]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.912+0000 7f12ee000700 1 -- 172.21.15.162:0/3544485873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 msgr2=0x7f12e81051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.912+0000 7f12ee000700 1 --2- 172.21.15.162:0/3544485873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e81051c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f12d8009910 tx=0x7f12d8009c20 comp rx=0 tx=0).stop 2024-05-27T07:07:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.912+0000 7f12ee000700 1 -- 172.21.15.162:0/3544485873 shutdown_connections 2024-05-27T07:07:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.912+0000 7f12ee000700 1 --2- 172.21.15.162:0/3544485873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e81051c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.912+0000 7f12ee000700 1 -- 172.21.15.162:0/3544485873 >> 172.21.15.162:0/3544485873 conn(0x7f12e80fe660 msgr2=0x7f12e8100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.912+0000 7f12ee000700 1 -- 172.21.15.162:0/3544485873 shutdown_connections 2024-05-27T07:07:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.912+0000 7f12ee000700 1 -- 172.21.15.162:0/3544485873 wait complete. 2024-05-27T07:07:50.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.913+0000 7f12ee000700 1 Processor -- start 2024-05-27T07:07:50.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.914+0000 7f12ee000700 1 -- start start 2024-05-27T07:07:50.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.914+0000 7f12ee000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e8102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:50.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.914+0000 7f12ee000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12e810e0c0 con 0x7f12e8104db0 2024-05-27T07:07:50.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.915+0000 7f12e7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e8102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:50.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.915+0000 7f12e7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e8102aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.162:42880/0 (socket says 172.21.15.162:42880) 2024-05-27T07:07:50.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.915+0000 7f12e7400700 1 -- 172.21.15.162:0/1017052412 learned_addr learned my addr 172.21.15.162:0/1017052412 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:07:50.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.915+0000 7f12e7400700 1 -- 172.21.15.162:0/1017052412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e81048f0 con 0x7f12e8104db0 2024-05-27T07:07:50.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.916+0000 7f12e7400700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e8102aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f12d8005060 tx=0x7f12d8005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:50.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.916+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12d8026070 con 0x7f12e8104db0 2024-05-27T07:07:50.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.916+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e8102fe0 con 0x7f12e8104db0 2024-05-27T07:07:50.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.917+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e81033e0 con 0x7f12e8104db0 2024-05-27T07:07:50.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.917+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f12d800b750 con 0x7f12e8104db0 2024-05-27T07:07:50.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.917+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12d8003810 con 0x7f12e8104db0 2024-05-27T07:07:50.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.918+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c8005330 con 0x7f12e8104db0 2024-05-27T07:07:50.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.919+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f12d8029070 con 0x7f12e8104db0 2024-05-27T07:07:50.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.919+0000 7f12dfe00700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f12cc03e570 0x7f12cc040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:50.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.919+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f12d801e070 con 0x7f12e8104db0 2024-05-27T07:07:50.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.920+0000 7f12e6a00700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f12cc03e570 0x7f12cc040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:50.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.921+0000 7f12e6a00700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f12cc03e570 0x7f12cc040a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006fd0 tx=0x7f12d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:50.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.937+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f12d8013070 con 0x7f12e8104db0 2024-05-27T07:07:50.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:50.987+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f12d80142b0 con 0x7f12e8104db0 2024-05-27T07:07:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:50 smithi012 ceph-mon[31473]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:51.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:51.258+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f12c8005620 con 0x7f12e8104db0 2024-05-27T07:07:56.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.020+0000 7f12dfe00700 1 -- 172.21.15.162:0/1017052412 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f12d800b900 con 0x7f12e8104db0 2024-05-27T07:07:56.013 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:07:56.013 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:07:50.986051Z","created":"2024-05-27T07:05:19.787338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T07:07:56.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f12cc03e570 msgr2=0x7f12cc040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:56.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f12cc03e570 0x7f12cc040a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006fd0 tx=0x7f12d4006e40 comp rx=0 tx=0).stop 2024-05-27T07:07:56.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 msgr2=0x7f12e8102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:56.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e8102aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f12d8005060 tx=0x7f12d8005710 comp rx=0 tx=0).stop 2024-05-27T07:07:56.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 shutdown_connections 2024-05-27T07:07:56.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f12cc03e570 0x7f12cc040a80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:56.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 --2- 172.21.15.162:0/1017052412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12e8104db0 0x7f12e8102aa0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:56.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 >> 172.21.15.162:0/1017052412 conn(0x7f12e80fe660 msgr2=0x7f12e810a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:56.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 shutdown_connections 2024-05-27T07:07:56.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:07:56.024+0000 7f12ee000700 1 -- 172.21.15.162:0/1017052412 wait complete. 2024-05-27T07:07:56.018 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-05-27T07:07:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:07:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: mon.a calling monitor election 2024-05-27T07:07:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/1017052412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:07:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: mon.c calling monitor election 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: monmap epoch 2 2024-05-27T07:07:56.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: last_changed 2024-05-27T07:07:50.986051+0000 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: created 2024-05-27T07:05:19.787338+0000 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: min_mon_release 19 (squid) 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: election_strategy: 1 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: fsmap 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: mgrmap e13: x(active, since 70s) 2024-05-27T07:07:56.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: overall HEALTH_OK 2024-05-27T07:07:56.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:07:56.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:07:56.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:56.364 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:07:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:07:57 smithi162 ceph-mon[35676]: Deploying daemon mon.b on smithi062 2024-05-27T07:07:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:07:57 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:57.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:57 smithi012 ceph-mon[31473]: Deploying daemon mon.b on smithi062 2024-05-27T07:07:57.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:57 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:07:57.953 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:07:57.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:07:58.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:07:58 smithi012 ceph-mon[31473]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:07:58.400 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:07:58.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:07:58 smithi162 ceph-mon[35676]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:00.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.323+0000 7f87f0a00700 1 -- 172.21.15.162:0/3115363637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 msgr2=0x7f87ec075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:00.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.323+0000 7f87f0a00700 1 --2- 172.21.15.162:0/3115363637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec075f60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009910 tx=0x7f87dc009c20 comp rx=0 tx=0).stop 2024-05-27T07:08:00.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.324+0000 7f87f0a00700 1 -- 172.21.15.162:0/3115363637 shutdown_connections 2024-05-27T07:08:00.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.324+0000 7f87f0a00700 1 --2- 172.21.15.162:0/3115363637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec075f60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:00.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.324+0000 7f87f0a00700 1 -- 172.21.15.162:0/3115363637 >> 172.21.15.162:0/3115363637 conn(0x7f87ec0ff580 msgr2=0x7f87ec1019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:00.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.324+0000 7f87f0a00700 1 -- 172.21.15.162:0/3115363637 shutdown_connections 2024-05-27T07:08:00.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.324+0000 7f87f0a00700 1 -- 172.21.15.162:0/3115363637 wait complete. 2024-05-27T07:08:00.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.325+0000 7f87f0a00700 1 Processor -- start 2024-05-27T07:08:00.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.325+0000 7f87f0a00700 1 -- start start 2024-05-27T07:08:00.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.326+0000 7f87f0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec19ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:00.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.326+0000 7f87f0a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 0x7f87ec1a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:00.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.326+0000 7f87f0a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1144e0 con 0x7f87ec075b50 2024-05-27T07:08:00.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.326+0000 7f87f0a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec114360 con 0x7f87ec1a0520 2024-05-27T07:08:00.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87ea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec19ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:00.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87ea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec19ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.162:55692/0 (socket says 172.21.15.162:55692) 2024-05-27T07:08:00.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87ea800700 1 -- 172.21.15.162:0/1337477883 learned_addr learned my addr 172.21.15.162:0/1337477883 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:08:00.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87e9e00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 0x7f87ec1a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:00.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87e9e00700 1 -- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 msgr2=0x7f87ec1a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-27T07:08:00.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87e9e00700 1 -- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 msgr2=0x7f87ec1a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T07:08:00.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87e9e00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 0x7f87ec1a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T07:08:00.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87e9e00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 0x7f87ec1a4810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:08:00.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87ea800700 1 -- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 msgr2=0x7f87ec1a4810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:08:00.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87ea800700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 0x7f87ec1a4810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:00.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.327+0000 7f87ea800700 1 -- 172.21.15.162:0/1337477883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ec1a0b20 con 0x7f87ec075b50 2024-05-27T07:08:00.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.328+0000 7f87ea800700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec19ffe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005b40 tx=0x7f87dc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:00.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.328+0000 7f87e3400700 1 -- 172.21.15.162:0/1337477883 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f87dc00b960 con 0x7f87ec075b50 2024-05-27T07:08:00.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.328+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ec1a4d50 con 0x7f87ec075b50 2024-05-27T07:08:00.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.329+0000 7f87e3400700 1 -- 172.21.15.162:0/1337477883 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f87dc022470 con 0x7f87ec075b50 2024-05-27T07:08:00.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.329+0000 7f87e3400700 1 -- 172.21.15.162:0/1337477883 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f87dc00f660 con 0x7f87ec075b50 2024-05-27T07:08:00.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.329+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ec1a5280 con 0x7f87ec075b50 2024-05-27T07:08:00.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.330+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87ec077290 con 0x7f87ec075b50 2024-05-27T07:08:00.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.331+0000 7f87e3400700 1 -- 172.21.15.162:0/1337477883 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f87dc02a070 con 0x7f87ec075b50 2024-05-27T07:08:00.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.331+0000 7f87e3400700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87d003e2b0 0x7f87d00407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:00.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.331+0000 7f87e3400700 1 -- 172.21.15.162:0/1337477883 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f87dc01e030 con 0x7f87ec075b50 2024-05-27T07:08:00.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.332+0000 7f87e9e00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87d003e2b0 0x7f87d00407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:00.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.333+0000 7f87e9e00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87d003e2b0 0x7f87d00407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87ec102750 tx=0x7f87d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:00.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.348+0000 7f87e3400700 1 -- 172.21.15.162:0/1337477883 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f87dc055e20 con 0x7f87ec075b50 2024-05-27T07:08:00.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.671+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f87ec1a55c0 con 0x7f87ec075b50 2024-05-27T07:08:00.665 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:00 smithi162 ceph-mon[35676]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:00.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.673+0000 7f87e3400700 1 -- 172.21.15.162:0/1337477883 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f87dc013070 con 0x7f87ec075b50 2024-05-27T07:08:00.665 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:08:00.665 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:07:50.986051Z","created":"2024-05-27T07:05:19.787338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T07:08:00.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87d003e2b0 msgr2=0x7f87d00407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87d003e2b0 0x7f87d00407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87ec102750 tx=0x7f87d8008040 comp rx=0 tx=0).stop 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 msgr2=0x7f87ec19ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec19ffe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005b40 tx=0x7f87dc005780 comp rx=0 tx=0).stop 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 shutdown_connections 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87d003e2b0 0x7f87d00407c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87ec075b50 0x7f87ec19ffe0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 --2- 172.21.15.162:0/1337477883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87ec1a0520 0x7f87ec1a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 >> 172.21.15.162:0/1337477883 conn(0x7f87ec0ff580 msgr2=0x7f87ec100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:00.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 shutdown_connections 2024-05-27T07:08:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:00.676+0000 7f87f0a00700 1 -- 172.21.15.162:0/1337477883 wait complete. 2024-05-27T07:08:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-05-27T07:08:00.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:00 smithi012 ceph-mon[31473]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:01.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:01 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/1337477883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:01.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 podman[35285]: 2024-05-27 07:08:01.471903201 +0000 UTC m=+0.137809816 container create ccbee1f2bf47cdf85b84ec85569f1ee6169e9ac42c65ac202f777ce3009b000d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:08:01.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 podman[35285]: 2024-05-27 07:08:01.744389341 +0000 UTC m=+0.410295983 container init ccbee1f2bf47cdf85b84ec85569f1ee6169e9ac42c65ac202f777ce3009b000d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-27T07:08:01.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:01 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/1337477883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 podman[35285]: 2024-05-27 07:08:01.756743051 +0000 UTC m=+0.422649673 container start ccbee1f2bf47cdf85b84ec85569f1ee6169e9ac42c65ac202f777ce3009b000d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 bash[35285]: ccbee1f2bf47cdf85b84ec85569f1ee6169e9ac42c65ac202f777ce3009b000d 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 systemd[1]: Started Ceph mon.b for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pidfile_write: ignore empty --pid-file 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: load: jerasure load: lrc 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:08:01.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Git sha 0 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: DB SUMMARY 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: DB Session ID: 1JLQBMBTAAX2KGMMY2VB 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:08:01.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.env: 0x5626f77af500 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.info_log: 0x56270d0a6040 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.statistics: (nil) 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.use_fsync: 0 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:08:01.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:08:01.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:08:01.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:08:01.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:08:01.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:08:01.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.db_log_dir: 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.wal_dir: 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:08:01.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.write_buffer_manager: 0x56270cf708c0 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.unordered_write: 0 2024-05-27T07:08:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.row_cache: None 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.wal_filter: None 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:08:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.wal_compression: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:08:01.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:08:01.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_open_files: -1 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Compression algorithms supported: 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kZSTD supported: 0 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:08:01.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.merge_operator: 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_filter: None 2024-05-27T07:08:01.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56270d0a6200) 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: index_type: 0 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: data_block_index_type: 0 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: index_shortening: 1 2024-05-27T07:08:01.999 INFO:journalctl@ceph.mon.b.smithi062.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: checksum: 4 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: no_block_cache: 0 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_cache: 0x56270cfa31f0 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_cache_options: 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: capacity : 536870912 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: num_shard_bits : 4 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: strict_capacity_limit : 0 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_cache_compressed: (nil) 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: persistent_cache: (nil) 2024-05-27T07:08:02.000 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_size: 4096 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_size_deviation: 10 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_restart_interval: 16 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: index_block_restart_interval: 1 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: metadata_block_size: 4096 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: partition_filters: 0 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: use_delta_encoding: 1 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: filter_policy: bloomfilter 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: whole_key_filtering: 1 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: verify_compression: 0 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: format_version: 5 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: enable_index_compression: 1 2024-05-27T07:08:02.001 INFO:journalctl@ceph.mon.b.smithi062.stdout: block_align: 0 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout: max_auto_readahead_size: 262144 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout: prepopulate_block_cache: 0 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression: NoCompression 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:08:02.002 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.num_levels: 7 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:08:02.004 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:08:02.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:08:02.006 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:08:02.007 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:08:02.008 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.ttl: 2592000 2024-05-27T07:08:02.009 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:08:02.010 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ba00f296-e3ef-42b4-9c7d-ee49461ab2a5 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793681890774, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793681891650, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716793681, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ba00f296-e3ef-42b4-9c7d-ee49461ab2a5", "db_session_id": "1JLQBMBTAAX2KGMMY2VB", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793681891944, "job": 1, "event": "recovery_finished"} 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56270cf87c00 2024-05-27T07:08:02.011 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: DB pointer 0x562711d7a000 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: ** DB Stats ** 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: 2024-05-27T07:08:02.012 INFO:journalctl@ceph.mon.b.smithi062.stdout: ** Compaction Stats [default] ** 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: ** Compaction Stats [default] ** 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:08:02.013 INFO:journalctl@ceph.mon.b.smithi062.stdout: 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: Cumulative compaction: 0.00 GB write, 0.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: Interval compaction: 0.00 GB write, 0.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: Block cache BinnedLRUCache@0x56270cfa31f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-05-27T07:08:02.014 INFO:journalctl@ceph.mon.b.smithi062.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout: 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: using public_addr v2:172.21.15.62:0/0 -> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: starting mon.b rank -1 at public addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] at bind addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(???) e0 preinit fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).mds e1 new map 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout: e1 2024-05-27T07:08:02.015 INFO:journalctl@ceph.mon.b.smithi062.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout: legacy client fscid: -1 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout: 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout: No filesystems configured 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T07:08:02.016 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi062", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Deploying cephadm binary to smithi062 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.017 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Added host smithi062 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:02.018 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Deploying cephadm binary to smithi162 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.019 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Added host smithi162 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.020 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2407117628' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2407117628' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.021 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.022 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.022 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.022 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:08:02.022 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:02.022 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:02.022 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:08:02.022 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi062:172.21.15.62=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi062:172.21.15.62=b;smithi162:172.21.15.162=c;count:3 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.026 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Deploying daemon mon.c on smithi162 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/3809154028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.a calling monitor election 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:02.027 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/1017052412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.c calling monitor election 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: monmap epoch 2 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:02.028 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: last_changed 2024-05-27T07:07:50.986051+0000 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: created 2024-05-27T07:05:19.787338+0000 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: min_mon_release 19 (squid) 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: election_strategy: 1 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: fsmap 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mgrmap e13: x(active, since 70s) 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: overall HEALTH_OK 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:02.029 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: Deploying daemon mon.b on smithi062 2024-05-27T07:08:02.030 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:02.030 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.030 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:02.030 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/1337477883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:02.030 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:01 smithi062 ceph-mon[35304]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-05-27T07:08:02.434 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:08:02.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:08:02.875 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:08:07.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: mon.a calling monitor election 2024-05-27T07:08:07.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:08:07.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:07.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: mon.c calling monitor election 2024-05-27T07:08:07.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: mon.b calling monitor election 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: monmap epoch 3 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: last_changed 2024-05-27T07:08:01.959164+0000 2024-05-27T07:08:07.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: created 2024-05-27T07:05:19.787338+0000 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: min_mon_release 19 (squid) 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: election_strategy: 1 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: 2: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.b 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: fsmap 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: mgrmap e13: x(active, since 81s) 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: overall HEALTH_OK 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: mon.a calling monitor election 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: mon.c calling monitor election 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: mon.b calling monitor election 2024-05-27T07:08:07.442 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: monmap epoch 3 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: last_changed 2024-05-27T07:08:01.959164+0000 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: created 2024-05-27T07:05:19.787338+0000 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: min_mon_release 19 (squid) 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: election_strategy: 1 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-05-27T07:08:07.443 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: 2: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.b 2024-05-27T07:08:07.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: fsmap 2024-05-27T07:08:07.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:08:07.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: mgrmap e13: x(active, since 81s) 2024-05-27T07:08:07.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: overall HEALTH_OK 2024-05-27T07:08:07.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: mon.a calling monitor election 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: mon.c calling monitor election 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: mon.b calling monitor election 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: monmap epoch 3 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: last_changed 2024-05-27T07:08:01.959164+0000 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: created 2024-05-27T07:05:19.787338+0000 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: min_mon_release 19 (squid) 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: election_strategy: 1 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: 2: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.b 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: fsmap 2024-05-27T07:08:07.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:08:07.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: mgrmap e13: x(active, since 81s) 2024-05-27T07:08:07.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: overall HEALTH_OK 2024-05-27T07:08:07.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:07.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.964+0000 7ff5bda00700 1 -- 172.21.15.162:0/3751800269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 msgr2=0x7ff59c005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.965+0000 7ff5bda00700 1 --2- 172.21.15.162:0/3751800269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 0x7ff59c005aa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8105b70 tx=0x7ff5a800feb0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.965+0000 7ff5bda00700 1 -- 172.21.15.162:0/3751800269 shutdown_connections 2024-05-27T07:08:07.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.965+0000 7ff5bda00700 1 --2- 172.21.15.162:0/3751800269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 0x7ff59c005aa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.965+0000 7ff5bda00700 1 -- 172.21.15.162:0/3751800269 >> 172.21.15.162:0/3751800269 conn(0x7ff5b81017c0 msgr2=0x7ff5b8103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:07.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.965+0000 7ff5bda00700 1 -- 172.21.15.162:0/3751800269 shutdown_connections 2024-05-27T07:08:07.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.965+0000 7ff5bda00700 1 -- 172.21.15.162:0/3751800269 wait complete. 2024-05-27T07:08:07.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.967+0000 7ff5bda00700 1 Processor -- start 2024-05-27T07:08:07.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.967+0000 7ff5bda00700 1 -- start start 2024-05-27T07:08:07.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5bda00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 0x7ff5b81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5bda00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b81a1ca0 0x7ff5b819b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5bda00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5b819bec0 0x7ff5b819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5bda00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b810daa0 con 0x7ff59c0035c0 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5bda00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b810d920 con 0x7ff5b819bec0 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5bda00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b810dc20 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5b7400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b81a1ca0 0x7ff5b819b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5b7400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b81a1ca0 0x7ff5b819b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:48500/0 (socket says 172.21.15.162:48500) 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.968+0000 7ff5b7400700 1 -- 172.21.15.162:0/1578566916 learned_addr learned my addr 172.21.15.162:0/1578566916 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b7400700 1 -- 172.21.15.162:0/1578566916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5b819bec0 msgr2=0x7ff5b819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b7400700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5b819bec0 0x7ff5b819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b7e00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 0x7ff5b81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:07.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b7400700 1 -- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 msgr2=0x7ff5b81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b7400700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 0x7ff5b81a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b7400700 1 -- 172.21.15.162:0/1578566916 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b81a85f0 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b7400700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b81a1ca0 0x7ff5b819b980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400b780 tx=0x7ff5a400bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.969+0000 7ff5b4c00700 1 -- 172.21.15.162:0/1578566916 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4004230 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.970+0000 7ff5b4c00700 1 -- 172.21.15.162:0/1578566916 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5a40043d0 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.970+0000 7ff5b4c00700 1 -- 172.21.15.162:0/1578566916 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4011ac0 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.970+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b81a8880 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.970+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b81a8d40 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.971+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b8107380 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.972+0000 7ff5b4c00700 1 -- 172.21.15.162:0/1578566916 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7ff5a4011c80 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.972+0000 7ff5b4c00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff59803e9a0 0x7ff598040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.972+0000 7ff5b4c00700 1 -- 172.21.15.162:0/1578566916 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff5a4052390 con 0x7ff5b81a1ca0 2024-05-27T07:08:07.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.973+0000 7ff5b7e00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff59803e9a0 0x7ff598040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:07.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.974+0000 7ff5b7e00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff59803e9a0 0x7ff598040eb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8000c00 tx=0x7ff5a800f4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:07.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:07.979+0000 7ff5b4c00700 1 -- 172.21.15.162:0/1578566916 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff5a4019e20 con 0x7ff5b81a1ca0 2024-05-27T07:08:08.182 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:08.183 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:08 smithi062 ceph-mon[35304]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:08.183 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:08.183 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:08.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:08.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:08 smithi162 ceph-mon[35676]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:08.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:08.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:08.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.308+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff5b81a9180 con 0x7ff5b81a1ca0 2024-05-27T07:08:08.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.308+0000 7ff5b4c00700 1 -- 172.21.15.162:0/1578566916 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7ff5a4014d90 con 0x7ff5b81a1ca0 2024-05-27T07:08:08.300 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-05-27T07:08:08.301 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":3,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:08:01.959164Z","created":"2024-05-27T07:05:19.787338Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-27T07:08:08.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff59803e9a0 msgr2=0x7ff598040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:08.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff59803e9a0 0x7ff598040eb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8000c00 tx=0x7ff5a800f4a0 comp rx=0 tx=0).stop 2024-05-27T07:08:08.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b81a1ca0 msgr2=0x7ff5b819b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b81a1ca0 0x7ff5b819b980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400b780 tx=0x7ff5a400bb40 comp rx=0 tx=0).stop 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 shutdown_connections 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff59803e9a0 0x7ff598040eb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5b819bec0 0x7ff5b819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff59c0035c0 0x7ff5b81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 --2- 172.21.15.162:0/1578566916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b81a1ca0 0x7ff5b819b980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.311+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 >> 172.21.15.162:0/1578566916 conn(0x7ff5b81017c0 msgr2=0x7ff5b8108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.312+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 shutdown_connections 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:08.312+0000 7ff5bda00700 1 -- 172.21.15.162:0/1578566916 wait complete. 2024-05-27T07:08:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 3 2024-05-27T07:08:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:08 smithi012 ceph-mon[31473]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:08:09.032 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-27T07:08:09.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-05-27T07:08:09.149 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:09 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/1578566916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:09.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:09 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/1578566916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:09.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:08:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:08:08.960+0000 7f57ba400700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-27T07:08:09.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:09 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/1578566916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:08:09.471 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:10.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: Updating smithi012:/etc/ceph/ceph.conf 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:08:10.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:10 smithi012 ceph-mon[31473]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:08:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:10.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:10.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: Updating smithi012:/etc/ceph/ceph.conf 2024-05-27T07:08:10.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:08:10.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:10 smithi162 ceph-mon[35676]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:08:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:10.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:10.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:10.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: Updating smithi012:/etc/ceph/ceph.conf 2024-05-27T07:08:10.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:08:10.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:10 smithi062 ceph-mon[35304]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:08:11.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.587+0000 7f7fd6c00700 1 -- 172.21.15.12:0/2342660916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0103de0 msgr2=0x7f7fd01041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.587+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/2342660916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0103de0 0x7f7fd01041f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009910 tx=0x7f7fc0009c20 comp rx=0 tx=0).stop 2024-05-27T07:08:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.587+0000 7f7fd6c00700 1 -- 172.21.15.12:0/2342660916 shutdown_connections 2024-05-27T07:08:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.587+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/2342660916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd0103de0 0x7f7fd01041f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.587+0000 7f7fd6c00700 1 -- 172.21.15.12:0/2342660916 >> 172.21.15.12:0/2342660916 conn(0x7f7fd00ff690 msgr2=0x7f7fd0101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.587+0000 7f7fd6c00700 1 -- 172.21.15.12:0/2342660916 shutdown_connections 2024-05-27T07:08:11.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.587+0000 7f7fd6c00700 1 -- 172.21.15.12:0/2342660916 wait complete. 2024-05-27T07:08:11.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 Processor -- start 2024-05-27T07:08:11.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 -- start start 2024-05-27T07:08:11.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 0x7f7fd019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fd019c290 0x7f7fd01a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd01a0ac0 0x7f7fd01a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd010df80 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd010de00 con 0x7f7fd0103de0 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.589+0000 7f7fd6c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd010e100 con 0x7f7fd019c290 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd01a0ac0 0x7f7fd01a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd01a0ac0 0x7f7fd01a0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38864/0 (socket says 172.21.15.12:38864) 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 -- 172.21.15.12:0/1539060795 learned_addr learned my addr 172.21.15.12:0/1539060795 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:08:11.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fcfe00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 0x7f7fd019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fc7400700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fd019c290 0x7f7fd01a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fcfe00700 1 -- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 msgr2=0x7f7fd019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fcfe00700 1 -- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 msgr2=0x7f7fd019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fcfe00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 0x7f7fd019bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fcfe00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 0x7f7fd019bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 -- 172.21.15.12:0/1539060795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fd019c290 msgr2=0x7f7fd01a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fd019c290 0x7f7fd01a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 -- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 msgr2=0x7f7fd019bd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 0x7f7fd019bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.590+0000 7f7fd4c00700 1 -- 172.21.15.12:0/1539060795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd01a8830 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.591+0000 7f7fc7400700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fd019c290 0x7f7fd01a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.591+0000 7f7fd4c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd01a0ac0 0x7f7fd01a0ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800ba50 tx=0x7f7fc800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.591+0000 7f7fcd600700 1 -- 172.21.15.12:0/1539060795 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8019070 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.591+0000 7f7fcd600700 1 -- 172.21.15.12:0/1539060795 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fc8014440 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.591+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd01a8a60 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.591+0000 7f7fcd600700 1 -- 172.21.15.12:0/1539060795 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc80075b0 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.592+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd01a8f50 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.593+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd0064e10 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.593+0000 7f7fcd600700 1 -- 172.21.15.12:0/1539060795 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f7fc801c070 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.594+0000 7f7fcd600700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7fb403e9a0 0x7f7fb4040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.594+0000 7f7fcd600700 1 -- 172.21.15.12:0/1539060795 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7fc8051fa0 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.595+0000 7f7fcfe00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7fb403e9a0 0x7f7fb4040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.595+0000 7f7fcfe00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7fb403e9a0 0x7f7fb4040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000b5c0 tx=0x7f7fc0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:11.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.600+0000 7f7fcd600700 1 -- 172.21.15.12:0/1539060795 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7fc8022170 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.848+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f7fd002e7a0 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.849+0000 7f7fcd600700 1 -- 172.21.15.12:0/1539060795 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f7fc8012c40 con 0x7f7fd01a0ac0 2024-05-27T07:08:11.842 INFO:teuthology.orchestra.run.smithi012.stdout:# minimal ceph.conf for 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:11.842 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-05-27T07:08:11.842 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:08:11.842 INFO:teuthology.orchestra.run.smithi012.stdout: mon_host = [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-05-27T07:08:11.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.852+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7fb403e9a0 msgr2=0x7f7fb4040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7fb403e9a0 0x7f7fb4040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000b5c0 tx=0x7f7fc0005fb0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd01a0ac0 msgr2=0x7f7fd01a0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd01a0ac0 0x7f7fd01a0ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800ba50 tx=0x7f7fc800be10 comp rx=0 tx=0).stop 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 shutdown_connections 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7fb403e9a0 0x7f7fb4040eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0103de0 0x7f7fd019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fd01a0ac0 0x7f7fd01a0ed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 --2- 172.21.15.12:0/1539060795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fd019c290 0x7f7fd01a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 >> 172.21.15.12:0/1539060795 conn(0x7f7fd00ff690 msgr2=0x7f7fd0075f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 shutdown_connections 2024-05-27T07:08:11.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:11.853+0000 7f7fd6c00700 1 -- 172.21.15.12:0/1539060795 wait complete. 2024-05-27T07:08:11.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.520 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-27T07:08:12.520 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:08:12.520 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:08:12.588 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:08:12.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:08:12.690 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:08:12.690 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1539060795' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:12.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:12.738 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:12 smithi012 ceph-mon[31473]: Reconfiguring daemon mon.a on smithi012 2024-05-27T07:08:12.790 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:08:12.790 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:08:12.909 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:12.909 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1539060795' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:12.909 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:12.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:12 smithi062 ceph-mon[35304]: Reconfiguring daemon mon.a on smithi012 2024-05-27T07:08:12.924 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:08:12.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:08:12.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:12.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1539060795' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:12.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:12.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:12 smithi162 ceph-mon[35676]: Reconfiguring daemon mon.a on smithi012 2024-05-27T07:08:13.018 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:08:13.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:08:13.146 INFO:tasks.cephadm:Adding mgr.x on smithi012 2024-05-27T07:08:13.146 INFO:tasks.cephadm:Adding mgr.y on smithi062 2024-05-27T07:08:13.146 INFO:tasks.cephadm:Adding mgr.z on smithi162 2024-05-27T07:08:13.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi012=x;smithi062=y;smithi162=z' 2024-05-27T07:08:13.628 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:08:15.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:14 smithi162 ceph-mon[35676]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:15.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:15.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:15.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:15.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:15.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:15.181 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:14 smithi062 ceph-mon[35304]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:15.181 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:15.181 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:15.181 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:15.181 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:15.181 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:14 smithi012 ceph-mon[31473]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:15.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.513+0000 7f8935c00700 1 -- 172.21.15.162:0/569699051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89301047e0 msgr2=0x7f8930106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.513+0000 7f8935c00700 1 --2- 172.21.15.162:0/569699051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89301047e0 0x7f8930106c00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f892000b0a0 tx=0x7f892000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.513+0000 7f8935c00700 1 -- 172.21.15.162:0/569699051 shutdown_connections 2024-05-27T07:08:15.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.513+0000 7f8935c00700 1 --2- 172.21.15.162:0/569699051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89301047e0 0x7f8930106c00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.513+0000 7f8935c00700 1 -- 172.21.15.162:0/569699051 >> 172.21.15.162:0/569699051 conn(0x7f89300fe240 msgr2=0x7f89301006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:15.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.513+0000 7f8935c00700 1 -- 172.21.15.162:0/569699051 shutdown_connections 2024-05-27T07:08:15.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.513+0000 7f8935c00700 1 -- 172.21.15.162:0/569699051 wait complete. 2024-05-27T07:08:15.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.515+0000 7f8935c00700 1 Processor -- start 2024-05-27T07:08:15.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.515+0000 7f8935c00700 1 -- start start 2024-05-27T07:08:15.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f8935c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89301047e0 0x7f8930104040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f8935c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89301026f0 0x7f8930102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f8935c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8930103040 0x7f89301a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f8935c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8930109b00 con 0x7f8930103040 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f8935c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8930109980 con 0x7f89301026f0 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f8935c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8930109c80 con 0x7f89301047e0 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f892ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89301047e0 0x7f8930104040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f892ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89301047e0 0x7f8930104040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55028/0 (socket says 172.21.15.162:55028) 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f892ea00700 1 -- 172.21.15.162:0/3165066688 learned_addr learned my addr 172.21.15.162:0/3165066688 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.516+0000 7f892f400700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8930103040 0x7f89301a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f892e000700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89301026f0 0x7f8930102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f892ea00700 1 -- 172.21.15.162:0/3165066688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89301026f0 msgr2=0x7f8930102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f892ea00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89301026f0 0x7f8930102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f892ea00700 1 -- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8930103040 msgr2=0x7f89301a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f892ea00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8930103040 0x7f89301a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f892ea00700 1 -- 172.21.15.162:0/3165066688 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89301a3e00 con 0x7f89301047e0 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f892ea00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89301047e0 0x7f8930104040 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f892000d010 tx=0x7f8920007990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f8927400700 1 -- 172.21.15.162:0/3165066688 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892001f020 con 0x7f89301047e0 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f8927400700 1 -- 172.21.15.162:0/3165066688 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f892000bdd0 con 0x7f89301047e0 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f8927400700 1 -- 172.21.15.162:0/3165066688 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920017040 con 0x7f89301047e0 2024-05-27T07:08:15.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.517+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89301a4030 con 0x7f89301047e0 2024-05-27T07:08:15.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.518+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89301a4570 con 0x7f89301047e0 2024-05-27T07:08:15.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.519+0000 7f8927400700 1 -- 172.21.15.162:0/3165066688 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f8920022020 con 0x7f89301047e0 2024-05-27T07:08:15.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.519+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8930064e10 con 0x7f89301047e0 2024-05-27T07:08:15.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.519+0000 7f8927400700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f891403e9a0 0x7f8914040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.520+0000 7f8927400700 1 -- 172.21.15.162:0/3165066688 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f892000f020 con 0x7f89301047e0 2024-05-27T07:08:15.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.520+0000 7f892e000700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f891403e9a0 0x7f8914040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.521+0000 7f892e000700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f891403e9a0 0x7f8914040eb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f89301039f0 tx=0x7f891c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:15.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.526+0000 7f8927400700 1 -- 172.21.15.162:0/3165066688 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f892001a350 con 0x7f89301047e0 2024-05-27T07:08:15.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.768+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi062=y;smithi162=z", "target": ["mon-mgr", ""]}) -- 0x7f8930100090 con 0x7f891403e9a0 2024-05-27T07:08:15.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.785+0000 7f8927400700 1 -- 172.21.15.162:0/3165066688 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8930100090 con 0x7f891403e9a0 2024-05-27T07:08:15.776 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mgr update... 2024-05-27T07:08:15.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f891403e9a0 msgr2=0x7f8914040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f891403e9a0 0x7f8914040eb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f89301039f0 tx=0x7f891c008040 comp rx=0 tx=0).stop 2024-05-27T07:08:15.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89301047e0 msgr2=0x7f8930104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89301047e0 0x7f8930104040 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f892000d010 tx=0x7f8920007990 comp rx=0 tx=0).stop 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 shutdown_connections 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f891403e9a0 0x7f8914040eb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8930103040 0x7f89301a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89301047e0 0x7f8930104040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 --2- 172.21.15.162:0/3165066688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89301026f0 0x7f8930102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 >> 172.21.15.162:0/3165066688 conn(0x7f89300fe240 msgr2=0x7f89300ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 shutdown_connections 2024-05-27T07:08:15.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:08:15.788+0000 7f8935c00700 1 -- 172.21.15.162:0/3165066688 wait complete. 2024-05-27T07:08:16.091 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:15 smithi062 ceph-mon[35304]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:08:16.091 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:15 smithi062 ceph-mon[35304]: Reconfiguring daemon mon.b on smithi062 2024-05-27T07:08:16.091 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:15 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:16.147 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:15 smithi162 ceph-mon[35676]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:08:16.147 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:15 smithi162 ceph-mon[35676]: Reconfiguring daemon mon.b on smithi062 2024-05-27T07:08:16.147 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:15 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:15 smithi012 ceph-mon[31473]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:08:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:15 smithi012 ceph-mon[31473]: Reconfiguring daemon mon.b on smithi062 2024-05-27T07:08:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:15 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:16.522 DEBUG:teuthology.orchestra.run.smithi062:mgr.y> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.y.service 2024-05-27T07:08:16.530 DEBUG:teuthology.orchestra.run.smithi162:mgr.z> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.z.service 2024-05-27T07:08:16.536 INFO:tasks.cephadm:Deploying OSDs... 2024-05-27T07:08:16.537 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:08:16.537 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:08:16.597 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:08:16.597 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-05-27T07:08:16.652 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-05-27 07:07:02.254166677 +0000 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-05-27 07:00:46.166221720 +0000 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-05-27 07:00:46.166221720 +0000 2024-05-27T07:08:16.653 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-05-27 07:00:46.166221720 +0000 2024-05-27T07:08:16.653 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:08:16.741 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-05-27T07:08:16.741 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-05-27T07:08:16.742 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00041054 s, 1.2 MB/s 2024-05-27T07:08:16.744 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:08:16.803 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-05-27T07:08:16.855 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:08:16.855 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:16.855 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-05-27T07:08:16.855 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:16.855 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:16.856 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-05-27 07:07:02.327166061 +0000 2024-05-27T07:08:16.856 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-05-27 07:00:46.779216503 +0000 2024-05-27T07:08:16.856 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-05-27 07:00:46.779216503 +0000 2024-05-27T07:08:16.856 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-05-27 07:00:46.779216503 +0000 2024-05-27T07:08:16.856 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:08:16.932 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-05-27T07:08:16.933 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-05-27T07:08:16.933 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000356006 s, 1.4 MB/s 2024-05-27T07:08:16.935 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:08:16.990 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-05-27 07:07:02.404165410 +0000 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-05-27 07:00:47.361211549 +0000 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-05-27 07:00:47.361211549 +0000 2024-05-27T07:08:17.042 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-05-27 07:00:47.361211549 +0000 2024-05-27T07:08:17.043 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:08:17.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi062=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: Saving service mgr spec with placement smithi012=x;smithi062=y;smithi162=z;count:3 2024-05-27T07:08:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:17.125 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-05-27T07:08:17.125 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-05-27T07:08:17.126 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000376635 s, 1.4 MB/s 2024-05-27T07:08:17.129 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:08:17.187 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi062=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: Saving service mgr spec with placement smithi012=x;smithi062=y;smithi162=z;count:3 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi062=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: Saving service mgr spec with placement smithi012=x;smithi062=y;smithi162=z;count:3 2024-05-27T07:08:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:08:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:08:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:08:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-05-27T07:08:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:17.241 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-05-27 07:07:02.472164836 +0000 2024-05-27T07:08:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-05-27 07:00:47.942206604 +0000 2024-05-27T07:08:17.242 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-05-27 07:00:47.942206604 +0000 2024-05-27T07:08:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-05-27 07:00:47.942206604 +0000 2024-05-27T07:08:17.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:08:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-05-27T07:08:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-05-27T07:08:17.325 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000379921 s, 1.3 MB/s 2024-05-27T07:08:17.328 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:08:17.385 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-05-27T07:08:17.385 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:08:17.446 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:08:17.447 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-05-27 07:08:00.924407122 +0000 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-05-27 07:01:02.854051858 +0000 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-05-27 07:01:02.854051858 +0000 2024-05-27T07:08:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-05-27 07:01:02.854051858 +0000 2024-05-27T07:08:17.506 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:08:17.592 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-05-27T07:08:17.592 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-05-27T07:08:17.592 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000401923 s, 1.3 MB/s 2024-05-27T07:08:17.596 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:08:17.655 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-05-27T07:08:17.710 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:08:17.710 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:17.711 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-05-27T07:08:17.711 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:17.711 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:17.711 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-05-27 07:08:00.925407107 +0000 2024-05-27T07:08:17.711 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-05-27 07:01:03.447042685 +0000 2024-05-27T07:08:17.711 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-05-27 07:01:03.447042685 +0000 2024-05-27T07:08:17.711 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-05-27 07:01:03.447042685 +0000 2024-05-27T07:08:17.711 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:08:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-05-27T07:08:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-05-27T07:08:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000337338 s, 1.5 MB/s 2024-05-27T07:08:17.797 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:08:17.854 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-05-27T07:08:17.906 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-05-27 07:08:00.926407091 +0000 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-05-27 07:01:04.072033017 +0000 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-05-27 07:01:04.072033017 +0000 2024-05-27T07:08:17.907 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-05-27 07:01:04.072033017 +0000 2024-05-27T07:08:17.907 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:08:17.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:17 smithi162 ceph-mon[35676]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:08:17.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:17 smithi162 ceph-mon[35676]: Reconfiguring daemon mon.c on smithi162 2024-05-27T07:08:17.989 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-05-27T07:08:17.989 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-05-27T07:08:17.989 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000435638 s, 1.2 MB/s 2024-05-27T07:08:17.993 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:08:18.055 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-05-27T07:08:18.107 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-05-27 07:08:00.927407076 +0000 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-05-27 07:01:04.681023597 +0000 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-05-27 07:01:04.681023597 +0000 2024-05-27T07:08:18.108 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-05-27 07:01:04.681023597 +0000 2024-05-27T07:08:18.108 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:08:18.182 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:17 smithi062 ceph-mon[35304]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:08:18.182 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:17 smithi062 ceph-mon[35304]: Reconfiguring daemon mon.c on smithi162 2024-05-27T07:08:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-05-27T07:08:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-05-27T07:08:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000416731 s, 1.2 MB/s 2024-05-27T07:08:18.195 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:08:18.256 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-05-27T07:08:18.257 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:08:18.306 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:08:18.306 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-05-27T07:08:18.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:17 smithi012 ceph-mon[31473]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:08:18.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:17 smithi012 ceph-mon[31473]: Reconfiguring daemon mon.c on smithi162 2024-05-27T07:08:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 636 Links: 1 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-05-27 07:07:49.418043115 +0000 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-05-27 07:00:46.157521919 +0000 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-05-27 07:00:46.157521919 +0000 2024-05-27T07:08:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-05-27 07:00:46.157521919 +0000 2024-05-27T07:08:18.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:08:18.466 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-05-27T07:08:18.466 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-05-27T07:08:18.466 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000307726 s, 1.7 MB/s 2024-05-27T07:08:18.469 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:08:18.513 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-05-27T07:08:18.596 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:08:18.596 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:18.596 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 655 Links: 1 2024-05-27T07:08:18.596 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:18.597 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:18.597 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-05-27 07:07:49.420043075 +0000 2024-05-27T07:08:18.597 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-05-27 07:00:46.738510217 +0000 2024-05-27T07:08:18.597 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-05-27 07:00:46.738510217 +0000 2024-05-27T07:08:18.597 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-05-27 07:00:46.738510217 +0000 2024-05-27T07:08:18.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:08:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-05-27T07:08:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-05-27T07:08:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000380187 s, 1.3 MB/s 2024-05-27T07:08:18.716 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:08:18.762 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-05-27 07:07:49.421043055 +0000 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-05-27 07:00:47.339498111 +0000 2024-05-27T07:08:18.850 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-05-27 07:00:47.339498111 +0000 2024-05-27T07:08:18.851 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-05-27 07:00:47.339498111 +0000 2024-05-27T07:08:18.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:08:18.918 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-05-27T07:08:18.918 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-05-27T07:08:18.918 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000292583 s, 1.7 MB/s 2024-05-27T07:08:18.922 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:08:18.969 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-05-27T07:08:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:18.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:18.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:18.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:08:18.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:18.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:19.012 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:08:19.012 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:08:19.013 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 693 Links: 1 2024-05-27T07:08:19.013 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:08:19.013 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:08:19.013 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-05-27 07:07:49.421043055 +0000 2024-05-27T07:08:19.013 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-05-27 07:00:47.919486428 +0000 2024-05-27T07:08:19.013 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-05-27 07:00:47.919486428 +0000 2024-05-27T07:08:19.013 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-05-27 07:00:47.919486428 +0000 2024-05-27T07:08:19.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:08:19.116 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-05-27T07:08:19.116 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-05-27T07:08:19.116 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000337255 s, 1.5 MB/s 2024-05-27T07:08:19.119 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:08:19.163 INFO:tasks.cephadm:Deploying osd.0 on smithi012 with /dev/vg_nvme/lv_4... 2024-05-27T07:08:19.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:19.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:19.248 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:19.249 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:08:19.249 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:19.249 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:19.625 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:20.046 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:19 smithi012 ceph-mon[31473]: Deploying daemon mgr.z on smithi162 2024-05-27T07:08:20.184 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:19 smithi162 ceph-mon[35676]: Deploying daemon mgr.z on smithi162 2024-05-27T07:08:20.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:19 smithi062 ceph-mon[35304]: Deploying daemon mgr.z on smithi162 2024-05-27T07:08:21.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:20 smithi012 ceph-mon[31473]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:21.135 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:20 smithi162 ceph-mon[35676]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:20 smithi062 ceph-mon[35304]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:21.487 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 systemd[1]: Starting Ceph mgr.z for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:08:21.849 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 podman[37540]: 2024-05-27 07:08:21.438328305 +0000 UTC m=+0.034893702 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:08:21.849 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 podman[37540]: 2024-05-27 07:08:21.547121382 +0000 UTC m=+0.143686817 container create f3a2800d7af695fb2de69802e1df8cc32f5b1c3bf15ed0d210c0d1319bcb0070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:08:21.849 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 podman[37540]: 2024-05-27 07:08:21.767397173 +0000 UTC m=+0.363962648 container init f3a2800d7af695fb2de69802e1df8cc32f5b1c3bf15ed0d210c0d1319bcb0070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:08:21.849 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 podman[37540]: 2024-05-27 07:08:21.777305362 +0000 UTC m=+0.373870807 container start f3a2800d7af695fb2de69802e1df8cc32f5b1c3bf15ed0d210c0d1319bcb0070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:08:22.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:21 smithi162 ceph-mon[35676]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:22.225 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 bash[37540]: f3a2800d7af695fb2de69802e1df8cc32f5b1c3bf15ed0d210c0d1319bcb0070 2024-05-27T07:08:22.225 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 systemd[1]: Started Ceph mgr.z for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:08:22.225 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:21 smithi162 ceph-mgr[37559]: -- 172.21.15.162:0/570842947 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5589c06b45a0 con 0x5589c543e000 2024-05-27T07:08:22.225 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:22.077+0000 7f0dd50e6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:08:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:21 smithi062 ceph-mon[35304]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:22.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:21 smithi012 ceph-mon[31473]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:08:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:08:22.348 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-05-27T07:08:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-05-27T07:08:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0386899 s, 271 MB/s 2024-05-27T07:08:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-05-27T07:08:22.724 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:22.323+0000 7f0dd50e6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:08:22.725 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:22.457+0000 7f0dd50e6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:08:23.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_4 2024-05-27T07:08:23.224 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:22.867+0000 7f0dd50e6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:08:23.224 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:22.985+0000 7f0dd50e6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:23.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:22 smithi162 ceph-mon[35676]: Deploying daemon mgr.y on smithi062 2024-05-27T07:08:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:23.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:08:23.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:23.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:23.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:22 smithi062 ceph-mon[35304]: Deploying daemon mgr.y on smithi062 2024-05-27T07:08:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:08:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:22 smithi012 ceph-mon[31473]: Deploying daemon mgr.y on smithi062 2024-05-27T07:08:23.567 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:24.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:24 smithi062 ceph-mon[35304]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:24.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:24 smithi012 ceph-mon[31473]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:24.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:24 smithi162 ceph-mon[35676]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:24.974 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:24.576+0000 7f0dd50e6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:08:24.975 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:24.804+0000 7f0dd50e6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:08:25.397 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:25.023+0000 7f0dd50e6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:08:25.643 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:25 smithi062 systemd[1]: Starting Ceph mgr.y for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:08:25.724 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:25.406+0000 7f0dd50e6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:08:25.725 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:25.529+0000 7f0dd50e6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:08:25.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 -- 172.21.15.12:0/1652694900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10109f00 msgr2=0x7f5f1010c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:25.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/1652694900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10109f00 0x7f5f1010c320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5f08009990 tx=0x7f5f08009ca0 comp rx=0 tx=0).stop 2024-05-27T07:08:25.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 -- 172.21.15.12:0/1652694900 shutdown_connections 2024-05-27T07:08:25.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/1652694900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10109f00 0x7f5f1010c320 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:25.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/1652694900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f10104c40 0x7f5f10107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:25.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/1652694900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f101075a0 0x7f5f101099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:25.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 -- 172.21.15.12:0/1652694900 >> 172.21.15.12:0/1652694900 conn(0x7f5f100fe680 msgr2=0x7f5f10100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:25.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 -- 172.21.15.12:0/1652694900 shutdown_connections 2024-05-27T07:08:25.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.734+0000 7f5f0fe00700 1 -- 172.21.15.12:0/1652694900 wait complete. 2024-05-27T07:08:25.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 Processor -- start 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 -- start start 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10104c40 0x7f5f1019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f101075a0 0x7f5f1019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f10109f00 0x7f5f101a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1010fa70 con 0x7f5f10104c40 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1010f8f0 con 0x7f5f101075a0 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0fe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1010fbf0 con 0x7f5f10109f00 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.736+0000 7f5f0ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10104c40 0x7f5f1019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10104c40 0x7f5f1019c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42442/0 (socket says 172.21.15.12:42442) 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0ea00700 1 -- 172.21.15.12:0/4017848037 learned_addr learned my addr 172.21.15.12:0/4017848037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f06000700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f101075a0 0x7f5f1019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:25.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0f400700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f10109f00 0x7f5f101a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:25.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0ea00700 1 -- 172.21.15.12:0/4017848037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f10109f00 msgr2=0x7f5f101a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:25.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0ea00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f10109f00 0x7f5f101a3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:25.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0ea00700 1 -- 172.21.15.12:0/4017848037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f101075a0 msgr2=0x7f5f1019cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:25.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0ea00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f101075a0 0x7f5f1019cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:25.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f0ea00700 1 -- 172.21.15.12:0/4017848037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f101a4330 con 0x7f5f10104c40 2024-05-27T07:08:25.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.737+0000 7f5f06000700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f101075a0 0x7f5f1019cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.738+0000 7f5f0ea00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10104c40 0x7f5f1019c7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5f00007ee0 tx=0x7f5f0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.738+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00004bb0 con 0x7f5f10104c40 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.738+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f00004d50 con 0x7f5f10104c40 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.738+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f101a4620 con 0x7f5f10104c40 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.738+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00015400 con 0x7f5f10104c40 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.738+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f101a4b60 con 0x7f5f10104c40 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.739+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f100512b0 con 0x7f5f10104c40 2024-05-27T07:08:25.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.739+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5f00005020 con 0x7f5f10104c40 2024-05-27T07:08:25.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.740+0000 7f5f07e00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5ef403e750 0x7f5ef4040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:25.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.740+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5f00052580 con 0x7f5f10104c40 2024-05-27T07:08:25.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.740+0000 7f5f06000700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5ef403e750 0x7f5ef4040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:25.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.741+0000 7f5f06000700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5ef403e750 0x7f5ef4040c60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8005950 tx=0x7f5ef80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.746+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5f00010070 con 0x7f5f10104c40 2024-05-27T07:08:25.944 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:25 smithi062 podman[36663]: 2024-05-27 07:08:25.55771976 +0000 UTC m=+0.036822268 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:08:25.944 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:25 smithi062 podman[36663]: 2024-05-27 07:08:25.682524092 +0000 UTC m=+0.161626630 container create 9873a08d4594b6e194919b5376619a247de8591ecbf8abdcf5106602b012166a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:08:25.944 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:25 smithi062 podman[36663]: 2024-05-27 07:08:25.953050904 +0000 UTC m=+0.432153476 container init 9873a08d4594b6e194919b5376619a247de8591ecbf8abdcf5106602b012166a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:08:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:25.996+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5f10103140 con 0x7f5ef403e750 2024-05-27T07:08:26.030 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:25.785+0000 7f0dd50e6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:08:26.031 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:26.039+0000 7f0dd50e6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:08:26.237 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:25 smithi062 podman[36663]: 2024-05-27 07:08:25.966850668 +0000 UTC m=+0.445953189 container start 9873a08d4594b6e194919b5376619a247de8591ecbf8abdcf5106602b012166a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:08:26.237 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:26 smithi062 bash[36663]: 9873a08d4594b6e194919b5376619a247de8591ecbf8abdcf5106602b012166a 2024-05-27T07:08:26.237 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:26 smithi062 systemd[1]: Started Ceph mgr.y for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:08:26.237 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mgr[36682]: -- 172.21.15.62:0/1834541215 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563eabc9e5a0 con 0x563eb0a28000 2024-05-27T07:08:26.557 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:26.296+0000 7fcc7e169200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:08:26.557 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:26.565+0000 7fcc7e169200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:08:26.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:26.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:08:26.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:08:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:26.986 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:26.703+0000 7fcc7e169200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:26.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:27.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:27.158 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:26.863+0000 7f0dd50e6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:08:27.474 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:27.167+0000 7f0dd50e6200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:08:27.486 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:27.132+0000 7fcc7e169200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:08:27.486 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:27.252+0000 7fcc7e169200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:08:27.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:27 smithi162 ceph-mon[35676]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:27.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:27 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:27.875 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:27.751+0000 7f0dd50e6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:08:27.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:27 smithi062 ceph-mon[35304]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:27.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:27 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:28.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:27 smithi012 ceph-mon[31473]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:08:28.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:27 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:28.221 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:27.882+0000 7f0dd50e6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:08:28.474 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:28.376+0000 7f0dd50e6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:08:28.724 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:28 smithi012 ceph-mon[31473]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:28.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:28 smithi162 ceph-mon[35676]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:28.927 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:28 smithi062 ceph-mon[35304]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:29.224 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:28.859+0000 7f0dd50e6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:08:29.224 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:28.992+0000 7f0dd50e6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:08:29.237 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:28.934+0000 7fcc7e169200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:08:29.237 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:29.178+0000 7fcc7e169200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:08:29.634 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:29.323+0000 7f0dd50e6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:08:29.634 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:29.508+0000 7f0dd50e6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:08:29.634 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:29.642+0000 7f0dd50e6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:08:29.646 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:29.403+0000 7fcc7e169200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:08:29.929 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:29.806+0000 7fcc7e169200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:08:29.929 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:29.929 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:29.929 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:29.929 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.190 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:29.936+0000 7fcc7e169200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:08:30.190 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:30.198+0000 7fcc7e169200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:08:30.474 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:30.058+0000 7f0dd50e6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:08:30.474 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:30.181+0000 7f0dd50e6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:08:30.711 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:30.464+0000 7fcc7e169200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:08:30.977 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:30.958+0000 7f0dd50e6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/4020847106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5c0c75c-b9be-4c85-8ad2-8c4555dfce22"}]: dispatch 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/4020847106' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5c0c75c-b9be-4c85-8ad2-8c4555dfce22"}]': finished 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:30.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:30.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:30.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:30.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:30.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:30.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:30.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/4020847106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5c0c75c-b9be-4c85-8ad2-8c4555dfce22"}]: dispatch 2024-05-27T07:08:30.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/4020847106' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5c0c75c-b9be-4c85-8ad2-8c4555dfce22"}]': finished 2024-05-27T07:08:30.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:08:30.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:30.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:30.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:30.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:30.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/4020847106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5c0c75c-b9be-4c85-8ad2-8c4555dfce22"}]: dispatch 2024-05-27T07:08:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/4020847106' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5c0c75c-b9be-4c85-8ad2-8c4555dfce22"}]': finished 2024-05-27T07:08:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:08:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:08:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:08:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:31.598 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:31.296+0000 7fcc7e169200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:08:31.599 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:31.599+0000 7fcc7e169200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:08:31.732 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:08:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:08:31.249+0000 7f0dd50e6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:08:31.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:31.748+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7f5f000155a0 con 0x7f5f10104c40 2024-05-27T07:08:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:08:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: Reconfiguring daemon mgr.x on smithi012 2024-05-27T07:08:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1295021098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:08:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: Standby manager daemon z started 2024-05-27T07:08:31.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:08:31.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:08:31.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:08:31.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:31 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:08:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:08:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: Reconfiguring daemon mgr.x on smithi012 2024-05-27T07:08:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1295021098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:08:32.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: Standby manager daemon z started 2024-05-27T07:08:32.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:08:32.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:08:32.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:08:32.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:31 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:08:32.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:08:32.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: Reconfiguring daemon mgr.x on smithi012 2024-05-27T07:08:32.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1295021098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:08:32.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: Standby manager daemon z started 2024-05-27T07:08:32.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:08:32.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:08:32.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:08:32.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:31 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/670493296' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:08:32.487 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:32.183+0000 7fcc7e169200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:08:32.487 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:32.316+0000 7fcc7e169200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:08:32.987 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:32.801+0000 7fcc7e169200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:08:32.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:32 smithi062 ceph-mon[35304]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:32.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:32 smithi062 ceph-mon[35304]: mgrmap e14: x(active, since 106s), standbys: z 2024-05-27T07:08:32.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:08:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:32 smithi012 ceph-mon[31473]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:32 smithi012 ceph-mon[31473]: mgrmap e14: x(active, since 106s), standbys: z 2024-05-27T07:08:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:08:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:32 smithi162 ceph-mon[35676]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:32 smithi162 ceph-mon[35676]: mgrmap e14: x(active, since 106s), standbys: z 2024-05-27T07:08:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:08:33.736 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:33.278+0000 7fcc7e169200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:08:33.736 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:33.410+0000 7fcc7e169200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:08:34.066 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:33.751+0000 7fcc7e169200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:08:34.066 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:33.938+0000 7fcc7e169200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:08:34.066 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:34.073+0000 7fcc7e169200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:08:34.735 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:34.485+0000 7fcc7e169200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:08:34.735 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:34.613+0000 7fcc7e169200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:08:34.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:34 smithi062 ceph-mon[35304]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:34 smithi012 ceph-mon[31473]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:35.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:34 smithi162 ceph-mon[35676]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:35.710 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:35.414+0000 7fcc7e169200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:08:35.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:35.767+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f5f00021120 con 0x7f5f10104c40 2024-05-27T07:08:35.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:35 smithi012 ceph-mon[31473]: Standby manager daemon y started 2024-05-27T07:08:35.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:35 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:08:35.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:35 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:08:35.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:35 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:08:35.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:35 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:08:35.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:35 smithi062 ceph-mon[35304]: Standby manager daemon y started 2024-05-27T07:08:35.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:35 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:08:35.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:35 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:08:35.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:35 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:08:35.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:35 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:08:35.987 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:08:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:08:35.717+0000 7fcc7e169200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:08:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:35 smithi162 ceph-mon[35676]: Standby manager daemon y started 2024-05-27T07:08:36.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:35 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:08:36.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:35 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:08:36.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:35 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:08:36.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:35 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3383517380' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:08:36.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:36 smithi012 ceph-mon[31473]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:36.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:36 smithi012 ceph-mon[31473]: mgrmap e15: x(active, since 110s), standbys: z, y 2024-05-27T07:08:36.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:08:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:36 smithi162 ceph-mon[35676]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:36 smithi162 ceph-mon[35676]: mgrmap e15: x(active, since 110s), standbys: z, y 2024-05-27T07:08:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:08:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:36 smithi062 ceph-mon[35304]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:37.281 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:36 smithi062 ceph-mon[35304]: mgrmap e15: x(active, since 110s), standbys: z, y 2024-05-27T07:08:37.282 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:08:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:38 smithi012 ceph-mon[31473]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:38.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:38.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:38.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:38.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:38 smithi162 ceph-mon[35676]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:38.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:38.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:38.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:38.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:38 smithi062 ceph-mon[35304]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:39.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:39 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:39.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:39 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:39.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:39 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:39.394 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:39 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:39.395 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:39 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:39.395 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:39 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:39.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:39 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:39.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:39 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:39.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:39 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:40.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:40 smithi162 ceph-mon[35676]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:40.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:40 smithi062 ceph-mon[35304]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:40.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:40 smithi012 ceph-mon[31473]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:41.373 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:08:41.373 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:41.373 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:41 smithi012 ceph-mon[31473]: Deploying daemon osd.0 on smithi012 2024-05-27T07:08:41.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:41 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:08:41.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:41 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:41.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:41 smithi162 ceph-mon[35676]: Deploying daemon osd.0 on smithi012 2024-05-27T07:08:41.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:41 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:08:41.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:41 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:41.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:41 smithi062 ceph-mon[35304]: Deploying daemon osd.0 on smithi012 2024-05-27T07:08:42.220 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:42 smithi012 ceph-mon[31473]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:42.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:42 smithi162 ceph-mon[35676]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:42.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:42 smithi062 ceph-mon[35304]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:44.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:44 smithi012 ceph-mon[31473]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:44.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:44 smithi162 ceph-mon[35676]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:44 smithi062 ceph-mon[35304]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:46 smithi162 ceph-mon[35676]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:46.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:46 smithi062 ceph-mon[35304]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:46 smithi012 ceph-mon[31473]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:48.818 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:48 smithi012 ceph-mon[31473]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:48 smithi162 ceph-mon[35676]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:48.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:48 smithi062 ceph-mon[35304]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:50.485 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:50.485 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:50.485 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:50.485 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:50 smithi012 ceph-mon[31473]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:50.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:50.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:50.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:50.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:50 smithi162 ceph-mon[35676]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:50.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:50.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:50.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:50 smithi062 ceph-mon[35304]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:52 smithi162 ceph-mon[35676]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:52 smithi062 ceph-mon[35304]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:52 smithi012 ceph-mon[31473]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.786 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:53 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:53 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:53.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:53 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:53.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:53 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:53 smithi012 ceph-mon[31473]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:08:53.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:53 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:53 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:53.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:53 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:53.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:53 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:53 smithi162 ceph-mon[35676]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:08:53.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:53 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:53 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:53.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:53 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:53.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:53 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:53.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:53 smithi062 ceph-mon[35304]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:08:54.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.656+0000 7f5f07e00700 1 -- 172.21.15.12:0/4017848037 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5f10103140 con 0x7f5ef403e750 2024-05-27T07:08:54.648 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 0 on host 'smithi012' 2024-05-27T07:08:54.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.660+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5ef403e750 msgr2=0x7f5ef4040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:54.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.660+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5ef403e750 0x7f5ef4040c60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8005950 tx=0x7f5ef80058e0 comp rx=0 tx=0).stop 2024-05-27T07:08:54.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.660+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10104c40 msgr2=0x7f5f1019c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:54.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.660+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10104c40 0x7f5f1019c7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5f00007ee0 tx=0x7f5f0000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:08:54.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.661+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 shutdown_connections 2024-05-27T07:08:54.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.661+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5ef403e750 0x7f5ef4040c60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:54.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.661+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f10109f00 0x7f5f101a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:54.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.661+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f10104c40 0x7f5f1019c7d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:54.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.661+0000 7f5f0fe00700 1 --2- 172.21.15.12:0/4017848037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f101075a0 0x7f5f1019cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:54.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.661+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 >> 172.21.15.12:0/4017848037 conn(0x7f5f100fe680 msgr2=0x7f5f10102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:54.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.661+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 shutdown_connections 2024-05-27T07:08:54.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:08:54.662+0000 7f5f0fe00700 1 -- 172.21.15.12:0/4017848037 wait complete. 2024-05-27T07:08:54.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:54 smithi012 ceph-mon[31473]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:54.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:54 smithi012 ceph-mon[31473]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:08:54.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:54 smithi012 ceph-mon[31473]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:08:54.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:54 smithi012 ceph-mon[31473]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:08:54.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:54 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:54.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:54 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:54.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:54 smithi162 ceph-mon[35676]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:54.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:54 smithi162 ceph-mon[35676]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:08:54.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:54 smithi162 ceph-mon[35676]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:08:54.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:54 smithi162 ceph-mon[35676]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:08:54.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:54 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:54.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:54 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:54.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:54 smithi062 ceph-mon[35304]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:54.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:54 smithi062 ceph-mon[35304]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:08:54.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:54 smithi062 ceph-mon[35304]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:08:54.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:54 smithi062 ceph-mon[35304]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:08:54.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:54 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:54.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:54 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:55.462 DEBUG:teuthology.orchestra.run.smithi012:osd.0> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.0.service 2024-05-27T07:08:55.467 INFO:tasks.cephadm:Deploying osd.1 on smithi062 with /dev/vg_nvme/lv_4... 2024-05-27T07:08:55.467 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:08:55.711 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:55.711 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:55.711 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:55 smithi012 ceph-mon[31473]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:08:55.711 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:55 smithi012 ceph-mon[31473]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:08:55.711 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:55.711 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:55.930 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:08:55.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:55.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:55.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:55 smithi162 ceph-mon[35676]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:08:55.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:55 smithi162 ceph-mon[35676]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:08:55.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:55.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:55.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:08:55.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:55.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:55 smithi062 ceph-mon[35304]: from='osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:08:55.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:55 smithi062 ceph-mon[35304]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:08:55.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:55.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:56 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:08:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:56 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:08:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:56 smithi012 ceph-mon[31473]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:56 smithi012 ceph-mon[31473]: osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] boot 2024-05-27T07:08:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:56 smithi012 ceph-mon[31473]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:08:56.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:56.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:56 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:08:56.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:56 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:08:56.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:56 smithi162 ceph-mon[35676]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:56.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:56 smithi162 ceph-mon[35676]: osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] boot 2024-05-27T07:08:56.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:56 smithi162 ceph-mon[35676]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:08:56.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:56 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:08:56.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:56 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:08:56.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:56 smithi062 ceph-mon[35304]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:08:56.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:56 smithi062 ceph-mon[35304]: osd.0 [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] boot 2024-05-27T07:08:56.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:56 smithi062 ceph-mon[35304]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:08:56.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:08:57.828 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:57 smithi062 ceph-mon[35304]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:08:57.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:57 smithi012 ceph-mon[31473]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:08:57.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:57 smithi162 ceph-mon[35676]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:08:58.797 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:08:58.797 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:08:58.797 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-05-27T07:08:58.797 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-05-27T07:08:58.797 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0539515 s, 194 MB/s 2024-05-27T07:08:58.797 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-05-27T07:08:58.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:08:58 smithi162 ceph-mon[35676]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:08:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:08:58 smithi062 ceph-mon[35304]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:08:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:08:58 smithi012 ceph-mon[31473]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:08:59.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_4 2024-05-27T07:09:00.003 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:00.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:00.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:00.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:00.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:09:00.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:00.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:00.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:00.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:09:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:01.926 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:01 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi012 2024-05-27T07:09:01.926 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:01 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi012 to 17094M 2024-05-27T07:09:01.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:01 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi012 2024-05-27T07:09:01.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:01 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi012 to 17094M 2024-05-27T07:09:02.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:01 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi012 2024-05-27T07:09:02.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:01 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi012 to 17094M 2024-05-27T07:09:02.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.185+0000 7f30aca00700 1 -- 172.21.15.62:0/1248185055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a80776d0 msgr2=0x7f30a8075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:02.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.185+0000 7f30aca00700 1 --2- 172.21.15.62:0/1248185055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a80776d0 0x7f30a8075ba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3098009990 tx=0x7f3098009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:02.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.185+0000 7f30aca00700 1 -- 172.21.15.62:0/1248185055 shutdown_connections 2024-05-27T07:09:02.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.185+0000 7f30aca00700 1 --2- 172.21.15.62:0/1248185055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a80776d0 0x7f30a8075ba0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:02.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.185+0000 7f30aca00700 1 --2- 172.21.15.62:0/1248185055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a8110790 0x7f30a8112bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:02.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.185+0000 7f30aca00700 1 --2- 172.21.15.62:0/1248185055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80760e0 0x7f30a8076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:02.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.185+0000 7f30aca00700 1 -- 172.21.15.62:0/1248185055 >> 172.21.15.62:0/1248185055 conn(0x7f30a80ff7a0 msgr2=0x7f30a8101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:02.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.186+0000 7f30aca00700 1 -- 172.21.15.62:0/1248185055 shutdown_connections 2024-05-27T07:09:02.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.186+0000 7f30aca00700 1 -- 172.21.15.62:0/1248185055 wait complete. 2024-05-27T07:09:02.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.188+0000 7f30aca00700 1 Processor -- start 2024-05-27T07:09:02.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.188+0000 7f30aca00700 1 -- start start 2024-05-27T07:09:02.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.188+0000 7f30aca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80760e0 0x7f30a81a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:02.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.188+0000 7f30aca00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a80776d0 0x7f30a81a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:02.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30aca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a8110790 0x7f30a81a8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:02.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30aca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f30a8116080 con 0x7f30a80760e0 2024-05-27T07:09:02.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30aca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f30a8115f00 con 0x7f30a8110790 2024-05-27T07:09:02.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30aca00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30a8116200 con 0x7f30a80776d0 2024-05-27T07:09:02.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30a7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a8110790 0x7f30a81a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:02.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30a7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a8110790 0x7f30a81a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57786/0 (socket says 172.21.15.62:57786) 2024-05-27T07:09:02.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30a7e00700 1 -- 172.21.15.62:0/2380847128 learned_addr learned my addr 172.21.15.62:0/2380847128 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-05-27T07:09:02.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f309ea00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a80776d0 0x7f30a81a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:02.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.189+0000 7f30a7400700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80760e0 0x7f30a81a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a7e00700 1 -- 172.21.15.62:0/2380847128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a80776d0 msgr2=0x7f30a81a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a7e00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a80776d0 0x7f30a81a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a7e00700 1 -- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80760e0 msgr2=0x7f30a81a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a7e00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80760e0 0x7f30a81a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a7e00700 1 -- 172.21.15.62:0/2380847128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a81a87f0 con 0x7f30a8110790 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a7e00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a8110790 0x7f30a81a8120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f30a0007ee0 tx=0x7f30a000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a4c00700 1 -- 172.21.15.62:0/2380847128 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a0004bb0 con 0x7f30a8110790 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a4c00700 1 -- 172.21.15.62:0/2380847128 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30a0004d50 con 0x7f30a8110790 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a81a8ae0 con 0x7f30a8110790 2024-05-27T07:09:02.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30a4c00700 1 -- 172.21.15.62:0/2380847128 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a00058b0 con 0x7f30a8110790 2024-05-27T07:09:02.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.190+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a81a8fd0 con 0x7f30a8110790 2024-05-27T07:09:02.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.192+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a802e5e0 con 0x7f30a8110790 2024-05-27T07:09:02.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.194+0000 7f30a4c00700 1 -- 172.21.15.62:0/2380847128 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f30a001a020 con 0x7f30a8110790 2024-05-27T07:09:02.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.196+0000 7f30a4c00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f308c0b27f0 0x7f308c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:02.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.196+0000 7f30a4c00700 1 -- 172.21.15.62:0/2380847128 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f30a00dfdf0 con 0x7f30a8110790 2024-05-27T07:09:02.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.197+0000 7f30a7400700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f308c0b27f0 0x7f308c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:02.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.198+0000 7f30a7400700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f308c0b27f0 0x7f308c0b4d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30a81a2080 tx=0x7f309801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:02.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.202+0000 7f30a4c00700 1 -- 172.21.15.62:0/2380847128 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f30a0010070 con 0x7f30a8110790 2024-05-27T07:09:02.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:02.448+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f30a8063bf0 con 0x7f308c0b27f0 2024-05-27T07:09:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:02 smithi162 ceph-mon[35676]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:02.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:09:02.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:09:02.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:02 smithi062 ceph-mon[35304]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:09:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:09:02.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:02 smithi012 ceph-mon[31473]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:09:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:09:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:03.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:03 smithi062 ceph-mon[35304]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:09:04.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:03 smithi012 ceph-mon[31473]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:09:04.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:03 smithi162 ceph-mon[35676]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:09:04.981 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:04 smithi062 ceph-mon[35304]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:05.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:04 smithi012 ceph-mon[31473]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:04 smithi162 ceph-mon[35676]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:06.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:06 smithi062 ceph-mon[35304]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:06 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/2864564723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]: dispatch 2024-05-27T07:09:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:06 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]: dispatch 2024-05-27T07:09:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:06 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]': finished 2024-05-27T07:09:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:06 smithi062 ceph-mon[35304]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:09:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:06 smithi012 ceph-mon[31473]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:06 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/2864564723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]: dispatch 2024-05-27T07:09:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:06 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]: dispatch 2024-05-27T07:09:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:06 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]': finished 2024-05-27T07:09:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:06 smithi012 ceph-mon[31473]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:09:07.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:06 smithi162 ceph-mon[35676]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:06 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/2864564723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]: dispatch 2024-05-27T07:09:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:06 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]: dispatch 2024-05-27T07:09:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:06 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afd3e256-9590-46d2-b846-c3b3539d8de5"}]': finished 2024-05-27T07:09:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:06 smithi162 ceph-mon[35676]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:09:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:08.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:07 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/3834732055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:09:08.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:07 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/3834732055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:09:09.285 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:08 smithi012 ceph-mon[31473]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:09.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:08 smithi162 ceph-mon[35676]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:09.285 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:07 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/3834732055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:09:09.285 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:08 smithi062 ceph-mon[35304]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:10 smithi162 ceph-mon[35676]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:10 smithi012 ceph-mon[31473]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:11.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:10 smithi062 ceph-mon[35304]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:12 smithi012 ceph-mon[31473]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:12 smithi162 ceph-mon[35676]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:13.225 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:12 smithi062 ceph-mon[35304]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:14.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:14 smithi012 ceph-mon[31473]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:14.949 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:14 smithi062 ceph-mon[35304]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:14 smithi162 ceph-mon[35676]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:16 smithi012 ceph-mon[31473]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:09:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:16 smithi162 ceph-mon[35676]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:09:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:16 smithi062 ceph-mon[35304]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:09:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:17.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:17 smithi062 ceph-mon[35304]: Deploying daemon osd.1 on smithi062 2024-05-27T07:09:18.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:17 smithi012 ceph-mon[31473]: Deploying daemon osd.1 on smithi062 2024-05-27T07:09:18.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:17 smithi162 ceph-mon[35676]: Deploying daemon osd.1 on smithi062 2024-05-27T07:09:18.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:18 smithi062 ceph-mon[35304]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:19.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:18 smithi012 ceph-mon[31473]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:18 smithi162 ceph-mon[35676]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:20 smithi012 ceph-mon[31473]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:21.159 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:20 smithi062 ceph-mon[35304]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:20 smithi162 ceph-mon[35676]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:22 smithi012 ceph-mon[31473]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:22 smithi162 ceph-mon[35676]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:22 smithi062 ceph-mon[35304]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:25.069 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:24 smithi062 ceph-mon[35304]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:25.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:24 smithi012 ceph-mon[31473]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:25.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:24 smithi162 ceph-mon[35676]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:26.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:26.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:26.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:26.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:26 smithi062 ceph-mon[35304]: pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:26.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:26.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:26.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:26.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:26 smithi012 ceph-mon[31473]: pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:26.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:26.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:26.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:26.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:26 smithi162 ceph-mon[35676]: pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:28.907 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:28 smithi062 ceph-mon[35304]: pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:28.907 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:28.907 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:28.907 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:28.908 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:28.908 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:28 smithi162 ceph-mon[35676]: pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:29.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:28 smithi012 ceph-mon[31473]: pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:29.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:29.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:29.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:29.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:29.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:29.684 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:29 smithi062 ceph-mon[35304]: from='osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:09:29.684 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:29 smithi062 ceph-mon[35304]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:09:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:29 smithi162 ceph-mon[35676]: from='osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:09:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:29 smithi162 ceph-mon[35676]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:09:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:29 smithi012 ceph-mon[31473]: from='osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:09:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:29 smithi012 ceph-mon[31473]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:09:30.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.564+0000 7f30a4c00700 1 -- 172.21.15.62:0/2380847128 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f30a8063bf0 con 0x7f308c0b27f0 2024-05-27T07:09:30.556 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 1 on host 'smithi062' 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f308c0b27f0 msgr2=0x7f308c0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f308c0b27f0 0x7f308c0b4d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30a81a2080 tx=0x7f309801a040 comp rx=0 tx=0).stop 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a8110790 msgr2=0x7f30a81a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a8110790 0x7f30a81a8120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f30a0007ee0 tx=0x7f30a000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 shutdown_connections 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f308c0b27f0 0x7f308c0b4d00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30a8110790 0x7f30a81a8120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30a80760e0 0x7f30a81a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 --2- 172.21.15.62:0/2380847128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a80776d0 0x7f30a81a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 >> 172.21.15.62:0/2380847128 conn(0x7f30a80ff7a0 msgr2=0x7f30a8103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 shutdown_connections 2024-05-27T07:09:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:09:30.568+0000 7f30aca00700 1 -- 172.21.15.62:0/2380847128 wait complete. 2024-05-27T07:09:30.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:30.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:09:30.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:09:30.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: from='osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:09:30.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:30.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:09:30.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:30.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:30.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: from='osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:09:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:09:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: from='osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:09:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:09:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:31.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:31.375 DEBUG:teuthology.orchestra.run.smithi062:osd.1> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.1.service 2024-05-27T07:09:31.379 INFO:tasks.cephadm:Deploying osd.2 on smithi162 with /dev/vg_nvme/lv_4... 2024-05-27T07:09:31.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:09:31.641 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:09:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1[39878]: 2024-05-27T07:09:31.501+0000 7f63d9000700 -1 osd.1 0 waiting for initial osdmap 2024-05-27T07:09:31.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:09:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1[39878]: 2024-05-27T07:09:31.527+0000 7f63cee00700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:09:31.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:31 smithi162 ceph-mon[35676]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:09:31.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:31 smithi162 ceph-mon[35676]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:09:31.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:31 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:31.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:31 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:31.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:31 smithi162 ceph-mon[35676]: from='osd.1 ' entity='osd.1' 2024-05-27T07:09:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:09:31.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:31 smithi062 ceph-mon[35304]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:09:31.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:31 smithi062 ceph-mon[35304]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:09:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:31 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:31 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:31 smithi062 ceph-mon[35304]: from='osd.1 ' entity='osd.1' 2024-05-27T07:09:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:31 smithi012 ceph-mon[31473]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:09:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:31 smithi012 ceph-mon[31473]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:09:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:31 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:31 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:32.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:31 smithi012 ceph-mon[31473]: from='osd.1 ' entity='osd.1' 2024-05-27T07:09:32.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:32 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:09:32.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:32 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:32 smithi162 ceph-mon[35676]: pgmap v85: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:32 smithi162 ceph-mon[35676]: osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] boot 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:32 smithi162 ceph-mon[35676]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:32 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:32 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:32 smithi062 ceph-mon[35304]: pgmap v85: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:32.932 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:32.933 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:32 smithi062 ceph-mon[35304]: osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] boot 2024-05-27T07:09:32.933 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:32 smithi062 ceph-mon[35304]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:09:32.933 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:32 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:09:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:32 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:09:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:32 smithi012 ceph-mon[31473]: pgmap v85: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:09:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:32 smithi012 ceph-mon[31473]: osd.1 [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] boot 2024-05-27T07:09:33.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:32 smithi012 ceph-mon[31473]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:09:33.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:09:34.397 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:09:34.398 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:09:34.398 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-05-27T07:09:34.398 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-05-27T07:09:34.398 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0329011 s, 319 MB/s 2024-05-27T07:09:34.398 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-05-27T07:09:34.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:34 smithi062 ceph-mon[35304]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:34.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:34 smithi062 ceph-mon[35304]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:09:34.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:34 smithi162 ceph-mon[35676]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:34.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:34 smithi162 ceph-mon[35676]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:09:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:34 smithi012 ceph-mon[31473]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:34 smithi012 ceph-mon[31473]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:09:35.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-05-27T07:09:35.587 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:09:36.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:36 smithi162 ceph-mon[35676]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:36.957 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:36 smithi062 ceph-mon[35304]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:37.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:36 smithi012 ceph-mon[31473]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:37.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 -- 172.21.15.162:0/137427768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0109fe0 msgr2=0x7fa0b010c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 --2- 172.21.15.162:0/137427768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0109fe0 0x7fa0b010c4f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009990 tx=0x7fa0a8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 -- 172.21.15.162:0/137427768 shutdown_connections 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 --2- 172.21.15.162:0/137427768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b006c2c0 0x7fa0b0106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 --2- 172.21.15.162:0/137427768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0109fe0 0x7fa0b010c4f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 --2- 172.21.15.162:0/137427768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b0107470 0x7fa0b0109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 -- 172.21.15.162:0/137427768 >> 172.21.15.162:0/137427768 conn(0x7fa0b00fe870 msgr2=0x7fa0b0100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 -- 172.21.15.162:0/137427768 shutdown_connections 2024-05-27T07:09:37.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.542+0000 7fa0b6200700 1 -- 172.21.15.162:0/137427768 wait complete. 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.543+0000 7fa0b6200700 1 Processor -- start 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.543+0000 7fa0b6200700 1 -- start start 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0b6200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b006c2c0 0x7fa0b019c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0b6200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0107470 0x7fa0b019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0b6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b0109fe0 0x7fa0b01a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0b6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b010fc70 con 0x7fa0b0109fe0 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0b6200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b010faf0 con 0x7fa0b006c2c0 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0b6200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b010fdf0 con 0x7fa0b0107470 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0a6a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0107470 0x7fa0b019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0a6a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0107470 0x7fa0b019cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34330/0 (socket says 172.21.15.162:34330) 2024-05-27T07:09:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0a6a00700 1 -- 172.21.15.162:0/2400320607 learned_addr learned my addr 172.21.15.162:0/2400320607 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.544+0000 7fa0afe00700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b0109fe0 0x7fa0b01a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0af400700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b006c2c0 0x7fa0b019c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0a6a00700 1 -- 172.21.15.162:0/2400320607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b006c2c0 msgr2=0x7fa0b019c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0a6a00700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b006c2c0 0x7fa0b019c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0a6a00700 1 -- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b0109fe0 msgr2=0x7fa0b01a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0a6a00700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b0109fe0 0x7fa0b01a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0a6a00700 1 -- 172.21.15.162:0/2400320607 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b01a4550 con 0x7fa0b0107470 2024-05-27T07:09:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0a6a00700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0107470 0x7fa0b019cf30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa09800d950 tx=0x7fa09800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:37.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0acc00700 1 -- 172.21.15.162:0/2400320607 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098016070 con 0x7fa0b0107470 2024-05-27T07:09:37.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b01a4840 con 0x7fa0b0107470 2024-05-27T07:09:37.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.545+0000 7fa0acc00700 1 -- 172.21.15.162:0/2400320607 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa098009dd0 con 0x7fa0b0107470 2024-05-27T07:09:37.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.546+0000 7fa0acc00700 1 -- 172.21.15.162:0/2400320607 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098020060 con 0x7fa0b0107470 2024-05-27T07:09:37.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.546+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b01a4dd0 con 0x7fa0b0107470 2024-05-27T07:09:37.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.547+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b00512b0 con 0x7fa0b0107470 2024-05-27T07:09:37.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.549+0000 7fa0acc00700 1 -- 172.21.15.162:0/2400320607 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fa098019070 con 0x7fa0b0107470 2024-05-27T07:09:37.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.551+0000 7fa0acc00700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa0940b2870 0x7fa0940b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.551+0000 7fa0acc00700 1 -- 172.21.15.162:0/2400320607 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fa0980e09c0 con 0x7fa0b0107470 2024-05-27T07:09:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.551+0000 7fa0af400700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa0940b2870 0x7fa0940b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:37.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.552+0000 7fa0af400700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa0940b2870 0x7fa0940b4d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d10 tx=0x7fa0a0009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.554+0000 7fa0acc00700 1 -- 172.21.15.162:0/2400320607 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa0980a9e90 con 0x7fa0b0107470 2024-05-27T07:09:37.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:09:37.794+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa0b010aad0 con 0x7fa0940b2870 2024-05-27T07:09:38.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi062 2024-05-27T07:09:38.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:09:38.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi062 to 17089M 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:09:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:38 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi062 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi062 to 17089M 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:38.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:09:38.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:09:38.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:09:38.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:38 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:38.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi062 2024-05-27T07:09:38.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi062 to 17089M 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:09:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:38 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:40.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:40 smithi162 ceph-mon[35676]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:40.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:40 smithi062 ceph-mon[35304]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:41.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:40 smithi012 ceph-mon[31473]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:42.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:42 smithi012 ceph-mon[31473]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:42.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:42 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/2081502130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]: dispatch 2024-05-27T07:09:42.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:42 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]: dispatch 2024-05-27T07:09:42.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:42 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]': finished 2024-05-27T07:09:42.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:42 smithi012 ceph-mon[31473]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:09:42.894 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:09:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:42 smithi162 ceph-mon[35676]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:42 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/2081502130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]: dispatch 2024-05-27T07:09:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:42 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]: dispatch 2024-05-27T07:09:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:42 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]': finished 2024-05-27T07:09:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:42 smithi162 ceph-mon[35676]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:09:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:09:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:42 smithi062 ceph-mon[35304]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:42 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/2081502130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]: dispatch 2024-05-27T07:09:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:42 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]: dispatch 2024-05-27T07:09:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:42 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3814d7d7-2bd7-4e53-958f-3fa857d074dd"}]': finished 2024-05-27T07:09:42.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:42 smithi062 ceph-mon[35304]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:09:42.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:09:43.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:43 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/3283799095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:09:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:43 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/3283799095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:09:44.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:43 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/3283799095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:09:44.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:44 smithi162 ceph-mon[35676]: pgmap v94: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:44 smithi062 ceph-mon[35304]: pgmap v94: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:45.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:44 smithi012 ceph-mon[31473]: pgmap v94: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:46 smithi162 ceph-mon[35676]: pgmap v95: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:46.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:46 smithi062 ceph-mon[35304]: pgmap v95: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:47.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:46 smithi012 ceph-mon[31473]: pgmap v95: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:48 smithi162 ceph-mon[35676]: pgmap v96: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:48.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:48 smithi062 ceph-mon[35304]: pgmap v96: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:48 smithi012 ceph-mon[31473]: pgmap v96: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:50.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:50 smithi062 ceph-mon[35304]: pgmap v97: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:50.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:50 smithi162 ceph-mon[35676]: pgmap v97: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:50 smithi012 ceph-mon[31473]: pgmap v97: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:51.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:09:51.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:51.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:09:51.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:52.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:09:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:52.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:52 smithi162 ceph-mon[35676]: Deploying daemon osd.2 on smithi162 2024-05-27T07:09:52.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:52 smithi162 ceph-mon[35676]: pgmap v98: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:52 smithi062 ceph-mon[35304]: Deploying daemon osd.2 on smithi162 2024-05-27T07:09:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:52 smithi062 ceph-mon[35304]: pgmap v98: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:52 smithi012 ceph-mon[31473]: Deploying daemon osd.2 on smithi162 2024-05-27T07:09:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:52 smithi012 ceph-mon[31473]: pgmap v98: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:54.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:54 smithi162 ceph-mon[35676]: pgmap v99: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:54.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:54 smithi062 ceph-mon[35304]: pgmap v99: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:54 smithi012 ceph-mon[31473]: pgmap v99: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:56.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:56 smithi162 ceph-mon[35676]: pgmap v100: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:56 smithi062 ceph-mon[35304]: pgmap v100: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:57.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:56 smithi012 ceph-mon[31473]: pgmap v100: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:58.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:09:58 smithi162 ceph-mon[35676]: pgmap v101: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:09:58 smithi062 ceph-mon[35304]: pgmap v101: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:09:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:09:58 smithi012 ceph-mon[31473]: pgmap v101: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:01.044 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:00 smithi162 ceph-mon[35676]: pgmap v102: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:00 smithi162 ceph-mon[35676]: overall HEALTH_OK 2024-05-27T07:10:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:00 smithi012 ceph-mon[31473]: pgmap v102: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:00 smithi012 ceph-mon[31473]: overall HEALTH_OK 2024-05-27T07:10:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:00 smithi062 ceph-mon[35304]: pgmap v102: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:01.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:01.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:01.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:01.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:00 smithi062 ceph-mon[35304]: overall HEALTH_OK 2024-05-27T07:10:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:02 smithi162 ceph-mon[35676]: pgmap v103: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:03.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:02 smithi012 ceph-mon[31473]: pgmap v103: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:02 smithi062 ceph-mon[35304]: pgmap v103: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:04.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:03 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:04.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:03 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:04.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:03 smithi162 ceph-mon[35676]: pgmap v104: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:04.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:03 smithi162 ceph-mon[35676]: from='osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:10:04.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:03 smithi162 ceph-mon[35676]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:10:04.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:03 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:04.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:03 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:04.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:03 smithi062 ceph-mon[35304]: pgmap v104: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:04.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:03 smithi062 ceph-mon[35304]: from='osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:10:04.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:03 smithi062 ceph-mon[35304]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:10:04.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:03 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:04.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:03 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:04.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:03 smithi012 ceph-mon[31473]: pgmap v104: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:04.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:03 smithi012 ceph-mon[31473]: from='osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:10:04.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:03 smithi012 ceph-mon[31473]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:10:04.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.878+0000 7fa0acc00700 1 -- 172.21.15.162:0/2400320607 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa0b010aad0 con 0x7fa0940b2870 2024-05-27T07:10:04.870 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 2 on host 'smithi162' 2024-05-27T07:10:04.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.882+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa0940b2870 msgr2=0x7fa0940b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:04.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa0940b2870 0x7fa0940b4d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d10 tx=0x7fa0a0009040 comp rx=0 tx=0).stop 2024-05-27T07:10:04.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0107470 msgr2=0x7fa0b019cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:04.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0107470 0x7fa0b019cf30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa09800d950 tx=0x7fa09800dc60 comp rx=0 tx=0).stop 2024-05-27T07:10:04.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 shutdown_connections 2024-05-27T07:10:04.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa0940b2870 0x7fa0940b4d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:04.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b006c2c0 0x7fa0b019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:04.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0b0109fe0 0x7fa0b01a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:04.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 --2- 172.21.15.162:0/2400320607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0b0107470 0x7fa0b019cf30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:04.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 >> 172.21.15.162:0/2400320607 conn(0x7fa0b00fe870 msgr2=0x7fa0b0100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:04.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 shutdown_connections 2024-05-27T07:10:04.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:10:04.883+0000 7fa0b6200700 1 -- 172.21.15.162:0/2400320607 wait complete. 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:10:05.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:10:05.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:10:05.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:05.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:10:05.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:05.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:05.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:05.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:05.711 DEBUG:teuthology.orchestra.run.smithi162:osd.2> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.2.service 2024-05-27T07:10:05.716 INFO:tasks.cephadm:Deploying osd.3 on smithi012 with /dev/vg_nvme/lv_3... 2024-05-27T07:10:05.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:10:05.921 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2[40827]: 2024-05-27T07:10:05.915+0000 7f9b3e000700 -1 osd.2 0 waiting for initial osdmap 2024-05-27T07:10:05.921 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2[40827]: 2024-05-27T07:10:05.930+0000 7f9b33400700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:10:06.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:06 smithi012 ceph-mon[31473]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:10:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:06 smithi012 ceph-mon[31473]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:10:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:06 smithi012 ceph-mon[31473]: pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:06 smithi012 ceph-mon[31473]: from='osd.2 ' entity='osd.2' 2024-05-27T07:10:06.185 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:06.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:06 smithi162 ceph-mon[35676]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:10:06.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:06 smithi162 ceph-mon[35676]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:10:06.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:06.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:06.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:06 smithi162 ceph-mon[35676]: pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:06.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:06 smithi162 ceph-mon[35676]: from='osd.2 ' entity='osd.2' 2024-05-27T07:10:06.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:06 smithi062 ceph-mon[35304]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:10:06.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:06 smithi062 ceph-mon[35304]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:10:06.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:06.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:06.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:06 smithi062 ceph-mon[35304]: pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:10:06.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:06 smithi062 ceph-mon[35304]: from='osd.2 ' entity='osd.2' 2024-05-27T07:10:07.098 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:07 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:10:07.098 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:07 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:10:07.098 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:07.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:07 smithi162 ceph-mon[35676]: osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] boot 2024-05-27T07:10:07.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:07 smithi162 ceph-mon[35676]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:10:07.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:07 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:10:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:07 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:10:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:07 smithi012 ceph-mon[31473]: osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] boot 2024-05-27T07:10:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:07 smithi012 ceph-mon[31473]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:10:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:07.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:07 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:10:07.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:07 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:10:07.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:07.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:07 smithi062 ceph-mon[35304]: osd.2 [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] boot 2024-05-27T07:10:07.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:07 smithi062 ceph-mon[35304]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:10:07.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:10:08.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:08 smithi012 ceph-mon[31473]: pgmap v109: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:08.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:10:08.388 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:08 smithi062 ceph-mon[35304]: pgmap v109: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:08.388 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:10:08.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:08 smithi162 ceph-mon[35676]: pgmap v109: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:08.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:10:09.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:10:09.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 ceph-mon[31473]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:10:09.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:10:09.123 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:10:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:10:09.124 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-05-27T07:10:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-05-27T07:10:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.03848 s, 272 MB/s 2024-05-27T07:10:09.124 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-05-27T07:10:09.358 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:09 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:10:09.358 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:09 smithi062 ceph-mon[35304]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:10:09.358 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:09 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:10:09.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:10:09.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 ceph-mon[35676]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:10:09.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:10:09.489 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50357]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:10:09.489 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50357]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:09.489 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50357]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:09.489 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50357]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:09.489 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50371]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:10:09.489 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50371]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:09.490 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50371]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:09.490 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50371]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:09.736 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:10:09.737 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:09.737 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43778]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:09.737 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43778]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:09.737 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:10:09.737 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:09.737 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43781]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:09.737 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43781]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:09.738 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50378]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:10:09.836 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[43872]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:10:09.837 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[43872]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:09.837 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[43872]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:09.837 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[43872]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:09.837 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44192]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:10:09.837 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44192]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:09.837 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44192]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:09.837 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44192]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:09.896 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_3 2024-05-27T07:10:10.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50378]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:10.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50378]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:10.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50378]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:10.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50381]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:10:10.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50381]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:10.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50381]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:10.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:09 smithi012 sudo[50381]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:10.174 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44934]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:10:10.174 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44934]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:10.174 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44934]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44934]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44938]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44938]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44938]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:09 smithi162 sudo[44938]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: pgmap v112: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.175 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.176 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.177 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.177 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:10 smithi162 ceph-mon[35676]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43784]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:10:10.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43784]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:10.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43784]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:10.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:09 smithi062 sudo[43784]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:10.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 sudo[43787]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:10:10.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 sudo[43787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 sudo[43787]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 sudo[43787]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: pgmap v112: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.240 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.241 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:10 smithi062 ceph-mon[35304]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: pgmap v112: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:10:10.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:10:10.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:10:10.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:10:10.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:10 smithi012 ceph-mon[31473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:10:10.349 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:10:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-05-27T07:10:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi162 2024-05-27T07:10:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi162 to 17089M 2024-05-27T07:10:11.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:11.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:11.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:10:11.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-05-27T07:10:11.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi162 2024-05-27T07:10:11.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:11.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi162 to 17089M 2024-05-27T07:10:11.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:11.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:11.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi162 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi162 to 17089M 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:11.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:12 smithi012 ceph-mon[31473]: pgmap v114: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:12.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:12 smithi162 ceph-mon[35676]: pgmap v114: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:12.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:12 smithi062 ceph-mon[35304]: pgmap v114: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:12.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 -- 172.21.15.12:0/2690674048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec105e00 msgr2=0x7f5fec10c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:12.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 --2- 172.21.15.12:0/2690674048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec105e00 0x7f5fec10c6b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009990 tx=0x7f5fe4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:12.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 -- 172.21.15.12:0/2690674048 shutdown_connections 2024-05-27T07:10:12.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 --2- 172.21.15.12:0/2690674048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fec104330 0x7f5fec104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:12.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 --2- 172.21.15.12:0/2690674048 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec105e00 0x7f5fec10c6b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:12.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 --2- 172.21.15.12:0/2690674048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fec105460 0x7f5fec105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:12.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 -- 172.21.15.12:0/2690674048 >> 172.21.15.12:0/2690674048 conn(0x7f5fec0ff9b0 msgr2=0x7f5fec101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:12.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 -- 172.21.15.12:0/2690674048 shutdown_connections 2024-05-27T07:10:12.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.730+0000 7f5ff1600700 1 -- 172.21.15.12:0/2690674048 wait complete. 2024-05-27T07:10:12.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.731+0000 7f5ff1600700 1 Processor -- start 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5ff1600700 1 -- start start 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5ff1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec104330 0x7f5fec19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5ff1600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fec105460 0x7f5fec19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5ff1600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fec105e00 0x7f5fec1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5ff1600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec10fca0 con 0x7f5fec104330 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5ff1600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec10fb20 con 0x7f5fec105e00 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5ff1600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec10fe20 con 0x7f5fec105460 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5feaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec104330 0x7f5fec19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5feaa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec104330 0x7f5fec19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55840/0 (socket says 172.21.15.12:55840) 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.732+0000 7f5feaa00700 1 -- 172.21.15.12:0/4114603854 learned_addr learned my addr 172.21.15.12:0/4114603854 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:10:12.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5fe2000700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fec105460 0x7f5fec19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:12.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5feaa00700 1 -- 172.21.15.12:0/4114603854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fec105460 msgr2=0x7f5fec19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:12.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5feaa00700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fec105460 0x7f5fec19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:12.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5feaa00700 1 -- 172.21.15.12:0/4114603854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fec105e00 msgr2=0x7f5fec1a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:12.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5feaa00700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fec105e00 0x7f5fec1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:12.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5feaa00700 1 -- 172.21.15.12:0/4114603854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec1a22e0 con 0x7f5fec104330 2024-05-27T07:10:12.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5feaa00700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec104330 0x7f5fec19a810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00d950 tx=0x7f5fdc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:12.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.733+0000 7f5fe3e00700 1 -- 172.21.15.12:0/4114603854 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc017070 con 0x7f5fec104330 2024-05-27T07:10:12.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.734+0000 7f5fe3e00700 1 -- 172.21.15.12:0/4114603854 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fdc00de30 con 0x7f5fec104330 2024-05-27T07:10:12.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.734+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fec1a25d0 con 0x7f5fec104330 2024-05-27T07:10:12.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.734+0000 7f5fe3e00700 1 -- 172.21.15.12:0/4114603854 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc003aa0 con 0x7f5fec104330 2024-05-27T07:10:12.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.734+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fec1a2b10 con 0x7f5fec104330 2024-05-27T07:10:12.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.735+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fec0512b0 con 0x7f5fec104330 2024-05-27T07:10:12.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.737+0000 7f5fe3e00700 1 -- 172.21.15.12:0/4114603854 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5fdc003c40 con 0x7f5fec104330 2024-05-27T07:10:12.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.739+0000 7f5fe3e00700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5fd00b2880 0x7f5fd00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:12.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.739+0000 7f5fe3e00700 1 -- 172.21.15.12:0/4114603854 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f5fdc0e1210 con 0x7f5fec104330 2024-05-27T07:10:12.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.739+0000 7f5fe2000700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5fd00b2880 0x7f5fd00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:12.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.740+0000 7f5fe2000700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5fd00b2880 0x7f5fd00b4d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5fec19b8c0 tx=0x7f5fd4009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:12.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.744+0000 7f5fe3e00700 1 -- 172.21.15.12:0/4114603854 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5fdc020030 con 0x7f5fec104330 2024-05-27T07:10:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:12.996+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5fec063bf0 con 0x7f5fd00b2880 2024-05-27T07:10:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:10:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:10:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:13.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:13 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:10:13.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:13 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:10:13.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:13 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:13.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:10:13.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:10:13.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:14.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:14 smithi012 ceph-mon[31473]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:14.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:14 smithi012 ceph-mon[31473]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:14.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:14 smithi162 ceph-mon[35676]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:14.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:14 smithi162 ceph-mon[35676]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:14.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:14 smithi062 ceph-mon[35304]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:14.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:14 smithi062 ceph-mon[35304]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:16.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:16 smithi162 ceph-mon[35676]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:16.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:16 smithi062 ceph-mon[35304]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:16 smithi012 ceph-mon[31473]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:17.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:17 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/273600938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6c89349-9353-4f0c-9a8d-07e7305be298"}]: dispatch 2024-05-27T07:10:17.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:17 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/273600938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6c89349-9353-4f0c-9a8d-07e7305be298"}]': finished 2024-05-27T07:10:17.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:17 smithi162 ceph-mon[35676]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:10:17.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:17 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:17.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:17 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/273600938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6c89349-9353-4f0c-9a8d-07e7305be298"}]: dispatch 2024-05-27T07:10:17.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:17 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/273600938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6c89349-9353-4f0c-9a8d-07e7305be298"}]': finished 2024-05-27T07:10:17.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:17 smithi062 ceph-mon[35304]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:10:17.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:17 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:18.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:17 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/273600938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6c89349-9353-4f0c-9a8d-07e7305be298"}]: dispatch 2024-05-27T07:10:18.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:17 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/273600938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6c89349-9353-4f0c-9a8d-07e7305be298"}]': finished 2024-05-27T07:10:18.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:17 smithi012 ceph-mon[31473]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:10:18.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:17 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:18 smithi162 ceph-mon[35676]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:18.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:18 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1683614855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:10:18.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:18 smithi062 ceph-mon[35304]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:18.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:18 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1683614855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:10:19.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:18 smithi012 ceph-mon[31473]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:19.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:18 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1683614855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:10:20.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:20 smithi162 ceph-mon[35676]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:20.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:20 smithi062 ceph-mon[35304]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:20 smithi012 ceph-mon[31473]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:22.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:22 smithi012 ceph-mon[31473]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:22 smithi162 ceph-mon[35676]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:22 smithi062 ceph-mon[35304]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:24.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:24 smithi162 ceph-mon[35676]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:24.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:24 smithi062 ceph-mon[35304]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:25.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:24 smithi012 ceph-mon[31473]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:26.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:26 smithi162 ceph-mon[35676]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:26.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:26 smithi062 ceph-mon[35304]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:27.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:26 smithi012 ceph-mon[31473]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:28.952 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:28 smithi012 ceph-mon[31473]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:28.953 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:10:28.953 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:28.953 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:28 smithi012 ceph-mon[31473]: Deploying daemon osd.3 on smithi012 2024-05-27T07:10:28.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:28 smithi162 ceph-mon[35676]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:28.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:10:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:28.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:28 smithi162 ceph-mon[35676]: Deploying daemon osd.3 on smithi012 2024-05-27T07:10:28.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:28 smithi062 ceph-mon[35304]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:28.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:10:28.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:28.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:28 smithi062 ceph-mon[35304]: Deploying daemon osd.3 on smithi012 2024-05-27T07:10:30.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:30 smithi162 ceph-mon[35676]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:30.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:30 smithi062 ceph-mon[35304]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:31.042 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:30 smithi012 ceph-mon[31473]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:32.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:32 smithi012 ceph-mon[31473]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:32.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:32 smithi162 ceph-mon[35676]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:32.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:32 smithi062 ceph-mon[35304]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:34.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:34 smithi162 ceph-mon[35676]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:34.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:34 smithi062 ceph-mon[35304]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:34 smithi012 ceph-mon[31473]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:36.951 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:36 smithi012 ceph-mon[31473]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:36.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:36 smithi162 ceph-mon[35676]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:36.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:36 smithi062 ceph-mon[35304]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:37.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:37.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:37.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:37.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:37.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:37.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:38.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:38.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:38.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:38.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:38 smithi162 ceph-mon[35676]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:38.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:38 smithi062 ceph-mon[35304]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:39.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:38 smithi012 ceph-mon[31473]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:40.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:40 smithi162 ceph-mon[35676]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:40.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:40 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:40.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:40 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:40.757 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:40 smithi012 ceph-mon[31473]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:40.757 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:40 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:40.757 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:40 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:40.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:40 smithi062 ceph-mon[35304]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:40.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:40 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:40.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:40 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:41.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:41 smithi162 ceph-mon[35676]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:10:41.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:41 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:41.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:41 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:41.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:41 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:41.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:41 smithi062 ceph-mon[35304]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:10:41.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:41 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:41.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:41 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:41.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:41 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:42.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:41 smithi012 ceph-mon[31473]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:10:42.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:42.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:42.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:42.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.813+0000 7f5fe3e00700 1 -- 172.21.15.12:0/4114603854 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5fec063bf0 con 0x7f5fd00b2880 2024-05-27T07:10:42.805 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 3 on host 'smithi012' 2024-05-27T07:10:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.816+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5fd00b2880 msgr2=0x7f5fd00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.816+0000 7f5ff1600700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5fd00b2880 0x7f5fd00b4d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5fec19b8c0 tx=0x7f5fd4009040 comp rx=0 tx=0).stop 2024-05-27T07:10:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec104330 msgr2=0x7f5fec19a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec104330 0x7f5fec19a810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00d950 tx=0x7f5fdc00dd10 comp rx=0 tx=0).stop 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 shutdown_connections 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f5fd00b2880 0x7f5fd00b4d90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fec105e00 0x7f5fec1a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fec104330 0x7f5fec19a810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 --2- 172.21.15.12:0/4114603854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fec105460 0x7f5fec19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 >> 172.21.15.12:0/4114603854 conn(0x7f5fec0ff9b0 msgr2=0x7f5fec108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 shutdown_connections 2024-05-27T07:10:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:10:42.817+0000 7f5ff1600700 1 -- 172.21.15.12:0/4114603854 wait complete. 2024-05-27T07:10:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:42 smithi062 ceph-mon[35304]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:42 smithi062 ceph-mon[35304]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:10:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:42 smithi062 ceph-mon[35304]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:10:42.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:42 smithi062 ceph-mon[35304]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:10:42.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:43.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:42 smithi012 ceph-mon[31473]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:43.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:42 smithi012 ceph-mon[31473]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:10:43.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:42 smithi012 ceph-mon[31473]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:10:43.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:42 smithi012 ceph-mon[31473]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:10:43.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:43.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:42 smithi162 ceph-mon[35676]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:43.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:42 smithi162 ceph-mon[35676]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:10:43.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:42 smithi162 ceph-mon[35676]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:10:43.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:42 smithi162 ceph-mon[35676]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:10:43.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:43.786 DEBUG:teuthology.orchestra.run.smithi012:osd.3> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.3.service 2024-05-27T07:10:43.793 INFO:tasks.cephadm:Deploying osd.4 on smithi062 with /dev/vg_nvme/lv_3... 2024-05-27T07:10:43.794 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:10:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:43 smithi062 ceph-mon[35304]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:10:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:43 smithi062 ceph-mon[35304]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:10:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:44.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:43 smithi012 ceph-mon[31473]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:10:44.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:43 smithi012 ceph-mon[31473]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:10:44.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:44.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:44.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:44.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:44.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:44.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:43 smithi162 ceph-mon[35676]: from='osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:10:44.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:43 smithi162 ceph-mon[35676]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:10:44.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:44.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:44.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:44.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:44.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:44.259 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:10:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:44 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:10:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:44 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:10:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:44 smithi062 ceph-mon[35304]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:44 smithi062 ceph-mon[35304]: osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] boot 2024-05-27T07:10:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:44 smithi062 ceph-mon[35304]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T07:10:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:44 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:10:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:44 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:10:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:44 smithi012 ceph-mon[31473]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:44 smithi012 ceph-mon[31473]: osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] boot 2024-05-27T07:10:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:44 smithi012 ceph-mon[31473]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T07:10:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:44 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:10:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:44 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:10:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:44 smithi162 ceph-mon[35676]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:10:45.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:45.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:44 smithi162 ceph-mon[35676]: osd.3 [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] boot 2024-05-27T07:10:45.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:44 smithi162 ceph-mon[35676]: osdmap e25: 4 total, 4 up, 4 in 2024-05-27T07:10:45.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:10:46.082 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:45 smithi062 ceph-mon[35304]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:10:46.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:45 smithi012 ceph-mon[31473]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:10:46.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:45 smithi162 ceph-mon[35676]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:10:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:46 smithi012 ceph-mon[31473]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:10:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:46 smithi012 ceph-mon[31473]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:10:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:46 smithi162 ceph-mon[35676]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:10:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:46 smithi162 ceph-mon[35676]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:10:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:46 smithi062 ceph-mon[35304]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:10:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:46 smithi062 ceph-mon[35304]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:10:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:10:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:10:47.364 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-05-27T07:10:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-05-27T07:10:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.035933 s, 292 MB/s 2024-05-27T07:10:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-05-27T07:10:48.310 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_3 2024-05-27T07:10:48.768 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:10:49.080 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:10:49.080 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.080 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.080 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:49.081 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:49.081 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.081 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:49.081 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:49.081 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:10:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:49.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:49.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:10:49.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:49.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:10:49.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:49.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:49.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:49.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:10:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:49 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi012 2024-05-27T07:10:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:49 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi012 to 8547M 2024-05-27T07:10:50.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:49 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi012 2024-05-27T07:10:50.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:49 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi012 to 8547M 2024-05-27T07:10:50.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:49 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi012 2024-05-27T07:10:50.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:49 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi012 to 8547M 2024-05-27T07:10:51.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:50 smithi012 ceph-mon[31473]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:10:51.091 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:50 smithi062 ceph-mon[35304]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:10:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:50 smithi162 ceph-mon[35676]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:10:51.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.246+0000 7f0e85000700 1 -- 172.21.15.62:0/2024608703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80107660 msgr2=0x7f0e80107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.246+0000 7f0e85000700 1 --2- 172.21.15.62:0/2024608703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80107660 0x7f0e80107a70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009990 tx=0x7f0e6c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.246+0000 7f0e85000700 1 -- 172.21.15.62:0/2024608703 shutdown_connections 2024-05-27T07:10:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.246+0000 7f0e85000700 1 --2- 172.21.15.62:0/2024608703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e80108000 0x7f0e8010e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.246+0000 7f0e85000700 1 --2- 172.21.15.62:0/2024608703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e80106530 0x7f0e80106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.246+0000 7f0e85000700 1 --2- 172.21.15.62:0/2024608703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80107660 0x7f0e80107a70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.247+0000 7f0e85000700 1 -- 172.21.15.62:0/2024608703 >> 172.21.15.62:0/2024608703 conn(0x7f0e80101b90 msgr2=0x7f0e80104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.247+0000 7f0e85000700 1 -- 172.21.15.62:0/2024608703 shutdown_connections 2024-05-27T07:10:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.247+0000 7f0e85000700 1 -- 172.21.15.62:0/2024608703 wait complete. 2024-05-27T07:10:51.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.248+0000 7f0e85000700 1 Processor -- start 2024-05-27T07:10:51.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.248+0000 7f0e85000700 1 -- start start 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e85000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80106530 0x7f0e8019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e85000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e80107660 0x7f0e8019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e85000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e80108000 0x7f0e801a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e85000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0e80111e50 con 0x7f0e80107660 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e85000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e80111cd0 con 0x7f0e80106530 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e85000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e80111fd0 con 0x7f0e80108000 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e7de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80106530 0x7f0e8019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e7de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80106530 0x7f0e8019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43552/0 (socket says 172.21.15.62:43552) 2024-05-27T07:10:51.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.249+0000 7f0e7de00700 1 -- 172.21.15.62:0/1703233950 learned_addr learned my addr 172.21.15.62:0/1703233950 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-05-27T07:10:51.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7d400700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e80107660 0x7f0e8019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7e800700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e80108000 0x7f0e801a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7de00700 1 -- 172.21.15.62:0/1703233950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e80108000 msgr2=0x7f0e801a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7de00700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e80108000 0x7f0e801a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7de00700 1 -- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e80107660 msgr2=0x7f0e8019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7de00700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e80107660 0x7f0e8019cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7de00700 1 -- 172.21.15.62:0/1703233950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e801a44d0 con 0x7f0e80106530 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7e800700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e80108000 0x7f0e801a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7d400700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e80107660 0x7f0e8019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:10:51.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.250+0000 7f0e7de00700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80106530 0x7f0e8019ca50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000d950 tx=0x7f0e7000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:51.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.251+0000 7f0e76a00700 1 -- 172.21.15.62:0/1703233950 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70017070 con 0x7f0e80106530 2024-05-27T07:10:51.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.251+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e801a47c0 con 0x7f0e80106530 2024-05-27T07:10:51.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.251+0000 7f0e76a00700 1 -- 172.21.15.62:0/1703233950 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e7000de30 con 0x7f0e80106530 2024-05-27T07:10:51.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.251+0000 7f0e76a00700 1 -- 172.21.15.62:0/1703233950 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70003aa0 con 0x7f0e80106530 2024-05-27T07:10:51.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.251+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e801a4d50 con 0x7f0e80106530 2024-05-27T07:10:51.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.252+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e800512b0 con 0x7f0e80106530 2024-05-27T07:10:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.255+0000 7f0e76a00700 1 -- 172.21.15.62:0/1703233950 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0e70003c40 con 0x7f0e80106530 2024-05-27T07:10:51.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.258+0000 7f0e76a00700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e640b2880 0x7f0e640b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.258+0000 7f0e76a00700 1 -- 172.21.15.62:0/1703233950 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7f0e700e1340 con 0x7f0e80106530 2024-05-27T07:10:51.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.258+0000 7f0e7d400700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e640b2880 0x7f0e640b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.259+0000 7f0e7d400700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e640b2880 0x7f0e640b4d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00b5c0 tx=0x7f0e6c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.260+0000 7f0e76a00700 1 -- 172.21.15.62:0/1703233950 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0e70020030 con 0x7f0e80106530 2024-05-27T07:10:51.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:10:51.514+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0e80063bf0 con 0x7f0e640b2880 2024-05-27T07:10:52.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:10:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:10:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:52.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:10:52.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:10:52.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:10:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:10:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:53.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:52 smithi062 ceph-mon[35304]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:53.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:52 smithi062 ceph-mon[35304]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:10:53.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:52 smithi012 ceph-mon[31473]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:52 smithi012 ceph-mon[31473]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:10:53.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:52 smithi162 ceph-mon[35676]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:53.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:52 smithi162 ceph-mon[35676]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:10:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:54 smithi012 ceph-mon[31473]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-27T07:10:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:54 smithi162 ceph-mon[35676]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-27T07:10:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:54 smithi062 ceph-mon[35304]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-27T07:10:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:56 smithi062 ceph-mon[35304]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:10:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:56 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/2469610489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]: dispatch 2024-05-27T07:10:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:56 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]: dispatch 2024-05-27T07:10:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:56 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]': finished 2024-05-27T07:10:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:56 smithi062 ceph-mon[35304]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T07:10:56.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:10:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:56 smithi012 ceph-mon[31473]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:10:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:56 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/2469610489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]: dispatch 2024-05-27T07:10:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:56 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]: dispatch 2024-05-27T07:10:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:56 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]': finished 2024-05-27T07:10:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:56 smithi012 ceph-mon[31473]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T07:10:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:10:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:56 smithi162 ceph-mon[35676]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:10:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:56 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/2469610489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]: dispatch 2024-05-27T07:10:57.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:56 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]: dispatch 2024-05-27T07:10:57.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:56 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50344032-dab9-44e9-a549-f57d219d7959"}]': finished 2024-05-27T07:10:57.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:56 smithi162 ceph-mon[35676]: osdmap e28: 5 total, 4 up, 5 in 2024-05-27T07:10:57.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:10:58.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:57 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/1980386335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:10:58.126 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:57 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/1980386335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:10:58.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:57 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/1980386335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:10:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:10:58 smithi012 ceph-mon[31473]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:10:59.162 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:10:58 smithi162 ceph-mon[35676]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:10:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:10:58 smithi062 ceph-mon[35304]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:11:00.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:00 smithi162 ceph-mon[35676]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:01.195 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:00 smithi062 ceph-mon[35304]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:01.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:00 smithi012 ceph-mon[31473]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:03.086 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:02 smithi062 ceph-mon[35304]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:03.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:02 smithi162 ceph-mon[35676]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:03.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:02 smithi012 ceph-mon[31473]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:04.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:04 smithi062 ceph-mon[35304]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:04 smithi162 ceph-mon[35676]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:05.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:04 smithi012 ceph-mon[31473]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:07.197 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:06 smithi062 ceph-mon[35304]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:07.197 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:11:07.197 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:06 smithi162 ceph-mon[35676]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:11:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:06 smithi012 ceph-mon[31473]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:11:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:08.176 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:07 smithi062 ceph-mon[35304]: Deploying daemon osd.4 on smithi062 2024-05-27T07:11:08.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:07 smithi162 ceph-mon[35676]: Deploying daemon osd.4 on smithi062 2024-05-27T07:11:08.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:07 smithi012 ceph-mon[31473]: Deploying daemon osd.4 on smithi062 2024-05-27T07:11:09.152 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:08 smithi062 ceph-mon[35304]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:08 smithi162 ceph-mon[35676]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:09.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:08 smithi012 ceph-mon[31473]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:10 smithi162 ceph-mon[35676]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:11.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:10 smithi062 ceph-mon[35304]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:11.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:10 smithi012 ceph-mon[31473]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:13.142 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:12 smithi012 ceph-mon[31473]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:12 smithi162 ceph-mon[35676]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:12 smithi062 ceph-mon[35304]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:15.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:14 smithi012 ceph-mon[31473]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:14 smithi162 ceph-mon[35676]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:14 smithi062 ceph-mon[35304]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:17.167 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:16 smithi062 ceph-mon[35304]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:17.167 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:17.167 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:17.167 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:16 smithi162 ceph-mon[35676]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:17.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:16 smithi012 ceph-mon[31473]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:17.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:17.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:17.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:19.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:18 smithi062 ceph-mon[35304]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:18 smithi162 ceph-mon[35676]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:19.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:18 smithi012 ceph-mon[31473]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:20.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:20.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:20.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:20.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: from='osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:11:20.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:20 smithi012 ceph-mon[31473]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:11:20.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: from='osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:11:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:20 smithi162 ceph-mon[35676]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: from='osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:11:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:20 smithi062 ceph-mon[35304]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:11:21.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.538+0000 7f0e76a00700 1 -- 172.21.15.62:0/1703233950 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0e80063bf0 con 0x7f0e640b2880 2024-05-27T07:11:21.530 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 4 on host 'smithi062' 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.542+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e640b2880 msgr2=0x7f0e640b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.542+0000 7f0e85000700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e640b2880 0x7f0e640b4d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00b5c0 tx=0x7f0e6c01a040 comp rx=0 tx=0).stop 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.542+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80106530 msgr2=0x7f0e8019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.542+0000 7f0e85000700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80106530 0x7f0e8019ca50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000d950 tx=0x7f0e7000dd10 comp rx=0 tx=0).stop 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 shutdown_connections 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e640b2880 0x7f0e640b4d90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e80106530 0x7f0e8019ca50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e80108000 0x7f0e801a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 --2- 172.21.15.62:0/1703233950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e80107660 0x7f0e8019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 >> 172.21.15.62:0/1703233950 conn(0x7f0e80101b90 msgr2=0x7f0e8010a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:21.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 shutdown_connections 2024-05-27T07:11:21.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:11:21.543+0000 7f0e85000700 1 -- 172.21.15.62:0/1703233950 wait complete. 2024-05-27T07:11:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:11:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:11:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: from='osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:11:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:11:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:21.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:21.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:11:21.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:11:21.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:21.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: from='osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:11:21.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:11:21.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:21.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:21.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: from='osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:22.500 DEBUG:teuthology.orchestra.run.smithi062:osd.4> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.4.service 2024-05-27T07:11:22.504 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-05-27T07:11:22.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:11:22.719 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:22 smithi062 ceph-mon[35304]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:22.719 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:22 smithi062 ceph-mon[35304]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:11:22.719 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:22 smithi062 ceph-mon[35304]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:11:22.719 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:22.719 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:22.719 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:22 smithi062 ceph-mon[35304]: from='osd.4 ' entity='osd.4' 2024-05-27T07:11:22.720 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:11:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:11:22.583+0000 7febed400700 -1 osd.4 0 waiting for initial osdmap 2024-05-27T07:11:22.720 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:11:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:11:22.606+0000 7febe3200700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:11:22.953 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:11:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:22 smithi162 ceph-mon[35676]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:22 smithi162 ceph-mon[35676]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:11:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:22 smithi162 ceph-mon[35676]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:11:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:22 smithi162 ceph-mon[35676]: from='osd.4 ' entity='osd.4' 2024-05-27T07:11:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:22 smithi012 ceph-mon[31473]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:22 smithi012 ceph-mon[31473]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:11:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:22 smithi012 ceph-mon[31473]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:11:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:22 smithi012 ceph-mon[31473]: from='osd.4 ' entity='osd.4' 2024-05-27T07:11:23.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:23 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:11:23.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:23 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:11:23.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:23 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:23.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:23 smithi162 ceph-mon[35676]: osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] boot 2024-05-27T07:11:23.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:23 smithi162 ceph-mon[35676]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T07:11:23.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:23 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:23 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:11:23.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:23 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:11:23.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:23 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:23.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:23 smithi062 ceph-mon[35304]: osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] boot 2024-05-27T07:11:23.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:23 smithi062 ceph-mon[35304]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T07:11:23.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:23 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:24.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:23 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:11:24.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:23 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:11:24.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:23 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:24.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:23 smithi012 ceph-mon[31473]: osd.4 [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] boot 2024-05-27T07:11:24.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:23 smithi012 ceph-mon[31473]: osdmap e31: 5 total, 5 up, 5 in 2024-05-27T07:11:24.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:23 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:11:24.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:24 smithi162 ceph-mon[35676]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:24.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:24 smithi012 ceph-mon[31473]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:24.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:24 smithi062 ceph-mon[35304]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:11:25.802 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:11:25.802 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:11:25.803 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-05-27T07:11:25.803 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-05-27T07:11:25.803 INFO:teuthology.orchestra.run.smithi162.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0369527 s, 284 MB/s 2024-05-27T07:11:25.803 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-05-27T07:11:25.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:25 smithi162 ceph-mon[35676]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:11:25.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:25 smithi062 ceph-mon[35304]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:11:26.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:25 smithi012 ceph-mon[31473]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:11:26.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-05-27T07:11:26.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:26 smithi162 ceph-mon[35676]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:26.979 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:11:26.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:26 smithi062 ceph-mon[35304]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:27.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:26 smithi012 ceph-mon[31473]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi062 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi062 to 8544M 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:28.712 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:28.712 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi062 2024-05-27T07:11:28.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:28.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:11:28.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:11:28.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi062 to 8544M 2024-05-27T07:11:28.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:28.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:28.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi062 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi062 to 8544M 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:28.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 -- 172.21.15.162:0/671818921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104b10 msgr2=0x7f179810b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 --2- 172.21.15.162:0/671818921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104b10 0x7f179810b3c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f179000b0a0 tx=0x7f179000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 -- 172.21.15.162:0/671818921 shutdown_connections 2024-05-27T07:11:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 --2- 172.21.15.162:0/671818921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1798103040 0x7f1798103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 --2- 172.21.15.162:0/671818921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104b10 0x7f179810b3c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 --2- 172.21.15.162:0/671818921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1798104170 0x7f1798104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 -- 172.21.15.162:0/671818921 >> 172.21.15.162:0/671818921 conn(0x7f17980fe680 msgr2=0x7f1798100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:29.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 -- 172.21.15.162:0/671818921 shutdown_connections 2024-05-27T07:11:29.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.057+0000 7f1797e00700 1 -- 172.21.15.162:0/671818921 wait complete. 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.058+0000 7f1797e00700 1 Processor -- start 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.058+0000 7f1797e00700 1 -- start start 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1797e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1798103040 0x7f179819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1797e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104170 0x7f179819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1797e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1798104b10 0x7f17981a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1797e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f179810e9b0 con 0x7f1798103040 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1797e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f179810e830 con 0x7f1798104b10 2024-05-27T07:11:29.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1797e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f179810eb30 con 0x7f1798104170 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1796000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104170 0x7f179819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1796000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104170 0x7f179819ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58204/0 (socket says 172.21.15.162:58204) 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.059+0000 7f1796000700 1 -- 172.21.15.162:0/1986088969 learned_addr learned my addr 172.21.15.162:0/1986088969 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1796a00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1798103040 0x7f179819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1797400700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1798104b10 0x7f17981a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1796000700 1 -- 172.21.15.162:0/1986088969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1798104b10 msgr2=0x7f17981a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1796000700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1798104b10 0x7f17981a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1796000700 1 -- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1798103040 msgr2=0x7f179819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1796000700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1798103040 0x7f179819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1796000700 1 -- 172.21.15.162:0/1986088969 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17981a2180 con 0x7f1798104170 2024-05-27T07:11:29.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1797400700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1798104b10 0x7f17981a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:11:29.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f1796000700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104170 0x7f179819ab60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f178400d950 tx=0x7f178400dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:29.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f178f400700 1 -- 172.21.15.162:0/1986088969 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784016070 con 0x7f1798104170 2024-05-27T07:11:29.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f178f400700 1 -- 172.21.15.162:0/1986088969 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1784010460 con 0x7f1798104170 2024-05-27T07:11:29.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.060+0000 7f178f400700 1 -- 172.21.15.162:0/1986088969 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178400f620 con 0x7f1798104170 2024-05-27T07:11:29.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.061+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17981a2470 con 0x7f1798104170 2024-05-27T07:11:29.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.061+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17981a2960 con 0x7f1798104170 2024-05-27T07:11:29.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.062+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17980512b0 con 0x7f1798104170 2024-05-27T07:11:29.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.065+0000 7f178f400700 1 -- 172.21.15.162:0/1986088969 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1784019070 con 0x7f1798104170 2024-05-27T07:11:29.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.067+0000 7f178f400700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f177c0b2880 0x7f177c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.067+0000 7f178f400700 1 -- 172.21.15.162:0/1986088969 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f17840e0e50 con 0x7f1798104170 2024-05-27T07:11:29.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.067+0000 7f1796a00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f177c0b2880 0x7f177c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.069+0000 7f1796a00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f177c0b2880 0x7f177c0b4d90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1788005950 tx=0x7f17880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:29.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.069+0000 7f178f400700 1 -- 172.21.15.162:0/1986088969 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f17840a9b40 con 0x7f1798104170 2024-05-27T07:11:29.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:29.311+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f17981a2c80 con 0x7f177c0b2880 2024-05-27T07:11:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:11:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:11:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:29.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:11:29.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:11:29.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:30.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:11:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:11:30.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:30.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:30 smithi162 ceph-mon[35676]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:11:30.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:30 smithi162 ceph-mon[35676]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:30.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:30 smithi062 ceph-mon[35304]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:11:30.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:30 smithi062 ceph-mon[35304]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:30 smithi012 ceph-mon[31473]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:11:31.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:30 smithi012 ceph-mon[31473]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:32.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:32 smithi162 ceph-mon[35676]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:32.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:32 smithi062 ceph-mon[35304]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:33.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:32 smithi012 ceph-mon[31473]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:33 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/3327168416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]: dispatch 2024-05-27T07:11:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:33 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]: dispatch 2024-05-27T07:11:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:33 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]': finished 2024-05-27T07:11:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:33 smithi162 ceph-mon[35676]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T07:11:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:33 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:33.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:33 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/3327168416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]: dispatch 2024-05-27T07:11:33.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:33 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]: dispatch 2024-05-27T07:11:33.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:33 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]': finished 2024-05-27T07:11:33.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:33 smithi062 ceph-mon[35304]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T07:11:33.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:33 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:34.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:33 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/3327168416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]: dispatch 2024-05-27T07:11:34.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:33 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]: dispatch 2024-05-27T07:11:34.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:33 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7e9391c-de97-406f-aa88-e12252972a6b"}]': finished 2024-05-27T07:11:34.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:33 smithi012 ceph-mon[31473]: osdmap e33: 6 total, 5 up, 6 in 2024-05-27T07:11:34.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:33 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:34.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:34 smithi162 ceph-mon[35676]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:34.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:34 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/3544614670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:11:34.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:34 smithi062 ceph-mon[35304]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:34.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:34 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/3544614670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:11:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:34 smithi012 ceph-mon[31473]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:34 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/3544614670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:11:36.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:36 smithi162 ceph-mon[35676]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:36.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:36 smithi062 ceph-mon[35304]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:37.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:36 smithi012 ceph-mon[31473]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:38.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:38 smithi162 ceph-mon[35676]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:38.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:38 smithi062 ceph-mon[35304]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:39.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:38 smithi012 ceph-mon[31473]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:40.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:40 smithi062 ceph-mon[35304]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:40.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:40 smithi162 ceph-mon[35676]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:41.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:40 smithi012 ceph-mon[31473]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:42 smithi162 ceph-mon[35676]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:42 smithi062 ceph-mon[35304]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:43.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:42 smithi012 ceph-mon[31473]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:43.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:11:43.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:43.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:43 smithi162 ceph-mon[35676]: Deploying daemon osd.5 on smithi162 2024-05-27T07:11:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:11:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:43 smithi062 ceph-mon[35304]: Deploying daemon osd.5 on smithi162 2024-05-27T07:11:43.992 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:11:43.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:43.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:43 smithi012 ceph-mon[31473]: Deploying daemon osd.5 on smithi162 2024-05-27T07:11:44.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:44 smithi162 ceph-mon[35676]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:44 smithi062 ceph-mon[35304]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:45.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:44 smithi012 ceph-mon[31473]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:46 smithi162 ceph-mon[35676]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:46.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:46 smithi062 ceph-mon[35304]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:47.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:46 smithi012 ceph-mon[31473]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:48 smithi162 ceph-mon[35676]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:48.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:48 smithi062 ceph-mon[35304]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:48 smithi012 ceph-mon[31473]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:50.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:50 smithi162 ceph-mon[35676]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:50.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:50 smithi062 ceph-mon[35304]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:51.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:50 smithi012 ceph-mon[31473]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:52.952 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:52 smithi162 ceph-mon[35676]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:52.952 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:52.952 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:52.953 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:52 smithi062 ceph-mon[35304]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:52.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:53.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:52 smithi012 ceph-mon[31473]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:54.777 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:54 smithi162 ceph-mon[35676]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:54.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:54 smithi062 ceph-mon[35304]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:54 smithi012 ceph-mon[31473]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: from='osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:11:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:55 smithi162 ceph-mon[35676]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:11:56.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:56.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: from='osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:11:56.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:55 smithi012 ceph-mon[31473]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:11:56.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:56.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:56.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:56.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:56.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: from='osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:11:56.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:55 smithi062 ceph-mon[35304]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:11:57.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:57 smithi162 ceph-mon[35676]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:11:57.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:57 smithi162 ceph-mon[35676]: from='osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:11:57.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:57 smithi162 ceph-mon[35676]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:11:57.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:57 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:57.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:57 smithi162 ceph-mon[35676]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:11:57.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.293+0000 7f178f400700 1 -- 172.21.15.162:0/1986088969 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f17981a2c80 con 0x7f177c0b2880 2024-05-27T07:11:57.284 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-05-27T07:11:57.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f177c0b2880 msgr2=0x7f177c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:57.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f177c0b2880 0x7f177c0b4d90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1788005950 tx=0x7f17880058e0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104170 msgr2=0x7f179819ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:57.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104170 0x7f179819ab60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f178400d950 tx=0x7f178400dd10 comp rx=0 tx=0).stop 2024-05-27T07:11:57.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 shutdown_connections 2024-05-27T07:11:57.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f177c0b2880 0x7f177c0b4d90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1798104b10 0x7f17981a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1798103040 0x7f179819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 --2- 172.21.15.162:0/1986088969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1798104170 0x7f179819ab60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 >> 172.21.15.162:0/1986088969 conn(0x7f17980fe680 msgr2=0x7f17980ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:57.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 shutdown_connections 2024-05-27T07:11:57.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:11:57.297+0000 7f1797e00700 1 -- 172.21.15.162:0/1986088969 wait complete. 2024-05-27T07:11:57.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:57 smithi012 ceph-mon[31473]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:11:57.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:57 smithi012 ceph-mon[31473]: from='osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:11:57.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:57 smithi012 ceph-mon[31473]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:11:57.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:57 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:57.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:57 smithi012 ceph-mon[31473]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:11:57.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:57 smithi062 ceph-mon[35304]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:11:57.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:57 smithi062 ceph-mon[35304]: from='osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:11:57.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:57 smithi062 ceph-mon[35304]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:11:57.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:57 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:57.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:57 smithi062 ceph-mon[35304]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:11:58.108 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.5.service 2024-05-27T07:11:58.112 INFO:tasks.cephadm:Deploying osd.6 on smithi012 with /dev/vg_nvme/lv_2... 2024-05-27T07:11:58.112 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] boot 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:11:58.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] boot 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T07:11:58.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: osd.5 [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] boot 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: osdmap e36: 6 total, 6 up, 6 in 2024-05-27T07:11:58.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:11:58.589 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:59.447 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:59 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:11:59.447 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:59 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:11:59.447 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:11:59 smithi162 ceph-mon[35676]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:11:59.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:59 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:11:59.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:59 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:11:59.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:11:59 smithi012 ceph-mon[31473]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:11:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:59 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:11:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:59 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:11:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:11:59 smithi062 ceph-mon[35304]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:12:00.547 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:00 smithi012 ceph-mon[31473]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:00.547 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:00 smithi012 ceph-mon[31473]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:12:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:00 smithi162 ceph-mon[35676]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:00.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:00 smithi162 ceph-mon[35676]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:12:00.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:00 smithi062 ceph-mon[35304]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:00.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:00 smithi062 ceph-mon[35304]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:12:01.609 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:12:01.609 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:12:01.609 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-05-27T07:12:01.609 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-05-27T07:12:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0336408 s, 312 MB/s 2024-05-27T07:12:01.610 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-05-27T07:12:02.671 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_2 2024-05-27T07:12:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: pgmap v186: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:02.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:02.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:02.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: pgmap v186: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:02.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:02.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:02.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:02.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:02.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:03.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: pgmap v186: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:03.098 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:03.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:03 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi162 2024-05-27T07:12:03.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:03 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi162 to 8544M 2024-05-27T07:12:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:03 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi162 2024-05-27T07:12:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:03 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi162 to 8544M 2024-05-27T07:12:03.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:03 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi162 2024-05-27T07:12:03.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:03 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi162 to 8544M 2024-05-27T07:12:04.944 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:04 smithi012 ceph-mon[31473]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:12:04.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:04 smithi162 ceph-mon[35676]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:12:04.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:04 smithi062 ceph-mon[35304]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:12:05.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 -- 172.21.15.12:0/978823720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 msgr2=0x7fc10c109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 --2- 172.21.15.12:0/978823720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c109760 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009990 tx=0x7fc0f8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 -- 172.21.15.12:0/978823720 shutdown_connections 2024-05-27T07:12:05.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 --2- 172.21.15.12:0/978823720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc10c06c260 0x7fc10c106d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 --2- 172.21.15.12:0/978823720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c109df0 0x7fc10c10c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 --2- 172.21.15.12:0/978823720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c109760 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 -- 172.21.15.12:0/978823720 >> 172.21.15.12:0/978823720 conn(0x7fc10c0fe680 msgr2=0x7fc10c100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:05.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 -- 172.21.15.12:0/978823720 shutdown_connections 2024-05-27T07:12:05.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.353+0000 7fc111e00700 1 -- 172.21.15.12:0/978823720 wait complete. 2024-05-27T07:12:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.354+0000 7fc111e00700 1 Processor -- start 2024-05-27T07:12:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc111e00700 1 -- start start 2024-05-27T07:12:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc111e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c06c260 0x7fc10c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc111e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc111e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc10c109df0 0x7fc10c1a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc111e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c10fab0 con 0x7fc10c1072e0 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc111e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c10f930 con 0x7fc10c109df0 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc111e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c10fc30 con 0x7fc10c06c260 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc10be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc10be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c19ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53468/0 (socket says 172.21.15.12:53468) 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.355+0000 7fc10be00700 1 -- 172.21.15.12:0/23799374 learned_addr learned my addr 172.21.15.12:0/23799374 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc110a00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c06c260 0x7fc10c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc111400700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc10c109df0 0x7fc10c1a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc10be00700 1 -- 172.21.15.12:0/23799374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c06c260 msgr2=0x7fc10c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc10be00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c06c260 0x7fc10c19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc10be00700 1 -- 172.21.15.12:0/23799374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc10c109df0 msgr2=0x7fc10c1a3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc10be00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc10c109df0 0x7fc10c1a3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc10be00700 1 -- 172.21.15.12:0/23799374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c1a4430 con 0x7fc10c1072e0 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc111400700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc10c109df0 0x7fc10c1a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.356+0000 7fc10be00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c19ce10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8000c00 tx=0x7fc0f80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.357+0000 7fc109600700 1 -- 172.21.15.12:0/23799374 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8004830 con 0x7fc10c1072e0 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.357+0000 7fc109600700 1 -- 172.21.15.12:0/23799374 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0f8003b90 con 0x7fc10c1072e0 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.357+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10c1a46c0 con 0x7fc10c1072e0 2024-05-27T07:12:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.357+0000 7fc109600700 1 -- 172.21.15.12:0/23799374 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8005280 con 0x7fc10c1072e0 2024-05-27T07:12:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.357+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10c1a4b10 con 0x7fc10c1072e0 2024-05-27T07:12:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.358+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10c0512b0 con 0x7fc10c1072e0 2024-05-27T07:12:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.361+0000 7fc109600700 1 -- 172.21.15.12:0/23799374 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc0f8005420 con 0x7fc10c1072e0 2024-05-27T07:12:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.362+0000 7fc109600700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0f40b2900 0x7fc0f40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.363+0000 7fc109600700 1 -- 172.21.15.12:0/23799374 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7fc0f80e27b0 con 0x7fc10c1072e0 2024-05-27T07:12:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.363+0000 7fc110a00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0f40b2900 0x7fc0f40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.364+0000 7fc110a00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0f40b2900 0x7fc0f40b4e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc100005950 tx=0x7fc10000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.365+0000 7fc109600700 1 -- 172.21.15.12:0/23799374 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc0f8013070 con 0x7fc10c1072e0 2024-05-27T07:12:05.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:05.614+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc10c10a9b0 con 0x7fc0f40b2900 2024-05-27T07:12:05.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:05 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:12:05.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:05 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:12:05.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:05 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:05.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:05 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:12:05.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:05 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:12:05.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:05 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:06.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:05 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:12:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:05 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:12:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:05 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:06.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:06 smithi012 ceph-mon[31473]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:12:06.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:06 smithi012 ceph-mon[31473]: from='client.14373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:06.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:06 smithi162 ceph-mon[35676]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:12:06.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:06 smithi162 ceph-mon[35676]: from='client.14373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:06 smithi062 ceph-mon[35304]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:12:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:06 smithi062 ceph-mon[35304]: from='client.14373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:08.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:08 smithi012 ceph-mon[31473]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:12:08.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:08 smithi162 ceph-mon[35676]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:12:08.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:08 smithi062 ceph-mon[35304]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:12:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:10 smithi162 ceph-mon[35676]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:12:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:10 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1371272448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9511154-fa9d-432a-b0be-c67dacf0d0f9"}]: dispatch 2024-05-27T07:12:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:10 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1371272448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9511154-fa9d-432a-b0be-c67dacf0d0f9"}]': finished 2024-05-27T07:12:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:10 smithi162 ceph-mon[35676]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T07:12:10.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:10 smithi062 ceph-mon[35304]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:12:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:10 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1371272448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9511154-fa9d-432a-b0be-c67dacf0d0f9"}]: dispatch 2024-05-27T07:12:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:10 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1371272448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9511154-fa9d-432a-b0be-c67dacf0d0f9"}]': finished 2024-05-27T07:12:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:10 smithi062 ceph-mon[35304]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T07:12:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:10 smithi012 ceph-mon[31473]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:12:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:10 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1371272448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9511154-fa9d-432a-b0be-c67dacf0d0f9"}]: dispatch 2024-05-27T07:12:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:10 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1371272448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9511154-fa9d-432a-b0be-c67dacf0d0f9"}]': finished 2024-05-27T07:12:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:10 smithi012 ceph-mon[31473]: osdmap e39: 7 total, 6 up, 7 in 2024-05-27T07:12:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:11.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:11 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3511780794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:12:11.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:11 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3511780794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:12:12.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:11 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3511780794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:12:12.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:12 smithi162 ceph-mon[35676]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:12:12.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:12 smithi062 ceph-mon[35304]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:12:13.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:12 smithi012 ceph-mon[31473]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:12:14.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:14 smithi162 ceph-mon[35676]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:14.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:14 smithi062 ceph-mon[35304]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:15.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:14 smithi012 ceph-mon[31473]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:16.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:16 smithi062 ceph-mon[35304]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:16.998 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:16 smithi012 ceph-mon[31473]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:16 smithi162 ceph-mon[35676]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:18.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:18 smithi012 ceph-mon[31473]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:18.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:18 smithi062 ceph-mon[35304]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:18 smithi162 ceph-mon[35676]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:20 smithi012 ceph-mon[31473]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:12:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:20 smithi162 ceph-mon[35676]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:12:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:20 smithi062 ceph-mon[35304]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:12:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:21.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:21 smithi012 ceph-mon[31473]: Deploying daemon osd.6 on smithi012 2024-05-27T07:12:22.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:21 smithi162 ceph-mon[35676]: Deploying daemon osd.6 on smithi012 2024-05-27T07:12:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:21 smithi062 ceph-mon[35304]: Deploying daemon osd.6 on smithi012 2024-05-27T07:12:23.062 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:22 smithi012 ceph-mon[31473]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:22 smithi162 ceph-mon[35676]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:22 smithi062 ceph-mon[35304]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:25.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:24 smithi012 ceph-mon[31473]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:25.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:24 smithi162 ceph-mon[35676]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:25.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:24 smithi062 ceph-mon[35304]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:26.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:26 smithi012 ceph-mon[31473]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:26 smithi162 ceph-mon[35676]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:26 smithi062 ceph-mon[35304]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:29.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:28 smithi162 ceph-mon[35676]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:29.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:28 smithi012 ceph-mon[31473]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:28 smithi062 ceph-mon[35304]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:30.882 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:30 smithi012 ceph-mon[31473]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:30.883 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:30.883 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:30.883 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:30.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:30 smithi162 ceph-mon[35676]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:30.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:30.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:30.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:30 smithi062 ceph-mon[35304]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:30.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:30.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:30.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:32.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:32 smithi162 ceph-mon[35676]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:32.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:32 smithi062 ceph-mon[35304]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:32.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:32 smithi012 ceph-mon[31473]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:34.712 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.712 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.712 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:34.712 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:34.712 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.712 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:34 smithi012 ceph-mon[31473]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:34.712 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:34 smithi012 ceph-mon[31473]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:12:34.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:34.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:34.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:34 smithi162 ceph-mon[35676]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:34.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:34 smithi162 ceph-mon[35676]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:12:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:34.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:34.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:34 smithi062 ceph-mon[35304]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:34.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:34 smithi062 ceph-mon[35304]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:12:35.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.722+0000 7fc109600700 1 -- 172.21.15.12:0/23799374 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc10c10a9b0 con 0x7fc0f40b2900 2024-05-27T07:12:35.714 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 6 on host 'smithi012' 2024-05-27T07:12:35.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:35 smithi012 ceph-mon[31473]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:12:35.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:35 smithi012 ceph-mon[31473]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:12:35.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:35 smithi012 ceph-mon[31473]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:12:35.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:35 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:35.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.726+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0f40b2900 msgr2=0x7fc0f40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:35.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.726+0000 7fc111e00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0f40b2900 0x7fc0f40b4e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc100005950 tx=0x7fc10000b410 comp rx=0 tx=0).stop 2024-05-27T07:12:35.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.726+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 msgr2=0x7fc10c19ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:35.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.726+0000 7fc111e00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c19ce10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8000c00 tx=0x7fc0f80047b0 comp rx=0 tx=0).stop 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 shutdown_connections 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0f40b2900 0x7fc0f40b4e10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc10c109df0 0x7fc10c1a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c06c260 0x7fc10c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 --2- 172.21.15.12:0/23799374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc10c1072e0 0x7fc10c19ce10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 >> 172.21.15.12:0/23799374 conn(0x7fc10c0fe680 msgr2=0x7fc10c100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 shutdown_connections 2024-05-27T07:12:35.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:12:35.727+0000 7fc111e00700 1 -- 172.21.15.12:0/23799374 wait complete. 2024-05-27T07:12:35.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:35 smithi162 ceph-mon[35676]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:12:35.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:35 smithi162 ceph-mon[35676]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:12:35.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:35 smithi162 ceph-mon[35676]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:12:35.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:35 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:35.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:35 smithi062 ceph-mon[35304]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:12:35.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:35 smithi062 ceph-mon[35304]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:12:35.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:35 smithi062 ceph-mon[35304]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:12:35.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:35 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:36.431 DEBUG:teuthology.orchestra.run.smithi012:osd.6> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.6.service 2024-05-27T07:12:36.436 INFO:tasks.cephadm:Deploying osd.7 on smithi062 with /dev/vg_nvme/lv_2... 2024-05-27T07:12:36.436 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:36.663 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:36.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:12:36.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:12:36.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:36.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:36.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:36.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:36.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:36.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: from='osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:12:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:12:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:36.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:36.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:12:36.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:36.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:36.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:36.909 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:12:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:37 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:12:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:37 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:12:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:37 smithi162 ceph-mon[35676]: osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] boot 2024-05-27T07:12:37.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:37 smithi162 ceph-mon[35676]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T07:12:37.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:37.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:37 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:12:37.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:37 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:12:37.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:37.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:37 smithi062 ceph-mon[35304]: osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] boot 2024-05-27T07:12:37.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:37 smithi062 ceph-mon[35304]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T07:12:37.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:37.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:37 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:12:37.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:37 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:12:37.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:37.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:37 smithi012 ceph-mon[31473]: osd.6 [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] boot 2024-05-27T07:12:37.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:37 smithi012 ceph-mon[31473]: osdmap e42: 7 total, 7 up, 7 in 2024-05-27T07:12:37.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:12:38.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:38 smithi012 ceph-mon[31473]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:12:38.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:38 smithi012 ceph-mon[31473]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:38.822 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:38 smithi062 ceph-mon[35304]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:12:38.823 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:38 smithi062 ceph-mon[35304]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:38.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:38 smithi162 ceph-mon[35676]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:12:38.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:38 smithi162 ceph-mon[35676]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:40.050 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:12:40.050 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:12:40.050 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-05-27T07:12:40.050 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-05-27T07:12:40.050 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0364339 s, 288 MB/s 2024-05-27T07:12:40.050 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-05-27T07:12:40.821 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_2 2024-05-27T07:12:40.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:40 smithi062 ceph-mon[35304]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:40.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:40 smithi162 ceph-mon[35676]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:41.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:40 smithi012 ceph-mon[31473]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:41.252 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:12:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi012 2024-05-27T07:12:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi012 to 5698M 2024-05-27T07:12:42.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:42.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:42.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:42 smithi012 ceph-mon[31473]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi012 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi012 to 5698M 2024-05-27T07:12:42.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.967 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:42.967 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:42.967 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.967 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:42 smithi062 ceph-mon[35304]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi012 2024-05-27T07:12:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi012 to 5698M 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:12:42.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:42 smithi162 ceph-mon[35676]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 -- 172.21.15.62:0/2581091319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0776d0 msgr2=0x7fe7dc075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/2581091319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0776d0 0x7fe7dc075ba0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009990 tx=0x7fe7c4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 -- 172.21.15.62:0/2581091319 shutdown_connections 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/2581091319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0776d0 0x7fe7dc075ba0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/2581091319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7dc110980 0x7fe7dc112da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/2581091319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7dc0760e0 0x7fe7dc076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 -- 172.21.15.62:0/2581091319 >> 172.21.15.62:0/2581091319 conn(0x7fe7dc0ff9b0 msgr2=0x7fe7dc101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 -- 172.21.15.62:0/2581091319 shutdown_connections 2024-05-27T07:12:43.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.621+0000 7fe7e3c00700 1 -- 172.21.15.62:0/2581091319 wait complete. 2024-05-27T07:12:43.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.622+0000 7fe7e3c00700 1 Processor -- start 2024-05-27T07:12:43.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e3c00700 1 -- start start 2024-05-27T07:12:43.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e3c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0760e0 0x7fe7dc1a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e3c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7dc0776d0 0x7fe7dc1a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e3c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7dc110980 0x7fe7dc1a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e3c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc116220 con 0x7fe7dc110980 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e3c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc1160a0 con 0x7fe7dc0760e0 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e3c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc1163a0 con 0x7fe7dc0776d0 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0760e0 0x7fe7dc1a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0760e0 0x7fe7dc1a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38266/0 (socket says 172.21.15.62:38266) 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.623+0000 7fe7e1200700 1 -- 172.21.15.62:0/850703634 learned_addr learned my addr 172.21.15.62:0/850703634 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7d3e00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7dc0776d0 0x7fe7dc1a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7e1c00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7dc110980 0x7fe7dc1a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:43.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7e1200700 1 -- 172.21.15.62:0/850703634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7dc0776d0 msgr2=0x7fe7dc1a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7e1200700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7dc0776d0 0x7fe7dc1a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7e1200700 1 -- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7dc110980 msgr2=0x7fe7dc1a82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7e1200700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7dc110980 0x7fe7dc1a82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7e1200700 1 -- 172.21.15.62:0/850703634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc1a8960 con 0x7fe7dc0760e0 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.624+0000 7fe7e1200700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0760e0 0x7fe7dc1a0e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40098e0 tx=0x7fe7c40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.625+0000 7fe7da000700 1 -- 172.21.15.62:0/850703634 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4003c10 con 0x7fe7dc0760e0 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.625+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7dc1a8bf0 con 0x7fe7dc0760e0 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.625+0000 7fe7da000700 1 -- 172.21.15.62:0/850703634 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7c4004830 con 0x7fe7dc0760e0 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.625+0000 7fe7da000700 1 -- 172.21.15.62:0/850703634 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4005260 con 0x7fe7dc0760e0 2024-05-27T07:12:43.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.625+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7dc1a8fe0 con 0x7fe7dc0760e0 2024-05-27T07:12:43.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.626+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7dc0512b0 con 0x7fe7dc0760e0 2024-05-27T07:12:43.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.629+0000 7fe7da000700 1 -- 172.21.15.62:0/850703634 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe7c4005420 con 0x7fe7dc0760e0 2024-05-27T07:12:43.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.631+0000 7fe7da000700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe7bc0b2880 0x7fe7bc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:43.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.632+0000 7fe7da000700 1 -- 172.21.15.62:0/850703634 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7fe7c40e2cc0 con 0x7fe7dc0760e0 2024-05-27T07:12:43.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.632+0000 7fe7d3e00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe7bc0b2880 0x7fe7bc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:43.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.633+0000 7fe7d3e00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe7bc0b2880 0x7fe7bc0b4d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005950 tx=0x7fe7cc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.635+0000 7fe7da000700 1 -- 172.21.15.62:0/850703634 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe7c4030020 con 0x7fe7dc0760e0 2024-05-27T07:12:43.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:12:43.881+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe7dc063cb0 con 0x7fe7bc0b2880 2024-05-27T07:12:44.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:44 smithi162 ceph-mon[35676]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:44.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:44 smithi162 ceph-mon[35676]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:44.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:12:44.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:12:44.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:44 smithi062 ceph-mon[35304]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:44 smithi062 ceph-mon[35304]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:12:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:12:44.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:44 smithi012 ceph-mon[31473]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:44 smithi012 ceph-mon[31473]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:12:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:12:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:46 smithi162 ceph-mon[35676]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:46.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:46 smithi062 ceph-mon[35304]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:46 smithi012 ceph-mon[31473]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:48 smithi162 ceph-mon[35676]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:48.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:48 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/842772026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]: dispatch 2024-05-27T07:12:48.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:48 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]: dispatch 2024-05-27T07:12:48.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:48 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]': finished 2024-05-27T07:12:48.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:48 smithi162 ceph-mon[35676]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T07:12:48.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:12:48.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:48 smithi062 ceph-mon[35304]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:48.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:48 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/842772026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]: dispatch 2024-05-27T07:12:48.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:48 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]: dispatch 2024-05-27T07:12:48.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:48 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]': finished 2024-05-27T07:12:48.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:48 smithi062 ceph-mon[35304]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T07:12:48.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:12:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:48 smithi012 ceph-mon[31473]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:48 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/842772026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]: dispatch 2024-05-27T07:12:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:48 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]: dispatch 2024-05-27T07:12:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:48 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff91153a-4b0f-4275-96a2-6bde3d6f192c"}]': finished 2024-05-27T07:12:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:48 smithi012 ceph-mon[31473]: osdmap e44: 8 total, 7 up, 8 in 2024-05-27T07:12:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:12:49.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:49 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/1497586183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:12:49.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:49 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/1497586183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:12:50.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:49 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/1497586183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:12:50.966 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:50 smithi062 ceph-mon[35304]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:50.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:50 smithi162 ceph-mon[35676]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:51.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:50 smithi012 ceph-mon[31473]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:52 smithi162 ceph-mon[35676]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:52 smithi062 ceph-mon[35304]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:53.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:52 smithi012 ceph-mon[31473]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:54.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:54 smithi162 ceph-mon[35676]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:54.978 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:54 smithi062 ceph-mon[35304]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:54 smithi012 ceph-mon[31473]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:56.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:56 smithi162 ceph-mon[35676]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:56 smithi062 ceph-mon[35304]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:57.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:56 smithi012 ceph-mon[31473]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:58 smithi062 ceph-mon[35304]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:12:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:58 smithi012 ceph-mon[31473]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:12:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:59.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:58 smithi162 ceph-mon[35676]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:12:59.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:12:59.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:59.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:12:59 smithi062 ceph-mon[35304]: Deploying daemon osd.7 on smithi062 2024-05-27T07:13:00.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:12:59 smithi012 ceph-mon[31473]: Deploying daemon osd.7 on smithi062 2024-05-27T07:13:00.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:12:59 smithi162 ceph-mon[35676]: Deploying daemon osd.7 on smithi062 2024-05-27T07:13:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:00 smithi162 ceph-mon[35676]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:00.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:00 smithi062 ceph-mon[35304]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:01.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:00 smithi012 ceph-mon[31473]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:02.961 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:02 smithi062 ceph-mon[35304]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:03.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:02 smithi012 ceph-mon[31473]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:03.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:02 smithi162 ceph-mon[35676]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:04 smithi012 ceph-mon[31473]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:04 smithi162 ceph-mon[35676]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:05.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:04 smithi062 ceph-mon[35304]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:06.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:06 smithi062 ceph-mon[35304]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:07.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:06 smithi012 ceph-mon[31473]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:06 smithi162 ceph-mon[35676]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:08 smithi162 ceph-mon[35676]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:09.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:08 smithi062 ceph-mon[35304]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:09.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:09.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:09.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:09.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:08 smithi012 ceph-mon[31473]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:10 smithi162 ceph-mon[35676]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:11.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:10 smithi062 ceph-mon[35304]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:11.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:10 smithi012 ceph-mon[31473]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:13.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:12 smithi012 ceph-mon[31473]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:12 smithi012 ceph-mon[31473]: from='osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:13:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:12 smithi012 ceph-mon[31473]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:13:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:12 smithi162 ceph-mon[35676]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:12 smithi162 ceph-mon[35676]: from='osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:13:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:12 smithi162 ceph-mon[35676]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:13:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:12 smithi062 ceph-mon[35304]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:12 smithi062 ceph-mon[35304]: from='osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:13:13.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:12 smithi062 ceph-mon[35304]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:13:13.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.768+0000 7fe7da000700 1 -- 172.21.15.62:0/850703634 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe7dc063cb0 con 0x7fe7bc0b2880 2024-05-27T07:13:13.759 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 7 on host 'smithi062' 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe7bc0b2880 msgr2=0x7fe7bc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe7bc0b2880 0x7fe7bc0b4d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005950 tx=0x7fe7cc0058e0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0760e0 msgr2=0x7fe7dc1a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0760e0 0x7fe7dc1a0e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40098e0 tx=0x7fe7c40047b0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 shutdown_connections 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe7bc0b2880 0x7fe7bc0b4d90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7dc0760e0 0x7fe7dc1a0e00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7dc110980 0x7fe7dc1a82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 --2- 172.21.15.62:0/850703634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7dc0776d0 0x7fe7dc1a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 >> 172.21.15.62:0/850703634 conn(0x7fe7dc0ff9b0 msgr2=0x7fe7dc103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:13.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 shutdown_connections 2024-05-27T07:13:13.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:13:13.772+0000 7fe7e3c00700 1 -- 172.21.15.62:0/850703634 wait complete. 2024-05-27T07:13:14.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:13:14.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:13:14.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:14.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:13:14.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:13:14.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:14.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:14.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:13 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:14.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: from='osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:13 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:14.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:13:14.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:13:14.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:14.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:13:14.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:13:14.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:14.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:14.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:14.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:13 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:14.587 DEBUG:teuthology.orchestra.run.smithi062:osd.7> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.7.service 2024-05-27T07:13:14.593 INFO:tasks.cephadm:Deploying osd.8 on smithi162 with /dev/vg_nvme/lv_2... 2024-05-27T07:13:14.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:13:15.040 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:13:15.090 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:15 smithi062 ceph-mon[35304]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:15.090 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:15 smithi062 ceph-mon[35304]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:13:15.090 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:15 smithi062 ceph-mon[35304]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:13:15.090 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:15 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:15.090 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:15 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:15.090 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:15 smithi062 ceph-mon[35304]: from='osd.7 ' entity='osd.7' 2024-05-27T07:13:15.091 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:13:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:13:14.851+0000 7f18a9200700 -1 osd.7 0 waiting for initial osdmap 2024-05-27T07:13:15.091 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:13:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:13:14.887+0000 7f189e600700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:13:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:15 smithi012 ceph-mon[31473]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:15 smithi012 ceph-mon[31473]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:13:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:15 smithi012 ceph-mon[31473]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:13:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:15 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:15 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:15 smithi012 ceph-mon[31473]: from='osd.7 ' entity='osd.7' 2024-05-27T07:13:15.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:15 smithi162 ceph-mon[35676]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:15.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:15 smithi162 ceph-mon[35676]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:13:15.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:15 smithi162 ceph-mon[35676]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:13:15.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:15 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:15.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:15 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:15.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:15 smithi162 ceph-mon[35676]: from='osd.7 ' entity='osd.7' 2024-05-27T07:13:16.209 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:16 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:13:16.209 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:16 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:13:16.209 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:16.209 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:16 smithi062 ceph-mon[35304]: osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] boot 2024-05-27T07:13:16.209 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:16 smithi062 ceph-mon[35304]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T07:13:16.209 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:16.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:16 smithi062 ceph-mon[35304]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:16 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:16 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:16 smithi012 ceph-mon[31473]: osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] boot 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:16 smithi012 ceph-mon[31473]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:16 smithi012 ceph-mon[31473]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:16.452 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:16 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:13:16.452 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:16 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:13:16.453 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:16.453 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:16 smithi162 ceph-mon[35676]: osd.7 [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] boot 2024-05-27T07:13:16.453 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:16 smithi162 ceph-mon[35676]: osdmap e47: 8 total, 8 up, 8 in 2024-05-27T07:13:16.453 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:16.453 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:16 smithi162 ceph-mon[35676]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:13:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:17 smithi062 ceph-mon[35304]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:13:17.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:17 smithi012 ceph-mon[31473]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:13:17.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:17 smithi162 ceph-mon[35676]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:13:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:13:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:13:18.038 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-05-27T07:13:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-05-27T07:13:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0347587 s, 302 MB/s 2024-05-27T07:13:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-05-27T07:13:18.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:18 smithi012 ceph-mon[31473]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:13:18.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:18 smithi012 ceph-mon[31473]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:18.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:18 smithi162 ceph-mon[35676]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:13:18.450 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:18 smithi162 ceph-mon[35676]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:18.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:18 smithi062 ceph-mon[35304]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:13:18.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:18 smithi062 ceph-mon[35304]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:18.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-05-27T07:13:19.395 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:13:20.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:20.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:20 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:20.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:20.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:20.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:20 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:21.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:20 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 -- 172.21.15.162:0/1968063734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 msgr2=0x7fbfd010af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1968063734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd010af10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009990 tx=0x7fbfc0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 -- 172.21.15.162:0/1968063734 shutdown_connections 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1968063734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfd010c410 0x7fbfd0112cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1968063734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd010af10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1968063734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd010ba70 0x7fbfd010be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 -- 172.21.15.162:0/1968063734 >> 172.21.15.162:0/1968063734 conn(0x7fbfd00799b0 msgr2=0x7fbfd007be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 -- 172.21.15.162:0/1968063734 shutdown_connections 2024-05-27T07:13:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.770+0000 7fbfd6600700 1 -- 172.21.15.162:0/1968063734 wait complete. 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.771+0000 7fbfd6600700 1 Processor -- start 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.771+0000 7fbfd6600700 1 -- start start 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfd6600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd01a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfd6600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd010ba70 0x7fbfd01a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfd6600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfd010c410 0x7fbfd01a8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfd6600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0116280 con 0x7fbfd010ba70 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfd6600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0116100 con 0x7fbfd010c410 2024-05-27T07:13:21.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfd6600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0116400 con 0x7fbfd010ab00 2024-05-27T07:13:21.763 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:21 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi062 2024-05-27T07:13:21.763 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:21 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi062 to 5696M 2024-05-27T07:13:21.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfcf400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd01a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfcf400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd01a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39164/0 (socket says 172.21.15.162:39164) 2024-05-27T07:13:21.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfcf400700 1 -- 172.21.15.162:0/1063756757 learned_addr learned my addr 172.21.15.162:0/1063756757 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:13:21.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfcfe00700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfd010c410 0x7fbfd01a8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.772+0000 7fbfc6a00700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd010ba70 0x7fbfd01a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfcf400700 1 -- 172.21.15.162:0/1063756757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfd010c410 msgr2=0x7fbfd01a8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:21.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfcf400700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfd010c410 0x7fbfd01a8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfcf400700 1 -- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd010ba70 msgr2=0x7fbfd01a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfcf400700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd010ba70 0x7fbfd01a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfcf400700 1 -- 172.21.15.162:0/1063756757 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd01a8920 con 0x7fbfd010ab00 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfcf400700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd01a0e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0006010 tx=0x7fbfc00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfccc00700 1 -- 172.21.15.162:0/1063756757 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0026070 con 0x7fbfd010ab00 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfccc00700 1 -- 172.21.15.162:0/1063756757 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbfc0004020 con 0x7fbfd010ab00 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.773+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd01a8bb0 con 0x7fbfd010ab00 2024-05-27T07:13:21.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.774+0000 7fbfccc00700 1 -- 172.21.15.162:0/1063756757 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc000fc00 con 0x7fbfd010ab00 2024-05-27T07:13:21.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.774+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd01a9060 con 0x7fbfd010ab00 2024-05-27T07:13:21.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.775+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd00512b0 con 0x7fbfd010ab00 2024-05-27T07:13:21.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.778+0000 7fbfccc00700 1 -- 172.21.15.162:0/1063756757 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fbfc0029070 con 0x7fbfd010ab00 2024-05-27T07:13:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.779+0000 7fbfccc00700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbfb40b2900 0x7fbfb40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.780+0000 7fbfccc00700 1 -- 172.21.15.162:0/1063756757 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5397+0+0 (secure 0 0 0) 0x7fbfc00e29e0 con 0x7fbfd010ab00 2024-05-27T07:13:21.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.780+0000 7fbfc6a00700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbfb40b2900 0x7fbfb40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.781+0000 7fbfc6a00700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbfb40b2900 0x7fbfb40b4e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbfd01a1f00 tx=0x7fbfb8009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:21.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:21.783+0000 7fbfccc00700 1 -- 172.21.15.162:0/1063756757 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbfc0013070 con 0x7fbfd010ab00 2024-05-27T07:13:21.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:21 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi062 2024-05-27T07:13:21.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:21 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi062 to 5696M 2024-05-27T07:13:22.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:22.021+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbfd0063bf0 con 0x7fbfb40b2900 2024-05-27T07:13:22.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:21 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi062 2024-05-27T07:13:22.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:21 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi062 to 5696M 2024-05-27T07:13:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:22 smithi162 ceph-mon[35676]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:13:22.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:13:22.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:13:22.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:22 smithi062 ceph-mon[35304]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:13:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:13:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:13:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:22 smithi012 ceph-mon[31473]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:13:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:13:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:13:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:24.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:23 smithi162 ceph-mon[35676]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:24.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:23 smithi012 ceph-mon[31473]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:24.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:23 smithi062 ceph-mon[35304]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:24.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:24 smithi162 ceph-mon[35676]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:13:25.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:24 smithi012 ceph-mon[31473]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:13:25.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:24 smithi062 ceph-mon[35304]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:13:26.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:26 smithi162 ceph-mon[35676]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:13:26.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:26 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/3577893750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]: dispatch 2024-05-27T07:13:26.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:26 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]: dispatch 2024-05-27T07:13:26.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:26 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]': finished 2024-05-27T07:13:26.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:26 smithi162 ceph-mon[35676]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T07:13:26.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:27.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:26 smithi012 ceph-mon[31473]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:13:27.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:26 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/3577893750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]: dispatch 2024-05-27T07:13:27.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:26 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]: dispatch 2024-05-27T07:13:27.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:26 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]': finished 2024-05-27T07:13:27.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:26 smithi012 ceph-mon[31473]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T07:13:27.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:26 smithi062 ceph-mon[35304]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:13:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:26 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/3577893750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]: dispatch 2024-05-27T07:13:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:26 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]: dispatch 2024-05-27T07:13:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:26 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0775d1a0-a6b0-4a31-9f58-7b310ad4eeba"}]': finished 2024-05-27T07:13:27.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:26 smithi062 ceph-mon[35304]: osdmap e50: 9 total, 8 up, 9 in 2024-05-27T07:13:27.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:28.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:27 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/1645633431' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:13:28.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:27 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/1645633431' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:13:28.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:27 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/1645633431' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:13:29.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:28 smithi012 ceph-mon[31473]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:13:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:28 smithi162 ceph-mon[35676]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:13:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:28 smithi062 ceph-mon[35304]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:13:30.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:30 smithi162 ceph-mon[35676]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:13:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:30 smithi012 ceph-mon[31473]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:13:31.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:30 smithi062 ceph-mon[35304]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:13:32.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:32 smithi162 ceph-mon[35676]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:33.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:32 smithi012 ceph-mon[31473]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:32 smithi062 ceph-mon[35304]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:35.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:34 smithi012 ceph-mon[31473]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:35.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:34 smithi162 ceph-mon[35676]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:35.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:34 smithi062 ceph-mon[35304]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:36.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:35 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:13:36.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:35 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:35 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:13:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:35 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:35 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:13:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:35 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:36.888 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:36 smithi162 ceph-mon[35676]: Deploying daemon osd.8 on smithi162 2024-05-27T07:13:36.888 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:36 smithi162 ceph-mon[35676]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:37.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:36 smithi012 ceph-mon[31473]: Deploying daemon osd.8 on smithi162 2024-05-27T07:13:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:36 smithi012 ceph-mon[31473]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:36 smithi062 ceph-mon[35304]: Deploying daemon osd.8 on smithi162 2024-05-27T07:13:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:36 smithi062 ceph-mon[35304]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:38.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:38 smithi162 ceph-mon[35676]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:39.064 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:38 smithi062 ceph-mon[35304]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:39.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:38 smithi012 ceph-mon[31473]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:40.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:40 smithi062 ceph-mon[35304]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:41.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:40 smithi162 ceph-mon[35676]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:41.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:40 smithi012 ceph-mon[31473]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:43.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:42 smithi162 ceph-mon[35676]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:43.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:42 smithi012 ceph-mon[31473]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:43.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:42 smithi062 ceph-mon[35304]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:45.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:44 smithi012 ceph-mon[31473]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:45.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:44 smithi162 ceph-mon[35676]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:45.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:44 smithi062 ceph-mon[35304]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:45.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:45.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:45.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:46 smithi162 ceph-mon[35676]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:47.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:46 smithi012 ceph-mon[31473]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:46 smithi062 ceph-mon[35304]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:48 smithi012 ceph-mon[31473]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:49.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:48 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:49.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:48 smithi162 ceph-mon[35676]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:49.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:49.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:48 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:48 smithi062 ceph-mon[35304]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:48 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:49.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:49.997+0000 7fbfccc00700 1 -- 172.21.15.162:0/1063756757 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbfd0063bf0 con 0x7fbfb40b2900 2024-05-27T07:13:49.989 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 8 on host 'smithi162' 2024-05-27T07:13:49.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbfb40b2900 msgr2=0x7fbfb40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:49.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbfb40b2900 0x7fbfb40b4e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbfd01a1f00 tx=0x7fbfb8009040 comp rx=0 tx=0).stop 2024-05-27T07:13:49.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 msgr2=0x7fbfd01a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:49.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd01a0e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0006010 tx=0x7fbfc00057c0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 shutdown_connections 2024-05-27T07:13:49.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbfb40b2900 0x7fbfb40b4e10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfd010c410 0x7fbfd01a8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfd010ab00 0x7fbfd01a0e50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 --2- 172.21.15.162:0/1063756757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd010ba70 0x7fbfd01a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.000+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 >> 172.21.15.162:0/1063756757 conn(0x7fbfd00799b0 msgr2=0x7fbfd010eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:49.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.001+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 shutdown_connections 2024-05-27T07:13:49.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:13:50.001+0000 7fbfd6600700 1 -- 172.21.15.162:0/1063756757 wait complete. 2024-05-27T07:13:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:49 smithi012 ceph-mon[31473]: from='osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:13:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:49 smithi012 ceph-mon[31473]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:13:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:49 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:49 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:49 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:50.125 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:49 smithi062 ceph-mon[35304]: from='osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:13:50.125 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:49 smithi062 ceph-mon[35304]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:13:50.125 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:49 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:50.125 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:49 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:50.125 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:49 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:50.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:49 smithi162 ceph-mon[35676]: from='osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:13:50.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:49 smithi162 ceph-mon[35676]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:13:50.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:49 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:50.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:49 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:50.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:49 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:50.725 DEBUG:teuthology.orchestra.run.smithi162:osd.8> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.8.service 2024-05-27T07:13:50.730 INFO:tasks.cephadm:Deploying osd.9 on smithi012 with /dev/vg_nvme/lv_1... 2024-05-27T07:13:50.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:13:50.938 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:50.938 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:13:50.938 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: from='osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:13:50.938 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:13:50.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:50.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:13:50.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:50.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:50.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:50.939 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:13:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:13:50.879+0000 7fe988c00700 -1 osd.8 0 waiting for initial osdmap 2024-05-27T07:13:50.939 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:13:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:13:50.897+0000 7fe983a00700 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:13:51.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: from='osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:51.193 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:13:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: from='osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:13:51.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:13:51.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:51.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:13:51.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:51.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:51.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:51 smithi012 ceph-mon[31473]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:13:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:51 smithi012 ceph-mon[31473]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:13:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:52.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:51 smithi162 ceph-mon[35676]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:13:52.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:51 smithi162 ceph-mon[35676]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:13:52.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:52.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:51 smithi062 ceph-mon[35304]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:13:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:51 smithi062 ceph-mon[35304]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:13:52.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:52.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:53.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:52 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:13:53.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:52 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:13:53.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:52 smithi012 ceph-mon[31473]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:53.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:53.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:52 smithi012 ceph-mon[31473]: osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] boot 2024-05-27T07:13:53.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:52 smithi012 ceph-mon[31473]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T07:13:53.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:53.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:52 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:13:53.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:52 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:13:53.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:52 smithi162 ceph-mon[35676]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:53.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:53.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:52 smithi162 ceph-mon[35676]: osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] boot 2024-05-27T07:13:53.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:52 smithi162 ceph-mon[35676]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T07:13:53.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:53.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:52 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:13:53.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:52 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:13:53.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:52 smithi062 ceph-mon[35304]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:13:53.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:53.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:52 smithi062 ceph-mon[35304]: osd.8 [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] boot 2024-05-27T07:13:53.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:52 smithi062 ceph-mon[35304]: osdmap e53: 9 total, 9 up, 9 in 2024-05-27T07:13:53.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:54.168 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:13:54.168 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:13:54.168 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-05-27T07:13:54.168 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-05-27T07:13:54.168 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0355161 s, 295 MB/s 2024-05-27T07:13:54.168 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-05-27T07:13:54.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:53 smithi012 ceph-mon[31473]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:13:54.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:53 smithi162 ceph-mon[35676]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:13:54.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:53 smithi062 ceph-mon[35304]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:13:55.110 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_1 2024-05-27T07:13:55.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:54 smithi012 ceph-mon[31473]: pgmap v258: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:13:55.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:54 smithi012 ceph-mon[31473]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:13:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:54 smithi162 ceph-mon[35676]: pgmap v258: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:13:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:54 smithi162 ceph-mon[35676]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:13:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:54 smithi062 ceph-mon[35304]: pgmap v258: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:13:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:54 smithi062 ceph-mon[35304]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:13:55.555 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi162 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi162 to 5696M 2024-05-27T07:13:56.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:56 smithi012 ceph-mon[31473]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi162 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi162 to 5696M 2024-05-27T07:13:56.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:56.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:56.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:56 smithi162 ceph-mon[35676]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:13:56.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi162 2024-05-27T07:13:56.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi162 to 5696M 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:13:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:56 smithi062 ceph-mon[35304]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:13:58.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 -- 172.21.15.12:0/1820408735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 msgr2=0x7f5628106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:58.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 --2- 172.21.15.12:0/1820408735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f5628106790 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5618009990 tx=0x7f5618009ca0 comp rx=0 tx=0).stop 2024-05-27T07:13:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 -- 172.21.15.12:0/1820408735 shutdown_connections 2024-05-27T07:13:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 --2- 172.21.15.12:0/1820408735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5628107e50 0x7f562810e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 --2- 172.21.15.12:0/1820408735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f5628106790 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 --2- 172.21.15.12:0/1820408735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56281074b0 0x7f56281078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 -- 172.21.15.12:0/1820408735 >> 172.21.15.12:0/1820408735 conn(0x7f5628101a00 msgr2=0x7f5628103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 -- 172.21.15.12:0/1820408735 shutdown_connections 2024-05-27T07:13:58.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.013+0000 7f562dc00700 1 -- 172.21.15.12:0/1820408735 wait complete. 2024-05-27T07:13:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 Processor -- start 2024-05-27T07:13:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 -- start start 2024-05-27T07:13:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f562819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:58.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56281074b0 0x7f562819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5628107e50 0x7f56281a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5628111cd0 con 0x7f5628106380 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5628111b50 con 0x7f5628107e50 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.015+0000 7f562dc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5628111e50 con 0x7f56281074b0 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f562819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f562819c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47202/0 (socket says 172.21.15.12:47202) 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 -- 172.21.15.12:0/2630919194 learned_addr learned my addr 172.21.15.12:0/2630919194 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f562d200700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5628107e50 0x7f56281a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627400700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56281074b0 0x7f562819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 -- 172.21.15.12:0/2630919194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56281074b0 msgr2=0x7f562819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56281074b0 0x7f562819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 -- 172.21.15.12:0/2630919194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5628107e50 msgr2=0x7f56281a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5628107e50 0x7f56281a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.016+0000 7f5627e00700 1 -- 172.21.15.12:0/2630919194 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56281a4410 con 0x7f5628106380 2024-05-27T07:13:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.017+0000 7f5627e00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f562819c8b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5618003820 tx=0x7f56180047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:58.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.017+0000 7f5624c00700 1 -- 172.21.15.12:0/2630919194 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618003c10 con 0x7f5628106380 2024-05-27T07:13:58.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.017+0000 7f5624c00700 1 -- 172.21.15.12:0/2630919194 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5618004830 con 0x7f5628106380 2024-05-27T07:13:58.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.017+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56281a46a0 con 0x7f5628106380 2024-05-27T07:13:58.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.017+0000 7f5624c00700 1 -- 172.21.15.12:0/2630919194 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618005260 con 0x7f5628106380 2024-05-27T07:13:58.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.017+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56281a4a90 con 0x7f5628106380 2024-05-27T07:13:58.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.018+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56280512b0 con 0x7f5628106380 2024-05-27T07:13:58.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.021+0000 7f5624c00700 1 -- 172.21.15.12:0/2630919194 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f56180054c0 con 0x7f5628106380 2024-05-27T07:13:58.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.024+0000 7f5624c00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f560c0b2880 0x7f560c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:58.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.024+0000 7f5624c00700 1 -- 172.21.15.12:0/2630919194 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5829+0+0 (secure 0 0 0) 0x7f56180e2a60 con 0x7f5628106380 2024-05-27T07:13:58.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.024+0000 7f5627400700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f560c0b2880 0x7f560c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:58.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.025+0000 7f5627400700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f560c0b2880 0x7f560c0b4d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5614005950 tx=0x7f56140058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:58.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.027+0000 7f5624c00700 1 -- 172.21.15.12:0/2630919194 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5618013070 con 0x7f5628106380 2024-05-27T07:13:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:13:58.279+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5628063bf0 con 0x7f560c0b2880 2024-05-27T07:13:58.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:58 smithi162 ceph-mon[35676]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:13:58.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:13:58.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:13:58.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:58 smithi062 ceph-mon[35304]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:13:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:13:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:13:58.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:58 smithi012 ceph-mon[31473]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:13:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:13:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:13:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:59.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:13:59 smithi012 ceph-mon[31473]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:59.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:13:59 smithi162 ceph-mon[35676]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:59.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:13:59 smithi062 ceph-mon[35304]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:00.970 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:00 smithi012 ceph-mon[31473]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:14:00.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:00 smithi162 ceph-mon[35676]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:14:00.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:00 smithi062 ceph-mon[35304]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:14:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:02 smithi162 ceph-mon[35676]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:14:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:02 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2967964667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7623211-c464-44f4-b7bd-9bd337ae5c80"}]: dispatch 2024-05-27T07:14:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:02 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2967964667' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7623211-c464-44f4-b7bd-9bd337ae5c80"}]': finished 2024-05-27T07:14:02.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:02 smithi162 ceph-mon[35676]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T07:14:02.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:02 smithi062 ceph-mon[35304]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:14:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:02 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2967964667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7623211-c464-44f4-b7bd-9bd337ae5c80"}]: dispatch 2024-05-27T07:14:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:02 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2967964667' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7623211-c464-44f4-b7bd-9bd337ae5c80"}]': finished 2024-05-27T07:14:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:02 smithi062 ceph-mon[35304]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T07:14:02.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:03.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:02 smithi012 ceph-mon[31473]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:14:03.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:02 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2967964667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7623211-c464-44f4-b7bd-9bd337ae5c80"}]: dispatch 2024-05-27T07:14:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:02 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2967964667' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7623211-c464-44f4-b7bd-9bd337ae5c80"}]': finished 2024-05-27T07:14:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:02 smithi012 ceph-mon[31473]: osdmap e56: 10 total, 9 up, 10 in 2024-05-27T07:14:03.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:03 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1444032518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:14:03.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:03 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1444032518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:14:04.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:03 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1444032518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:14:04.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:04 smithi062 ceph-mon[35304]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:14:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:04 smithi012 ceph-mon[31473]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:14:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:04 smithi162 ceph-mon[35676]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:14:07.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:06 smithi012 ceph-mon[31473]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:14:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:06 smithi162 ceph-mon[35676]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:14:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:06 smithi062 ceph-mon[35304]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:14:09.039 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:08 smithi062 ceph-mon[35304]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:09.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:08 smithi012 ceph-mon[31473]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:08 smithi162 ceph-mon[35676]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:10.973 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:10 smithi012 ceph-mon[31473]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:10 smithi162 ceph-mon[35676]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:11.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:10 smithi062 ceph-mon[35304]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:13.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:12 smithi012 ceph-mon[31473]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:14:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:12 smithi162 ceph-mon[35676]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:14:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:12 smithi062 ceph-mon[35304]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:14:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:13.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:13 smithi162 ceph-mon[35676]: Deploying daemon osd.9 on smithi012 2024-05-27T07:14:14.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:13 smithi012 ceph-mon[31473]: Deploying daemon osd.9 on smithi012 2024-05-27T07:14:14.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:13 smithi062 ceph-mon[35304]: Deploying daemon osd.9 on smithi012 2024-05-27T07:14:15.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:14 smithi012 ceph-mon[31473]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:14 smithi162 ceph-mon[35676]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:14 smithi062 ceph-mon[35304]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:17.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:16 smithi012 ceph-mon[31473]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:16 smithi162 ceph-mon[35676]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:16 smithi062 ceph-mon[35304]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:19.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:18 smithi012 ceph-mon[31473]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:18 smithi162 ceph-mon[35676]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:18 smithi062 ceph-mon[35304]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:21.063 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:20 smithi012 ceph-mon[31473]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:20 smithi162 ceph-mon[35676]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:20 smithi062 ceph-mon[35304]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:22 smithi012 ceph-mon[31473]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:22 smithi162 ceph-mon[35676]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:22 smithi062 ceph-mon[35304]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:25.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:24 smithi012 ceph-mon[31473]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:25.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:24 smithi162 ceph-mon[35676]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:25.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:24 smithi062 ceph-mon[35304]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:27.051 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:26 smithi012 ceph-mon[31473]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:27.051 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:27.051 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:27.051 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:26 smithi012 ceph-mon[31473]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:14:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:26 smithi162 ceph-mon[35676]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:27.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:26 smithi162 ceph-mon[35676]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:14:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:26 smithi062 ceph-mon[35304]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:26 smithi062 ceph-mon[35304]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:14:27.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.541+0000 7f5624c00700 1 -- 172.21.15.12:0/2630919194 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5628063bf0 con 0x7f560c0b2880 2024-05-27T07:14:27.533 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 9 on host 'smithi012' 2024-05-27T07:14:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f560c0b2880 msgr2=0x7f560c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f560c0b2880 0x7f560c0b4d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5614005950 tx=0x7f56140058e0 comp rx=0 tx=0).stop 2024-05-27T07:14:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 msgr2=0x7f562819c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f562819c8b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5618003820 tx=0x7f56180047b0 comp rx=0 tx=0).stop 2024-05-27T07:14:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 shutdown_connections 2024-05-27T07:14:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f560c0b2880 0x7f560c0b4d90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5628107e50 0x7f56281a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5628106380 0x7f562819c8b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 --2- 172.21.15.12:0/2630919194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56281074b0 0x7f562819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.545+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 >> 172.21.15.12:0/2630919194 conn(0x7f5628101a00 msgr2=0x7f562810a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.546+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 shutdown_connections 2024-05-27T07:14:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:14:27.546+0000 7f562dc00700 1 -- 172.21.15.12:0/2630919194 wait complete. 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:28.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:28 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:28.471 DEBUG:teuthology.orchestra.run.smithi012:osd.9> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.9.service 2024-05-27T07:14:28.477 INFO:tasks.cephadm:Deploying osd.10 on smithi062 with /dev/vg_nvme/lv_1... 2024-05-27T07:14:28.477 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:14:28.557 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:14:28.557 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:14:28.557 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:14:28.557 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:28.558 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:28.558 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:28.558 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.558 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.558 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.558 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:28.558 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:28 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:28.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:28 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:14:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:29 smithi012 ceph-mon[31473]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:14:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:29 smithi012 ceph-mon[31473]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:14:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:29 smithi012 ceph-mon[31473]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' 2024-05-27T07:14:29.341 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:14:29 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:14:29.135+0000 7fded8200700 -1 osd.9 0 waiting for initial osdmap 2024-05-27T07:14:29.342 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:14:29 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:14:29.168+0000 7fdece000700 -1 osd.9 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:14:29.359 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:29 smithi062 ceph-mon[35304]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:14:29.359 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:29 smithi062 ceph-mon[35304]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:14:29.359 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:29.359 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:29.359 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:29 smithi062 ceph-mon[35304]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' 2024-05-27T07:14:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:29 smithi162 ceph-mon[35676]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-05-27T07:14:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:29 smithi162 ceph-mon[35676]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:14:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:29.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:29 smithi162 ceph-mon[35676]: from='osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154]' entity='osd.9' 2024-05-27T07:14:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:30 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:14:30.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:30 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:14:30.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:30.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:30 smithi062 ceph-mon[35304]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:30.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:30 smithi062 ceph-mon[35304]: osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] boot 2024-05-27T07:14:30.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:30 smithi062 ceph-mon[35304]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T07:14:30.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:30 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:30.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:30 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:14:30.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:30 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:14:30.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:30.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:30 smithi012 ceph-mon[31473]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:30.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:30 smithi012 ceph-mon[31473]: osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] boot 2024-05-27T07:14:30.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:30 smithi012 ceph-mon[31473]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T07:14:30.555 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:30 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:30.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:30 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:14:30.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:30 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:14:30.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:30.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:30 smithi162 ceph-mon[35676]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:14:30.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:30 smithi162 ceph-mon[35676]: osd.9 [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] boot 2024-05-27T07:14:30.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:30 smithi162 ceph-mon[35676]: osdmap e59: 10 total, 10 up, 10 in 2024-05-27T07:14:30.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:30 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:14:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:14:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:14:32.158 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-05-27T07:14:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-05-27T07:14:32.158 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.032922 s, 319 MB/s 2024-05-27T07:14:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-05-27T07:14:32.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:32 smithi062 ceph-mon[35304]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:14:32.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:32 smithi062 ceph-mon[35304]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:32.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:32 smithi012 ceph-mon[31473]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:14:32.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:32 smithi012 ceph-mon[31473]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:32.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:32 smithi162 ceph-mon[35676]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:14:32.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:32 smithi162 ceph-mon[35676]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:33.094 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_1 2024-05-27T07:14:33.559 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:14:34.751 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:34.751 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi012 2024-05-27T07:14:34.751 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:34.751 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:34.751 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi012 to 4273M 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:34.752 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:34 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:34.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:34.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi012 2024-05-27T07:14:34.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:34.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi012 to 4273M 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:34.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:34 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi012 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:35.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:35.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi012 to 4273M 2024-05-27T07:14:35.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:35.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:35.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:35.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:34 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:14:35.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.934+0000 7fdac7e00700 1 -- 172.21.15.62:0/2860555571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac8107e50 msgr2=0x7fdac810e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:35.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.934+0000 7fdac7e00700 1 --2- 172.21.15.62:0/2860555571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac8107e50 0x7fdac810e700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdac000b0a0 tx=0x7fdac000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:14:35.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.935+0000 7fdac7e00700 1 -- 172.21.15.62:0/2860555571 shutdown_connections 2024-05-27T07:14:35.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.935+0000 7fdac7e00700 1 --2- 172.21.15.62:0/2860555571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac8107e50 0x7fdac810e700 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:35.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.935+0000 7fdac7e00700 1 --2- 172.21.15.62:0/2860555571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdac8106380 0x7fdac8106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:35.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.935+0000 7fdac7e00700 1 --2- 172.21.15.62:0/2860555571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac81074b0 0x7fdac81078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:35.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.935+0000 7fdac7e00700 1 -- 172.21.15.62:0/2860555571 >> 172.21.15.62:0/2860555571 conn(0x7fdac8101a00 msgr2=0x7fdac8103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:35.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.935+0000 7fdac7e00700 1 -- 172.21.15.62:0/2860555571 shutdown_connections 2024-05-27T07:14:35.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.935+0000 7fdac7e00700 1 -- 172.21.15.62:0/2860555571 wait complete. 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.936+0000 7fdac7e00700 1 Processor -- start 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.936+0000 7fdac7e00700 1 -- start start 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.936+0000 7fdac7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdac8106380 0x7fdac819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.936+0000 7fdac7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac81074b0 0x7fdac819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.936+0000 7fdac7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac8107e50 0x7fdac81a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdac8111cf0 con 0x7fdac8107e50 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdac8111b70 con 0x7fdac81074b0 2024-05-27T07:14:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac7e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdac8111e70 con 0x7fdac8106380 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac81074b0 0x7fdac819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac81074b0 0x7fdac819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43616/0 (socket says 172.21.15.62:43616) 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 -- 172.21.15.62:0/3372824302 learned_addr learned my addr 172.21.15.62:0/3372824302 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6a00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdac8106380 0x7fdac819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac7400700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac8107e50 0x7fdac81a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 -- 172.21.15.62:0/3372824302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdac8106380 msgr2=0x7fdac819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdac8106380 0x7fdac819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 -- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac8107e50 msgr2=0x7fdac81a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac8107e50 0x7fdac81a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.937+0000 7fdac6000700 1 -- 172.21.15.62:0/3372824302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac81a4360 con 0x7fdac81074b0 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.938+0000 7fdac6000700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac81074b0 0x7fdac819cd40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdab400d950 tx=0x7fdab400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.938+0000 7fdabf400700 1 -- 172.21.15.62:0/3372824302 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4016070 con 0x7fdac81074b0 2024-05-27T07:14:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.938+0000 7fdabf400700 1 -- 172.21.15.62:0/3372824302 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdab4009dd0 con 0x7fdac81074b0 2024-05-27T07:14:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.938+0000 7fdabf400700 1 -- 172.21.15.62:0/3372824302 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab400f3c0 con 0x7fdac81074b0 2024-05-27T07:14:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.938+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac81a4650 con 0x7fdac81074b0 2024-05-27T07:14:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.938+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac81a4be0 con 0x7fdac81074b0 2024-05-27T07:14:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.939+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac80512b0 con 0x7fdac81074b0 2024-05-27T07:14:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.945+0000 7fdabf400700 1 -- 172.21.15.62:0/3372824302 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdab4019070 con 0x7fdac81074b0 2024-05-27T07:14:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.947+0000 7fdabf400700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdaac0b2900 0x7fdaac0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.947+0000 7fdabf400700 1 -- 172.21.15.62:0/3372824302 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6261+0+0 (secure 0 0 0) 0x7fdab40e1670 con 0x7fdac81074b0 2024-05-27T07:14:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.948+0000 7fdac6a00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdaac0b2900 0x7fdaac0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.949+0000 7fdac6a00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdaac0b2900 0x7fdaac0b4e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdac81073b0 tx=0x7fdab800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:35.949+0000 7fdabf400700 1 -- 172.21.15.62:0/3372824302 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdab40a9af0 con 0x7fdac81074b0 2024-05-27T07:14:36.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:14:36.199+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdac8063bf0 con 0x7fdaac0b2900 2024-05-27T07:14:36.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:36 smithi062 ceph-mon[35304]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:36.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:14:36.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:14:36.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:36 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:37.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:36 smithi012 ceph-mon[31473]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:14:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:14:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:36 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:36 smithi162 ceph-mon[35676]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:14:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:14:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:36 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:37.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:37 smithi062 ceph-mon[35304]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:38.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:37 smithi012 ceph-mon[31473]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:38.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:37 smithi162 ceph-mon[35676]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:38.913 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:38 smithi062 ceph-mon[35304]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:39.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:38 smithi012 ceph-mon[31473]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:39.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:38 smithi162 ceph-mon[35676]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:41.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:40 smithi012 ceph-mon[31473]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:41.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:40 smithi162 ceph-mon[35676]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:41.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:40 smithi062 ceph-mon[35304]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:42.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:41 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/1536506566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]: dispatch 2024-05-27T07:14:42.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:41 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]: dispatch 2024-05-27T07:14:42.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:41 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]': finished 2024-05-27T07:14:42.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:41 smithi012 ceph-mon[31473]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T07:14:42.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:41 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:14:42.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:41 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/1536506566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]: dispatch 2024-05-27T07:14:42.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:41 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]: dispatch 2024-05-27T07:14:42.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:41 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]': finished 2024-05-27T07:14:42.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:41 smithi162 ceph-mon[35676]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T07:14:42.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:41 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:14:42.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:41 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/1536506566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]: dispatch 2024-05-27T07:14:42.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:41 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]: dispatch 2024-05-27T07:14:42.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:41 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9449f46e-d354-4305-9f57-ff89f196c55a"}]': finished 2024-05-27T07:14:42.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:41 smithi062 ceph-mon[35304]: osdmap e61: 11 total, 10 up, 11 in 2024-05-27T07:14:42.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:41 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:14:43.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:42 smithi012 ceph-mon[31473]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:43.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:42 smithi012 ceph-mon[31473]: from='client.? 172.21.15.62:0/487854105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:14:43.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:42 smithi162 ceph-mon[35676]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:43.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:42 smithi162 ceph-mon[35676]: from='client.? 172.21.15.62:0/487854105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:14:43.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:42 smithi062 ceph-mon[35304]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:43.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:42 smithi062 ceph-mon[35304]: from='client.? 172.21.15.62:0/487854105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:14:45.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:44 smithi012 ceph-mon[31473]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:44 smithi162 ceph-mon[35676]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:45.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:44 smithi062 ceph-mon[35304]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:47.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:46 smithi012 ceph-mon[31473]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:46 smithi162 ceph-mon[35676]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:46 smithi062 ceph-mon[35304]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:48.864 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:48 smithi062 ceph-mon[35304]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:48 smithi012 ceph-mon[31473]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:49.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:48 smithi162 ceph-mon[35676]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:50.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:50 smithi062 ceph-mon[35304]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:50.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:14:50.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:51.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:50 smithi012 ceph-mon[31473]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:51.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:14:51.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:50 smithi162 ceph-mon[35676]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:14:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:52.012 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:51 smithi062 ceph-mon[35304]: Deploying daemon osd.10 on smithi062 2024-05-27T07:14:52.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:51 smithi012 ceph-mon[31473]: Deploying daemon osd.10 on smithi062 2024-05-27T07:14:52.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:51 smithi162 ceph-mon[35676]: Deploying daemon osd.10 on smithi062 2024-05-27T07:14:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:52 smithi062 ceph-mon[35304]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:53.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:52 smithi012 ceph-mon[31473]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:53.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:52 smithi162 ceph-mon[35676]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:54.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:54 smithi062 ceph-mon[35304]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:54 smithi012 ceph-mon[31473]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:54 smithi162 ceph-mon[35676]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:57.069 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:56 smithi062 ceph-mon[35304]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:57.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:56 smithi012 ceph-mon[31473]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:56 smithi162 ceph-mon[35676]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:59.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:14:58 smithi012 ceph-mon[31473]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:59.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:14:58 smithi162 ceph-mon[35676]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:14:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:14:58 smithi062 ceph-mon[35304]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:01.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:00 smithi012 ceph-mon[31473]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:00 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:01.194 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:00 smithi062 ceph-mon[35304]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:01.194 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:01.194 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:01.194 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:00 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:00 smithi162 ceph-mon[35676]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:00 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:03.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:02 smithi012 ceph-mon[31473]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:03.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:02 smithi162 ceph-mon[35676]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:02 smithi062 ceph-mon[35304]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:04 smithi012 ceph-mon[31473]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:05.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:05.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:05.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:04 smithi162 ceph-mon[35676]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:04 smithi062 ceph-mon[35304]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:05.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:05.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:05.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:05.239 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:06.108 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:05 smithi062 ceph-mon[35304]: from='osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:15:06.108 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:05 smithi062 ceph-mon[35304]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:15:06.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:05 smithi162 ceph-mon[35676]: from='osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:15:06.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:05 smithi162 ceph-mon[35676]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:15:06.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:05 smithi012 ceph-mon[31473]: from='osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:15:06.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:05 smithi012 ceph-mon[31473]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:15:06.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.628+0000 7fdabf400700 1 -- 172.21.15.62:0/3372824302 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fdac8063bf0 con 0x7fdaac0b2900 2024-05-27T07:15:06.621 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 10 on host 'smithi062' 2024-05-27T07:15:06.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdaac0b2900 msgr2=0x7fdaac0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:06.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdaac0b2900 0x7fdaac0b4e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdac81073b0 tx=0x7fdab800b410 comp rx=0 tx=0).stop 2024-05-27T07:15:06.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac81074b0 msgr2=0x7fdac819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:06.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac81074b0 0x7fdac819cd40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdab400d950 tx=0x7fdab400dc60 comp rx=0 tx=0).stop 2024-05-27T07:15:06.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 shutdown_connections 2024-05-27T07:15:06.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdaac0b2900 0x7fdaac0b4e10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:06.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdac8107e50 0x7fdac81a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:06.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdac8106380 0x7fdac819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:06.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 --2- 172.21.15.62:0/3372824302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdac81074b0 0x7fdac819cd40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:06.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 >> 172.21.15.62:0/3372824302 conn(0x7fdac8101a00 msgr2=0x7fdac810a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:06.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 shutdown_connections 2024-05-27T07:15:06.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-05-27T07:15:06.633+0000 7fdac7e00700 1 -- 172.21.15.62:0/3372824302 wait complete. 2024-05-27T07:15:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:15:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:15:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: from='osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:15:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:15:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: from='osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: from='osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:07.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:07.551 DEBUG:teuthology.orchestra.run.smithi062:osd.10> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.10.service 2024-05-27T07:15:07.556 INFO:tasks.cephadm:Deploying osd.11 on smithi162 with /dev/vg_nvme/lv_1... 2024-05-27T07:15:07.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:15:07.757 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:15:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:15:07.736+0000 7fd37ee00700 -1 osd.10 0 waiting for initial osdmap 2024-05-27T07:15:07.757 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:15:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:15:07.765+0000 7fd374c00700 -1 osd.10 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:15:07.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:07 smithi162 ceph-mon[35676]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:15:07.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:07 smithi162 ceph-mon[35676]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:15:07.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:07.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:07 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:07.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:07 smithi162 ceph-mon[35676]: from='osd.10 ' entity='osd.10' 2024-05-27T07:15:08.000 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:15:08.066 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:07 smithi062 ceph-mon[35304]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:15:08.066 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:07 smithi062 ceph-mon[35304]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:15:08.066 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:08.066 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:07 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:08.066 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:07 smithi062 ceph-mon[35304]: from='osd.10 ' entity='osd.10' 2024-05-27T07:15:08.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:07 smithi012 ceph-mon[31473]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-05-27T07:15:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:07 smithi012 ceph-mon[31473]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:15:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:07 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:07 smithi012 ceph-mon[31473]: from='osd.10 ' entity='osd.10' 2024-05-27T07:15:08.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:08 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:15:08.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:08 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:15:08.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:08 smithi062 ceph-mon[35304]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:08.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:08.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:08 smithi062 ceph-mon[35304]: osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] boot 2024-05-27T07:15:08.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:08 smithi062 ceph-mon[35304]: osdmap e64: 11 total, 11 up, 11 in 2024-05-27T07:15:08.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:09.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:08 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:15:09.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:08 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:15:09.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:08 smithi162 ceph-mon[35676]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:09.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:09.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:08 smithi162 ceph-mon[35676]: osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] boot 2024-05-27T07:15:09.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:08 smithi162 ceph-mon[35676]: osdmap e64: 11 total, 11 up, 11 in 2024-05-27T07:15:09.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:09.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:08 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:15:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:08 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:15:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:08 smithi012 ceph-mon[31473]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:15:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:08 smithi012 ceph-mon[31473]: osd.10 [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] boot 2024-05-27T07:15:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:08 smithi012 ceph-mon[31473]: osdmap e64: 11 total, 11 up, 11 in 2024-05-27T07:15:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:15:09.925 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:09 smithi062 ceph-mon[35304]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:15:10.114 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:09 smithi162 ceph-mon[35676]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:15:10.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:09 smithi012 ceph-mon[31473]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:15:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:10 smithi162 ceph-mon[35676]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:10 smithi162 ceph-mon[35676]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:15:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:10 smithi062 ceph-mon[35304]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:10 smithi062 ceph-mon[35304]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:15:11.013 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:15:11.013 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:15:11.013 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-05-27T07:15:11.013 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-05-27T07:15:11.014 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0364473 s, 288 MB/s 2024-05-27T07:15:11.014 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-05-27T07:15:11.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:10 smithi012 ceph-mon[31473]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:11.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:10 smithi012 ceph-mon[31473]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:15:11.904 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-05-27T07:15:12.350 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-05-27T07:15:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:13.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:13.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:14.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:13 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi062 2024-05-27T07:15:14.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:13 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi062 to 4272M 2024-05-27T07:15:14.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:13 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi062 2024-05-27T07:15:14.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:13 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi062 to 4272M 2024-05-27T07:15:14.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:13 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi062 2024-05-27T07:15:14.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:13 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi062 to 4272M 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 -- 172.21.15.162:0/2955681961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0104140 msgr2=0x7ff0c0104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 --2- 172.21.15.162:0/2955681961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0104140 0x7ff0c0104550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009990 tx=0x7ff0b0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 -- 172.21.15.162:0/2955681961 shutdown_connections 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 --2- 172.21.15.162:0/2955681961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0c0105c10 0x7ff0c010c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 --2- 172.21.15.162:0/2955681961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0104140 0x7ff0c0104550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 --2- 172.21.15.162:0/2955681961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0c0105270 0x7ff0c0105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 -- 172.21.15.162:0/2955681961 >> 172.21.15.162:0/2955681961 conn(0x7ff0c00ff7a0 msgr2=0x7ff0c0101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 -- 172.21.15.162:0/2955681961 shutdown_connections 2024-05-27T07:15:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.471+0000 7ff0c5800700 1 -- 172.21.15.162:0/2955681961 wait complete. 2024-05-27T07:15:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 Processor -- start 2024-05-27T07:15:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 -- start start 2024-05-27T07:15:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0c0104140 0x7ff0c019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0105270 0x7ff0c019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0c0105c10 0x7ff0c01a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c010fa90 con 0x7ff0c0104140 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c010f910 con 0x7ff0c0105c10 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0c5800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c010fc10 con 0x7ff0c0105270 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0bf400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0105270 0x7ff0c019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0bf400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0105270 0x7ff0c019ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42058/0 (socket says 172.21.15.162:42058) 2024-05-27T07:15:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.473+0000 7ff0bf400700 1 -- 172.21.15.162:0/3376298636 learned_addr learned my addr 172.21.15.162:0/3376298636 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:15:14.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bfe00700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0c0104140 0x7ff0c019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bf400700 1 -- 172.21.15.162:0/3376298636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0c0105c10 msgr2=0x7ff0c01a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bf400700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0c0105c10 0x7ff0c01a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bf400700 1 -- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0c0104140 msgr2=0x7ff0c019c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bf400700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0c0104140 0x7ff0c019c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bf400700 1 -- 172.21.15.162:0/3376298636 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c01a4400 con 0x7ff0c0105270 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bf400700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0105270 0x7ff0c019ce70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d950 tx=0x7ff0ac00dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.474+0000 7ff0bcc00700 1 -- 172.21.15.162:0/3376298636 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac016070 con 0x7ff0c0105270 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.475+0000 7ff0bcc00700 1 -- 172.21.15.162:0/3376298636 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff0ac010460 con 0x7ff0c0105270 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.475+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c01a46f0 con 0x7ff0c0105270 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.475+0000 7ff0bcc00700 1 -- 172.21.15.162:0/3376298636 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00f620 con 0x7ff0c0105270 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.475+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c01a4c30 con 0x7ff0c0105270 2024-05-27T07:15:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.476+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c00512b0 con 0x7ff0c0105270 2024-05-27T07:15:14.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.479+0000 7ff0bcc00700 1 -- 172.21.15.162:0/3376298636 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff0ac019070 con 0x7ff0c0105270 2024-05-27T07:15:14.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.481+0000 7ff0bcc00700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff0a40b2880 0x7ff0a40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.482+0000 7ff0bcc00700 1 -- 172.21.15.162:0/3376298636 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6694+0+0 (secure 0 0 0) 0x7ff0ac0e1870 con 0x7ff0c0105270 2024-05-27T07:15:14.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.482+0000 7ff0bfe00700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff0a40b2880 0x7ff0a40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.483+0000 7ff0bfe00700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff0a40b2880 0x7ff0a40b4d90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0006010 tx=0x7ff0b001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.484+0000 7ff0bcc00700 1 -- 172.21.15.162:0/3376298636 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff0ac0a9b40 con 0x7ff0c0105270 2024-05-27T07:15:14.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:14.723+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff0c0077100 con 0x7ff0a40b2880 2024-05-27T07:15:14.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:14 smithi162 ceph-mon[35676]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:15:14.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:15:14.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:15:14.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:14 smithi062 ceph-mon[35304]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:15:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:15:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:15:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:14 smithi012 ceph-mon[31473]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:15:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:15:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:15:15.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:16.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:15 smithi162 ceph-mon[35676]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:16.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:15 smithi062 ceph-mon[35304]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:16.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:15 smithi012 ceph-mon[31473]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:16.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:16 smithi162 ceph-mon[35676]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:15:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:16 smithi062 ceph-mon[35304]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:15:17.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:16 smithi012 ceph-mon[31473]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:15:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:18 smithi162 ceph-mon[35676]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:15:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:18 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/2579977569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]: dispatch 2024-05-27T07:15:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:18 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]: dispatch 2024-05-27T07:15:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:18 smithi162 ceph-mon[35676]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]': finished 2024-05-27T07:15:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:18 smithi162 ceph-mon[35676]: osdmap e67: 12 total, 11 up, 12 in 2024-05-27T07:15:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:18 smithi062 ceph-mon[35304]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:15:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:18 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/2579977569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]: dispatch 2024-05-27T07:15:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:18 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]: dispatch 2024-05-27T07:15:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:18 smithi062 ceph-mon[35304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]': finished 2024-05-27T07:15:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:18 smithi062 ceph-mon[35304]: osdmap e67: 12 total, 11 up, 12 in 2024-05-27T07:15:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:19.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:18 smithi012 ceph-mon[31473]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:15:19.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:18 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/2579977569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]: dispatch 2024-05-27T07:15:19.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:18 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]: dispatch 2024-05-27T07:15:19.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:18 smithi012 ceph-mon[31473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6"}]': finished 2024-05-27T07:15:19.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:18 smithi012 ceph-mon[31473]: osdmap e67: 12 total, 11 up, 12 in 2024-05-27T07:15:19.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:20.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:19 smithi162 ceph-mon[35676]: from='client.? 172.21.15.162:0/1915018825' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:15:20.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:19 smithi062 ceph-mon[35304]: from='client.? 172.21.15.162:0/1915018825' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:15:20.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:19 smithi012 ceph-mon[31473]: from='client.? 172.21.15.162:0/1915018825' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:15:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:20 smithi162 ceph-mon[35676]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:15:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:20 smithi062 ceph-mon[35304]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:15:21.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:20 smithi012 ceph-mon[31473]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:15:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:22 smithi062 ceph-mon[35304]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:23.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:22 smithi012 ceph-mon[31473]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:23.470 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:22 smithi162 ceph-mon[35676]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:25.293 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:24 smithi162 ceph-mon[35676]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:25.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:24 smithi012 ceph-mon[31473]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:25.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:24 smithi062 ceph-mon[35304]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:27 smithi162 ceph-mon[35676]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:27.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:27 smithi012 ceph-mon[31473]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:27.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:27 smithi062 ceph-mon[35304]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:29.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:29 smithi012 ceph-mon[31473]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:29.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:15:29.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:29.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:29 smithi162 ceph-mon[35676]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:29.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:15:29.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:29.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:29 smithi062 ceph-mon[35304]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:29.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:15:29.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:30.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:30 smithi162 ceph-mon[35676]: Deploying daemon osd.11 on smithi162 2024-05-27T07:15:30.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:30 smithi012 ceph-mon[31473]: Deploying daemon osd.11 on smithi162 2024-05-27T07:15:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:30 smithi062 ceph-mon[35304]: Deploying daemon osd.11 on smithi162 2024-05-27T07:15:31.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:31 smithi012 ceph-mon[31473]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:31.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:31 smithi162 ceph-mon[35676]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:31.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:31 smithi062 ceph-mon[35304]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:32.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:32 smithi162 ceph-mon[35676]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:32 smithi012 ceph-mon[31473]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:32.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:32 smithi062 ceph-mon[35304]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:34.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:34 smithi062 ceph-mon[35304]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:35.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:34 smithi012 ceph-mon[31473]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:35.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:34 smithi162 ceph-mon[35676]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:36.965 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:36 smithi162 ceph-mon[35676]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:36.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:36 smithi062 ceph-mon[35304]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:37.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:36 smithi012 ceph-mon[31473]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:38.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:38.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:38.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:38.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:39.180 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:38 smithi162 ceph-mon[35676]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:39.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:38 smithi062 ceph-mon[35304]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:39.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:38 smithi012 ceph-mon[31473]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:41.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:40 smithi162 ceph-mon[35676]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:41.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:40 smithi062 ceph-mon[35304]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:41.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:40 smithi012 ceph-mon[31473]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:42.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: from='osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:15:42.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:42 smithi162 ceph-mon[35676]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: from='osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:15:42.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:42 smithi012 ceph-mon[31473]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:15:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:42.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:42.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: from='osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:15:42.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:42 smithi062 ceph-mon[35304]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:15:43.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.400+0000 7ff0bcc00700 1 -- 172.21.15.162:0/3376298636 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff0c0077100 con 0x7ff0a40b2880 2024-05-27T07:15:43.392 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 11 on host 'smithi162' 2024-05-27T07:15:43.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff0a40b2880 msgr2=0x7ff0a40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff0a40b2880 0x7ff0a40b4d90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0006010 tx=0x7ff0b001a040 comp rx=0 tx=0).stop 2024-05-27T07:15:43.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0105270 msgr2=0x7ff0c019ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0105270 0x7ff0c019ce70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d950 tx=0x7ff0ac00dd10 comp rx=0 tx=0).stop 2024-05-27T07:15:43.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 shutdown_connections 2024-05-27T07:15:43.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff0a40b2880 0x7ff0a40b4d90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0c0105c10 0x7ff0c01a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0c0104140 0x7ff0c019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 --2- 172.21.15.162:0/3376298636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0c0105270 0x7ff0c019ce70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 >> 172.21.15.162:0/3376298636 conn(0x7ff0c00ff7a0 msgr2=0x7ff0c00759e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:43.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 shutdown_connections 2024-05-27T07:15:43.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:43.405+0000 7ff0c5800700 1 -- 172.21.15.162:0/3376298636 wait complete. 2024-05-27T07:15:43.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:15:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:15:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: from='osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:15:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:15:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:43 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:43.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:15:43.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:15:43.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:43.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: from='osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:15:43.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:15:43.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:43.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:43.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:43 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: from='osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:43.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:43 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:44.198 DEBUG:teuthology.orchestra.run.smithi162:osd.11> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.11.service 2024-05-27T07:15:44.202 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-27T07:15:44.203 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-27T07:15:44.691 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:44 smithi162 ceph-mon[35676]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:15:44.691 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:44 smithi162 ceph-mon[35676]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:15:44.691 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:44.691 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:44 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:44.691 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:44 smithi162 ceph-mon[35676]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:44.691 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:44 smithi162 ceph-mon[35676]: from='osd.11 ' entity='osd.11' 2024-05-27T07:15:44.692 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:15:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:15:44.444+0000 7f9914600700 -1 osd.11 0 waiting for initial osdmap 2024-05-27T07:15:44.692 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:15:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:15:44.475+0000 7f990a400700 -1 osd.11 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:15:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:44.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:44 smithi012 ceph-mon[31473]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:15:44.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:44 smithi012 ceph-mon[31473]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:15:44.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:44.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:44 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:44.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:44 smithi012 ceph-mon[31473]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:44.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:44 smithi012 ceph-mon[31473]: from='osd.11 ' entity='osd.11' 2024-05-27T07:15:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:44 smithi062 ceph-mon[35304]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-05-27T07:15:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:44 smithi062 ceph-mon[35304]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:15:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:44 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:44 smithi062 ceph-mon[35304]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:15:44.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:44 smithi062 ceph-mon[35304]: from='osd.11 ' entity='osd.11' 2024-05-27T07:15:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:45 smithi012 ceph-mon[31473]: purged_snaps scrub starts 2024-05-27T07:15:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:45 smithi012 ceph-mon[31473]: purged_snaps scrub ok 2024-05-27T07:15:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:45 smithi012 ceph-mon[31473]: osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] boot 2024-05-27T07:15:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:45 smithi012 ceph-mon[31473]: osdmap e70: 12 total, 12 up, 12 in 2024-05-27T07:15:45.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:45.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:45 smithi162 ceph-mon[35676]: purged_snaps scrub starts 2024-05-27T07:15:45.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:45 smithi162 ceph-mon[35676]: purged_snaps scrub ok 2024-05-27T07:15:45.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:45.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:45 smithi162 ceph-mon[35676]: osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] boot 2024-05-27T07:15:45.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:45 smithi162 ceph-mon[35676]: osdmap e70: 12 total, 12 up, 12 in 2024-05-27T07:15:45.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:45.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:45 smithi062 ceph-mon[35304]: purged_snaps scrub starts 2024-05-27T07:15:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:45 smithi062 ceph-mon[35304]: purged_snaps scrub ok 2024-05-27T07:15:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:45 smithi062 ceph-mon[35304]: osd.11 [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] boot 2024-05-27T07:15:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:45 smithi062 ceph-mon[35304]: osdmap e70: 12 total, 12 up, 12 in 2024-05-27T07:15:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:15:46.756 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:46 smithi012 ceph-mon[31473]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:46 smithi162 ceph-mon[35676]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:46.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:46 smithi062 ceph-mon[35304]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.100+0000 7fc758200700 1 -- 172.21.15.12:0/418857333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 msgr2=0x7fc750107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.100+0000 7fc758200700 1 --2- 172.21.15.12:0/418857333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc750107a80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009990 tx=0x7fc73c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.100+0000 7fc758200700 1 -- 172.21.15.12:0/418857333 shutdown_connections 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.100+0000 7fc758200700 1 --2- 172.21.15.12:0/418857333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc750108010 0x7fc75010e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.100+0000 7fc758200700 1 --2- 172.21.15.12:0/418857333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc750106540 0x7fc750106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.100+0000 7fc758200700 1 --2- 172.21.15.12:0/418857333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc750107a80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.100+0000 7fc758200700 1 -- 172.21.15.12:0/418857333 >> 172.21.15.12:0/418857333 conn(0x7fc750101b80 msgr2=0x7fc750104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.101+0000 7fc758200700 1 -- 172.21.15.12:0/418857333 shutdown_connections 2024-05-27T07:15:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.101+0000 7fc758200700 1 -- 172.21.15.12:0/418857333 wait complete. 2024-05-27T07:15:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 Processor -- start 2024-05-27T07:15:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 -- start start 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc750106540 0x7fc75019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc75019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc750108010 0x7fc7501a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc750111ea0 con 0x7fc750107670 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc750111d20 con 0x7fc750106540 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.102+0000 7fc758200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc750112020 con 0x7fc750108010 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc754e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc75019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc754e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc75019cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49436/0 (socket says 172.21.15.12:49436) 2024-05-27T07:15:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc754e00700 1 -- 172.21.15.12:0/2646447470 learned_addr learned my addr 172.21.15.12:0/2646447470 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:15:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc756200700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc750108010 0x7fc7501a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc755800700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc750106540 0x7fc75019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc754e00700 1 -- 172.21.15.12:0/2646447470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc750108010 msgr2=0x7fc7501a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc754e00700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc750108010 0x7fc7501a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc754e00700 1 -- 172.21.15.12:0/2646447470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc750106540 msgr2=0x7fc75019caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.103+0000 7fc754e00700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc750106540 0x7fc75019caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.104+0000 7fc754e00700 1 -- 172.21.15.12:0/2646447470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7501a45b0 con 0x7fc750107670 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.104+0000 7fc754e00700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc75019cfe0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc73c000c00 tx=0x7fc73c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.104+0000 7fc74e000700 1 -- 172.21.15.12:0/2646447470 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c027070 con 0x7fc750107670 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.104+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7501a4840 con 0x7fc750107670 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.104+0000 7fc74e000700 1 -- 172.21.15.12:0/2646447470 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc73c0041a0 con 0x7fc750107670 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.104+0000 7fc74e000700 1 -- 172.21.15.12:0/2646447470 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c022e50 con 0x7fc750107670 2024-05-27T07:15:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.105+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7501a4c90 con 0x7fc750107670 2024-05-27T07:15:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.106+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7500512b0 con 0x7fc750107670 2024-05-27T07:15:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.108+0000 7fc74e000700 1 -- 172.21.15.12:0/2646447470 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc73c02a070 con 0x7fc750107670 2024-05-27T07:15:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.110+0000 7fc74e000700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc7380b2a20 0x7fc7380b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.110+0000 7fc74e000700 1 -- 172.21.15.12:0/2646447470 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fc73c0e3420 con 0x7fc750107670 2024-05-27T07:15:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.110+0000 7fc755800700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc7380b2a20 0x7fc7380b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.111+0000 7fc755800700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc7380b2a20 0x7fc7380b4f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc744005fd0 tx=0x7fc744005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.113+0000 7fc74e000700 1 -- 172.21.15.12:0/2646447470 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc73c013070 con 0x7fc750107670 2024-05-27T07:15:47.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.350+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fc7500643c0 con 0x7fc750107670 2024-05-27T07:15:47.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.351+0000 7fc74e000700 1 -- 172.21.15.12:0/2646447470 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7fc73c0ab5d0 con 0x7fc750107670 2024-05-27T07:15:47.343 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:15:47.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc7380b2a20 msgr2=0x7fc7380b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc7380b2a20 0x7fc7380b4f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc744005fd0 tx=0x7fc744005dd0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 msgr2=0x7fc75019cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc75019cfe0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc73c000c00 tx=0x7fc73c005ef0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 shutdown_connections 2024-05-27T07:15:47.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc7380b2a20 0x7fc7380b4f30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc750106540 0x7fc75019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc750108010 0x7fc7501a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 --2- 172.21.15.12:0/2646447470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc750107670 0x7fc75019cfe0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.354+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 >> 172.21.15.12:0/2646447470 conn(0x7fc750101b80 msgr2=0x7fc75010a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.355+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 shutdown_connections 2024-05-27T07:15:47.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:47.355+0000 7fc758200700 1 -- 172.21.15.12:0/2646447470 wait complete. 2024-05-27T07:15:47.911 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:47 smithi162 ceph-mon[35676]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:15:47.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:47 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2646447470' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:15:47.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:47 smithi062 ceph-mon[35304]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:15:47.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:47 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2646447470' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:15:48.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:47 smithi012 ceph-mon[31473]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:15:48.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:47 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2646447470' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:15:48.272 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":71,"num_osds":12,"num_up_osds":12,"osd_up_since":1716794145,"num_in_osds":12,"osd_in_since":1716794118,"num_remapped_pgs":0} 2024-05-27T07:15:48.273 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:15:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:48.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:48 smithi012 ceph-mon[31473]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:48 smithi162 ceph-mon[35676]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:48.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:48 smithi062 ceph-mon[35304]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:50.545 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: Detected new or changed devices on smithi162 2024-05-27T07:15:50.545 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.545 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.545 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.545 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi162 to 4272M 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.546 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:50 smithi012 ceph-mon[31473]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:50.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: Detected new or changed devices on smithi162 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi162 to 4272M 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:50.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:50.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:50 smithi162 ceph-mon[35676]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: Detected new or changed devices on smithi162 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi162 to 4272M 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:50.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:50.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:50 smithi062 ceph-mon[35304]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 -- 172.21.15.12:0/2385204844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510108ef0 msgr2=0x7f7510109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 --2- 172.21.15.12:0/2385204844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510108ef0 0x7f7510109300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f75000099c0 tx=0x7f7500009cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 -- 172.21.15.12:0/2385204844 shutdown_connections 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 --2- 172.21.15.12:0/2385204844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7510109840 0x7f75101111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 --2- 172.21.15.12:0/2385204844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75101045c0 0x7f75101049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 --2- 172.21.15.12:0/2385204844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510108ef0 0x7f7510109300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 -- 172.21.15.12:0/2385204844 >> 172.21.15.12:0/2385204844 conn(0x7f75100ffe90 msgr2=0x7f75101022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 -- 172.21.15.12:0/2385204844 shutdown_connections 2024-05-27T07:15:50.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.806+0000 7f7518800700 1 -- 172.21.15.12:0/2385204844 wait complete. 2024-05-27T07:15:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 Processor -- start 2024-05-27T07:15:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 -- start start 2024-05-27T07:15:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75101045c0 0x7f751019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7510108ef0 0x7f751019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510109840 0x7f75101a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7510114300 con 0x7f7510109840 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7510114180 con 0x7f75101045c0 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.808+0000 7f7518800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7510114480 con 0x7f7510108ef0 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510109840 0x7f75101a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510109840 0x7f75101a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49462/0 (socket says 172.21.15.12:49462) 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 -- 172.21.15.12:0/2974174658 learned_addr learned my addr 172.21.15.12:0/2974174658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7515e00700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75101045c0 0x7f751019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7515400700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7510108ef0 0x7f751019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 -- 172.21.15.12:0/2974174658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7510108ef0 msgr2=0x7f751019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7510108ef0 0x7f751019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 -- 172.21.15.12:0/2974174658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75101045c0 msgr2=0x7f751019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75101045c0 0x7f751019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:50.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7516800700 1 -- 172.21.15.12:0/2974174658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75101a1dc0 con 0x7f7510109840 2024-05-27T07:15:50.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7515400700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7510108ef0 0x7f751019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:50.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.809+0000 7f7515e00700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75101045c0 0x7f751019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:50.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.810+0000 7f7516800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510109840 0x7f75101a1720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f750c007ee0 tx=0x7f750c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:50.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.810+0000 7f750aa00700 1 -- 172.21.15.12:0/2974174658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c00dbd0 con 0x7f7510109840 2024-05-27T07:15:50.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.810+0000 7f750aa00700 1 -- 172.21.15.12:0/2974174658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f750c00dd70 con 0x7f7510109840 2024-05-27T07:15:50.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.810+0000 7f750aa00700 1 -- 172.21.15.12:0/2974174658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c0058d0 con 0x7f7510109840 2024-05-27T07:15:50.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.810+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75101a20b0 con 0x7f7510109840 2024-05-27T07:15:50.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.810+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75101a99d0 con 0x7f7510109840 2024-05-27T07:15:50.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.812+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7510105a70 con 0x7f7510109840 2024-05-27T07:15:50.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.816+0000 7f750aa00700 1 -- 172.21.15.12:0/2974174658 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f750c01a020 con 0x7f7510109840 2024-05-27T07:15:50.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.818+0000 7f750aa00700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f74f80b2900 0x7f74f80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.818+0000 7f750aa00700 1 -- 172.21.15.12:0/2974174658 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f750c0e1be0 con 0x7f7510109840 2024-05-27T07:15:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.818+0000 7f7515e00700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f74f80b2900 0x7f74f80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.819+0000 7f7515e00700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f74f80b2900 0x7f74f80b4e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7504005950 tx=0x7f75040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:50.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:50.820+0000 7f750aa00700 1 -- 172.21.15.12:0/2974174658 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f750c010070 con 0x7f7510109840 2024-05-27T07:15:51.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.061+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75101a9ba0 con 0x7f7510109840 2024-05-27T07:15:51.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.063+0000 7f750aa00700 1 -- 172.21.15.12:0/2974174658 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+18419 (secure 0 0 0) 0x7f750c0a9d00 con 0x7f7510109840 2024-05-27T07:15:51.055 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:15:51.055 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":71,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:15:46.591777+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:15:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.066+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f74f80b2900 msgr2=0x7f74f80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.066+0000 7f7518800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f74f80b2900 0x7f74f80b4e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7504005950 tx=0x7f75040058e0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.066+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510109840 msgr2=0x7f75101a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.066+0000 7f7518800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510109840 0x7f75101a1720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f750c007ee0 tx=0x7f750c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 shutdown_connections 2024-05-27T07:15:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f74f80b2900 0x7f74f80b4e10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75101045c0 0x7f751019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7510109840 0x7f75101a1720 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 --2- 172.21.15.12:0/2974174658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7510108ef0 0x7f751019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 >> 172.21.15.12:0/2974174658 conn(0x7f75100ffe90 msgr2=0x7f751010a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 shutdown_connections 2024-05-27T07:15:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:51.067+0000 7f7518800700 1 -- 172.21.15.12:0/2974174658 wait complete. 2024-05-27T07:15:51.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:51 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2974174658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:15:51.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:51 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2974174658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:15:51.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:51 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2974174658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:15:51.860 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:10:07.718700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T07:15:51.861 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-27T07:15:52.329 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:52.638 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:52 smithi012 ceph-mon[31473]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:52.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:52 smithi162 ceph-mon[35676]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:52.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:52 smithi062 ceph-mon[35304]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:54.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 -- 172.21.15.12:0/999952110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc848075b50 msgr2=0x7fc848075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:54.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 --2- 172.21.15.12:0/999952110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc848075b50 0x7fc848075f60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009990 tx=0x7fc83c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 -- 172.21.15.12:0/999952110 shutdown_connections 2024-05-27T07:15:54.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 --2- 172.21.15.12:0/999952110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc848115170 0x7fc848117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 --2- 172.21.15.12:0/999952110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc848075b50 0x7fc848075f60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 --2- 172.21.15.12:0/999952110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8480764a0 0x7fc848114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 -- 172.21.15.12:0/999952110 >> 172.21.15.12:0/999952110 conn(0x7fc848079d00 msgr2=0x7fc84807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:54.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 -- 172.21.15.12:0/999952110 shutdown_connections 2024-05-27T07:15:54.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.451+0000 7fc84e800700 1 -- 172.21.15.12:0/999952110 wait complete. 2024-05-27T07:15:54.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.452+0000 7fc84e800700 1 Processor -- start 2024-05-27T07:15:54.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.452+0000 7fc84e800700 1 -- start start 2024-05-27T07:15:54.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84e800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc848075b50 0x7fc8481a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:54.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8480764a0 0x7fc8481a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:54.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc848115170 0x7fc8481a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:54.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc84811a800 con 0x7fc8480764a0 2024-05-27T07:15:54.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84e800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc84811a680 con 0x7fc848115170 2024-05-27T07:15:54.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84e800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc84811a980 con 0x7fc848075b50 2024-05-27T07:15:54.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8480764a0 0x7fc8481a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:54.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8480764a0 0x7fc8481a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43432/0 (socket says 172.21.15.12:43432) 2024-05-27T07:15:54.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84ca00700 1 -- 172.21.15.12:0/1455304144 learned_addr learned my addr 172.21.15.12:0/1455304144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:15:54.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84d400700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc848075b50 0x7fc8481a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:54.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.453+0000 7fc84de00700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc848115170 0x7fc8481a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc84ca00700 1 -- 172.21.15.12:0/1455304144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc848075b50 msgr2=0x7fc8481a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc84ca00700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc848075b50 0x7fc8481a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc84ca00700 1 -- 172.21.15.12:0/1455304144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc848115170 msgr2=0x7fc8481a5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc84ca00700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc848115170 0x7fc8481a5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc84ca00700 1 -- 172.21.15.12:0/1455304144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8481a6030 con 0x7fc8480764a0 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc84ca00700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8480764a0 0x7fc8481a17b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c9b0 tx=0x7fc83400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc846000700 1 -- 172.21.15.12:0/1455304144 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834007b00 con 0x7fc8480764a0 2024-05-27T07:15:54.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.454+0000 7fc846000700 1 -- 172.21.15.12:0/1455304144 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc834007ca0 con 0x7fc8480764a0 2024-05-27T07:15:54.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.455+0000 7fc846000700 1 -- 172.21.15.12:0/1455304144 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834018660 con 0x7fc8480764a0 2024-05-27T07:15:54.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.455+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8481a6320 con 0x7fc8480764a0 2024-05-27T07:15:54.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.455+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8481a6810 con 0x7fc8480764a0 2024-05-27T07:15:54.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.456+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc848077290 con 0x7fc8480764a0 2024-05-27T07:15:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.459+0000 7fc846000700 1 -- 172.21.15.12:0/1455304144 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc83400f450 con 0x7fc8480764a0 2024-05-27T07:15:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.460+0000 7fc846000700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc8300b2900 0x7fc8300b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.460+0000 7fc846000700 1 -- 172.21.15.12:0/1455304144 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fc8340e1fe0 con 0x7fc8480764a0 2024-05-27T07:15:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.461+0000 7fc84d400700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc8300b2900 0x7fc8300b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:54.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.461+0000 7fc84d400700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc8300b2900 0x7fc8300b4e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc83c006010 tx=0x7fc83c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.463+0000 7fc846000700 1 -- 172.21.15.12:0/1455304144 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc834014030 con 0x7fc8480764a0 2024-05-27T07:15:54.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.700+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8481a25a0 con 0x7fc8480764a0 2024-05-27T07:15:54.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.701+0000 7fc846000700 1 -- 172.21.15.12:0/1455304144 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7fc8340b2020 con 0x7fc8480764a0 2024-05-27T07:15:54.693 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-05-27T07:15:54.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.704+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc8300b2900 msgr2=0x7fc8300b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:54.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.704+0000 7fc84e800700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc8300b2900 0x7fc8300b4e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc83c006010 tx=0x7fc83c00b540 comp rx=0 tx=0).stop 2024-05-27T07:15:54.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.704+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8480764a0 msgr2=0x7fc8481a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.704+0000 7fc84e800700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8480764a0 0x7fc8481a17b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c9b0 tx=0x7fc83400ccc0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 shutdown_connections 2024-05-27T07:15:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc8300b2900 0x7fc8300b4e10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc848115170 0x7fc8481a5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc848075b50 0x7fc8481a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 --2- 172.21.15.12:0/1455304144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8480764a0 0x7fc8481a17b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:54.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 >> 172.21.15.12:0/1455304144 conn(0x7fc848079d00 msgr2=0x7fc848100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 shutdown_connections 2024-05-27T07:15:54.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:15:54.705+0000 7fc84e800700 1 -- 172.21.15.12:0/1455304144 wait complete. 2024-05-27T07:15:54.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:54 smithi062 ceph-mon[35304]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:54.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:54 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1455304144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:15:55.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:54 smithi012 ceph-mon[31473]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:55.077 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:54 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1455304144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:15:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:54 smithi162 ceph-mon[35676]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:54 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1455304144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:15:55.453 INFO:tasks.cephadm:Adding mds.a on smithi012 2024-05-27T07:15:55.453 INFO:tasks.cephadm:Adding mds.d on smithi012 2024-05-27T07:15:55.453 INFO:tasks.cephadm:Adding mds.g on smithi012 2024-05-27T07:15:55.453 INFO:tasks.cephadm:Adding mds.j on smithi012 2024-05-27T07:15:55.453 INFO:tasks.cephadm:Adding mds.b on smithi062 2024-05-27T07:15:55.454 INFO:tasks.cephadm:Adding mds.e on smithi062 2024-05-27T07:15:55.454 INFO:tasks.cephadm:Adding mds.h on smithi062 2024-05-27T07:15:55.454 INFO:tasks.cephadm:Adding mds.k on smithi062 2024-05-27T07:15:55.454 INFO:tasks.cephadm:Adding mds.c on smithi162 2024-05-27T07:15:55.454 INFO:tasks.cephadm:Adding mds.f on smithi162 2024-05-27T07:15:55.454 INFO:tasks.cephadm:Adding mds.i on smithi162 2024-05-27T07:15:55.454 INFO:tasks.cephadm:Adding mds.l on smithi162 2024-05-27T07:15:55.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l' 2024-05-27T07:15:55.912 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:15:57.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:56 smithi012 ceph-mon[31473]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:56 smithi162 ceph-mon[35676]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:57.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:56 smithi062 ceph-mon[35304]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:58.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.146+0000 7feb03e00700 1 -- 172.21.15.162:0/2609570973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc10a0f0 msgr2=0x7feafc10c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.146+0000 7feb03e00700 1 --2- 172.21.15.162:0/2609570973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc10a0f0 0x7feafc10c510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feaf400b0a0 tx=0x7feaf400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.147+0000 7feb03e00700 1 -- 172.21.15.162:0/2609570973 shutdown_connections 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.147+0000 7feb03e00700 1 --2- 172.21.15.162:0/2609570973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feafc104e30 0x7feafc107250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.147+0000 7feb03e00700 1 --2- 172.21.15.162:0/2609570973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc10a0f0 0x7feafc10c510 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.147+0000 7feb03e00700 1 --2- 172.21.15.162:0/2609570973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feafc107790 0x7feafc109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.147+0000 7feb03e00700 1 -- 172.21.15.162:0/2609570973 >> 172.21.15.162:0/2609570973 conn(0x7feafc0fe870 msgr2=0x7feafc100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.147+0000 7feb03e00700 1 -- 172.21.15.162:0/2609570973 shutdown_connections 2024-05-27T07:15:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.147+0000 7feb03e00700 1 -- 172.21.15.162:0/2609570973 wait complete. 2024-05-27T07:15:58.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.148+0000 7feb03e00700 1 Processor -- start 2024-05-27T07:15:58.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.148+0000 7feb03e00700 1 -- start start 2024-05-27T07:15:58.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb03e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feafc104e30 0x7feafc19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:58.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb03e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc107790 0x7feafc19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:58.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb03e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feafc10a0f0 0x7feafc1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:58.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb03e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10fc80 con 0x7feafc10a0f0 2024-05-27T07:15:58.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb03e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10fb00 con 0x7feafc104e30 2024-05-27T07:15:58.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb03e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10fe00 con 0x7feafc107790 2024-05-27T07:15:58.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb00a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc107790 0x7feafc19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:58.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb00a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc107790 0x7feafc19ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:52044/0 (socket says 172.21.15.162:52044) 2024-05-27T07:15:58.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.149+0000 7feb00a00700 1 -- 172.21.15.162:0/1096057449 learned_addr learned my addr 172.21.15.162:0/1096057449 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-05-27T07:15:58.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb01400700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feafc104e30 0x7feafc19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:58.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb01e00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feafc10a0f0 0x7feafc1a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:58.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb00a00700 1 -- 172.21.15.162:0/1096057449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feafc104e30 msgr2=0x7feafc19a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:58.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb00a00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feafc104e30 0x7feafc19a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb00a00700 1 -- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feafc10a0f0 msgr2=0x7feafc1a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:58.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb00a00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feafc10a0f0 0x7feafc1a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb00a00700 1 -- 172.21.15.162:0/1096057449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc1a2330 con 0x7feafc107790 2024-05-27T07:15:58.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feb00a00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc107790 0x7feafc19ad10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feae800d950 tx=0x7feae800dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:58.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.150+0000 7feafa000700 1 -- 172.21.15.162:0/1096057449 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae8016070 con 0x7feafc107790 2024-05-27T07:15:58.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.151+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feafc1a2620 con 0x7feafc107790 2024-05-27T07:15:58.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.151+0000 7feafa000700 1 -- 172.21.15.162:0/1096057449 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feae8010460 con 0x7feafc107790 2024-05-27T07:15:58.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.151+0000 7feafa000700 1 -- 172.21.15.162:0/1096057449 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae800f620 con 0x7feafc107790 2024-05-27T07:15:58.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.151+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feafc1a2b60 con 0x7feafc107790 2024-05-27T07:15:58.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.152+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feafc0512b0 con 0x7feafc107790 2024-05-27T07:15:58.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.155+0000 7feafa000700 1 -- 172.21.15.162:0/1096057449 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7feae8019070 con 0x7feafc107790 2024-05-27T07:15:58.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.157+0000 7feafa000700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7feae40b2880 0x7feae40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:58.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.157+0000 7feafa000700 1 -- 172.21.15.162:0/1096057449 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7feae80e1980 con 0x7feafc107790 2024-05-27T07:15:58.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.158+0000 7feb01400700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7feae40b2880 0x7feae40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:58.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.159+0000 7feb01400700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7feae40b2880 0x7feae40b4d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feaf0005fd0 tx=0x7feaf000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:58.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.160+0000 7feafa000700 1 -- 172.21.15.162:0/1096057449 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feae80a9b40 con 0x7feafc107790 2024-05-27T07:15:58.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.405+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}) -- 0x7feafc063db0 con 0x7feae40b2880 2024-05-27T07:15:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.423+0000 7feafa000700 1 -- 172.21.15.162:0/1096057449 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7feafc063db0 con 0x7feae40b2880 2024-05-27T07:15:58.414 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mds.all update... 2024-05-27T07:15:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7feae40b2880 msgr2=0x7feae40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7feae40b2880 0x7feae40b4d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feaf0005fd0 tx=0x7feaf000a560 comp rx=0 tx=0).stop 2024-05-27T07:15:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc107790 msgr2=0x7feafc19ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc107790 0x7feafc19ad10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feae800d950 tx=0x7feae800dd10 comp rx=0 tx=0).stop 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 shutdown_connections 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7feae40b2880 0x7feae40b4d90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feafc104e30 0x7feafc19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feafc10a0f0 0x7feafc1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 --2- 172.21.15.162:0/1096057449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feafc107790 0x7feafc19ad10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 >> 172.21.15.162:0/1096057449 conn(0x7feafc0fe870 msgr2=0x7feafc105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 shutdown_connections 2024-05-27T07:15:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-05-27T07:15:58.427+0000 7feb03e00700 1 -- 172.21.15.162:0/1096057449 wait complete. 2024-05-27T07:15:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:15:59.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:15:59.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:59.211 DEBUG:teuthology.orchestra.run.smithi012:mds.a> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service 2024-05-27T07:15:59.220 DEBUG:teuthology.orchestra.run.smithi012:mds.d> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service 2024-05-27T07:15:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:15:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:15:59.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:15:59.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:15:59.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:59.265 DEBUG:teuthology.orchestra.run.smithi012:mds.g> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service 2024-05-27T07:15:59.311 DEBUG:teuthology.orchestra.run.smithi012:mds.j> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service 2024-05-27T07:15:59.358 DEBUG:teuthology.orchestra.run.smithi062:mds.b> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.b.service 2024-05-27T07:15:59.367 DEBUG:teuthology.orchestra.run.smithi062:mds.e> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.e.service 2024-05-27T07:15:59.413 DEBUG:teuthology.orchestra.run.smithi062:mds.h> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.h.service 2024-05-27T07:15:59.459 DEBUG:teuthology.orchestra.run.smithi062:mds.k> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service 2024-05-27T07:15:59.505 DEBUG:teuthology.orchestra.run.smithi162:mds.c> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service 2024-05-27T07:15:59.513 DEBUG:teuthology.orchestra.run.smithi162:mds.f> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service 2024-05-27T07:15:59.557 DEBUG:teuthology.orchestra.run.smithi162:mds.i> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service 2024-05-27T07:15:59.603 DEBUG:teuthology.orchestra.run.smithi162:mds.l> sudo journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service 2024-05-27T07:15:59.648 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-27T07:15:59.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:00.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:59 smithi012 ceph-mon[31473]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:00.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:59 smithi012 ceph-mon[31473]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-05-27T07:16:00.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:15:59 smithi012 ceph-mon[31473]: Deploying daemon mds.l on smithi162 2024-05-27T07:16:00.115 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:00.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:59 smithi162 ceph-mon[35676]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:00.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:59 smithi162 ceph-mon[35676]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-05-27T07:16:00.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:15:59 smithi162 ceph-mon[35676]: Deploying daemon mds.l on smithi162 2024-05-27T07:16:00.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:59 smithi062 ceph-mon[35304]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:00.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:59 smithi062 ceph-mon[35304]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-05-27T07:16:00.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:15:59 smithi062 ceph-mon[35304]: Deploying daemon mds.l on smithi162 2024-05-27T07:16:01.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:00 smithi012 ceph-mon[31473]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:01.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:00 smithi162 ceph-mon[35676]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:00 smithi062 ceph-mon[35304]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:01.970 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:01 smithi162 systemd[1]: Starting Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:02.224 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:01 smithi162 podman[65692]: 2024-05-27 07:16:01.882905503 +0000 UTC m=+0.033534509 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:02.224 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:01 smithi162 podman[65692]: 2024-05-27 07:16:01.999422961 +0000 UTC m=+0.150052005 container create e113b963d71c3b70e6706dca4fb32c103f93a75ac379fb0e06ac7bc04171b6d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 -- 172.21.15.12:0/196843667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 msgr2=0x7fa00c117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 --2- 172.21.15.12:0/196843667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c117800 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa00400b0a0 tx=0x7fa00400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 -- 172.21.15.12:0/196843667 shutdown_connections 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 --2- 172.21.15.12:0/196843667 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c117800 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 --2- 172.21.15.12:0/196843667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa00c110960 0x7fa00c110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 --2- 172.21.15.12:0/196843667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c10a980 0x7fa00c10ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 -- 172.21.15.12:0/196843667 >> 172.21.15.12:0/196843667 conn(0x7fa00c077c70 msgr2=0x7fa00c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:02.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 -- 172.21.15.12:0/196843667 shutdown_connections 2024-05-27T07:16:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.300+0000 7fa013800700 1 -- 172.21.15.12:0/196843667 wait complete. 2024-05-27T07:16:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.302+0000 7fa013800700 1 Processor -- start 2024-05-27T07:16:02.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.302+0000 7fa013800700 1 -- start start 2024-05-27T07:16:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa013800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa00c10a980 0x7fa00c1a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa013800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa013800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c110960 0x7fa00c1a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa013800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c11a9b0 con 0x7fa00c10b2d0 2024-05-27T07:16:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa013800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c11a830 con 0x7fa00c110960 2024-05-27T07:16:02.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa013800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c11ab30 con 0x7fa00c10a980 2024-05-27T07:16:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa00be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa00be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c1a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60300/0 (socket says 172.21.15.12:60300) 2024-05-27T07:16:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa00be00700 1 -- 172.21.15.12:0/336482984 learned_addr learned my addr 172.21.15.12:0/336482984 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa010e00700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa00c10a980 0x7fa00c1a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.303+0000 7fa011800700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c110960 0x7fa00c1a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.304+0000 7fa00be00700 1 -- 172.21.15.12:0/336482984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa00c10a980 msgr2=0x7fa00c1a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:02.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.304+0000 7fa00be00700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa00c10a980 0x7fa00c1a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.304+0000 7fa00be00700 1 -- 172.21.15.12:0/336482984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c110960 msgr2=0x7fa00c1a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.304+0000 7fa00be00700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c110960 0x7fa00c1a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.304+0000 7fa00be00700 1 -- 172.21.15.12:0/336482984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c1a6160 con 0x7fa00c10b2d0 2024-05-27T07:16:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.304+0000 7fa00be00700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c1a18e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c9e0 tx=0x7f9ff800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.304+0000 7fa009600700 1 -- 172.21.15.12:0/336482984 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8007aa0 con 0x7fa00c10b2d0 2024-05-27T07:16:02.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.305+0000 7fa009600700 1 -- 172.21.15.12:0/336482984 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9ff8007c40 con 0x7fa00c10b2d0 2024-05-27T07:16:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.305+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00c1a6450 con 0x7fa00c10b2d0 2024-05-27T07:16:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.305+0000 7fa009600700 1 -- 172.21.15.12:0/336482984 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8018660 con 0x7fa00c10b2d0 2024-05-27T07:16:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.305+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00c1a6990 con 0x7fa00c10b2d0 2024-05-27T07:16:02.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.306+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa00c0512b0 con 0x7fa00c10b2d0 2024-05-27T07:16:02.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.309+0000 7fa009600700 1 -- 172.21.15.12:0/336482984 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9ff800f450 con 0x7fa00c10b2d0 2024-05-27T07:16:02.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.310+0000 7fa009600700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ff40b2880 0x7f9ff40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:02.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.310+0000 7fa009600700 1 -- 172.21.15.12:0/336482984 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f9ff80e1df0 con 0x7fa00c10b2d0 2024-05-27T07:16:02.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.311+0000 7fa010e00700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ff40b2880 0x7f9ff40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:02.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.311+0000 7fa010e00700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ff40b2880 0x7f9ff40b4d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa00c07d0e0 tx=0x7fa00000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:02.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.315+0000 7fa009600700 1 -- 172.21.15.12:0/336482984 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9ff8014030 con 0x7fa00c10b2d0 2024-05-27T07:16:02.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.607+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa00c1a26a0 con 0x7fa00c10b2d0 2024-05-27T07:16:02.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.608+0000 7fa009600700 1 -- 172.21.15.12:0/336482984 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f9ff80a9f10 con 0x7fa00c10b2d0 2024-05-27T07:16:02.599 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:02.599 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.611+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ff40b2880 msgr2=0x7f9ff40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.611+0000 7fa013800700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ff40b2880 0x7f9ff40b4d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa00c07d0e0 tx=0x7fa00000b3c0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.611+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 msgr2=0x7fa00c1a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.611+0000 7fa013800700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c1a18e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c9e0 tx=0x7f9ff800ccf0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 shutdown_connections 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ff40b2880 0x7f9ff40b4d90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c110960 0x7fa00c1a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa00c10a980 0x7fa00c1a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 --2- 172.21.15.12:0/336482984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa00c10b2d0 0x7fa00c1a18e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 >> 172.21.15.12:0/336482984 conn(0x7fa00c077c70 msgr2=0x7fa00c07b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 shutdown_connections 2024-05-27T07:16:02.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:02.612+0000 7fa013800700 1 -- 172.21.15.12:0/336482984 wait complete. 2024-05-27T07:16:02.605 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 2 2024-05-27T07:16:02.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:02 smithi162 podman[65692]: 2024-05-27 07:16:02.287640732 +0000 UTC m=+0.438269783 container init e113b963d71c3b70e6706dca4fb32c103f93a75ac379fb0e06ac7bc04171b6d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:16:02.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:02 smithi162 podman[65692]: 2024-05-27 07:16:02.299442097 +0000 UTC m=+0.450071158 container start e113b963d71c3b70e6706dca4fb32c103f93a75ac379fb0e06ac7bc04171b6d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-27T07:16:02.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:02 smithi162 bash[65692]: e113b963d71c3b70e6706dca4fb32c103f93a75ac379fb0e06ac7bc04171b6d6 2024-05-27T07:16:02.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:02 smithi162 systemd[1]: Started Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:02.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mds[65711]: -- 172.21.15.162:0/3312352115 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56392c1b1a40 con 0x56392c1f9180 2024-05-27T07:16:02.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:16:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: starting mds.l at 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/1597502189,v1:172.21.15.162:6833/1597502189] up:boot 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: fsmap 1 up:standby 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:02.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:02.979 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:02 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/336482984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:03.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:03.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/1597502189,v1:172.21.15.162:6833/1597502189] up:boot 2024-05-27T07:16:03.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: fsmap 1 up:standby 2024-05-27T07:16:03.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:16:03.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:03.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:03.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:03.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:03.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:03.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:03.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:02 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/336482984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/1597502189,v1:172.21.15.162:6833/1597502189] up:boot 2024-05-27T07:16:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: fsmap 1 up:standby 2024-05-27T07:16:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:16:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:03.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:03.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:03.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:03.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:03.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:03.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:02 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/336482984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:03.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:03 smithi062 ceph-mon[35304]: Deploying daemon mds.k on smithi062 2024-05-27T07:16:04.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:03 smithi012 ceph-mon[31473]: Deploying daemon mds.k on smithi062 2024-05-27T07:16:04.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:03 smithi162 ceph-mon[35676]: Deploying daemon mds.k on smithi062 2024-05-27T07:16:04.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:04 smithi062 ceph-mon[35304]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:04 smithi012 ceph-mon[31473]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:05.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:04 smithi162 ceph-mon[35676]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:05.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:06.736 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:16:06 smithi062 podman[65214]: 2024-05-27 07:16:06.544448346 +0000 UTC m=+0.211654559 container create ff5ae73d73948dc6eed798250867092a4332530841a9ac50dc91f46d303a9f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:16:07.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:06 smithi012 ceph-mon[31473]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:06 smithi162 ceph-mon[35676]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:06 smithi062 ceph-mon[35304]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:07.237 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:16:06 smithi062 podman[65214]: 2024-05-27 07:16:06.806917613 +0000 UTC m=+0.474123869 container init ff5ae73d73948dc6eed798250867092a4332530841a9ac50dc91f46d303a9f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:16:07.237 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:16:06 smithi062 podman[65214]: 2024-05-27 07:16:06.819955509 +0000 UTC m=+0.487161693 container start ff5ae73d73948dc6eed798250867092a4332530841a9ac50dc91f46d303a9f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:16:07.237 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:16:06 smithi062 ceph-mds[65242]: -- 172.21.15.62:0/3706181113 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c802b1fa40 con 0x55c802b67600 2024-05-27T07:16:07.237 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:16:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: starting mds.k at 2024-05-27T07:16:07.237 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:16:06 smithi062 bash[65214]: ff5ae73d73948dc6eed798250867092a4332530841a9ac50dc91f46d303a9f20 2024-05-27T07:16:07.237 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:16:06 smithi062 systemd[1]: Started Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.013+0000 7f8cbce00700 1 -- 172.21.15.12:0/4214978535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 msgr2=0x7f8cb810c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.013+0000 7f8cbce00700 1 --2- 172.21.15.12:0/4214978535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb810c9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009990 tx=0x7f8ca8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.013+0000 7f8cbce00700 1 -- 172.21.15.12:0/4214978535 shutdown_connections 2024-05-27T07:16:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.013+0000 7f8cbce00700 1 --2- 172.21.15.12:0/4214978535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cb8106ec0 0x7f8cb8113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.013+0000 7f8cbce00700 1 --2- 172.21.15.12:0/4214978535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb810c9c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.013+0000 7f8cbce00700 1 --2- 172.21.15.12:0/4214978535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb8106570 0x7f8cb8106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.013+0000 7f8cbce00700 1 -- 172.21.15.12:0/4214978535 >> 172.21.15.12:0/4214978535 conn(0x7f8cb8102050 msgr2=0x7f8cb81044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.014+0000 7f8cbce00700 1 -- 172.21.15.12:0/4214978535 shutdown_connections 2024-05-27T07:16:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.014+0000 7f8cbce00700 1 -- 172.21.15.12:0/4214978535 wait complete. 2024-05-27T07:16:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.015+0000 7f8cbce00700 1 Processor -- start 2024-05-27T07:16:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.015+0000 7f8cbce00700 1 -- start start 2024-05-27T07:16:08.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.016+0000 7f8cbce00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cb8106570 0x7f8cb819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:08.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.016+0000 7f8cbce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb8106ec0 0x7f8cb819d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:08.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.016+0000 7f8cbce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:08.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.016+0000 7f8cbce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb81165c0 con 0x7f8cb810c5b0 2024-05-27T07:16:08.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.016+0000 7f8cbce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb8116440 con 0x7f8cb8106ec0 2024-05-27T07:16:08.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.016+0000 7f8cbce00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb8116740 con 0x7f8cb8106570 2024-05-27T07:16:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb81a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60310/0 (socket says 172.21.15.12:60310) 2024-05-27T07:16:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 -- 172.21.15.12:0/1829728335 learned_addr learned my addr 172.21.15.12:0/1829728335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb5e00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cb8106570 0x7f8cb819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb5400700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb8106ec0 0x7f8cb819d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 -- 172.21.15.12:0/1829728335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cb8106570 msgr2=0x7f8cb819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cb8106570 0x7f8cb819cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 -- 172.21.15.12:0/1829728335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb8106ec0 msgr2=0x7f8cb819d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb8106ec0 0x7f8cb819d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb6800700 1 -- 172.21.15.12:0/1829728335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb81a1e20 con 0x7f8cb810c5b0 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb5e00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cb8106570 0x7f8cb819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.017+0000 7f8cb5400700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb8106ec0 0x7f8cb819d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.018+0000 7f8cb6800700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb81a1780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0007ee0 tx=0x7f8cb000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.018+0000 7f8caea00700 1 -- 172.21.15.12:0/1829728335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb000dbd0 con 0x7f8cb810c5b0 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.018+0000 7f8caea00700 1 -- 172.21.15.12:0/1829728335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8cb000dd70 con 0x7f8cb810c5b0 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.018+0000 7f8caea00700 1 -- 172.21.15.12:0/1829728335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb00058d0 con 0x7f8cb810c5b0 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.018+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb81a2110 con 0x7f8cb810c5b0 2024-05-27T07:16:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.018+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb81a9a30 con 0x7f8cb810c5b0 2024-05-27T07:16:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.020+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb80512b0 con 0x7f8cb810c5b0 2024-05-27T07:16:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.022+0000 7f8caea00700 1 -- 172.21.15.12:0/1829728335 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8cb001a020 con 0x7f8cb810c5b0 2024-05-27T07:16:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.024+0000 7f8caea00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c9c0b2900 0x7f8c9c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.024+0000 7f8caea00700 1 -- 172.21.15.12:0/1829728335 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f8cb00e1c80 con 0x7f8cb810c5b0 2024-05-27T07:16:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.024+0000 7f8cb5e00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c9c0b2900 0x7f8c9c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.025+0000 7f8cb5e00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c9c0b2900 0x7f8c9c0b4e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8006010 tx=0x7f8ca800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.027+0000 7f8caea00700 1 -- 172.21.15.12:0/1829728335 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8cb0010070 con 0x7f8cb810c5b0 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: Deploying daemon mds.j on smithi012 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:08.312 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6832/1529598052,v1:172.21.15.62:6833/1529598052] up:boot 2024-05-27T07:16:08.313 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: fsmap 2 up:standby 2024-05-27T07:16:08.313 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:16:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.328+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cb802e5e0 con 0x7f8cb810c5b0 2024-05-27T07:16:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.329+0000 7f8caea00700 1 -- 172.21.15.12:0/1829728335 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f8cb00a9da0 con 0x7f8cb810c5b0 2024-05-27T07:16:08.321 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:08.321 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T07:16:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c9c0b2900 msgr2=0x7f8c9c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c9c0b2900 0x7f8c9c0b4e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8006010 tx=0x7f8ca800b540 comp rx=0 tx=0).stop 2024-05-27T07:16:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 msgr2=0x7f8cb81a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb81a1780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0007ee0 tx=0x7f8cb000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 shutdown_connections 2024-05-27T07:16:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c9c0b2900 0x7f8c9c0b4e10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb810c5b0 0x7f8cb81a1780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cb8106570 0x7f8cb819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 --2- 172.21.15.12:0/1829728335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb8106ec0 0x7f8cb819d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.332+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 >> 172.21.15.12:0/1829728335 conn(0x7f8cb8102050 msgr2=0x7f8cb8103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.333+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 shutdown_connections 2024-05-27T07:16:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:08.333+0000 7f8cbce00700 1 -- 172.21.15.12:0/1829728335 wait complete. 2024-05-27T07:16:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-05-27T07:16:08.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: Deploying daemon mds.j on smithi012 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6832/1529598052,v1:172.21.15.62:6833/1529598052] up:boot 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: fsmap 2 up:standby 2024-05-27T07:16:08.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:16:08.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:08.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:08.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:08.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:08.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: Deploying daemon mds.j on smithi012 2024-05-27T07:16:08.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:08.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6832/1529598052,v1:172.21.15.62:6833/1529598052] up:boot 2024-05-27T07:16:08.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: fsmap 2 up:standby 2024-05-27T07:16:08.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:16:09.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:09 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1829728335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:09.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:09 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1829728335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:09.518 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:09 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1829728335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:10.311 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:10 smithi012 ceph-mon[31473]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:10.408 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:10 smithi062 ceph-mon[35304]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:10.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:10 smithi162 ceph-mon[35676]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:10.591 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:10 smithi012 systemd[1]: Starting Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:11.020 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:10 smithi012 podman[71919]: 2024-05-27 07:16:10.659139818 +0000 UTC m=+0.033523577 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:11.021 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:10 smithi012 podman[71919]: 2024-05-27 07:16:10.766670208 +0000 UTC m=+0.141054005 container create f20e208d4b35181eec0e98b46cea3908edcd2325685a809881a08ebe75fac05f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:16:11.021 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:11 smithi012 podman[71919]: 2024-05-27 07:16:11.029187567 +0000 UTC m=+0.403571408 container init f20e208d4b35181eec0e98b46cea3908edcd2325685a809881a08ebe75fac05f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:16:11.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:11 smithi012 podman[71919]: 2024-05-27 07:16:11.041331506 +0000 UTC m=+0.415715329 container start f20e208d4b35181eec0e98b46cea3908edcd2325685a809881a08ebe75fac05f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:16:11.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:11 smithi012 bash[71919]: f20e208d4b35181eec0e98b46cea3908edcd2325685a809881a08ebe75fac05f 2024-05-27T07:16:11.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:11 smithi012 systemd[1]: Started Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:11.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:11 smithi012 ceph-mds[71938]: -- 172.21.15.12:0/2099804907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c07f46ba40 con 0x55c07f4b3a80 2024-05-27T07:16:11.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:16:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: starting mds.j at 2024-05-27T07:16:12.088 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:12.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/2219705283,v1:172.21.15.12:6835/2219705283] up:boot 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: fsmap 3 up:standby 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: Deploying daemon mds.d on smithi012 2024-05-27T07:16:12.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:12 smithi062 ceph-mon[35304]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:12.498 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/2219705283,v1:172.21.15.12:6835/2219705283] up:boot 2024-05-27T07:16:12.498 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: fsmap 3 up:standby 2024-05-27T07:16:12.498 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:16:12.498 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.498 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:12.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:12.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:12.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: Deploying daemon mds.d on smithi012 2024-05-27T07:16:12.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:12 smithi012 ceph-mon[31473]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:12.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/2219705283,v1:172.21.15.12:6835/2219705283] up:boot 2024-05-27T07:16:12.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: fsmap 3 up:standby 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: Deploying daemon mds.d on smithi012 2024-05-27T07:16:12.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:12 smithi162 ceph-mon[35676]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:14 smithi012 systemd[1]: Starting Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:14.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:14 smithi162 ceph-mon[35676]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:15.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:14 smithi012 ceph-mon[31473]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:14 smithi062 ceph-mon[35304]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:15.394 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 podman[72315]: 2024-05-27 07:16:15.018051173 +0000 UTC m=+0.035797896 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:15.394 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 podman[72315]: 2024-05-27 07:16:15.146842836 +0000 UTC m=+0.164589598 container create bdd79a7e3870a44300c72d72894968d4b4dd64f2e326d4ffa15510e1cece4146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:16:15.648 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 podman[72315]: 2024-05-27 07:16:15.646004552 +0000 UTC m=+0.663751331 container init bdd79a7e3870a44300c72d72894968d4b4dd64f2e326d4ffa15510e1cece4146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-27T07:16:15.953 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 podman[72315]: 2024-05-27 07:16:15.657784 +0000 UTC m=+0.675530733 container start bdd79a7e3870a44300c72d72894968d4b4dd64f2e326d4ffa15510e1cece4146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:16:15.954 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 ceph-mds[72352]: -- 172.21.15.12:0/3958816266 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560cbd03fa40 con 0x560cbd087a80 2024-05-27T07:16:15.954 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: starting mds.d at 2024-05-27T07:16:15.954 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 bash[72315]: bdd79a7e3870a44300c72d72894968d4b4dd64f2e326d4ffa15510e1cece4146 2024-05-27T07:16:15.954 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:16:15 smithi012 systemd[1]: Started Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 -- 172.21.15.12:0/2306906413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106ec0 msgr2=0x7f33d4113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 --2- 172.21.15.12:0/2306906413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106ec0 0x7f33d4113350 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00b0a0 tx=0x7f33cc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 -- 172.21.15.12:0/2306906413 shutdown_connections 2024-05-27T07:16:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 --2- 172.21.15.12:0/2306906413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106ec0 0x7f33d4113350 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 --2- 172.21.15.12:0/2306906413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d410c5b0 0x7f33d410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 --2- 172.21.15.12:0/2306906413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d4106570 0x7f33d4106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 -- 172.21.15.12:0/2306906413 >> 172.21.15.12:0/2306906413 conn(0x7f33d4102070 msgr2=0x7f33d41044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:16.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 -- 172.21.15.12:0/2306906413 shutdown_connections 2024-05-27T07:16:16.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.048+0000 7f33dc000700 1 -- 172.21.15.12:0/2306906413 wait complete. 2024-05-27T07:16:16.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.049+0000 7f33dc000700 1 Processor -- start 2024-05-27T07:16:16.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.049+0000 7f33dc000700 1 -- start start 2024-05-27T07:16:16.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33dc000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106570 0x7f33d419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:16.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33dc000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d4106ec0 0x7f33d419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:16.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33dc000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d410c5b0 0x7f33d41a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:16.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33dc000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f33d4116590 con 0x7f33d4106570 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33dc000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f33d4116410 con 0x7f33d410c5b0 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33dc000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f33d4116710 con 0x7f33d4106ec0 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33d9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106570 0x7f33d419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33d9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106570 0x7f33d419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55910/0 (socket says 172.21.15.12:55910) 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.050+0000 7f33d9600700 1 -- 172.21.15.12:0/288994521 learned_addr learned my addr 172.21.15.12:0/288994521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33d8c00700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d4106ec0 0x7f33d419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33da000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d410c5b0 0x7f33d41a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33d9600700 1 -- 172.21.15.12:0/288994521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d4106ec0 msgr2=0x7f33d419d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33d9600700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d4106ec0 0x7f33d419d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33d9600700 1 -- 172.21.15.12:0/288994521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d410c5b0 msgr2=0x7f33d41a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33d9600700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d410c5b0 0x7f33d41a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33d9600700 1 -- 172.21.15.12:0/288994521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d41a1cc0 con 0x7f33d4106570 2024-05-27T07:16:16.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.051+0000 7f33da000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d410c5b0 0x7f33d41a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:16:16.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.052+0000 7f33d9600700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106570 0x7f33d419cea0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c9e0 tx=0x7f33c800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:16.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.052+0000 7f33d2000700 1 -- 172.21.15.12:0/288994521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8007aa0 con 0x7f33d4106570 2024-05-27T07:16:16.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.052+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d41a1fb0 con 0x7f33d4106570 2024-05-27T07:16:16.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.052+0000 7f33d2000700 1 -- 172.21.15.12:0/288994521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33c8007c40 con 0x7f33d4106570 2024-05-27T07:16:16.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.052+0000 7f33d2000700 1 -- 172.21.15.12:0/288994521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8018660 con 0x7f33d4106570 2024-05-27T07:16:16.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.052+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d41a9980 con 0x7f33d4106570 2024-05-27T07:16:16.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.054+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d40512b0 con 0x7f33d4106570 2024-05-27T07:16:16.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.056+0000 7f33d2000700 1 -- 172.21.15.12:0/288994521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f33c800f450 con 0x7f33d4106570 2024-05-27T07:16:16.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.058+0000 7f33d2000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f33bc0b2880 0x7f33bc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:16.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.058+0000 7f33d2000700 1 -- 172.21.15.12:0/288994521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f33c80e1d30 con 0x7f33d4106570 2024-05-27T07:16:16.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.058+0000 7f33d8c00700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f33bc0b2880 0x7f33bc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:16.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.059+0000 7f33d8c00700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f33bc0b2880 0x7f33bc0b4d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f33d4103bf0 tx=0x7f33c000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:16.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.063+0000 7f33d2000700 1 -- 172.21.15.12:0/288994521 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f33c8014030 con 0x7f33d4106570 2024-05-27T07:16:16.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.351+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33d419e200 con 0x7f33d4106570 2024-05-27T07:16:16.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.352+0000 7f33d2000700 1 -- 172.21.15.12:0/288994521 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f33c80a9e50 con 0x7f33d4106570 2024-05-27T07:16:16.344 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:16.344 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.355+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f33bc0b2880 msgr2=0x7f33bc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.355+0000 7f33dc000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f33bc0b2880 0x7f33bc0b4d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f33d4103bf0 tx=0x7f33c000a380 comp rx=0 tx=0).stop 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.355+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106570 msgr2=0x7f33d419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.355+0000 7f33dc000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106570 0x7f33d419cea0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c9e0 tx=0x7f33c800ccf0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 shutdown_connections 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f33bc0b2880 0x7f33bc0b4d90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d410c5b0 0x7f33d41a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33d4106570 0x7f33d419cea0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 --2- 172.21.15.12:0/288994521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d4106ec0 0x7f33d419d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 >> 172.21.15.12:0/288994521 conn(0x7f33d4102070 msgr2=0x7f33d41046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 shutdown_connections 2024-05-27T07:16:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:16.356+0000 7f33dc000700 1 -- 172.21.15.12:0/288994521 wait complete. 2024-05-27T07:16:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 4 2024-05-27T07:16:17.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/288994521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6836/4064394372,v1:172.21.15.12:6837/4064394372] up:boot 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: fsmap 4 up:standby 2024-05-27T07:16:17.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:16:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/288994521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6836/4064394372,v1:172.21.15.12:6837/4064394372] up:boot 2024-05-27T07:16:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: fsmap 4 up:standby 2024-05-27T07:16:17.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:16:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:17.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:17.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:17.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:17.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/288994521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:17.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6836/4064394372,v1:172.21.15.12:6837/4064394372] up:boot 2024-05-27T07:16:17.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: fsmap 4 up:standby 2024-05-27T07:16:17.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:16:18.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:17 smithi012 ceph-mon[31473]: Deploying daemon mds.e on smithi062 2024-05-27T07:16:18.464 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:17 smithi062 ceph-mon[35304]: Deploying daemon mds.e on smithi062 2024-05-27T07:16:18.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:17 smithi162 ceph-mon[35676]: Deploying daemon mds.e on smithi062 2024-05-27T07:16:19.078 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:19.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:18 smithi012 ceph-mon[31473]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:19.352 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:18 smithi062 ceph-mon[35304]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:19.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:18 smithi162 ceph-mon[35676]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:19.567 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:19.736 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:19 smithi062 systemd[1]: Starting Ceph mds.e for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:20.005 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:19 smithi062 podman[65526]: 2024-05-27 07:16:19.653940731 +0000 UTC m=+0.038868610 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:20.005 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:19 smithi062 podman[65526]: 2024-05-27 07:16:19.80957138 +0000 UTC m=+0.194499294 container create c6bd01fa6bb2f14932df7b4e355ec8be844d3ad2ca4d49d54a7c816b524cc1a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:16:20.469 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:20 smithi062 podman[65526]: 2024-05-27 07:16:20.145079518 +0000 UTC m=+0.530007472 container init c6bd01fa6bb2f14932df7b4e355ec8be844d3ad2ca4d49d54a7c816b524cc1a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-27T07:16:20.469 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:20 smithi062 podman[65526]: 2024-05-27 07:16:20.157019254 +0000 UTC m=+0.541947152 container start c6bd01fa6bb2f14932df7b4e355ec8be844d3ad2ca4d49d54a7c816b524cc1a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:16:20.469 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:20 smithi062 bash[65526]: c6bd01fa6bb2f14932df7b4e355ec8be844d3ad2ca4d49d54a7c816b524cc1a2 2024-05-27T07:16:20.469 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:20 smithi062 systemd[1]: Started Ceph mds.e for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:20.469 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:20 smithi062 ceph-mds[65545]: -- 172.21.15.62:0/416962833 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624b2801a40 con 0x5624b2849180 2024-05-27T07:16:20.470 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:16:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e[65541]: starting mds.e at 2024-05-27T07:16:21.267 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:21 smithi012 ceph-mon[31473]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:21.267 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:21.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:21.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:21.471 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:21 smithi162 ceph-mon[35676]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:21.471 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:21.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:21.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:21.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:21 smithi062 ceph-mon[35304]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:21.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:21.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:21.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:21.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 -- 172.21.15.12:0/2179587479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4109840 msgr2=0x7f43d41111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 --2- 172.21.15.12:0/2179587479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4109840 0x7f43d41111a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00b0d0 tx=0x7f43cc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 -- 172.21.15.12:0/2179587479 shutdown_connections 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 --2- 172.21.15.12:0/2179587479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d41045c0 0x7f43d41049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 --2- 172.21.15.12:0/2179587479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4109840 0x7f43d41111a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 --2- 172.21.15.12:0/2179587479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43d4108ef0 0x7f43d4109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 -- 172.21.15.12:0/2179587479 >> 172.21.15.12:0/2179587479 conn(0x7f43d40ffe90 msgr2=0x7f43d41022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 -- 172.21.15.12:0/2179587479 shutdown_connections 2024-05-27T07:16:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.776+0000 7f43da400700 1 -- 172.21.15.12:0/2179587479 wait complete. 2024-05-27T07:16:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.778+0000 7f43da400700 1 Processor -- start 2024-05-27T07:16:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.778+0000 7f43da400700 1 -- start start 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43da400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43d41045c0 0x7f43d419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43da400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4108ef0 0x7f43d419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43da400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d4109840 0x7f43d41a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43da400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f43d4114320 con 0x7f43d4108ef0 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43da400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41141a0 con 0x7f43d4109840 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43da400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f43d41144a0 con 0x7f43d41045c0 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4108ef0 0x7f43d419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4108ef0 0x7f43d419d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55920/0 (socket says 172.21.15.12:55920) 2024-05-27T07:16:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 -- 172.21.15.12:0/3744311072 learned_addr learned my addr 172.21.15.12:0/3744311072 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43d3400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43d41045c0 0x7f43d419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43d3e00700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d4109840 0x7f43d41a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 -- 172.21.15.12:0/3744311072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43d41045c0 msgr2=0x7f43d419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43d41045c0 0x7f43d419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 -- 172.21.15.12:0/3744311072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d4109840 msgr2=0x7f43d41a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d4109840 0x7f43d41a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.779+0000 7f43cbe00700 1 -- 172.21.15.12:0/3744311072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d41a1c90 con 0x7f43d4108ef0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.780+0000 7f43cbe00700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4108ef0 0x7f43d419d410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00c9e0 tx=0x7f43bc00cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.780+0000 7f43d0c00700 1 -- 172.21.15.12:0/3744311072 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43bc007aa0 con 0x7f43d4108ef0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.780+0000 7f43d0c00700 1 -- 172.21.15.12:0/3744311072 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43bc007c40 con 0x7f43d4108ef0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.780+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d41a1f80 con 0x7f43d4108ef0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.780+0000 7f43d0c00700 1 -- 172.21.15.12:0/3744311072 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43bc018660 con 0x7f43d4108ef0 2024-05-27T07:16:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.780+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d41a24c0 con 0x7f43d4108ef0 2024-05-27T07:16:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.782+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d40512b0 con 0x7f43d4108ef0 2024-05-27T07:16:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.785+0000 7f43d0c00700 1 -- 172.21.15.12:0/3744311072 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f43bc018800 con 0x7f43d4108ef0 2024-05-27T07:16:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.786+0000 7f43d0c00700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f43b80b2880 0x7f43b80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.786+0000 7f43d0c00700 1 -- 172.21.15.12:0/3744311072 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f43bc0e1f30 con 0x7f43d4108ef0 2024-05-27T07:16:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.787+0000 7f43d3400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f43b80b2880 0x7f43b80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.787+0000 7f43d3400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f43b80b2880 0x7f43b80b4d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f43c4005950 tx=0x7f43c40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:21.790+0000 7f43d0c00700 1 -- 172.21.15.12:0/3744311072 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f43bc054df0 con 0x7f43d4108ef0 2024-05-27T07:16:22.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.088+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43d419e1d0 con 0x7f43d4108ef0 2024-05-27T07:16:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:22 smithi012 ceph-mon[31473]: Deploying daemon mds.f on smithi162 2024-05-27T07:16:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:22 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] up:boot 2024-05-27T07:16:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:22 smithi012 ceph-mon[31473]: fsmap 5 up:standby 2024-05-27T07:16:22.080 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:16:22.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.090+0000 7f43d0c00700 1 -- 172.21.15.12:0/3744311072 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f43bc014030 con 0x7f43d4108ef0 2024-05-27T07:16:22.081 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:22.081 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-27T07:16:22.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f43b80b2880 msgr2=0x7f43b80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:22.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f43b80b2880 0x7f43b80b4d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f43c4005950 tx=0x7f43c40058e0 comp rx=0 tx=0).stop 2024-05-27T07:16:22.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4108ef0 msgr2=0x7f43d419d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:22.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4108ef0 0x7f43d419d410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00c9e0 tx=0x7f43bc00cda0 comp rx=0 tx=0).stop 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 shutdown_connections 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f43b80b2880 0x7f43b80b4d90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d4109840 0x7f43d41a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43d41045c0 0x7f43d419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 --2- 172.21.15.12:0/3744311072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d4108ef0 0x7f43d419d410 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.093+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 >> 172.21.15.12:0/3744311072 conn(0x7f43d40ffe90 msgr2=0x7f43d4110ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.094+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 shutdown_connections 2024-05-27T07:16:22.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:22.094+0000 7f43da400700 1 -- 172.21.15.12:0/3744311072 wait complete. 2024-05-27T07:16:22.086 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-05-27T07:16:22.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:22 smithi162 ceph-mon[35676]: Deploying daemon mds.f on smithi162 2024-05-27T07:16:22.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:22 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] up:boot 2024-05-27T07:16:22.206 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:22 smithi162 ceph-mon[35676]: fsmap 5 up:standby 2024-05-27T07:16:22.206 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:16:22.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:22 smithi062 ceph-mon[35304]: Deploying daemon mds.f on smithi162 2024-05-27T07:16:22.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:22 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] up:boot 2024-05-27T07:16:22.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:22 smithi062 ceph-mon[35304]: fsmap 5 up:standby 2024-05-27T07:16:22.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:16:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:23 smithi012 ceph-mon[31473]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:23 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3744311072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:23.327 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:23 smithi162 ceph-mon[35676]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:23.328 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:23 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3744311072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:23.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:23 smithi062 ceph-mon[35304]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:23.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:23 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3744311072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:23.693 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:23 smithi162 systemd[1]: Starting Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:23.973 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:23 smithi162 podman[66007]: 2024-05-27 07:16:23.617300935 +0000 UTC m=+0.037294100 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:23.973 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:23 smithi162 podman[66007]: 2024-05-27 07:16:23.711869101 +0000 UTC m=+0.131862277 container create ebb2488adfa96edf6a986f3a5fdae35b7f280bde596321f51c981f12b3c6080d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-27T07:16:23.973 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:23 smithi162 podman[66007]: 2024-05-27 07:16:23.982705852 +0000 UTC m=+0.402699049 container init ebb2488adfa96edf6a986f3a5fdae35b7f280bde596321f51c981f12b3c6080d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:16:24.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:23 smithi162 podman[66007]: 2024-05-27 07:16:23.991904348 +0000 UTC m=+0.411897562 container start ebb2488adfa96edf6a986f3a5fdae35b7f280bde596321f51c981f12b3c6080d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:16:24.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:24 smithi162 bash[66007]: ebb2488adfa96edf6a986f3a5fdae35b7f280bde596321f51c981f12b3c6080d 2024-05-27T07:16:24.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:24 smithi162 systemd[1]: Started Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:24.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:24 smithi162 ceph-mds[66030]: -- 172.21.15.162:0/202759657 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583db7dba40 con 0x5583db823180 2024-05-27T07:16:24.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:16:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: starting mds.f at 2024-05-27T07:16:24.892 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:25.214 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:25 smithi162 ceph-mon[35676]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:25.214 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:25 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.214 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:25 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.214 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:25 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.214 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:25 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:25.214 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:25 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:25.214 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:25 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:25.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:25 smithi012 ceph-mon[31473]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:25.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:25 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:25.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:25 smithi062 ceph-mon[35304]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:25.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:25 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:25 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:25 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:25.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:25 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:25.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:25 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:25.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:25 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:26.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:26 smithi012 ceph-mon[31473]: Deploying daemon mds.c on smithi162 2024-05-27T07:16:26.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:26 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6834/2577053624,v1:172.21.15.162:6835/2577053624] up:boot 2024-05-27T07:16:26.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:26 smithi012 ceph-mon[31473]: fsmap 6 up:standby 2024-05-27T07:16:26.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:26 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:16:26.252 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:26 smithi162 ceph-mon[35676]: Deploying daemon mds.c on smithi162 2024-05-27T07:16:26.252 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:26 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6834/2577053624,v1:172.21.15.162:6835/2577053624] up:boot 2024-05-27T07:16:26.252 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:26 smithi162 ceph-mon[35676]: fsmap 6 up:standby 2024-05-27T07:16:26.252 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:26 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:16:26.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:26 smithi062 ceph-mon[35304]: Deploying daemon mds.c on smithi162 2024-05-27T07:16:26.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:26 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6834/2577053624,v1:172.21.15.162:6835/2577053624] up:boot 2024-05-27T07:16:26.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:26 smithi062 ceph-mon[35304]: fsmap 6 up:standby 2024-05-27T07:16:26.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:26 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:16:27.269 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:27 smithi012 ceph-mon[31473]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:27.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:27 smithi162 ceph-mon[35676]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:27.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:27 smithi062 ceph-mon[35304]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.525+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2652921530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8108580 msgr2=0x7f6dc810fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.525+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2652921530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8108580 0x7f6dc810fee0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000b0d0 tx=0x7f6dc000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.525+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2652921530 shutdown_connections 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.525+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2652921530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc8103300 0x7f6dc8103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.525+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2652921530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8108580 0x7f6dc810fee0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.525+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2652921530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc8107c30 0x7f6dc8108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.525+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2652921530 >> 172.21.15.12:0/2652921530 conn(0x7f6dc8078c30 msgr2=0x7f6dc8079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.526+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2652921530 shutdown_connections 2024-05-27T07:16:27.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.526+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2652921530 wait complete. 2024-05-27T07:16:27.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 Processor -- start 2024-05-27T07:16:27.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 -- start start 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc8103300 0x7f6dc8112cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8107c30 0x7f6dc81131f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc8108580 0x7f6dc8113730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc806c500 con 0x7f6dc8107c30 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc806c380 con 0x7f6dc8103300 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.527+0000 7f6dc7e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc806c680 con 0x7f6dc8108580 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.528+0000 7f6dc6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8107c30 0x7f6dc81131f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.528+0000 7f6dc6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8107c30 0x7f6dc81131f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38612/0 (socket says 172.21.15.12:38612) 2024-05-27T07:16:27.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.528+0000 7f6dc6000700 1 -- 172.21.15.12:0/2361610700 learned_addr learned my addr 172.21.15.12:0/2361610700 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:27.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.528+0000 7f6dc7400700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc8108580 0x7f6dc8113730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:27.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.528+0000 7f6dc6a00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc8103300 0x7f6dc8112cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:27.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc6000700 1 -- 172.21.15.12:0/2361610700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc8108580 msgr2=0x7f6dc8113730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc6000700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc8108580 0x7f6dc8113730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc6000700 1 -- 172.21.15.12:0/2361610700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc8103300 msgr2=0x7f6dc8112cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc6000700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc8103300 0x7f6dc8112cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc6000700 1 -- 172.21.15.12:0/2361610700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc81a8f40 con 0x7f6dc8107c30 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc7400700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc8108580 0x7f6dc8113730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc6000700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8107c30 0x7f6dc81131f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6db400d980 tx=0x7f6db400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dbf400700 1 -- 172.21.15.12:0/2361610700 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4017070 con 0x7f6dc8107c30 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dbf400700 1 -- 172.21.15.12:0/2361610700 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6db4009dc0 con 0x7f6dc8107c30 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc81a9230 con 0x7f6dc8107c30 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.529+0000 7f6dbf400700 1 -- 172.21.15.12:0/2361610700 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db400f3c0 con 0x7f6dc8107c30 2024-05-27T07:16:27.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.530+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc81a9770 con 0x7f6dc8107c30 2024-05-27T07:16:27.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.531+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc80512b0 con 0x7f6dc8107c30 2024-05-27T07:16:27.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.534+0000 7f6dbf400700 1 -- 172.21.15.12:0/2361610700 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f6db4009930 con 0x7f6dc8107c30 2024-05-27T07:16:27.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.536+0000 7f6dbf400700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6dac0b2880 0x7f6dac0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:27.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.536+0000 7f6dbf400700 1 -- 172.21.15.12:0/2361610700 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f6db40e18b0 con 0x7f6dc8107c30 2024-05-27T07:16:27.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.536+0000 7f6dc6a00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6dac0b2880 0x7f6dac0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:27.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.537+0000 7f6dc6a00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6dac0b2880 0x7f6dac0b4d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6db8005950 tx=0x7f6db80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:27.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.541+0000 7f6dbf400700 1 -- 172.21.15.12:0/2361610700 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6db40a99d0 con 0x7f6dc8107c30 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:27 smithi162 systemd[1]: Starting Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.826+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dc81a99d0 con 0x7f6dc8107c30 2024-05-27T07:16:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.827+0000 7f6dbf400700 1 -- 172.21.15.12:0/2361610700 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f6db40a92d0 con 0x7f6dc8107c30 2024-05-27T07:16:27.819 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:27.819 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-27T07:16:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6dac0b2880 msgr2=0x7f6dac0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6dac0b2880 0x7f6dac0b4d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6db8005950 tx=0x7f6db80058e0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8107c30 msgr2=0x7f6dc81131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8107c30 0x7f6dc81131f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6db400d980 tx=0x7f6db400dc90 comp rx=0 tx=0).stop 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 shutdown_connections 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6dac0b2880 0x7f6dac0b4d90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc8103300 0x7f6dc8112cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dc8108580 0x7f6dc8113730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 --2- 172.21.15.12:0/2361610700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6dc8107c30 0x7f6dc81131f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 >> 172.21.15.12:0/2361610700 conn(0x7f6dc8078c30 msgr2=0x7f6dc80fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 shutdown_connections 2024-05-27T07:16:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:27.831+0000 7f6dc7e00700 1 -- 172.21.15.12:0/2361610700 wait complete. 2024-05-27T07:16:27.824 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-05-27T07:16:28.047 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:27 smithi162 podman[66315]: 2024-05-27 07:16:27.744135753 +0000 UTC m=+0.126187925 container create 6506cb73b5802bfcc5f2990960e0fe055e2012aac24d358877c6401761fae3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:16:28.047 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:27 smithi162 podman[66315]: 2024-05-27 07:16:27.650713232 +0000 UTC m=+0.032765365 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:28.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:28 smithi012 ceph-mon[31473]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:28.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:28 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2361610700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:28.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:28 smithi162 ceph-mon[35676]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:28.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:28 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2361610700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:28.475 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:28 smithi162 podman[66315]: 2024-05-27 07:16:28.077642464 +0000 UTC m=+0.459694616 container init 6506cb73b5802bfcc5f2990960e0fe055e2012aac24d358877c6401761fae3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T07:16:28.475 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:28 smithi162 podman[66315]: 2024-05-27 07:16:28.087583766 +0000 UTC m=+0.469635945 container start 6506cb73b5802bfcc5f2990960e0fe055e2012aac24d358877c6401761fae3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:16:28.475 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:28 smithi162 bash[66315]: 6506cb73b5802bfcc5f2990960e0fe055e2012aac24d358877c6401761fae3d0 2024-05-27T07:16:28.475 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:28 smithi162 systemd[1]: Started Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:28.475 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:28 smithi162 ceph-mds[66334]: -- 172.21.15.162:0/3633383636 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631c7fc9a40 con 0x5631c8011600 2024-05-27T07:16:28.475 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:16:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: starting mds.c at 2024-05-27T07:16:28.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:28 smithi062 ceph-mon[35304]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:28.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:28 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2361610700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:29.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:29.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: Deploying daemon mds.h on smithi062 2024-05-27T07:16:29.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/1296782695,v1:172.21.15.162:6837/1296782695] up:boot 2024-05-27T07:16:29.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: fsmap 7 up:standby 2024-05-27T07:16:29.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:29 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:29.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: Deploying daemon mds.h on smithi062 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/1296782695,v1:172.21.15.162:6837/1296782695] up:boot 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: fsmap 7 up:standby 2024-05-27T07:16:29.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:29 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: Deploying daemon mds.h on smithi062 2024-05-27T07:16:29.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/1296782695,v1:172.21.15.162:6837/1296782695] up:boot 2024-05-27T07:16:29.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: fsmap 7 up:standby 2024-05-27T07:16:29.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:29 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:30.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:30 smithi012 ceph-mon[31473]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:30.660 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:30.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:30 smithi162 ceph-mon[35676]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:30.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:30 smithi062 ceph-mon[35304]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:31.986 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:31 smithi062 systemd[1]: Starting Ceph mds.h for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:32.454 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 podman[65824]: 2024-05-27 07:16:32.016934946 +0000 UTC m=+0.043322295 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:32.455 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 podman[65824]: 2024-05-27 07:16:32.144627909 +0000 UTC m=+0.171015272 container create 6a0fbf2d97ca3acdb56b1a6d99ad025ead46245de5c0da57c74d79ec05d01fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:16:32.455 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 podman[65824]: 2024-05-27 07:16:32.462822844 +0000 UTC m=+0.489210213 container init 6a0fbf2d97ca3acdb56b1a6d99ad025ead46245de5c0da57c74d79ec05d01fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-27T07:16:32.737 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 podman[65824]: 2024-05-27 07:16:32.47434091 +0000 UTC m=+0.500728259 container start 6a0fbf2d97ca3acdb56b1a6d99ad025ead46245de5c0da57c74d79ec05d01fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-27T07:16:32.737 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 bash[65824]: 6a0fbf2d97ca3acdb56b1a6d99ad025ead46245de5c0da57c74d79ec05d01fbf 2024-05-27T07:16:32.737 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 systemd[1]: Started Ceph mds.h for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:32.737 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 ceph-mds[65844]: -- 172.21.15.62:0/1374739208 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559d6d009a40 con 0x559d6d051a80 2024-05-27T07:16:32.737 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:16:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h[65840]: starting mds.h at 2024-05-27T07:16:33.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:32 smithi012 ceph-mon[31473]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:33.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:33.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:32 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:32 smithi162 ceph-mon[35676]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:32 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:32 smithi062 ceph-mon[35304]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:32 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.345+0000 7f6556e00700 1 -- 172.21.15.12:0/3747750543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65500764a0 msgr2=0x7f6550114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.345+0000 7f6556e00700 1 --2- 172.21.15.12:0/3747750543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65500764a0 0x7f6550114e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6538009990 tx=0x7f6538009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.345+0000 7f6556e00700 1 -- 172.21.15.12:0/3747750543 shutdown_connections 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.345+0000 7f6556e00700 1 --2- 172.21.15.12:0/3747750543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6550115360 0x7f6550117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.345+0000 7f6556e00700 1 --2- 172.21.15.12:0/3747750543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6550075b50 0x7f6550075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.345+0000 7f6556e00700 1 --2- 172.21.15.12:0/3747750543 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65500764a0 0x7f6550114e20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.345+0000 7f6556e00700 1 -- 172.21.15.12:0/3747750543 >> 172.21.15.12:0/3747750543 conn(0x7f6550100030 msgr2=0x7f6550102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:33.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.346+0000 7f6556e00700 1 -- 172.21.15.12:0/3747750543 shutdown_connections 2024-05-27T07:16:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.346+0000 7f6556e00700 1 -- 172.21.15.12:0/3747750543 wait complete. 2024-05-27T07:16:33.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.347+0000 7f6556e00700 1 Processor -- start 2024-05-27T07:16:33.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.347+0000 7f6556e00700 1 -- start start 2024-05-27T07:16:33.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6556e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6550075b50 0x7f65501a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6556e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65500764a0 0x7f65501a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6556e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6550115360 0x7f65501a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6556e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f655011a960 con 0x7f6550115360 2024-05-27T07:16:33.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6556e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f655011a7e0 con 0x7f65500764a0 2024-05-27T07:16:33.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6556e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f655011aae0 con 0x7f6550075b50 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6554e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6550115360 0x7f65501a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6554e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6550115360 0x7f65501a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49550/0 (socket says 172.21.15.12:49550) 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6554e00700 1 -- 172.21.15.12:0/3280720309 learned_addr learned my addr 172.21.15.12:0/3280720309 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f654fe00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6550075b50 0x7f65501a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.348+0000 7f6547400700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65500764a0 0x7f65501a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f6554e00700 1 -- 172.21.15.12:0/3280720309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6550075b50 msgr2=0x7f65501a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f6554e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6550075b50 0x7f65501a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f6554e00700 1 -- 172.21.15.12:0/3280720309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65500764a0 msgr2=0x7f65501a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f6554e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65500764a0 0x7f65501a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f6554e00700 1 -- 172.21.15.12:0/3280720309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65501a6240 con 0x7f6550115360 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f6554e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6550115360 0x7f65501a5ba0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f654800d980 tx=0x7f654800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f654d600700 1 -- 172.21.15.12:0/3280720309 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548017070 con 0x7f6550115360 2024-05-27T07:16:33.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f654d600700 1 -- 172.21.15.12:0/3280720309 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6548005020 con 0x7f6550115360 2024-05-27T07:16:33.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f654d600700 1 -- 172.21.15.12:0/3280720309 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654800f980 con 0x7f6550115360 2024-05-27T07:16:33.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.349+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65501a6530 con 0x7f6550115360 2024-05-27T07:16:33.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.350+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65501ade50 con 0x7f6550115360 2024-05-27T07:16:33.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.351+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6550077290 con 0x7f6550115360 2024-05-27T07:16:33.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.353+0000 7f654d600700 1 -- 172.21.15.12:0/3280720309 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f6548004a30 con 0x7f6550115360 2024-05-27T07:16:33.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.355+0000 7f654d600700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65340b2900 0x7f65340b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.355+0000 7f654d600700 1 -- 172.21.15.12:0/3280720309 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f65480e1ab0 con 0x7f6550115360 2024-05-27T07:16:33.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.355+0000 7f654fe00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65340b2900 0x7f65340b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.356+0000 7f654fe00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65340b2900 0x7f65340b4e10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6540009550 tx=0x7f65400091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:33.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.360+0000 7f654d600700 1 -- 172.21.15.12:0/3280720309 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6548020020 con 0x7f6550115360 2024-05-27T07:16:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.648+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65501a26d0 con 0x7f6550115360 2024-05-27T07:16:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.649+0000 7f654d600700 1 -- 172.21.15.12:0/3280720309 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f65480a9bd0 con 0x7f6550115360 2024-05-27T07:16:33.640 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:33.640 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34439,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65340b2900 msgr2=0x7f65340b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65340b2900 0x7f65340b4e10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6540009550 tx=0x7f65400091b0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6550115360 msgr2=0x7f65501a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6550115360 0x7f65501a5ba0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f654800d980 tx=0x7f654800dc90 comp rx=0 tx=0).stop 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 shutdown_connections 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65340b2900 0x7f65340b4e10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6550115360 0x7f65501a5ba0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6550075b50 0x7f65501a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 --2- 172.21.15.12:0/3280720309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65500764a0 0x7f65501a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.652+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 >> 172.21.15.12:0/3280720309 conn(0x7f6550100030 msgr2=0x7f6550100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.653+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 shutdown_connections 2024-05-27T07:16:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:33.653+0000 7f6556e00700 1 -- 172.21.15.12:0/3280720309 wait complete. 2024-05-27T07:16:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-05-27T07:16:34.002 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:33 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:34.003 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:33 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] up:boot 2024-05-27T07:16:34.003 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:33 smithi012 ceph-mon[31473]: fsmap 8 up:standby 2024-05-27T07:16:34.003 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:33 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:16:34.003 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:33 smithi012 ceph-mon[31473]: Deploying daemon mds.i on smithi162 2024-05-27T07:16:34.003 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:33 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3280720309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:34.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:33 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:34.110 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:33 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] up:boot 2024-05-27T07:16:34.110 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:33 smithi162 ceph-mon[35676]: fsmap 8 up:standby 2024-05-27T07:16:34.110 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:33 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:16:34.110 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:33 smithi162 ceph-mon[35676]: Deploying daemon mds.i on smithi162 2024-05-27T07:16:34.110 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:33 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3280720309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:33 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:33 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] up:boot 2024-05-27T07:16:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:33 smithi062 ceph-mon[35304]: fsmap 8 up:standby 2024-05-27T07:16:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:33 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:16:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:33 smithi062 ceph-mon[35304]: Deploying daemon mds.i on smithi162 2024-05-27T07:16:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:33 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3280720309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:34.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:34 smithi162 ceph-mon[35676]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:35.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:34 smithi012 ceph-mon[31473]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:35.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:34 smithi062 ceph-mon[35304]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:36.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:35 smithi162 systemd[1]: Starting Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:36.465 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:36.580 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 podman[66614]: 2024-05-27 07:16:36.177529534 +0000 UTC m=+0.035264027 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:36.580 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 podman[66614]: 2024-05-27 07:16:36.29345781 +0000 UTC m=+0.151192326 container create ce8edb43337e4d539418b9f61955d3cc9ab7e398a4e42d124eea1f21e14b9bc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T07:16:36.580 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 podman[66614]: 2024-05-27 07:16:36.58928204 +0000 UTC m=+0.447016609 container init ce8edb43337e4d539418b9f61955d3cc9ab7e398a4e42d124eea1f21e14b9bc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:16:36.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:36 smithi012 ceph-mon[31473]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:36.926 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:36.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:36 smithi162 ceph-mon[35676]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:36.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 podman[66614]: 2024-05-27 07:16:36.600615397 +0000 UTC m=+0.458349919 container start ce8edb43337e4d539418b9f61955d3cc9ab7e398a4e42d124eea1f21e14b9bc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:16:36.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 bash[66614]: ce8edb43337e4d539418b9f61955d3cc9ab7e398a4e42d124eea1f21e14b9bc4 2024-05-27T07:16:36.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 systemd[1]: Started Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:36.975 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 ceph-mds[66633]: -- 172.21.15.162:0/3936894556 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623a9c93a40 con 0x5623a9cdb180 2024-05-27T07:16:36.975 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:16:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: starting mds.i at 2024-05-27T07:16:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:36 smithi062 ceph-mon[35304]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6838/884400330,v1:172.21.15.162:6839/884400330] up:boot 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: fsmap 9 up:standby 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:38.073 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:37 smithi012 ceph-mon[31473]: Deploying daemon mds.g on smithi012 2024-05-27T07:16:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6838/884400330,v1:172.21.15.162:6839/884400330] up:boot 2024-05-27T07:16:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: fsmap 9 up:standby 2024-05-27T07:16:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:16:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:38.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:38.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:38.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:37 smithi162 ceph-mon[35676]: Deploying daemon mds.g on smithi012 2024-05-27T07:16:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6838/884400330,v1:172.21.15.162:6839/884400330] up:boot 2024-05-27T07:16:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: fsmap 9 up:standby 2024-05-27T07:16:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:16:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:38.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:38.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:38.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:38.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:37 smithi062 ceph-mon[35304]: Deploying daemon mds.g on smithi012 2024-05-27T07:16:38.872 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:38 smithi012 ceph-mon[31473]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:39.138 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:38 smithi062 ceph-mon[35304]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:39.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:38 smithi162 ceph-mon[35676]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.574+0000 7f4e02600700 1 -- 172.21.15.12:0/2515581468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 msgr2=0x7f4dfc10c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.574+0000 7f4e02600700 1 --2- 172.21.15.12:0/2515581468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc10c9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009990 tx=0x7f4dec009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.575+0000 7f4e02600700 1 -- 172.21.15.12:0/2515581468 shutdown_connections 2024-05-27T07:16:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.575+0000 7f4e02600700 1 --2- 172.21.15.12:0/2515581468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dfc106ec0 0x7f4dfc113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.575+0000 7f4e02600700 1 --2- 172.21.15.12:0/2515581468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc10c9c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.575+0000 7f4e02600700 1 --2- 172.21.15.12:0/2515581468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dfc106570 0x7f4dfc106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.575+0000 7f4e02600700 1 -- 172.21.15.12:0/2515581468 >> 172.21.15.12:0/2515581468 conn(0x7f4dfc102050 msgr2=0x7f4dfc1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.575+0000 7f4e02600700 1 -- 172.21.15.12:0/2515581468 shutdown_connections 2024-05-27T07:16:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.575+0000 7f4e02600700 1 -- 172.21.15.12:0/2515581468 wait complete. 2024-05-27T07:16:39.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.576+0000 7f4e02600700 1 Processor -- start 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.576+0000 7f4e02600700 1 -- start start 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4e02600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dfc106570 0x7f4dfc19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4e02600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dfc106ec0 0x7f4dfc19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4e02600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc1a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4e02600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4dfc1165c0 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4e02600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4dfc116440 con 0x7f4dfc106ec0 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4e02600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4dfc116740 con 0x7f4dfc106570 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4dfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc1a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4dfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc1a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49572/0 (socket says 172.21.15.12:49572) 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4dfbe00700 1 -- 172.21.15.12:0/2982900538 learned_addr learned my addr 172.21.15.12:0/2982900538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4dfb400700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dfc106570 0x7f4dfc19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.577+0000 7f4dfaa00700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dfc106ec0 0x7f4dfc19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.578+0000 7f4dfbe00700 1 -- 172.21.15.12:0/2982900538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dfc106570 msgr2=0x7f4dfc19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.578+0000 7f4dfbe00700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dfc106570 0x7f4dfc19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.578+0000 7f4dfbe00700 1 -- 172.21.15.12:0/2982900538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dfc106ec0 msgr2=0x7f4dfc19d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.578+0000 7f4dfbe00700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dfc106ec0 0x7f4dfc19d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.578+0000 7f4dfbe00700 1 -- 172.21.15.12:0/2982900538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dfc1a1e70 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.578+0000 7f4dfbe00700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc1a17d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4df4007ee0 tx=0x7f4df400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.578+0000 7f4df3e00700 1 -- 172.21.15.12:0/2982900538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df400dbd0 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.579+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dfc1a2160 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.579+0000 7f4df3e00700 1 -- 172.21.15.12:0/2982900538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df400dd70 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.579+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dfc1a9a80 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.579+0000 7f4df3e00700 1 -- 172.21.15.12:0/2982900538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df4005870 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.580+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dfc0512b0 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.583+0000 7f4df3e00700 1 -- 172.21.15.12:0/2982900538 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f4df401a020 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.584+0000 7f4df3e00700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4de00b2880 0x7f4de00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.584+0000 7f4df3e00700 1 -- 172.21.15.12:0/2982900538 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f4df40e1c10 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.584+0000 7f4dfb400700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4de00b2880 0x7f4de00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.585+0000 7f4dfb400700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4de00b2880 0x7f4de00b4d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00b5c0 tx=0x7f4dec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.590+0000 7f4df3e00700 1 -- 172.21.15.12:0/2982900538 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4df4010070 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.881+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dfc02e5e0 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.883+0000 7f4df3e00700 1 -- 172.21.15.12:0/2982900538 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f4df40a9d30 con 0x7f4dfc10c5b0 2024-05-27T07:16:39.874 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:39.875 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34439,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:16:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4de00b2880 msgr2=0x7f4de00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4de00b2880 0x7f4de00b4d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00b5c0 tx=0x7f4dec005fd0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 msgr2=0x7f4dfc1a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc1a17d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4df4007ee0 tx=0x7f4df400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 shutdown_connections 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4de00b2880 0x7f4de00b4d90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4dfc10c5b0 0x7f4dfc1a17d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dfc106570 0x7f4dfc19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 --2- 172.21.15.12:0/2982900538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dfc106ec0 0x7f4dfc19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.886+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 >> 172.21.15.12:0/2982900538 conn(0x7f4dfc102050 msgr2=0x7f4dfc103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.887+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 shutdown_connections 2024-05-27T07:16:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:39.887+0000 7f4e02600700 1 -- 172.21.15.12:0/2982900538 wait complete. 2024-05-27T07:16:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 10 2024-05-27T07:16:41.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:40 smithi012 ceph-mon[31473]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:41.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:40 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2982900538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:41.207 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:16:40 smithi012 podman[73305]: 2024-05-27 07:16:40.952730562 +0000 UTC m=+0.153815073 container create 49c7c89e3875cc510dee7921b9234441b4950243b659c04df6f6ee925e87c671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-05-27T07:16:41.207 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:16:41 smithi012 podman[73305]: 2024-05-27 07:16:41.215791807 +0000 UTC m=+0.416876370 container init 49c7c89e3875cc510dee7921b9234441b4950243b659c04df6f6ee925e87c671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T07:16:41.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:40 smithi162 ceph-mon[35676]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:41.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:40 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2982900538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:41.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:40 smithi062 ceph-mon[35304]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:41.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:40 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2982900538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:41.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:16:41 smithi012 podman[73305]: 2024-05-27 07:16:41.226936382 +0000 UTC m=+0.428020872 container start 49c7c89e3875cc510dee7921b9234441b4950243b659c04df6f6ee925e87c671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:16:41.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:16:41 smithi012 bash[73305]: 49c7c89e3875cc510dee7921b9234441b4950243b659c04df6f6ee925e87c671 2024-05-27T07:16:41.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:16:41 smithi012 systemd[1]: Started Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:41.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:16:41 smithi012 ceph-mds[73325]: -- 172.21.15.12:0/18964890 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56535a265a40 con 0x56535a2ada80 2024-05-27T07:16:41.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:16:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: starting mds.g at 2024-05-27T07:16:42.664 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: Deploying daemon mds.a on smithi012 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/2663671830,v1:172.21.15.12:6839/2663671830] up:boot 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: fsmap 10 up:standby 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:42 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:42.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: Deploying daemon mds.a on smithi012 2024-05-27T07:16:42.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:42.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/2663671830,v1:172.21.15.12:6839/2663671830] up:boot 2024-05-27T07:16:42.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: fsmap 10 up:standby 2024-05-27T07:16:42.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:42 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: Deploying daemon mds.a on smithi012 2024-05-27T07:16:42.747 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:42.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/2663671830,v1:172.21.15.12:6839/2663671830] up:boot 2024-05-27T07:16:42.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: fsmap 10 up:standby 2024-05-27T07:16:42.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:42 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:44.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:44 smithi012 ceph-mon[31473]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:45.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:44 smithi162 ceph-mon[35676]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:45.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:44 smithi062 ceph-mon[35304]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:45.551 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:45 smithi012 podman[73702]: 2024-05-27 07:16:45.400684492 +0000 UTC m=+0.037477486 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:45.551 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:45 smithi012 podman[73702]: 2024-05-27 07:16:45.559678375 +0000 UTC m=+0.196471401 container create de5f65ec6cc62b0b87a48521091354d24146bf437c44ade4c2d820a3036d39a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-27T07:16:46.091 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:46 smithi012 podman[73702]: 2024-05-27 07:16:46.041127702 +0000 UTC m=+0.677920697 container init de5f65ec6cc62b0b87a48521091354d24146bf437c44ade4c2d820a3036d39a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T07:16:46.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:46 smithi012 podman[73702]: 2024-05-27 07:16:46.050763712 +0000 UTC m=+0.687556719 container start de5f65ec6cc62b0b87a48521091354d24146bf437c44ade4c2d820a3036d39a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T07:16:46.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 -- 172.21.15.12:0/47195603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c109860 msgr2=0x7fe02c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 --2- 172.21.15.12:0/47195603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c109860 0x7fe02c1111c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe02400b0d0 tx=0x7fe02400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 -- 172.21.15.12:0/47195603 shutdown_connections 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 --2- 172.21.15.12:0/47195603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c109860 0x7fe02c1111c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 --2- 172.21.15.12:0/47195603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe02c1045e0 0x7fe02c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 --2- 172.21.15.12:0/47195603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe02c108f10 0x7fe02c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 -- 172.21.15.12:0/47195603 >> 172.21.15.12:0/47195603 conn(0x7fe02c100030 msgr2=0x7fe02c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 -- 172.21.15.12:0/47195603 shutdown_connections 2024-05-27T07:16:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.406+0000 7fe033000700 1 -- 172.21.15.12:0/47195603 wait complete. 2024-05-27T07:16:46.398 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mds[73745]: -- 172.21.15.12:0/3319068218 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e33df49a40 con 0x55e33df91180 2024-05-27T07:16:46.398 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: starting mds.a at 2024-05-27T07:16:46.398 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:46 smithi012 bash[73702]: de5f65ec6cc62b0b87a48521091354d24146bf437c44ade4c2d820a3036d39a8 2024-05-27T07:16:46.398 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:16:46 smithi012 systemd[1]: Started Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.407+0000 7fe033000700 1 Processor -- start 2024-05-27T07:16:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.407+0000 7fe033000700 1 -- start start 2024-05-27T07:16:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe033000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c1045e0 0x7fe02c10b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe033000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe02c108f10 0x7fe02c10b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe033000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe02c109860 0x7fe02c10bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe033000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe02c114390 con 0x7fe02c1045e0 2024-05-27T07:16:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe033000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe02c114210 con 0x7fe02c109860 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe033000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe02c114510 con 0x7fe02c108f10 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe02be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c1045e0 0x7fe02c10b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe02be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c1045e0 0x7fe02c10b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44870/0 (socket says 172.21.15.12:44870) 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.408+0000 7fe02be00700 1 -- 172.21.15.12:0/806431617 learned_addr learned my addr 172.21.15.12:0/806431617 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.409+0000 7fe02b400700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe02c108f10 0x7fe02c10b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.409+0000 7fe031000700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe02c109860 0x7fe02c10bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.409+0000 7fe02be00700 1 -- 172.21.15.12:0/806431617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe02c108f10 msgr2=0x7fe02c10b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.409+0000 7fe02be00700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe02c108f10 0x7fe02c10b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.409+0000 7fe02be00700 1 -- 172.21.15.12:0/806431617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe02c109860 msgr2=0x7fe02c10bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.409+0000 7fe02be00700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe02c109860 0x7fe02c10bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.409+0000 7fe02be00700 1 -- 172.21.15.12:0/806431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02c1ad540 con 0x7fe02c1045e0 2024-05-27T07:16:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.410+0000 7fe02be00700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c1045e0 0x7fe02c10b240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00d950 tx=0x7fe01c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.410+0000 7fe028c00700 1 -- 172.21.15.12:0/806431617 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c016070 con 0x7fe02c1045e0 2024-05-27T07:16:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.410+0000 7fe028c00700 1 -- 172.21.15.12:0/806431617 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe01c010460 con 0x7fe02c1045e0 2024-05-27T07:16:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.410+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02c1ad7d0 con 0x7fe02c1045e0 2024-05-27T07:16:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.410+0000 7fe028c00700 1 -- 172.21.15.12:0/806431617 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c00f620 con 0x7fe02c1045e0 2024-05-27T07:16:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.410+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe02c1add10 con 0x7fe02c1045e0 2024-05-27T07:16:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.412+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe02c105a90 con 0x7fe02c1045e0 2024-05-27T07:16:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.415+0000 7fe028c00700 1 -- 172.21.15.12:0/806431617 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe01c019070 con 0x7fe02c1045e0 2024-05-27T07:16:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.416+0000 7fe028c00700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0100b2880 0x7fe0100b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.416+0000 7fe028c00700 1 -- 172.21.15.12:0/806431617 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fe01c0e1ec0 con 0x7fe02c1045e0 2024-05-27T07:16:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.417+0000 7fe02b400700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0100b2880 0x7fe0100b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.418+0000 7fe02b400700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0100b2880 0x7fe0100b4d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe018005950 tx=0x7fe0180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.419+0000 7fe028c00700 1 -- 172.21.15.12:0/806431617 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe01c0a9fe0 con 0x7fe02c1045e0 2024-05-27T07:16:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.709+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe02c0512b0 con 0x7fe02c1045e0 2024-05-27T07:16:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.710+0000 7fe028c00700 1 -- 172.21.15.12:0/806431617 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fe02c0512b0 con 0x7fe02c1045e0 2024-05-27T07:16:46.701 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:46.701 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34439,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:16:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0100b2880 msgr2=0x7fe0100b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0100b2880 0x7fe0100b4d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe018005950 tx=0x7fe0180058e0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c1045e0 msgr2=0x7fe02c10b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c1045e0 0x7fe02c10b240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00d950 tx=0x7fe01c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:16:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 shutdown_connections 2024-05-27T07:16:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0100b2880 0x7fe0100b4d90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe02c109860 0x7fe02c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe02c1045e0 0x7fe02c10b240 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 --2- 172.21.15.12:0/806431617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe02c108f10 0x7fe02c10b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 >> 172.21.15.12:0/806431617 conn(0x7fe02c100030 msgr2=0x7fe02c110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.714+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 shutdown_connections 2024-05-27T07:16:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:46.715+0000 7fe033000700 1 -- 172.21.15.12:0/806431617 wait complete. 2024-05-27T07:16:46.706 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 11 2024-05-27T07:16:47.084 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:47.084 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.084 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.084 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.084 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:47.084 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:47.085 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:47.085 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:46 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/806431617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:47.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:47.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:47.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:46 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/806431617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:47.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:46 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/806431617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:48.038 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:47 smithi062 ceph-mon[35304]: Deploying daemon mds.b on smithi062 2024-05-27T07:16:48.039 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:47 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6840/3587822997,v1:172.21.15.12:6841/3587822997] up:boot 2024-05-27T07:16:48.039 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:47 smithi062 ceph-mon[35304]: fsmap 11 up:standby 2024-05-27T07:16:48.039 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:47 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:16:48.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:47 smithi012 ceph-mon[31473]: Deploying daemon mds.b on smithi062 2024-05-27T07:16:48.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:47 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6840/3587822997,v1:172.21.15.12:6841/3587822997] up:boot 2024-05-27T07:16:48.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:47 smithi012 ceph-mon[31473]: fsmap 11 up:standby 2024-05-27T07:16:48.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:47 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:16:48.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:47 smithi162 ceph-mon[35676]: Deploying daemon mds.b on smithi062 2024-05-27T07:16:48.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:47 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6840/3587822997,v1:172.21.15.12:6841/3587822997] up:boot 2024-05-27T07:16:48.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:47 smithi162 ceph-mon[35676]: fsmap 11 up:standby 2024-05-27T07:16:48.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:47 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:16:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:48 smithi012 ceph-mon[31473]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:49.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:48 smithi162 ceph-mon[35676]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:48 smithi062 ceph-mon[35304]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:49.498 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:49.969 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:49.986 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:49 smithi062 systemd[1]: Starting Ceph mds.b for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:16:50.393 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 podman[66137]: 2024-05-27 07:16:49.936876527 +0000 UTC m=+0.038709442 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:50.393 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 podman[66137]: 2024-05-27 07:16:50.08896021 +0000 UTC m=+0.190793114 container create 7e9040046a30aa48fda6d0795ba65937c9ac8ad47d9f0645d3e606a80f43e42b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T07:16:50.393 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 podman[66137]: 2024-05-27 07:16:50.401834334 +0000 UTC m=+0.503667320 container init 7e9040046a30aa48fda6d0795ba65937c9ac8ad47d9f0645d3e606a80f43e42b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T07:16:50.736 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 podman[66137]: 2024-05-27 07:16:50.413311476 +0000 UTC m=+0.515144436 container start 7e9040046a30aa48fda6d0795ba65937c9ac8ad47d9f0645d3e606a80f43e42b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:16:50.737 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 bash[66137]: 7e9040046a30aa48fda6d0795ba65937c9ac8ad47d9f0645d3e606a80f43e42b 2024-05-27T07:16:50.737 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 systemd[1]: Started Ceph mds.b for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:16:50.737 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-mds[66156]: -- 172.21.15.62:0/3395534725 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c2db0e9a40 con 0x55c2db131a80 2024-05-27T07:16:50.737 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b[66152]: starting mds.b at 2024-05-27T07:16:51.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:50 smithi012 ceph-mon[31473]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:51.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:50 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:50 smithi162 ceph-mon[35676]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:50 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-mon[35304]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:50 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:52.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:51 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] up:boot 2024-05-27T07:16:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:51 smithi012 ceph-mon[31473]: fsmap 12 up:standby 2024-05-27T07:16:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:16:52.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:51 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:52.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:51 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] up:boot 2024-05-27T07:16:52.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:51 smithi162 ceph-mon[35676]: fsmap 12 up:standby 2024-05-27T07:16:52.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:16:52.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:51 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:52.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:51 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] up:boot 2024-05-27T07:16:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:51 smithi062 ceph-mon[35304]: fsmap 12 up:standby 2024-05-27T07:16:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:16:52.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:51 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.360+0000 7f5146e00700 1 -- 172.21.15.12:0/3639842613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105e60 msgr2=0x7f514010e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.360+0000 7f5146e00700 1 --2- 172.21.15.12:0/3639842613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105e60 0x7f514010e3b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f512c009990 tx=0x7f512c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.361+0000 7f5146e00700 1 -- 172.21.15.12:0/3639842613 shutdown_connections 2024-05-27T07:16:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.361+0000 7f5146e00700 1 --2- 172.21.15.12:0/3639842613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f514010e8f0 0x7f5140110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.361+0000 7f5146e00700 1 --2- 172.21.15.12:0/3639842613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5140105510 0x7f5140105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.361+0000 7f5146e00700 1 --2- 172.21.15.12:0/3639842613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105e60 0x7f514010e3b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.361+0000 7f5146e00700 1 -- 172.21.15.12:0/3639842613 >> 172.21.15.12:0/3639842613 conn(0x7f51400fedb0 msgr2=0x7f51401011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.361+0000 7f5146e00700 1 -- 172.21.15.12:0/3639842613 shutdown_connections 2024-05-27T07:16:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.361+0000 7f5146e00700 1 -- 172.21.15.12:0/3639842613 wait complete. 2024-05-27T07:16:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.362+0000 7f5146e00700 1 Processor -- start 2024-05-27T07:16:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.362+0000 7f5146e00700 1 -- start start 2024-05-27T07:16:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f5146e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105510 0x7f514019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f5146e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5140105e60 0x7f514019d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f5146e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f514010e8f0 0x7f51401a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f5146e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5140114330 con 0x7f5140105510 2024-05-27T07:16:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f5146e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f51401141b0 con 0x7f5140105e60 2024-05-27T07:16:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f5146e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f51401144b0 con 0x7f514010e8f0 2024-05-27T07:16:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f513fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105510 0x7f514019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f513f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5140105e60 0x7f514019d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f513fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105510 0x7f514019cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37780/0 (socket says 172.21.15.12:37780) 2024-05-27T07:16:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f513fe00700 1 -- 172.21.15.12:0/2245922976 learned_addr learned my addr 172.21.15.12:0/2245922976 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.363+0000 7f5144e00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f514010e8f0 0x7f51401a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.364+0000 7f513f400700 1 -- 172.21.15.12:0/2245922976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f514010e8f0 msgr2=0x7f51401a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.364+0000 7f513f400700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f514010e8f0 0x7f51401a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.364+0000 7f513f400700 1 -- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105510 msgr2=0x7f514019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.364+0000 7f513f400700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105510 0x7f514019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.364+0000 7f513f400700 1 -- 172.21.15.12:0/2245922976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51401a1e70 con 0x7f5140105e60 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.364+0000 7f513fe00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105510 0x7f514019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.364+0000 7f513f400700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5140105e60 0x7f514019d540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b5c0 tx=0x7f512c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.365+0000 7f513cc00700 1 -- 172.21.15.12:0/2245922976 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c026070 con 0x7f5140105e60 2024-05-27T07:16:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.365+0000 7f513cc00700 1 -- 172.21.15.12:0/2245922976 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f512c00fb00 con 0x7f5140105e60 2024-05-27T07:16:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.365+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51401a2100 con 0x7f5140105e60 2024-05-27T07:16:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.365+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51401a25e0 con 0x7f5140105e60 2024-05-27T07:16:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.365+0000 7f513cc00700 1 -- 172.21.15.12:0/2245922976 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c021b40 con 0x7f5140105e60 2024-05-27T07:16:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.366+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51400512b0 con 0x7f5140105e60 2024-05-27T07:16:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.372+0000 7f513cc00700 1 -- 172.21.15.12:0/2245922976 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f512c029070 con 0x7f5140105e60 2024-05-27T07:16:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.374+0000 7f513cc00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f51240b2880 0x7f51240b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.374+0000 7f513cc00700 1 -- 172.21.15.12:0/2245922976 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f512c0e2b50 con 0x7f5140105e60 2024-05-27T07:16:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.374+0000 7f513fe00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f51240b2880 0x7f51240b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.375+0000 7f513fe00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f51240b2880 0x7f51240b4d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5130005950 tx=0x7f513000a400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.380+0000 7f513cc00700 1 -- 172.21.15.12:0/2245922976 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f512c013070 con 0x7f5140105e60 2024-05-27T07:16:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.675+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f514002e5e0 con 0x7f5140105e60 2024-05-27T07:16:52.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.677+0000 7f513cc00700 1 -- 172.21.15.12:0/2245922976 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7f512c0aad00 con 0x7f5140105e60 2024-05-27T07:16:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:52.667 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34439,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":34475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-27T07:16:52.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f51240b2880 msgr2=0x7f51240b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f51240b2880 0x7f51240b4d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5130005950 tx=0x7f513000a400 comp rx=0 tx=0).stop 2024-05-27T07:16:52.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5140105e60 msgr2=0x7f514019d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5140105e60 0x7f514019d540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b5c0 tx=0x7f512c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 shutdown_connections 2024-05-27T07:16:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f51240b2880 0x7f51240b4d90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f514010e8f0 0x7f51401a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5140105510 0x7f514019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 --2- 172.21.15.12:0/2245922976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5140105e60 0x7f514019d540 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 >> 172.21.15.12:0/2245922976 conn(0x7f51400fedb0 msgr2=0x7f5140103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 shutdown_connections 2024-05-27T07:16:52.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:52.680+0000 7f5146e00700 1 -- 172.21.15.12:0/2245922976 wait complete. 2024-05-27T07:16:52.672 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 13 2024-05-27T07:16:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:52 smithi062 ceph-mon[35304]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:52 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:52 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2245922976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:53.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:52 smithi012 ceph-mon[31473]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:53.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:53.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:52 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:53.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:52 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2245922976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:53.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:52 smithi162 ceph-mon[35676]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:53.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:53.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:52 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:53.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:52 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2245922976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:53.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-27T07:16:54.310 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:54 smithi012 ceph-mon[31473]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:54 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:54 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:54 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:54 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:54 smithi162 ceph-mon[35676]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:54 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:54 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:54 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:54 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:54 smithi062 ceph-mon[35304]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:54 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:54 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:54 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:54 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:56.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:56 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:56.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:56 smithi162 ceph-mon[35676]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:56 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:56.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:56 smithi062 ceph-mon[35304]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 -- 172.21.15.12:0/1282772262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548109860 msgr2=0x7fd5481111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 --2- 172.21.15.12:0/1282772262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548109860 0x7fd5481111c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd54000b0d0 tx=0x7fd54000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 -- 172.21.15.12:0/1282772262 shutdown_connections 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 --2- 172.21.15.12:0/1282772262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5481045e0 0x7fd5481049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 --2- 172.21.15.12:0/1282772262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548109860 0x7fd5481111c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 --2- 172.21.15.12:0/1282772262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd548108f10 0x7fd548109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 -- 172.21.15.12:0/1282772262 >> 172.21.15.12:0/1282772262 conn(0x7fd548100030 msgr2=0x7fd548102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 -- 172.21.15.12:0/1282772262 shutdown_connections 2024-05-27T07:16:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.997+0000 7fd54d800700 1 -- 172.21.15.12:0/1282772262 wait complete. 2024-05-27T07:16:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.999+0000 7fd54d800700 1 Processor -- start 2024-05-27T07:16:56.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:56 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:56.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:56 smithi012 ceph-mon[31473]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:56.999+0000 7fd54d800700 1 -- start start 2024-05-27T07:16:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd54d800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5481045e0 0x7fd54810b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd54d800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548108f10 0x7fd54810b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd54d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd548109860 0x7fd54810bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd54d800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd548114340 con 0x7fd548108f10 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd54d800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd5481141c0 con 0x7fd548109860 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd54d800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5481144c0 con 0x7fd5481045e0 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548108f10 0x7fd54810b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd53fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548108f10 0x7fd54810b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37794/0 (socket says 172.21.15.12:37794) 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.000+0000 7fd53fe00700 1 -- 172.21.15.12:0/3860595030 learned_addr learned my addr 172.21.15.12:0/3860595030 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.001+0000 7fd547400700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd548109860 0x7fd54810bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.001+0000 7fd546a00700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5481045e0 0x7fd54810b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.001+0000 7fd53fe00700 1 -- 172.21.15.12:0/3860595030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5481045e0 msgr2=0x7fd54810b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.001+0000 7fd53fe00700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5481045e0 0x7fd54810b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.001+0000 7fd53fe00700 1 -- 172.21.15.12:0/3860595030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd548109860 msgr2=0x7fd54810bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.001+0000 7fd53fe00700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd548109860 0x7fd54810bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.001+0000 7fd53fe00700 1 -- 172.21.15.12:0/3860595030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5481ad510 con 0x7fd548108f10 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.002+0000 7fd53fe00700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548108f10 0x7fd54810b780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd53000d980 tx=0x7fd53000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.002+0000 7fd53f400700 1 -- 172.21.15.12:0/3860595030 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530017070 con 0x7fd548108f10 2024-05-27T07:16:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.002+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5481ad740 con 0x7fd548108f10 2024-05-27T07:16:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.002+0000 7fd53f400700 1 -- 172.21.15.12:0/3860595030 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd53000fd20 con 0x7fd548108f10 2024-05-27T07:16:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.002+0000 7fd53f400700 1 -- 172.21.15.12:0/3860595030 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53000e3b0 con 0x7fd548108f10 2024-05-27T07:16:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.002+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5481adc80 con 0x7fd548108f10 2024-05-27T07:16:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.004+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd548105a90 con 0x7fd548108f10 2024-05-27T07:16:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.007+0000 7fd53f400700 1 -- 172.21.15.12:0/3860595030 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd53000f890 con 0x7fd548108f10 2024-05-27T07:16:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.009+0000 7fd53f400700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd52c0b2900 0x7fd52c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.009+0000 7fd53f400700 1 -- 172.21.15.12:0/3860595030 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fd5300e1780 con 0x7fd548108f10 2024-05-27T07:16:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.010+0000 7fd546a00700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd52c0b2900 0x7fd52c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.010+0000 7fd546a00700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd52c0b2900 0x7fd52c0b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd53800ac50 tx=0x7fd53800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.013+0000 7fd53f400700 1 -- 172.21.15.12:0/3860595030 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd5300a9920 con 0x7fd548108f10 2024-05-27T07:16:57.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.274+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd5481adee0 con 0x7fd548108f10 2024-05-27T07:16:57.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.274+0000 7fd53f400700 1 -- 172.21.15.12:0/3860595030 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v71) ==== 84+0+65 (secure 0 0 0) 0x7fd5300a9220 con 0x7fd548108f10 2024-05-27T07:16:57.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:16:57.265 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-05-27T07:16:57.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:16:57.265 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-05-27T07:16:57.266 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-05-27T07:16:57.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:16:57.266 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd52c0b2900 msgr2=0x7fd52c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd52c0b2900 0x7fd52c0b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd53800ac50 tx=0x7fd53800a380 comp rx=0 tx=0).stop 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548108f10 msgr2=0x7fd54810b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548108f10 0x7fd54810b780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd53000d980 tx=0x7fd53000dc90 comp rx=0 tx=0).stop 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 shutdown_connections 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd52c0b2900 0x7fd52c0b4e10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd548109860 0x7fd54810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5481045e0 0x7fd54810b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 --2- 172.21.15.12:0/3860595030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd548108f10 0x7fd54810b780 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 >> 172.21.15.12:0/3860595030 conn(0x7fd548100030 msgr2=0x7fd5481086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 shutdown_connections 2024-05-27T07:16:57.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:16:57.278+0000 7fd54d800700 1 -- 172.21.15.12:0/3860595030 wait complete. 2024-05-27T07:16:58.044 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-27T07:16:58.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:58 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:58 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3860595030' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:58 smithi012 ceph-mon[31473]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:58.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:58.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:58 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:58.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:58 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3860595030' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:16:58.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:58 smithi062 ceph-mon[35304]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:58.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:58.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:58 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:58.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:58 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3860595030' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:16:58.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:58 smithi162 ceph-mon[35676]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:16:59.569 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:59 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:59.570 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:59 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:59.570 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:59 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:59.570 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:16:59 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:59.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:59 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:59.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:59 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:59 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:16:59 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:59.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:59 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:59.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:59 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:59 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:16:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:16:59 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:00.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:00 smithi012 ceph-mon[31473]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:00.603 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:00 smithi062 ceph-mon[35304]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:00 smithi162 ceph-mon[35676]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.941+0000 7fe0c0600700 1 -- 172.21.15.12:0/2432872350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 msgr2=0x7fe0b8106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.941+0000 7fe0c0600700 1 --2- 172.21.15.12:0/2432872350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b8106970 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4009990 tx=0x7fe0a4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.942+0000 7fe0c0600700 1 -- 172.21.15.12:0/2432872350 shutdown_connections 2024-05-27T07:17:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.942+0000 7fe0c0600700 1 --2- 172.21.15.12:0/2432872350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0b8106eb0 0x7fe0b8113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.942+0000 7fe0c0600700 1 --2- 172.21.15.12:0/2432872350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0b810c5a0 0x7fe0b810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.942+0000 7fe0c0600700 1 --2- 172.21.15.12:0/2432872350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b8106970 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:00.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.942+0000 7fe0c0600700 1 -- 172.21.15.12:0/2432872350 >> 172.21.15.12:0/2432872350 conn(0x7fe0b8102040 msgr2=0x7fe0b8104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:00.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.942+0000 7fe0c0600700 1 -- 172.21.15.12:0/2432872350 shutdown_connections 2024-05-27T07:17:00.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.942+0000 7fe0c0600700 1 -- 172.21.15.12:0/2432872350 wait complete. 2024-05-27T07:17:00.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.943+0000 7fe0c0600700 1 Processor -- start 2024-05-27T07:17:00.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0c0600700 1 -- start start 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0c0600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0c0600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0b8106eb0 0x7fe0b819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0c0600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0b810c5a0 0x7fe0b81a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0c0600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8116530 con 0x7fe0b8106560 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0c0600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81163b0 con 0x7fe0b810c5a0 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0c0600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81166b0 con 0x7fe0b8106eb0 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0bdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.944+0000 7fe0bdc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b819cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37814/0 (socket says 172.21.15.12:37814) 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bdc00700 1 -- 172.21.15.12:0/1222947065 learned_addr learned my addr 172.21.15.12:0/1222947065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bd200700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0b8106eb0 0x7fe0b819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:00.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0be600700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0b810c5a0 0x7fe0b81a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bdc00700 1 -- 172.21.15.12:0/1222947065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0b8106eb0 msgr2=0x7fe0b819d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bdc00700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0b8106eb0 0x7fe0b819d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bdc00700 1 -- 172.21.15.12:0/1222947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0b810c5a0 msgr2=0x7fe0b81a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:00.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bdc00700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0b810c5a0 0x7fe0b81a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bdc00700 1 -- 172.21.15.12:0/1222947065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b81a1dd0 con 0x7fe0b8106560 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.945+0000 7fe0bd200700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0b8106eb0 0x7fe0b819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.946+0000 7fe0bdc00700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b819cf00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00d950 tx=0x7fe0ac00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.946+0000 7fe0b6000700 1 -- 172.21.15.12:0/1222947065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac016070 con 0x7fe0b8106560 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.946+0000 7fe0b6000700 1 -- 172.21.15.12:0/1222947065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0ac004830 con 0x7fe0b8106560 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.946+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b81a20c0 con 0x7fe0b8106560 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.946+0000 7fe0b6000700 1 -- 172.21.15.12:0/1222947065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0053e0 con 0x7fe0b8106560 2024-05-27T07:17:00.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.946+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b81a99e0 con 0x7fe0b8106560 2024-05-27T07:17:00.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.947+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b80512b0 con 0x7fe0b8106560 2024-05-27T07:17:00.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.952+0000 7fe0b6000700 1 -- 172.21.15.12:0/1222947065 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe0ac005580 con 0x7fe0b8106560 2024-05-27T07:17:00.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.953+0000 7fe0b6000700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0a00b2900 0x7fe0a00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:00.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.954+0000 7fe0b6000700 1 -- 172.21.15.12:0/1222947065 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fe0ac0e2060 con 0x7fe0b8106560 2024-05-27T07:17:00.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.954+0000 7fe0bd200700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0a00b2900 0x7fe0a00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:00.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.955+0000 7fe0bd200700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0a00b2900 0x7fe0a00b4e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4006010 tx=0x7fe0a4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:00.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:00.955+0000 7fe0b6000700 1 -- 172.21.15.12:0/1222947065 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe0ac01f020 con 0x7fe0b8106560 2024-05-27T07:17:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.195+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe0b819e260 con 0x7fe0b8106560 2024-05-27T07:17:01.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.285+0000 7fe0b6000700 1 -- 172.21.15.12:0/1222947065 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v72) ==== 138+0+0 (secure 0 0 0) 0x7fe0ac0b2030 con 0x7fe0b8106560 2024-05-27T07:17:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.289+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0a00b2900 msgr2=0x7fe0a00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.289+0000 7fe0c0600700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0a00b2900 0x7fe0a00b4e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4006010 tx=0x7fe0a4005960 comp rx=0 tx=0).stop 2024-05-27T07:17:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.289+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 msgr2=0x7fe0b819cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.289+0000 7fe0c0600700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b819cf00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00d950 tx=0x7fe0ac00dc60 comp rx=0 tx=0).stop 2024-05-27T07:17:01.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 shutdown_connections 2024-05-27T07:17:01.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe0a00b2900 0x7fe0a00b4e10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0b810c5a0 0x7fe0b81a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0b8106560 0x7fe0b819cf00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 --2- 172.21.15.12:0/1222947065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0b8106eb0 0x7fe0b819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 >> 172.21.15.12:0/1222947065 conn(0x7fe0b8102040 msgr2=0x7fe0b81038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 shutdown_connections 2024-05-27T07:17:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:01.291+0000 7fe0c0600700 1 -- 172.21.15.12:0/1222947065 wait complete. 2024-05-27T07:17:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-05-27T07:17:01.519 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:01 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1222947065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:17:01.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:01 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1222947065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:17:01.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1222947065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:17:02.294 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:02 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1222947065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:17:02.294 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:02 smithi012 ceph-mon[31473]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:17:02.547 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:02 smithi012 ceph-mon[31473]: pgmap v370: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:02.548 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.548 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.548 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.548 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:02 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-27T07:17:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:02 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1222947065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:17:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:02 smithi162 ceph-mon[35676]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:17:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:02 smithi162 ceph-mon[35676]: pgmap v370: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:02 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:02 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1222947065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:17:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:02 smithi062 ceph-mon[35304]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:17:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:02 smithi062 ceph-mon[35304]: pgmap v370: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:02 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:03.011 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:03.468 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:03 smithi012 ceph-mon[31473]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:17:03.468 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:03 smithi012 ceph-mon[31473]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:17:03.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:03 smithi162 ceph-mon[35676]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:17:03.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:03 smithi162 ceph-mon[35676]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:17:03.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:03 smithi062 ceph-mon[35304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:17:03.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:03 smithi062 ceph-mon[35304]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:17:04.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:04 smithi012 ceph-mon[31473]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:17:04.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:04 smithi012 ceph-mon[31473]: pgmap v373: 33 pgs: 3 creating+peering, 19 active+clean, 11 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:04.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:04.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:04.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:04 smithi162 ceph-mon[35676]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:17:04.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:04 smithi162 ceph-mon[35676]: pgmap v373: 33 pgs: 3 creating+peering, 19 active+clean, 11 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:04.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:04.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:04.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:04 smithi062 ceph-mon[35304]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:17:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:04 smithi062 ceph-mon[35304]: pgmap v373: 33 pgs: 3 creating+peering, 19 active+clean, 11 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:17:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 -- 172.21.15.12:0/1291395690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 msgr2=0x7f8c64106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 --2- 172.21.15.12:0/1291395690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c64106a40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8c540099c0 tx=0x7f8c54009cd0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 -- 172.21.15.12:0/1291395690 shutdown_connections 2024-05-27T07:17:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 --2- 172.21.15.12:0/1291395690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c6410b860 0x7f8c641131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 --2- 172.21.15.12:0/1291395690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c64106a40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 --2- 172.21.15.12:0/1291395690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6410af10 0x7f8c6410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 -- 172.21.15.12:0/1291395690 >> 172.21.15.12:0/1291395690 conn(0x7f8c64102020 msgr2=0x7f8c64104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 -- 172.21.15.12:0/1291395690 shutdown_connections 2024-05-27T07:17:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.374+0000 7f8c6a200700 1 -- 172.21.15.12:0/1291395690 wait complete. 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.375+0000 7f8c6a200700 1 Processor -- start 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c6a200700 1 -- start start 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c6a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c6419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c6a200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6410af10 0x7f8c6419d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c6a200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c6410b860 0x7f8c641a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c6a200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64116390 con 0x7f8c64106630 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c6a200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64116210 con 0x7f8c6410af10 2024-05-27T07:17:05.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c6a200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64116510 con 0x7f8c6410b860 2024-05-27T07:17:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c68e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c6419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c68e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c6419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39600/0 (socket says 172.21.15.12:39600) 2024-05-27T07:17:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.376+0000 7f8c68e00700 1 -- 172.21.15.12:0/192516373 learned_addr learned my addr 172.21.15.12:0/192516373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c5be00700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6410af10 0x7f8c6419d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c69800700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c6410b860 0x7f8c641a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c68e00700 1 -- 172.21.15.12:0/192516373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c6410b860 msgr2=0x7f8c641a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c68e00700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c6410b860 0x7f8c641a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c68e00700 1 -- 172.21.15.12:0/192516373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6410af10 msgr2=0x7f8c6419d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c68e00700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6410af10 0x7f8c6419d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c68e00700 1 -- 172.21.15.12:0/192516373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c641a1b60 con 0x7f8c64106630 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.377+0000 7f8c68e00700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c6419cd40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8c54000c00 tx=0x7f8c5400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.378+0000 7f8c62000700 1 -- 172.21.15.12:0/192516373 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54027070 con 0x7f8c64106630 2024-05-27T07:17:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.378+0000 7f8c62000700 1 -- 172.21.15.12:0/192516373 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c540056d0 con 0x7f8c64106630 2024-05-27T07:17:05.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.378+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c641a1df0 con 0x7f8c64106630 2024-05-27T07:17:05.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.378+0000 7f8c62000700 1 -- 172.21.15.12:0/192516373 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c540173f0 con 0x7f8c64106630 2024-05-27T07:17:05.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.378+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c641a2320 con 0x7f8c64106630 2024-05-27T07:17:05.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.379+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c6406a610 con 0x7f8c64106630 2024-05-27T07:17:05.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.382+0000 7f8c62000700 1 -- 172.21.15.12:0/192516373 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8c5402a070 con 0x7f8c64106630 2024-05-27T07:17:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.383+0000 7f8c62000700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c480b2900 0x7f8c480b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.384+0000 7f8c62000700 1 -- 172.21.15.12:0/192516373 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7497+0+0 (secure 0 0 0) 0x7f8c540e2e60 con 0x7f8c64106630 2024-05-27T07:17:05.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.384+0000 7f8c5be00700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c480b2900 0x7f8c480b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:05.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.385+0000 7f8c5be00700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c480b2900 0x7f8c480b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8c64109850 tx=0x7f8c4c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:05.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.388+0000 7f8c62000700 1 -- 172.21.15.12:0/192516373 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8c54013070 con 0x7f8c64106630 2024-05-27T07:17:05.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.626+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f8c6402e5e0 con 0x7f8c64106630 2024-05-27T07:17:05.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.703+0000 7f8c62000700 1 -- 172.21.15.12:0/192516373 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v75) ==== 171+0+0 (secure 0 0 0) 0x7f8c540aaea0 con 0x7f8c64106630 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c480b2900 msgr2=0x7f8c480b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c480b2900 0x7f8c480b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8c64109850 tx=0x7f8c4c00a380 comp rx=0 tx=0).stop 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 msgr2=0x7f8c6419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c6419cd40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8c54000c00 tx=0x7f8c5400f690 comp rx=0 tx=0).stop 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 shutdown_connections 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c480b2900 0x7f8c480b4e10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c6410b860 0x7f8c641a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c64106630 0x7f8c6419cd40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 --2- 172.21.15.12:0/192516373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6410af10 0x7f8c6419d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.708+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 >> 172.21.15.12:0/192516373 conn(0x7f8c64102020 msgr2=0x7f8c6410a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.709+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 shutdown_connections 2024-05-27T07:17:05.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:05.709+0000 7f8c6a200700 1 -- 172.21.15.12:0/192516373 wait complete. 2024-05-27T07:17:05.701 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-05-27T07:17:05.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:05 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/192516373' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:17:05.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:05 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/192516373' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:17:06.084 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:05 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/192516373' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:17:06.512 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-27T07:17:06.735 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:06 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/192516373' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:17:06.735 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:06 smithi012 ceph-mon[31473]: pgmap v375: 97 pgs: 3 creating+peering, 27 active+clean, 67 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:06.993 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:07.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:06 smithi012 ceph-mon[31473]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:17:07.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:06 smithi012 ceph-mon[31473]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:17:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:06 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/192516373' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:17:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:06 smithi162 ceph-mon[35676]: pgmap v375: 97 pgs: 3 creating+peering, 27 active+clean, 67 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:06 smithi162 ceph-mon[35676]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:17:07.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:06 smithi162 ceph-mon[35676]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:17:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:06 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/192516373' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:17:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:06 smithi062 ceph-mon[35304]: pgmap v375: 97 pgs: 3 creating+peering, 27 active+clean, 67 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:06 smithi062 ceph-mon[35304]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:17:07.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:06 smithi062 ceph-mon[35304]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:17:08.912 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:08 smithi012 ceph-mon[31473]: pgmap v377: 97 pgs: 16 creating+peering, 43 active+clean, 38 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:08.912 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:08 smithi012 ceph-mon[31473]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:17:09.039 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:08 smithi062 ceph-mon[35304]: pgmap v377: 97 pgs: 16 creating+peering, 43 active+clean, 38 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:09.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:08 smithi062 ceph-mon[35304]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:17:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:08 smithi162 ceph-mon[35676]: pgmap v377: 97 pgs: 16 creating+peering, 43 active+clean, 38 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:08 smithi162 ceph-mon[35676]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 -- 172.21.15.12:0/968485424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106570 msgr2=0x7fe2b0106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/968485424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106570 0x7fe2b0106980 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe298009990 tx=0x7fe298009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 -- 172.21.15.12:0/968485424 shutdown_connections 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/968485424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b010c5b0 0x7fe2b010c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/968485424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0106ec0 0x7fe2b0113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/968485424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106570 0x7fe2b0106980 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 -- 172.21.15.12:0/968485424 >> 172.21.15.12:0/968485424 conn(0x7fe2b0102070 msgr2=0x7fe2b01044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 -- 172.21.15.12:0/968485424 shutdown_connections 2024-05-27T07:17:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.525+0000 7fe2b4c00700 1 -- 172.21.15.12:0/968485424 wait complete. 2024-05-27T07:17:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.527+0000 7fe2b4c00700 1 Processor -- start 2024-05-27T07:17:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.528+0000 7fe2b4c00700 1 -- start start 2024-05-27T07:17:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.528+0000 7fe2b4c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0106570 0x7fe2b019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.528+0000 7fe2b4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106ec0 0x7fe2b019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.528+0000 7fe2b4c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b010c5b0 0x7fe2b01a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.528+0000 7fe2b4c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0116590 con 0x7fe2b0106ec0 2024-05-27T07:17:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2b4c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0116410 con 0x7fe2b010c5b0 2024-05-27T07:17:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2b4c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0116710 con 0x7fe2b0106570 2024-05-27T07:17:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106ec0 0x7fe2b019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106ec0 0x7fe2b019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39618/0 (socket says 172.21.15.12:39618) 2024-05-27T07:17:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 -- 172.21.15.12:0/2714932271 learned_addr learned my addr 172.21.15.12:0/2714932271 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2ae800700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b010c5b0 0x7fe2b01a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2ade00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0106570 0x7fe2b019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 -- 172.21.15.12:0/2714932271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0106570 msgr2=0x7fe2b019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0106570 0x7fe2b019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 -- 172.21.15.12:0/2714932271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b010c5b0 msgr2=0x7fe2b01a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b010c5b0 0x7fe2b01a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.529+0000 7fe2a7e00700 1 -- 172.21.15.12:0/2714932271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b01a1d40 con 0x7fe2b0106ec0 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.530+0000 7fe2a7e00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106ec0 0x7fe2b019d4c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe2980098e0 tx=0x7fe298003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.530+0000 7fe2a6a00700 1 -- 172.21.15.12:0/2714932271 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298027070 con 0x7fe2b0106ec0 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.530+0000 7fe2a6a00700 1 -- 172.21.15.12:0/2714932271 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe29800fc00 con 0x7fe2b0106ec0 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.530+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b01a1fd0 con 0x7fe2b0106ec0 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.530+0000 7fe2a6a00700 1 -- 172.21.15.12:0/2714932271 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298022ce0 con 0x7fe2b0106ec0 2024-05-27T07:17:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.530+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b01a2500 con 0x7fe2b0106ec0 2024-05-27T07:17:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.531+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b010e970 con 0x7fe2b0106ec0 2024-05-27T07:17:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.534+0000 7fe2a6a00700 1 -- 172.21.15.12:0/2714932271 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe29802a070 con 0x7fe2b0106ec0 2024-05-27T07:17:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.536+0000 7fe2a6a00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe2940b2900 0x7fe2940b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:09.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.536+0000 7fe2a6a00700 1 -- 172.21.15.12:0/2714932271 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7879+0+0 (secure 0 0 0) 0x7fe2980e33a0 con 0x7fe2b0106ec0 2024-05-27T07:17:09.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.536+0000 7fe2ade00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe2940b2900 0x7fe2940b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:09.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.537+0000 7fe2ade00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe2940b2900 0x7fe2940b4e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0005ea0 tx=0x7fe2a0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.539+0000 7fe2a6a00700 1 -- 172.21.15.12:0/2714932271 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe29805fdf0 con 0x7fe2b0106ec0 2024-05-27T07:17:09.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:09.823+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe2b019e280 con 0x7fe2b0106ec0 2024-05-27T07:17:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.783+0000 7fe2a6a00700 1 -- 172.21.15.12:0/2714932271 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fe298013070 con 0x7fe2b0106ec0 2024-05-27T07:17:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe2940b2900 msgr2=0x7fe2940b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe2940b2900 0x7fe2940b4e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0005ea0 tx=0x7fe2a0005e30 comp rx=0 tx=0).stop 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106ec0 msgr2=0x7fe2b019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106ec0 0x7fe2b019d4c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe2980098e0 tx=0x7fe298003680 comp rx=0 tx=0).stop 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 shutdown_connections 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe2940b2900 0x7fe2940b4e10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b010c5b0 0x7fe2b01a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0106570 0x7fe2b019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 --2- 172.21.15.12:0/2714932271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0106ec0 0x7fe2b019d4c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.787+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 >> 172.21.15.12:0/2714932271 conn(0x7fe2b0102070 msgr2=0x7fe2b0104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.788+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 shutdown_connections 2024-05-27T07:17:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:10.788+0000 7fe2b4c00700 1 -- 172.21.15.12:0/2714932271 wait complete. 2024-05-27T07:17:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 2 and data pool 3 2024-05-27T07:17:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:10 smithi062 ceph-mon[35304]: pgmap v379: 97 pgs: 16 creating+peering, 64 active+clean, 17 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:10 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2714932271' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:17:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:10 smithi062 ceph-mon[35304]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:17:11.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:10 smithi012 ceph-mon[31473]: pgmap v379: 97 pgs: 16 creating+peering, 64 active+clean, 17 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:10 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2714932271' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:17:11.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:10 smithi012 ceph-mon[31473]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:17:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:10 smithi162 ceph-mon[35676]: pgmap v379: 97 pgs: 16 creating+peering, 64 active+clean, 17 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:10 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2714932271' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:17:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:10 smithi162 ceph-mon[35676]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:17:11.566 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-27T07:17:11.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-27T07:17:12.039 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:12.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:11 smithi012 ceph-mon[31473]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:17:12.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:11 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2714932271' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:17:12.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:11 smithi012 ceph-mon[31473]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:17:12.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:11 smithi012 ceph-mon[31473]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:17:12.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:11 smithi162 ceph-mon[35676]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:17:12.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:11 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2714932271' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:17:12.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:11 smithi162 ceph-mon[35676]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:17:12.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:11 smithi162 ceph-mon[35676]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:17:12.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:11 smithi062 ceph-mon[35304]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:17:12.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:11 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2714932271' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:17:12.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:11 smithi062 ceph-mon[35304]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:17:12.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:11 smithi062 ceph-mon[35304]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:17:12.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:12 smithi012 ceph-mon[31473]: pgmap v381: 97 pgs: 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:12.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:12 smithi012 ceph-mon[31473]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:17:12.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:12 smithi012 ceph-mon[31473]: Cluster is now healthy 2024-05-27T07:17:12.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:12 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] up:active 2024-05-27T07:17:12.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:12 smithi012 ceph-mon[31473]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:12.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:12 smithi012 ceph-mon[31473]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:12 smithi162 ceph-mon[35676]: pgmap v381: 97 pgs: 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:12 smithi162 ceph-mon[35676]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:17:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:12 smithi162 ceph-mon[35676]: Cluster is now healthy 2024-05-27T07:17:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:12 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] up:active 2024-05-27T07:17:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:12 smithi162 ceph-mon[35676]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:13.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:12 smithi162 ceph-mon[35676]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:12 smithi062 ceph-mon[35304]: pgmap v381: 97 pgs: 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:12 smithi062 ceph-mon[35304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:17:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:12 smithi062 ceph-mon[35304]: Cluster is now healthy 2024-05-27T07:17:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:12 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] up:active 2024-05-27T07:17:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:12 smithi062 ceph-mon[35304]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:13.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:12 smithi062 ceph-mon[35304]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 -- 172.21.15.12:0/1593770769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c10e8e0 msgr2=0x7f840c110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 --2- 172.21.15.12:0/1593770769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c10e8e0 0x7f840c110dc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f840400b0a0 tx=0x7f840400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 -- 172.21.15.12:0/1593770769 shutdown_connections 2024-05-27T07:17:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 --2- 172.21.15.12:0/1593770769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f840c105470 0x7f840c105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 --2- 172.21.15.12:0/1593770769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c10e8e0 0x7f840c110dc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 --2- 172.21.15.12:0/1593770769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f840c105dc0 0x7f840c10e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 -- 172.21.15.12:0/1593770769 >> 172.21.15.12:0/1593770769 conn(0x7f840c0fef20 msgr2=0x7f840c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 -- 172.21.15.12:0/1593770769 shutdown_connections 2024-05-27T07:17:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.380+0000 7f8411e00700 1 -- 172.21.15.12:0/1593770769 wait complete. 2024-05-27T07:17:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.382+0000 7f8411e00700 1 Processor -- start 2024-05-27T07:17:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.382+0000 7f8411e00700 1 -- start start 2024-05-27T07:17:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f8411e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c105470 0x7f840c19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f8411e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f840c105dc0 0x7f840c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f8411e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f840c10e8e0 0x7f840c1a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f8411e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f840c114340 con 0x7f840c105470 2024-05-27T07:17:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f8411e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f840c1141c0 con 0x7f840c105dc0 2024-05-27T07:17:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f8411e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f840c1144c0 con 0x7f840c10e8e0 2024-05-27T07:17:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f840aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c105470 0x7f840c19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f840aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c105470 0x7f840c19ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48434/0 (socket says 172.21.15.12:48434) 2024-05-27T07:17:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f840aa00700 1 -- 172.21.15.12:0/1893305218 learned_addr learned my addr 172.21.15.12:0/1893305218 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.383+0000 7f8403e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f840c105dc0 0x7f840c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.384+0000 7f840b400700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f840c10e8e0 0x7f840c1a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.384+0000 7f840aa00700 1 -- 172.21.15.12:0/1893305218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f840c10e8e0 msgr2=0x7f840c1a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.384+0000 7f840aa00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f840c10e8e0 0x7f840c1a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.384+0000 7f840aa00700 1 -- 172.21.15.12:0/1893305218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f840c105dc0 msgr2=0x7f840c19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.384+0000 7f840aa00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f840c105dc0 0x7f840c19d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.384+0000 7f840aa00700 1 -- 172.21.15.12:0/1893305218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840c1a1cf0 con 0x7f840c105470 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.384+0000 7f840b400700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f840c10e8e0 0x7f840c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.385+0000 7f840aa00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c105470 0x7f840c19ced0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00c9b0 tx=0x7f83fc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.385+0000 7f8410a00700 1 -- 172.21.15.12:0/1893305218 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc007b00 con 0x7f840c105470 2024-05-27T07:17:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.385+0000 7f8410a00700 1 -- 172.21.15.12:0/1893305218 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f83fc007ca0 con 0x7f840c105470 2024-05-27T07:17:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.385+0000 7f8410a00700 1 -- 172.21.15.12:0/1893305218 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc018660 con 0x7f840c105470 2024-05-27T07:17:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.385+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840c1a1fe0 con 0x7f840c105470 2024-05-27T07:17:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.385+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f840c1a2520 con 0x7f840c105470 2024-05-27T07:17:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.387+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f840c0512b0 con 0x7f840c105470 2024-05-27T07:17:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.389+0000 7f8410a00700 1 -- 172.21.15.12:0/1893305218 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f83fc00f450 con 0x7f840c105470 2024-05-27T07:17:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.391+0000 7f8410a00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f83f00b2880 0x7f83f00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.391+0000 7f8410a00700 1 -- 172.21.15.12:0/1893305218 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f83fc0e2330 con 0x7f840c105470 2024-05-27T07:17:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.391+0000 7f8403e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f83f00b2880 0x7f83f00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.392+0000 7f8403e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f83f00b2880 0x7f83f00b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f840c104ae0 tx=0x7f83f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:14.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.394+0000 7f8410a00700 1 -- 172.21.15.12:0/1893305218 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f83fc014030 con 0x7f840c105470 2024-05-27T07:17:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.644+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f840c19e230 con 0x7f840c105470 2024-05-27T07:17:14.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.793+0000 7f8410a00700 1 -- 172.21.15.12:0/1893305218 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v79) ==== 157+0+0 (secure 0 0 0) 0x7f840c19e230 con 0x7f840c105470 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f83f00b2880 msgr2=0x7f83f00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f83f00b2880 0x7f83f00b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f840c104ae0 tx=0x7f83f400b410 comp rx=0 tx=0).stop 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c105470 msgr2=0x7f840c19ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c105470 0x7f840c19ced0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00c9b0 tx=0x7f83fc00ccc0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 shutdown_connections 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f83f00b2880 0x7f83f00b4d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f840c10e8e0 0x7f840c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f840c105470 0x7f840c19ced0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 --2- 172.21.15.12:0/1893305218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f840c105dc0 0x7f840c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:14.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.798+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 >> 172.21.15.12:0/1893305218 conn(0x7f840c0fef20 msgr2=0x7f840c103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:14.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.799+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 shutdown_connections 2024-05-27T07:17:14.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:14.799+0000 7f8411e00700 1 -- 172.21.15.12:0/1893305218 wait complete. 2024-05-27T07:17:15.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:14 smithi012 ceph-mon[31473]: pgmap v382: 97 pgs: 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-27T07:17:15.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:14 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1893305218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:17:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:14 smithi162 ceph-mon[35676]: pgmap v382: 97 pgs: 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-27T07:17:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:14 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1893305218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:17:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:14 smithi062 ceph-mon[35304]: pgmap v382: 97 pgs: 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-27T07:17:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:14 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1893305218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:17:15.664 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-27T07:17:16.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:15 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1893305218' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:17:16.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:15 smithi012 ceph-mon[31473]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:17:16.126 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:16.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:15 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1893305218' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:17:16.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:15 smithi162 ceph-mon[35676]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:17:16.236 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:17:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b[66152]: 2024-05-27T07:17:15.822+0000 7f6e0c000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T07:17:16.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:15 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1893305218' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:17:16.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:15 smithi062 ceph-mon[35304]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:17:17.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:16 smithi012 ceph-mon[31473]: pgmap v384: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:17:17.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:16 smithi162 ceph-mon[35676]: pgmap v384: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:17:17.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:16 smithi062 ceph-mon[35304]: pgmap v384: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 -- 172.21.15.12:0/1873012272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff41052e0 msgr2=0x7efff41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 --2- 172.21.15.12:0/1873012272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff41052e0 0x7efff41056f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7effe4009990 tx=0x7effe4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 -- 172.21.15.12:0/1873012272 shutdown_connections 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 --2- 172.21.15.12:0/1873012272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff410e710 0x7efff4110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 --2- 172.21.15.12:0/1873012272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff41052e0 0x7efff41056f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 --2- 172.21.15.12:0/1873012272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff4105c30 0x7efff410e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 -- 172.21.15.12:0/1873012272 >> 172.21.15.12:0/1873012272 conn(0x7efff4078af0 msgr2=0x7efff4078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 -- 172.21.15.12:0/1873012272 shutdown_connections 2024-05-27T07:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.613+0000 7efff3e00700 1 -- 172.21.15.12:0/1873012272 wait complete. 2024-05-27T07:17:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.614+0000 7efff3e00700 1 Processor -- start 2024-05-27T07:17:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.614+0000 7efff3e00700 1 -- start start 2024-05-27T07:17:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.614+0000 7efff3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff41052e0 0x7efff41a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.614+0000 7efff3e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff4105c30 0x7efff41a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff410e710 0x7efff419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efff4114160 con 0x7efff410e710 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efff4113fe0 con 0x7efff41052e0 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff3e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efff41142e0 con 0x7efff4105c30 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff410e710 0x7efff419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff410e710 0x7efff419a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48448/0 (socket says 172.21.15.12:48448) 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff3400700 1 -- 172.21.15.12:0/2036142363 learned_addr learned my addr 172.21.15.12:0/2036142363 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7effebe00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff4105c30 0x7efff41a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.615+0000 7efff2a00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff41052e0 0x7efff41a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7efff3400700 1 -- 172.21.15.12:0/2036142363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff4105c30 msgr2=0x7efff41a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7efff3400700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff4105c30 0x7efff41a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7efff3400700 1 -- 172.21.15.12:0/2036142363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff41052e0 msgr2=0x7efff41a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7efff3400700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff41052e0 0x7efff41a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7efff3400700 1 -- 172.21.15.12:0/2036142363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff419af60 con 0x7efff410e710 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7efff3400700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff410e710 0x7efff419a920 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7effec007d70 tx=0x7effec00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7effeb400700 1 -- 172.21.15.12:0/2036142363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec016020 con 0x7efff410e710 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7effeb400700 1 -- 172.21.15.12:0/2036142363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effec00ac90 con 0x7efff410e710 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7effeb400700 1 -- 172.21.15.12:0/2036142363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec0115e0 con 0x7efff410e710 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.616+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff419b250 con 0x7efff410e710 2024-05-27T07:17:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.617+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff419b740 con 0x7efff410e710 2024-05-27T07:17:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.618+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff406c8b0 con 0x7efff410e710 2024-05-27T07:17:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.620+0000 7effeb400700 1 -- 172.21.15.12:0/2036142363 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7effec019020 con 0x7efff410e710 2024-05-27T07:17:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.622+0000 7effeb400700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7effd80b2880 0x7effd80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.622+0000 7effeb400700 1 -- 172.21.15.12:0/2036142363 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8277+0+0 (secure 0 0 0) 0x7effec0e2730 con 0x7efff410e710 2024-05-27T07:17:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.622+0000 7efff2a00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7effd80b2880 0x7effd80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.623+0000 7efff2a00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7effd80b2880 0x7effd80b4d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7effe4003760 tx=0x7effe4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:18.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.625+0000 7effeb400700 1 -- 172.21.15.12:0/2036142363 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7effec020020 con 0x7efff410e710 2024-05-27T07:17:18.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:18.868+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7efff419be80 con 0x7efff410e710 2024-05-27T07:17:18.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:18 smithi012 ceph-mon[31473]: pgmap v385: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:17:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:18 smithi162 ceph-mon[35676]: pgmap v385: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:17:19.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:18 smithi062 ceph-mon[35304]: pgmap v385: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:17:20.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:19 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2036142363' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:17:20.151 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:19 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2036142363' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:17:20.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:19 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2036142363' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:17:20.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.837+0000 7effeb400700 1 -- 172.21.15.12:0/2036142363 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v81) ==== 228+0+0 (secure 0 0 0) 0x7efff419be80 con 0x7efff410e710 2024-05-27T07:17:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.842+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7effd80b2880 msgr2=0x7effd80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.842+0000 7efff3e00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7effd80b2880 0x7effd80b4d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7effe4003760 tx=0x7effe4005fd0 comp rx=0 tx=0).stop 2024-05-27T07:17:20.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.842+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff410e710 msgr2=0x7efff419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.842+0000 7efff3e00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff410e710 0x7efff419a920 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7effec007d70 tx=0x7effec00a340 comp rx=0 tx=0).stop 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 shutdown_connections 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7effd80b2880 0x7effd80b4d90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff41052e0 0x7efff41a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff410e710 0x7efff419a920 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 --2- 172.21.15.12:0/2036142363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff4105c30 0x7efff41a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 >> 172.21.15.12:0/2036142363 conn(0x7efff4078af0 msgr2=0x7efff40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 shutdown_connections 2024-05-27T07:17:20.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:20.843+0000 7efff3e00700 1 -- 172.21.15.12:0/2036142363 wait complete. 2024-05-27T07:17:20.836 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data_ec' created 2024-05-27T07:17:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:20 smithi012 ceph-mon[31473]: pgmap v386: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:17:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:20 smithi012 ceph-mon[31473]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:17:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:20 smithi162 ceph-mon[35676]: pgmap v386: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:17:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:20 smithi162 ceph-mon[35676]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:17:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:20 smithi062 ceph-mon[35304]: pgmap v386: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:17:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:20 smithi062 ceph-mon[35304]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:17:21.870 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-27T07:17:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:21 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2036142363' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:17:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:21 smithi012 ceph-mon[31473]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:17:22.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:21 smithi012 ceph-mon[31473]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:17:22.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:21 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2036142363' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:17:22.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:21 smithi162 ceph-mon[35676]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:17:22.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:21 smithi162 ceph-mon[35676]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:17:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:21 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2036142363' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:17:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:21 smithi062 ceph-mon[35304]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:17:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:21 smithi062 ceph-mon[35304]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:17:22.315 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:22 smithi012 ceph-mon[31473]: pgmap v389: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s wr, 0 op/s 2024-05-27T07:17:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:22 smithi012 ceph-mon[31473]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:17:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:22 smithi162 ceph-mon[35676]: pgmap v389: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s wr, 0 op/s 2024-05-27T07:17:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:22 smithi162 ceph-mon[35676]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:17:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:22 smithi062 ceph-mon[35304]: pgmap v389: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s wr, 0 op/s 2024-05-27T07:17:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:22 smithi062 ceph-mon[35304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:17:24.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:23 smithi012 ceph-mon[31473]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:17:24.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:23 smithi162 ceph-mon[35676]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:17:24.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:23 smithi062 ceph-mon[35304]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 -- 172.21.15.12:0/2647128601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4076670 msgr2=0x7fb5a4110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 --2- 172.21.15.12:0/2647128601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4076670 0x7fb5a4110650 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb594009990 tx=0x7fb594009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 -- 172.21.15.12:0/2647128601 shutdown_connections 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 --2- 172.21.15.12:0/2647128601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5a4075d20 0x7fb5a4076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 --2- 172.21.15.12:0/2647128601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5a4110b90 0x7fb5a4112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 --2- 172.21.15.12:0/2647128601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4076670 0x7fb5a4110650 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 -- 172.21.15.12:0/2647128601 >> 172.21.15.12:0/2647128601 conn(0x7fb5a4100030 msgr2=0x7fb5a4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.741+0000 7fb5ab400700 1 -- 172.21.15.12:0/2647128601 shutdown_connections 2024-05-27T07:17:24.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.742+0000 7fb5ab400700 1 -- 172.21.15.12:0/2647128601 wait complete. 2024-05-27T07:17:24.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.743+0000 7fb5ab400700 1 Processor -- start 2024-05-27T07:17:24.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.743+0000 7fb5ab400700 1 -- start start 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5ab400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5a4075d20 0x7fb5a419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5ab400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5a4076670 0x7fb5a419d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5ab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4110b90 0x7fb5a41a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5ab400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a4116540 con 0x7fb5a4110b90 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5ab400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41163c0 con 0x7fb5a4076670 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5ab400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41166c0 con 0x7fb5a4075d20 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5a9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4110b90 0x7fb5a41a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:24.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5a9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4110b90 0x7fb5a41a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49028/0 (socket says 172.21.15.12:49028) 2024-05-27T07:17:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5a9400700 1 -- 172.21.15.12:0/1640625747 learned_addr learned my addr 172.21.15.12:0/1640625747 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:24.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb59be00700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5a4076670 0x7fb5a419d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.744+0000 7fb5a8a00700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5a4075d20 0x7fb5a419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a9400700 1 -- 172.21.15.12:0/1640625747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5a4075d20 msgr2=0x7fb5a419cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a9400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5a4075d20 0x7fb5a419cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a9400700 1 -- 172.21.15.12:0/1640625747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5a4076670 msgr2=0x7fb5a419d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a9400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5a4076670 0x7fb5a419d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a9400700 1 -- 172.21.15.12:0/1640625747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a41a1e20 con 0x7fb5a4110b90 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb59be00700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5a4076670 0x7fb5a419d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a9400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4110b90 0x7fb5a41a1780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0094f0 tx=0x7fb59c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a2000700 1 -- 172.21.15.12:0/1640625747 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c004db0 con 0x7fb5a4110b90 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a2000700 1 -- 172.21.15.12:0/1640625747 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb59c005020 con 0x7fb5a4110b90 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a41a2110 con 0x7fb5a4110b90 2024-05-27T07:17:24.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.745+0000 7fb5a2000700 1 -- 172.21.15.12:0/1640625747 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c015660 con 0x7fb5a4110b90 2024-05-27T07:17:24.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.746+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a41a9a30 con 0x7fb5a4110b90 2024-05-27T07:17:24.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.747+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a40512b0 con 0x7fb5a4110b90 2024-05-27T07:17:24.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.751+0000 7fb5a2000700 1 -- 172.21.15.12:0/1640625747 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb59c00db60 con 0x7fb5a4110b90 2024-05-27T07:17:24.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.753+0000 7fb5a2000700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5840b2880 0x7fb5840b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:24.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.753+0000 7fb5a2000700 1 -- 172.21.15.12:0/1640625747 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8770+0+0 (secure 0 0 0) 0x7fb59c0e26f0 con 0x7fb5a4110b90 2024-05-27T07:17:24.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.753+0000 7fb5a8a00700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5840b2880 0x7fb5840b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:24.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.754+0000 7fb5a8a00700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5840b2880 0x7fb5840b4d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb58c005950 tx=0x7fb58c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:24.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.756+0000 7fb5a2000700 1 -- 172.21.15.12:0/1640625747 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb59c010070 con 0x7fb5a4110b90 2024-05-27T07:17:24.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:24.993+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb5a402e5e0 con 0x7fb5a4110b90 2024-05-27T07:17:24.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:24 smithi012 ceph-mon[31473]: pgmap v392: 129 pgs: 1 creating+peering, 11 unknown, 117 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:25.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:24 smithi162 ceph-mon[35676]: pgmap v392: 129 pgs: 1 creating+peering, 11 unknown, 117 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:25.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:24 smithi062 ceph-mon[35304]: pgmap v392: 129 pgs: 1 creating+peering, 11 unknown, 117 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:25.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.922+0000 7fb5a2000700 1 -- 172.21.15.12:0/1640625747 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v84) ==== 169+0+0 (secure 0 0 0) 0x7fb59c0a5020 con 0x7fb5a4110b90 2024-05-27T07:17:25.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.926+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5840b2880 msgr2=0x7fb5840b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:25.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.926+0000 7fb5ab400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5840b2880 0x7fb5840b4d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb58c005950 tx=0x7fb58c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:17:25.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.926+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4110b90 msgr2=0x7fb5a41a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:25.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.926+0000 7fb5ab400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4110b90 0x7fb5a41a1780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0094f0 tx=0x7fb59c007f90 comp rx=0 tx=0).stop 2024-05-27T07:17:25.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.927+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 shutdown_connections 2024-05-27T07:17:25.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.927+0000 7fb5ab400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5840b2880 0x7fb5840b4d90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:25.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.927+0000 7fb5ab400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb5a4110b90 0x7fb5a41a1780 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:25.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.927+0000 7fb5ab400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5a4075d20 0x7fb5a419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:25.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.927+0000 7fb5ab400700 1 --2- 172.21.15.12:0/1640625747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5a4076670 0x7fb5a419d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:25.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.927+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 >> 172.21.15.12:0/1640625747 conn(0x7fb5a4100030 msgr2=0x7fb5a4100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:25.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.928+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 shutdown_connections 2024-05-27T07:17:25.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:25.928+0000 7fb5ab400700 1 -- 172.21.15.12:0/1640625747 wait complete. 2024-05-27T07:17:25.921 INFO:teuthology.orchestra.run.smithi012.stderr:set pool 4 allow_ec_overwrites to true 2024-05-27T07:17:26.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:25 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1640625747' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:17:26.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:25 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1640625747' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:17:26.308 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:25 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1640625747' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:17:26.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-27T07:17:27.198 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:26 smithi162 ceph-mon[35676]: pgmap v393: 129 pgs: 1 creating+peering, 2 unknown, 126 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:26 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1640625747' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:17:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:26 smithi162 ceph-mon[35676]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:17:27.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:26 smithi062 ceph-mon[35304]: pgmap v393: 129 pgs: 1 creating+peering, 2 unknown, 126 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:27.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:26 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1640625747' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:17:27.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:26 smithi062 ceph-mon[35304]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:17:27.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:26 smithi012 ceph-mon[31473]: pgmap v393: 129 pgs: 1 creating+peering, 2 unknown, 126 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:27.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:26 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1640625747' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:17:27.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:26 smithi012 ceph-mon[31473]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:17:29.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:29 smithi012 ceph-mon[31473]: pgmap v395: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:29.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:29 smithi162 ceph-mon[35676]: pgmap v395: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:29.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:29 smithi062 ceph-mon[35304]: pgmap v395: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.725+0000 7f23b7200700 1 -- 172.21.15.12:0/344206734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b81025d0 msgr2=0x7f23b81029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.725+0000 7f23b7200700 1 --2- 172.21.15.12:0/344206734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b81025d0 0x7f23b81029e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009990 tx=0x7f23a4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.725+0000 7f23b7200700 1 -- 172.21.15.12:0/344206734 shutdown_connections 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.725+0000 7f23b7200700 1 --2- 172.21.15.12:0/344206734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b8102f20 0x7f23b81110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.725+0000 7f23b7200700 1 --2- 172.21.15.12:0/344206734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23b8109250 0x7f23b8109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.725+0000 7f23b7200700 1 --2- 172.21.15.12:0/344206734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b81025d0 0x7f23b81029e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.725+0000 7f23b7200700 1 -- 172.21.15.12:0/344206734 >> 172.21.15.12:0/344206734 conn(0x7f23b8078360 msgr2=0x7f23b80787a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.726+0000 7f23b7200700 1 -- 172.21.15.12:0/344206734 shutdown_connections 2024-05-27T07:17:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.726+0000 7f23b7200700 1 -- 172.21.15.12:0/344206734 wait complete. 2024-05-27T07:17:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 Processor -- start 2024-05-27T07:17:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 -- start start 2024-05-27T07:17:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b81025d0 0x7f23b819ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23b8102f20 0x7f23b819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b8109250 0x7f23b819f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f23b81143f0 con 0x7f23b8109250 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f23b8114270 con 0x7f23b81025d0 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.727+0000 7f23b7200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f23b8114570 con 0x7f23b8102f20 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b8109250 0x7f23b819f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b8109250 0x7f23b819f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49038/0 (socket says 172.21.15.12:49038) 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 -- 172.21.15.12:0/2937206786 learned_addr learned my addr 172.21.15.12:0/2937206786 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b5e00700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b81025d0 0x7f23b819ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b5400700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23b8102f20 0x7f23b819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 -- 172.21.15.12:0/2937206786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23b8102f20 msgr2=0x7f23b819b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23b8102f20 0x7f23b819b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 -- 172.21.15.12:0/2937206786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b81025d0 msgr2=0x7f23b819ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b81025d0 0x7f23b819ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b6800700 1 -- 172.21.15.12:0/2937206786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b819fcd0 con 0x7f23b8109250 2024-05-27T07:17:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.728+0000 7f23b5e00700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b81025d0 0x7f23b819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.729+0000 7f23b6800700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b8109250 0x7f23b819f630 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007ee0 tx=0x7f23b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.729+0000 7f23aea00700 1 -- 172.21.15.12:0/2937206786 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b000dbd0 con 0x7f23b8109250 2024-05-27T07:17:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.729+0000 7f23aea00700 1 -- 172.21.15.12:0/2937206786 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23b000dd70 con 0x7f23b8109250 2024-05-27T07:17:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.729+0000 7f23aea00700 1 -- 172.21.15.12:0/2937206786 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b00058d0 con 0x7f23b8109250 2024-05-27T07:17:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.729+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b819ffc0 con 0x7f23b8109250 2024-05-27T07:17:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.729+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b81a78e0 con 0x7f23b8109250 2024-05-27T07:17:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.731+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23b810c7f0 con 0x7f23b8109250 2024-05-27T07:17:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.734+0000 7f23aea00700 1 -- 172.21.15.12:0/2937206786 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f23b001a020 con 0x7f23b8109250 2024-05-27T07:17:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.736+0000 7f23aea00700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f239c0b2900 0x7f239c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.737+0000 7f23aea00700 1 -- 172.21.15.12:0/2937206786 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8770+0+0 (secure 0 0 0) 0x7f23b00e22a0 con 0x7f23b8109250 2024-05-27T07:17:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.737+0000 7f23b5e00700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f239c0b2900 0x7f239c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.737+0000 7f23b5e00700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f239c0b2900 0x7f239c0b4e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f23a8005950 tx=0x7f23a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:29.739+0000 7f23aea00700 1 -- 172.21.15.12:0/2937206786 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f23b0010070 con 0x7f23b8109250 2024-05-27T07:17:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:30.039+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f23b819c160 con 0x7f23b8109250 2024-05-27T07:17:31.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.029+0000 7f23aea00700 1 -- 172.21.15.12:0/2937206786 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f23b00a9da0 con 0x7f23b8109250 2024-05-27T07:17:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f239c0b2900 msgr2=0x7f239c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f239c0b2900 0x7f239c0b4e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f23a8005950 tx=0x7f23a80058e0 comp rx=0 tx=0).stop 2024-05-27T07:17:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b8109250 msgr2=0x7f23b819f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b8109250 0x7f23b819f630 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007ee0 tx=0x7f23b000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 shutdown_connections 2024-05-27T07:17:31.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f239c0b2900 0x7f239c0b4e10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b81025d0 0x7f23b819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f23b8109250 0x7f23b819f630 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 --2- 172.21.15.12:0/2937206786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23b8102f20 0x7f23b819b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 >> 172.21.15.12:0/2937206786 conn(0x7f23b8078360 msgr2=0x7f23b80fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 shutdown_connections 2024-05-27T07:17:31.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:31.034+0000 7f23b7200700 1 -- 172.21.15.12:0/2937206786 wait complete. 2024-05-27T07:17:31.027 INFO:teuthology.orchestra.run.smithi012.stderr:added data pool 4 to fsmap 2024-05-27T07:17:31.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:31 smithi012 ceph-mon[31473]: pgmap v396: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:31.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:31 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2937206786' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:17:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:31 smithi162 ceph-mon[35676]: pgmap v396: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:31 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2937206786' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:17:31.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:31 smithi062 ceph-mon[35304]: pgmap v396: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:31.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:31 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2937206786' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:17:31.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:17:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:32 smithi012 ceph-mon[31473]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:17:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:32 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2937206786' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:17:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:32 smithi012 ceph-mon[31473]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:32.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:32 smithi162 ceph-mon[35676]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:17:32.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:32 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2937206786' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:17:32.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:32 smithi162 ceph-mon[35676]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:32.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:32 smithi062 ceph-mon[35304]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:17:32.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:32 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2937206786' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:17:32.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:32 smithi062 ceph-mon[35304]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:17:33.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:33 smithi012 ceph-mon[31473]: pgmap v398: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:33.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:33 smithi012 ceph-mon[31473]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:17:33.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:33 smithi012 ceph-mon[31473]: Cluster is now healthy 2024-05-27T07:17:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:33 smithi162 ceph-mon[35676]: pgmap v398: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:33 smithi162 ceph-mon[35676]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:17:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:33 smithi162 ceph-mon[35676]: Cluster is now healthy 2024-05-27T07:17:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:33 smithi062 ceph-mon[35304]: pgmap v398: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:33 smithi062 ceph-mon[35304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:17:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:33 smithi062 ceph-mon[35304]: Cluster is now healthy 2024-05-27T07:17:34.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 -- 172.21.15.12:0/1708688444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c8103a40 msgr2=0x7f42c8110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 --2- 172.21.15.12:0/1708688444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c8103a40 0x7f42c8110760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009990 tx=0x7f42b4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 -- 172.21.15.12:0/1708688444 shutdown_connections 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 --2- 172.21.15.12:0/1708688444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42c8110ca0 0x7f42c81130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 --2- 172.21.15.12:0/1708688444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c81030f0 0x7f42c8103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 --2- 172.21.15.12:0/1708688444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c8103a40 0x7f42c8110760 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 -- 172.21.15.12:0/1708688444 >> 172.21.15.12:0/1708688444 conn(0x7f42c80fedb0 msgr2=0x7f42c81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 -- 172.21.15.12:0/1708688444 shutdown_connections 2024-05-27T07:17:34.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.666+0000 7f42cfe00700 1 -- 172.21.15.12:0/1708688444 wait complete. 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.667+0000 7f42cfe00700 1 Processor -- start 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.667+0000 7f42cfe00700 1 -- start start 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c81030f0 0x7f42c819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cfe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c8103a40 0x7f42c819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42c8110ca0 0x7f42c81a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cfe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8116500 con 0x7f42c81030f0 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cfe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8116380 con 0x7f42c8110ca0 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cfe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8116680 con 0x7f42c8103a40 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c81030f0 0x7f42c819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c81030f0 0x7f42c819cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39616/0 (socket says 172.21.15.12:39616) 2024-05-27T07:17:34.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cd400700 1 -- 172.21.15.12:0/3610082104 learned_addr learned my addr 172.21.15.12:0/3610082104 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cca00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c8103a40 0x7f42c819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.668+0000 7f42cde00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42c8110ca0 0x7f42c81a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cd400700 1 -- 172.21.15.12:0/3610082104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c8103a40 msgr2=0x7f42c819d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cd400700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c8103a40 0x7f42c819d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cd400700 1 -- 172.21.15.12:0/3610082104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42c8110ca0 msgr2=0x7f42c81a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cd400700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42c8110ca0 0x7f42c81a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cd400700 1 -- 172.21.15.12:0/3610082104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c81a1e40 con 0x7f42c81030f0 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cde00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42c8110ca0 0x7f42c81a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:17:34.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cd400700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c81030f0 0x7f42c819cf70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00c920 tx=0x7f42bc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42c6000700 1 -- 172.21.15.12:0/3610082104 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc007b00 con 0x7f42c81030f0 2024-05-27T07:17:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42c6000700 1 -- 172.21.15.12:0/3610082104 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42bc004cc0 con 0x7f42c81030f0 2024-05-27T07:17:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c81a2130 con 0x7f42c81030f0 2024-05-27T07:17:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.669+0000 7f42c6000700 1 -- 172.21.15.12:0/3610082104 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0054f0 con 0x7f42c81030f0 2024-05-27T07:17:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.670+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c81a9a50 con 0x7f42c81030f0 2024-05-27T07:17:34.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.671+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c80512b0 con 0x7f42c81030f0 2024-05-27T07:17:34.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.673+0000 7f42c6000700 1 -- 172.21.15.12:0/3610082104 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f42bc007ca0 con 0x7f42c81030f0 2024-05-27T07:17:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.675+0000 7f42c6000700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f42b00b2900 0x7f42b00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:34.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.675+0000 7f42c6000700 1 -- 172.21.15.12:0/3610082104 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f42bc0e2450 con 0x7f42c81030f0 2024-05-27T07:17:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.675+0000 7f42cca00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f42b00b2900 0x7f42b00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.676+0000 7f42cca00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f42b00b2900 0x7f42b00b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f42b4006010 tx=0x7f42b400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:34.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.678+0000 7f42c6000700 1 -- 172.21.15.12:0/3610082104 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f42bc014030 con 0x7f42c81030f0 2024-05-27T07:17:34.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.969+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42c819e300 con 0x7f42c81030f0 2024-05-27T07:17:34.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.971+0000 7f42c6000700 1 -- 172.21.15.12:0/3610082104 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7f42c819e300 con 0x7f42c81030f0 2024-05-27T07:17:34.963 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:17:34.963 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34439,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:17:30.041943+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34475,"qdb_cluster":[34475]},"id":1}]} 2024-05-27T07:17:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f42b00b2900 msgr2=0x7f42b00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f42b00b2900 0x7f42b00b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f42b4006010 tx=0x7f42b400b540 comp rx=0 tx=0).stop 2024-05-27T07:17:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c81030f0 msgr2=0x7f42c819cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c81030f0 0x7f42c819cf70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00c920 tx=0x7f42bc00cce0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 shutdown_connections 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f42b00b2900 0x7f42b00b4e10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42c8110ca0 0x7f42c81a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42c81030f0 0x7f42c819cf70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 --2- 172.21.15.12:0/3610082104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c8103a40 0x7f42c819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 >> 172.21.15.12:0/3610082104 conn(0x7f42c80fedb0 msgr2=0x7f42c8101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 shutdown_connections 2024-05-27T07:17:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:34.975+0000 7f42cfe00700 1 -- 172.21.15.12:0/3610082104 wait complete. 2024-05-27T07:17:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 17 2024-05-27T07:17:35.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:35 smithi012 ceph-mon[31473]: pgmap v399: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:35.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:35 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3610082104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:35.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:35 smithi162 ceph-mon[35676]: pgmap v399: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:35.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:35 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3610082104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:35.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:35 smithi062 ceph-mon[35304]: pgmap v399: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:35.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:35 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3610082104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:35.926 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:17:36.405 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:37.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:37 smithi012 ceph-mon[31473]: pgmap v400: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:37.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:37 smithi162 ceph-mon[35676]: pgmap v400: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:37.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:37 smithi062 ceph-mon[35304]: pgmap v400: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 -- 172.21.15.12:0/2970995744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810a7b0 msgr2=0x7f194810abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 --2- 172.21.15.12:0/2970995744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810a7b0 0x7f194810abc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1934009990 tx=0x7f1934009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 -- 172.21.15.12:0/2970995744 shutdown_connections 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 --2- 172.21.15.12:0/2970995744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810b100 0x7f1948117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 --2- 172.21.15.12:0/2970995744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f194810f6b0 0x7f194810fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 --2- 172.21.15.12:0/2970995744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810a7b0 0x7f194810abc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 -- 172.21.15.12:0/2970995744 >> 172.21.15.12:0/2970995744 conn(0x7f1948076af0 msgr2=0x7f1948076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 -- 172.21.15.12:0/2970995744 shutdown_connections 2024-05-27T07:17:38.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.914+0000 7f1947200700 1 -- 172.21.15.12:0/2970995744 wait complete. 2024-05-27T07:17:38.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 Processor -- start 2024-05-27T07:17:38.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 -- start start 2024-05-27T07:17:38.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810a7b0 0x7f19481a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810b100 0x7f19481a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f194810f6b0 0x7f19481a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f194811a7a0 con 0x7f194810b100 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f194811a620 con 0x7f194810a7b0 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.916+0000 7f1947200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f194811a920 con 0x7f194810f6b0 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810b100 0x7f19481a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810b100 0x7f19481a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39642/0 (socket says 172.21.15.12:39642) 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 -- 172.21.15.12:0/2401695225 learned_addr learned my addr 172.21.15.12:0/2401695225 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:38.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945e00700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810a7b0 0x7f19481a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:38.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1946800700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f194810f6b0 0x7f19481a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:38.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 -- 172.21.15.12:0/2401695225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f194810f6b0 msgr2=0x7f19481a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:38.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f194810f6b0 0x7f19481a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:38.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 -- 172.21.15.12:0/2401695225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810a7b0 msgr2=0x7f19481a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:38.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810a7b0 0x7f19481a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.917+0000 7f1945400700 1 -- 172.21.15.12:0/2401695225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19481a6040 con 0x7f194810b100 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.918+0000 7f1945400700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810b100 0x7f19481a1710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f193400b650 tx=0x7f193400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.918+0000 7f193ea00700 1 -- 172.21.15.12:0/2401695225 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934027070 con 0x7f194810b100 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.918+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19481a62d0 con 0x7f194810b100 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.918+0000 7f193ea00700 1 -- 172.21.15.12:0/2401695225 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1934022470 con 0x7f194810b100 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.918+0000 7f193ea00700 1 -- 172.21.15.12:0/2401695225 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934017650 con 0x7f194810b100 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.918+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19481a67b0 con 0x7f194810b100 2024-05-27T07:17:38.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.919+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19480512b0 con 0x7f194810b100 2024-05-27T07:17:38.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.922+0000 7f193ea00700 1 -- 172.21.15.12:0/2401695225 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1934009d10 con 0x7f194810b100 2024-05-27T07:17:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.923+0000 7f193ea00700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f192c0b2880 0x7f192c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:38.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.924+0000 7f193ea00700 1 -- 172.21.15.12:0/2401695225 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f19340e3420 con 0x7f194810b100 2024-05-27T07:17:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.924+0000 7f1945e00700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f192c0b2880 0x7f192c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:38.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.924+0000 7f1945e00700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f192c0b2880 0x7f192c0b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f193800dad0 tx=0x7f193800d4b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:38.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:38.927+0000 7f193ea00700 1 -- 172.21.15.12:0/2401695225 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1934013070 con 0x7f194810b100 2024-05-27T07:17:39.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.167+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f194802e5e0 con 0x7f194810b100 2024-05-27T07:17:39.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:39 smithi012 ceph-mon[31473]: pgmap v401: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.171+0000 7f193ea00700 1 -- 172.21.15.12:0/2401695225 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24091 (secure 0 0 0) 0x7f19340aaf30 con 0x7f194810b100 2024-05-27T07:17:39.163 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:17:39.163 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":85,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:17:30.041692+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:17:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.175+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f192c0b2880 msgr2=0x7f192c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.175+0000 7f1947200700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f192c0b2880 0x7f192c0b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f193800dad0 tx=0x7f193800d4b0 comp rx=0 tx=0).stop 2024-05-27T07:17:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.175+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810b100 msgr2=0x7f19481a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.175+0000 7f1947200700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810b100 0x7f19481a1710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f193400b650 tx=0x7f193400bfc0 comp rx=0 tx=0).stop 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.175+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 shutdown_connections 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.176+0000 7f1947200700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f192c0b2880 0x7f192c0b4d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.176+0000 7f1947200700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810a7b0 0x7f19481a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.176+0000 7f1947200700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f194810f6b0 0x7f19481a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.176+0000 7f1947200700 1 --2- 172.21.15.12:0/2401695225 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f194810b100 0x7f19481a1710 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.176+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 >> 172.21.15.12:0/2401695225 conn(0x7f1948076af0 msgr2=0x7f19481088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.176+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 shutdown_connections 2024-05-27T07:17:39.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:39.176+0000 7f1947200700 1 -- 172.21.15.12:0/2401695225 wait complete. 2024-05-27T07:17:39.289 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:39 smithi062 ceph-mon[35304]: pgmap v401: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:39.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:39 smithi162 ceph-mon[35676]: pgmap v401: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:39.933 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:17:40.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:40 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2401695225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:40.404 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:40.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:40 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2401695225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:40 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2401695225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:41.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:41 smithi012 ceph-mon[31473]: pgmap v402: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:41.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:41 smithi162 ceph-mon[35676]: pgmap v402: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:41.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:41 smithi062 ceph-mon[35304]: pgmap v402: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:42.299 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:42 smithi012 ceph-mon[31473]: pgmap v403: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:42.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:42 smithi162 ceph-mon[35676]: pgmap v403: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:42.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:42 smithi062 ceph-mon[35304]: pgmap v403: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.907+0000 7f3450e00700 1 -- 172.21.15.12:0/2620827046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 msgr2=0x7f344c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.907+0000 7f3450e00700 1 --2- 172.21.15.12:0/2620827046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c10b320 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f34340099c0 tx=0x7f3434009cd0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.907+0000 7f3450e00700 1 -- 172.21.15.12:0/2620827046 shutdown_connections 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.907+0000 7f3450e00700 1 --2- 172.21.15.12:0/2620827046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f344c106630 0x7f344c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.907+0000 7f3450e00700 1 --2- 172.21.15.12:0/2620827046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f344c10b860 0x7f344c1131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.907+0000 7f3450e00700 1 --2- 172.21.15.12:0/2620827046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c10b320 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.907+0000 7f3450e00700 1 -- 172.21.15.12:0/2620827046 >> 172.21.15.12:0/2620827046 conn(0x7f344c102020 msgr2=0x7f344c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.908+0000 7f3450e00700 1 -- 172.21.15.12:0/2620827046 shutdown_connections 2024-05-27T07:17:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.908+0000 7f3450e00700 1 -- 172.21.15.12:0/2620827046 wait complete. 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 Processor -- start 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 -- start start 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f344c106630 0x7f344c1a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c1a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f344c10b860 0x7f344c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f344c116320 con 0x7f344c10af10 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1161a0 con 0x7f344c106630 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.909+0000 7f3450e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1164a0 con 0x7f344c10b860 2024-05-27T07:17:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c1a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c1a2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47560/0 (socket says 172.21.15.12:47560) 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 -- 172.21.15.12:0/88063125 learned_addr learned my addr 172.21.15.12:0/88063125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f344b400700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f344c106630 0x7f344c1a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f344be00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f344c10b860 0x7f344c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 -- 172.21.15.12:0/88063125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f344c10b860 msgr2=0x7f344c19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f344c10b860 0x7f344c19cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 -- 172.21.15.12:0/88063125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f344c106630 msgr2=0x7f344c1a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f344c106630 0x7f344c1a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.910+0000 7f3442a00700 1 -- 172.21.15.12:0/88063125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c19d270 con 0x7f344c10af10 2024-05-27T07:17:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.911+0000 7f3442a00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c1a2ef0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f343400b580 tx=0x7f343400f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.911+0000 7f3448c00700 1 -- 172.21.15.12:0/88063125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434026070 con 0x7f344c10af10 2024-05-27T07:17:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.911+0000 7f3448c00700 1 -- 172.21.15.12:0/88063125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3434005030 con 0x7f344c10af10 2024-05-27T07:17:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.911+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c19d500 con 0x7f344c10af10 2024-05-27T07:17:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.911+0000 7f3448c00700 1 -- 172.21.15.12:0/88063125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340049f0 con 0x7f344c10af10 2024-05-27T07:17:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.911+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c1a9760 con 0x7f344c10af10 2024-05-27T07:17:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.913+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344c107a90 con 0x7f344c10af10 2024-05-27T07:17:42.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.917+0000 7f3448c00700 1 -- 172.21.15.12:0/88063125 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3434029070 con 0x7f344c10af10 2024-05-27T07:17:42.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.919+0000 7f3448c00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f34300b2900 0x7f34300b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:42.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.919+0000 7f3448c00700 1 -- 172.21.15.12:0/88063125 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8802+0+0 (secure 0 0 0) 0x7f34340e3250 con 0x7f344c10af10 2024-05-27T07:17:42.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.919+0000 7f344b400700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f34300b2900 0x7f34300b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:42.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.920+0000 7f344b400700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f34300b2900 0x7f34300b4e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f343c009d10 tx=0x7f343c009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:42.922+0000 7f3448c00700 1 -- 172.21.15.12:0/88063125 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3434030080 con 0x7f344c10af10 2024-05-27T07:17:43.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.160+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f344c19dfe0 con 0x7f344c10af10 2024-05-27T07:17:43.156 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:17:43.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.165+0000 7f3448c00700 1 -- 172.21.15.12:0/88063125 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24091 (secure 0 0 0) 0x7f3434013070 con 0x7f344c10af10 2024-05-27T07:17:43.157 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":85,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:17:30.041692+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:17:43.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.168+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f34300b2900 msgr2=0x7f34300b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:43.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.168+0000 7f3450e00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f34300b2900 0x7f34300b4e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f343c009d10 tx=0x7f343c009040 comp rx=0 tx=0).stop 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.168+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 msgr2=0x7f344c1a2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.168+0000 7f3450e00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c1a2ef0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f343400b580 tx=0x7f343400f510 comp rx=0 tx=0).stop 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 shutdown_connections 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f34300b2900 0x7f34300b4e10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f344c106630 0x7f344c1a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f344c10b860 0x7f344c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 --2- 172.21.15.12:0/88063125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f344c10af10 0x7f344c1a2ef0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 >> 172.21.15.12:0/88063125 conn(0x7f344c102020 msgr2=0x7f344c10a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 shutdown_connections 2024-05-27T07:17:43.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:43.169+0000 7f3450e00700 1 -- 172.21.15.12:0/88063125 wait complete. 2024-05-27T07:17:43.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:43 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/88063125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:43.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:43 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/88063125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:43.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:43 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/88063125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:43.986 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:17:44.458 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:44.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:44 smithi062 ceph-mon[35304]: pgmap v404: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:44 smithi012 ceph-mon[31473]: pgmap v404: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:44.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:44 smithi162 ceph-mon[35676]: pgmap v404: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:45.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-27T07:17:45.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-05-27T07:17:45.808 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:45 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-05-27T07:17:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-27T07:17:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-27T07:17:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-27T07:17:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-05-27T07:17:45.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-27T07:17:45.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-05-27T07:17:45.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-05-27T07:17:45.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-05-27T07:17:45.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-05-27T07:17:45.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:45 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-05-27T07:17:45.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-27T07:17:45.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-27T07:17:45.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-27T07:17:45.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-05-27T07:17:45.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-27T07:17:45.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-05-27T07:17:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-05-27T07:17:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-05-27T07:17:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-05-27T07:17:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:45 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-05-27T07:17:46.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.698+0000 7fe208400700 1 -- 172.21.15.12:0/2614149383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 msgr2=0x7fe200117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:46.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.698+0000 7fe208400700 1 --2- 172.21.15.12:0/2614149383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 0x7fe200117780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800b0a0 tx=0x7fe1f800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.699+0000 7fe208400700 1 -- 172.21.15.12:0/2614149383 shutdown_connections 2024-05-27T07:17:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.699+0000 7fe208400700 1 --2- 172.21.15.12:0/2614149383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 0x7fe200117780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.699+0000 7fe208400700 1 --2- 172.21.15.12:0/2614149383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe200075b50 0x7fe200075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.699+0000 7fe208400700 1 --2- 172.21.15.12:0/2614149383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2000764a0 0x7fe200114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.699+0000 7fe208400700 1 -- 172.21.15.12:0/2614149383 >> 172.21.15.12:0/2614149383 conn(0x7fe200100030 msgr2=0x7fe200102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.699+0000 7fe208400700 1 -- 172.21.15.12:0/2614149383 shutdown_connections 2024-05-27T07:17:46.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.699+0000 7fe208400700 1 -- 172.21.15.12:0/2614149383 wait complete. 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.700+0000 7fe208400700 1 Processor -- start 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.700+0000 7fe208400700 1 -- start start 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.701+0000 7fe208400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200075b50 0x7fe2001a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.701+0000 7fe208400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2000764a0 0x7fe2001a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.701+0000 7fe208400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 0x7fe2001a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.701+0000 7fe208400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe20011aa60 con 0x7fe200115360 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.701+0000 7fe208400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe20011a8e0 con 0x7fe200075b50 2024-05-27T07:17:46.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.701+0000 7fe208400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe20011abe0 con 0x7fe2000764a0 2024-05-27T07:17:46.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.702+0000 7fe205000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2000764a0 0x7fe2001a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:46.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.702+0000 7fe205000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2000764a0 0x7fe2001a1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:49484/0 (socket says 172.21.15.12:49484) 2024-05-27T07:17:46.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.702+0000 7fe205000700 1 -- 172.21.15.12:0/2735959429 learned_addr learned my addr 172.21.15.12:0/2735959429 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:46.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.702+0000 7fe206400700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 0x7fe2001a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.702+0000 7fe205a00700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200075b50 0x7fe2001a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.703+0000 7fe205000700 1 -- 172.21.15.12:0/2735959429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200075b50 msgr2=0x7fe2001a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.703+0000 7fe205000700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200075b50 0x7fe2001a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.703+0000 7fe205000700 1 -- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 msgr2=0x7fe2001a5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.703+0000 7fe205000700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 0x7fe2001a5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:46.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.703+0000 7fe205000700 1 -- 172.21.15.12:0/2735959429 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2001a6360 con 0x7fe2000764a0 2024-05-27T07:17:46.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-27T07:17:46.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-27T07:17:46.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-27T07:17:46.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-05-27T07:17:46.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.704+0000 7fe205a00700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200075b50 0x7fe2001a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:17:46.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.704+0000 7fe205000700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2000764a0 0x7fe2001a1a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00b530 tx=0x7fe1ec00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.711+0000 7fe1fe000700 1 -- 172.21.15.12:0/2735959429 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec015020 con 0x7fe2000764a0 2024-05-27T07:17:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.711+0000 7fe1fe000700 1 -- 172.21.15.12:0/2735959429 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1ec011cc0 con 0x7fe2000764a0 2024-05-27T07:17:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.711+0000 7fe1fe000700 1 -- 172.21.15.12:0/2735959429 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec010dc0 con 0x7fe2000764a0 2024-05-27T07:17:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.711+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2001a65f0 con 0x7fe2000764a0 2024-05-27T07:17:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.711+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2001a6ae0 con 0x7fe2000764a0 2024-05-27T07:17:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.713+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2000512b0 con 0x7fe2000764a0 2024-05-27T07:17:46.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.715+0000 7fe1fe000700 1 -- 172.21.15.12:0/2735959429 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe1ec018020 con 0x7fe2000764a0 2024-05-27T07:17:46.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.717+0000 7fe1fe000700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1e80b2900 0x7fe1e80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:46.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.717+0000 7fe1fe000700 1 -- 172.21.15.12:0/2735959429 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fe1ec0e2330 con 0x7fe2000764a0 2024-05-27T07:17:46.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.717+0000 7fe205a00700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1e80b2900 0x7fe1e80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:46.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.718+0000 7fe205a00700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1e80b2900 0x7fe1e80b4e10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4007900 tx=0x7fe1f4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:46.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:46.725+0000 7fe1fe000700 1 -- 172.21.15.12:0/2735959429 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe1ec01f020 con 0x7fe2000764a0 2024-05-27T07:17:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: pgmap v406: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:46 smithi162 ceph-mon[35676]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:17:46.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-27T07:17:46.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: pgmap v406: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:46.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:46 smithi062 ceph-mon[35304]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:17:47.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.024+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe20002e7a0 con 0x7fe2000764a0 2024-05-27T07:17:47.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-27T07:17:47.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-05-27T07:17:47.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-05-27T07:17:47.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-05-27T07:17:47.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-05-27T07:17:47.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-05-27T07:17:47.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:17:47.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: pgmap v406: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:47.016 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:46 smithi012 ceph-mon[31473]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:17:47.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.026+0000 7fe1fe000700 1 -- 172.21.15.12:0/2735959429 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7fe1ec0a9cf0 con 0x7fe2000764a0 2024-05-27T07:17:47.017 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:17:47.017 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34391,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34415,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34439,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:17:30.041943+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34475,"qdb_cluster":[34475]},"id":1}]} 2024-05-27T07:17:47.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1e80b2900 msgr2=0x7fe1e80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:47.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1e80b2900 0x7fe1e80b4e10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4007900 tx=0x7fe1f4008040 comp rx=0 tx=0).stop 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2000764a0 msgr2=0x7fe2001a1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2000764a0 0x7fe2001a1a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00b530 tx=0x7fe1ec00b840 comp rx=0 tx=0).stop 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 shutdown_connections 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1e80b2900 0x7fe1e80b4e10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200075b50 0x7fe2001a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe200115360 0x7fe2001a5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.029+0000 7fe208400700 1 --2- 172.21.15.12:0/2735959429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2000764a0 0x7fe2001a1a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.030+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 >> 172.21.15.12:0/2735959429 conn(0x7fe200100030 msgr2=0x7fe200101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.030+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 shutdown_connections 2024-05-27T07:17:47.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:47.030+0000 7fe208400700 1 -- 172.21.15.12:0/2735959429 wait complete. 2024-05-27T07:17:47.022 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 17 2024-05-27T07:17:47.800 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:17:47.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:47 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2735959429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:47.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:47 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2735959429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:48.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:47 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2735959429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:48.235 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:48 smithi162 ceph-mon[35676]: pgmap v408: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:48.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:48 smithi062 ceph-mon[35304]: pgmap v408: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:48 smithi012 ceph-mon[31473]: pgmap v408: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:17:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.463+0000 7f397ac00700 1 -- 172.21.15.12:0/1214798777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3974105da0 msgr2=0x7f397410e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.463+0000 7f397ac00700 1 --2- 172.21.15.12:0/1214798777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3974105da0 0x7f397410e2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3960009990 tx=0x7f3960009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.464+0000 7f397ac00700 1 -- 172.21.15.12:0/1214798777 shutdown_connections 2024-05-27T07:17:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.464+0000 7f397ac00700 1 --2- 172.21.15.12:0/1214798777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f397410e8c0 0x7f3974110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.464+0000 7f397ac00700 1 --2- 172.21.15.12:0/1214798777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3974105450 0x7f3974105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.464+0000 7f397ac00700 1 --2- 172.21.15.12:0/1214798777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3974105da0 0x7f397410e2f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.464+0000 7f397ac00700 1 -- 172.21.15.12:0/1214798777 >> 172.21.15.12:0/1214798777 conn(0x7f39740fedb0 msgr2=0x7f39741011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.464+0000 7f397ac00700 1 -- 172.21.15.12:0/1214798777 shutdown_connections 2024-05-27T07:17:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.464+0000 7f397ac00700 1 -- 172.21.15.12:0/1214798777 wait complete. 2024-05-27T07:17:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 Processor -- start 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 -- start start 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974105450 0x7f397419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3974105da0 0x7f397419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397410e8c0 0x7f39741a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3974114320 con 0x7f397410e8c0 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f39741141a0 con 0x7f3974105450 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.465+0000 7f397ac00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39741144a0 con 0x7f3974105da0 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397410e8c0 0x7f39741a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397410e8c0 0x7f39741a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47596/0 (socket says 172.21.15.12:47596) 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 -- 172.21.15.12:0/791213163 learned_addr learned my addr 172.21.15.12:0/791213163 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3973e00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974105450 0x7f397419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3973400700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3974105da0 0x7f397419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 -- 172.21.15.12:0/791213163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3974105da0 msgr2=0x7f397419d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3974105da0 0x7f397419d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 -- 172.21.15.12:0/791213163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974105450 msgr2=0x7f397419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974105450 0x7f397419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3978c00700 1 -- 172.21.15.12:0/791213163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39741a1df0 con 0x7f397410e8c0 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3973e00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974105450 0x7f397419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.466+0000 7f3973400700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3974105da0 0x7f397419d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.467+0000 7f3978c00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397410e8c0 0x7f39741a1800 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f396c007ee0 tx=0x7f396c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.467+0000 7f3970c00700 1 -- 172.21.15.12:0/791213163 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c00dbd0 con 0x7f397410e8c0 2024-05-27T07:17:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.467+0000 7f3970c00700 1 -- 172.21.15.12:0/791213163 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f396c00dd70 con 0x7f397410e8c0 2024-05-27T07:17:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.467+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39741a20e0 con 0x7f397410e8c0 2024-05-27T07:17:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.467+0000 7f3970c00700 1 -- 172.21.15.12:0/791213163 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c0058d0 con 0x7f397410e8c0 2024-05-27T07:17:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.467+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39741a25d0 con 0x7f397410e8c0 2024-05-27T07:17:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.469+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39740512b0 con 0x7f397410e8c0 2024-05-27T07:17:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.471+0000 7f3970c00700 1 -- 172.21.15.12:0/791213163 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f396c01a020 con 0x7f397410e8c0 2024-05-27T07:17:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.473+0000 7f3970c00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f39580b2880 0x7f39580b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.473+0000 7f3970c00700 1 -- 172.21.15.12:0/791213163 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f396c0e2490 con 0x7f397410e8c0 2024-05-27T07:17:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.473+0000 7f3973e00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f39580b2880 0x7f39580b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.474+0000 7f3973e00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f39580b2880 0x7f39580b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3964005fd0 tx=0x7f3964005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.475+0000 7f3970c00700 1 -- 172.21.15.12:0/791213163 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f396c010070 con 0x7f397410e8c0 2024-05-27T07:17:50.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.713+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f397402e5e0 con 0x7f397410e8c0 2024-05-27T07:17:50.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.715+0000 7f3970c00700 1 -- 172.21.15.12:0/791213163 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24618 (secure 0 0 0) 0x7f396c0b0e20 con 0x7f397410e8c0 2024-05-27T07:17:50.707 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:17:50.707 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":87,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:17:46.671419+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:17:50.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.718+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f39580b2880 msgr2=0x7f39580b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.718+0000 7f397ac00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f39580b2880 0x7f39580b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3964005fd0 tx=0x7f3964005e30 comp rx=0 tx=0).stop 2024-05-27T07:17:50.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.718+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397410e8c0 msgr2=0x7f39741a1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.718+0000 7f397ac00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397410e8c0 0x7f39741a1800 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f396c007ee0 tx=0x7f396c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 shutdown_connections 2024-05-27T07:17:50.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f39580b2880 0x7f39580b4d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974105450 0x7f397419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f397410e8c0 0x7f39741a1800 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 --2- 172.21.15.12:0/791213163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3974105da0 0x7f397419d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 >> 172.21.15.12:0/791213163 conn(0x7f39740fedb0 msgr2=0x7f3974103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:50.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 shutdown_connections 2024-05-27T07:17:50.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:50.719+0000 7f397ac00700 1 -- 172.21.15.12:0/791213163 wait complete. 2024-05-27T07:17:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:50 smithi012 ceph-mon[31473]: pgmap v409: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 0 objects/s recovering 2024-05-27T07:17:51.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:50 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/791213163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:50 smithi162 ceph-mon[35676]: pgmap v409: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 0 objects/s recovering 2024-05-27T07:17:51.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:50 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/791213163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:50 smithi062 ceph-mon[35304]: pgmap v409: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 0 objects/s recovering 2024-05-27T07:17:51.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:50 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/791213163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:17:51.564 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-27T07:17:09.825542+0000', 'modified': '2024-05-27T07:17:30.041943+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34475': {'gid': 34475, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.62:6839/994488749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6838', 'nonce': 994488749}, {'type': 'v1', 'addr': '172.21.15.62:6839', 'nonce': 994488749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 34475, 'qdb_cluster': [34475]} 2024-05-27T07:17:51.564 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-27T07:17:51.565 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:17:52.060 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:52.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:52 smithi012 ceph-mon[31473]: pgmap v410: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 274 B/s, 1 objects/s recovering 2024-05-27T07:17:53.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:52 smithi162 ceph-mon[35676]: pgmap v410: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 274 B/s, 1 objects/s recovering 2024-05-27T07:17:53.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:52 smithi062 ceph-mon[35304]: pgmap v410: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 274 B/s, 1 objects/s recovering 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.410+0000 7fdd6a200700 1 -- 172.21.15.12:0/650628672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 msgr2=0x7fdd64106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.410+0000 7fdd6a200700 1 --2- 172.21.15.12:0/650628672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd64106980 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009990 tx=0x7fdd4c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.410+0000 7fdd6a200700 1 -- 172.21.15.12:0/650628672 shutdown_connections 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.411+0000 7fdd6a200700 1 --2- 172.21.15.12:0/650628672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd64106ec0 0x7fdd64113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.411+0000 7fdd6a200700 1 --2- 172.21.15.12:0/650628672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd6410c5b0 0x7fdd6410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.411+0000 7fdd6a200700 1 --2- 172.21.15.12:0/650628672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd64106980 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.411+0000 7fdd6a200700 1 -- 172.21.15.12:0/650628672 >> 172.21.15.12:0/650628672 conn(0x7fdd64102070 msgr2=0x7fdd641044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.411+0000 7fdd6a200700 1 -- 172.21.15.12:0/650628672 shutdown_connections 2024-05-27T07:17:54.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.411+0000 7fdd6a200700 1 -- 172.21.15.12:0/650628672 wait complete. 2024-05-27T07:17:54.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.412+0000 7fdd6a200700 1 Processor -- start 2024-05-27T07:17:54.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.412+0000 7fdd6a200700 1 -- start start 2024-05-27T07:17:54.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd6a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd6419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd6a200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64106ec0 0x7fdd6419d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd6a200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd6410c5b0 0x7fdd641a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd6a200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdd64116590 con 0x7fdd64106570 2024-05-27T07:17:54.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd6a200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdd64116410 con 0x7fdd6410c5b0 2024-05-27T07:17:54.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd6a200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd64116710 con 0x7fdd64106ec0 2024-05-27T07:17:54.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd63400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd6419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd63400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd6419cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59134/0 (socket says 172.21.15.12:59134) 2024-05-27T07:17:54.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.413+0000 7fdd63400700 1 -- 172.21.15.12:0/3918461773 learned_addr learned my addr 172.21.15.12:0/3918461773 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd5be00700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64106ec0 0x7fdd6419d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd63e00700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd6410c5b0 0x7fdd641a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd63400700 1 -- 172.21.15.12:0/3918461773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64106ec0 msgr2=0x7fdd6419d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd63400700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64106ec0 0x7fdd6419d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd63400700 1 -- 172.21.15.12:0/3918461773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd6410c5b0 msgr2=0x7fdd641a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd63400700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd6410c5b0 0x7fdd641a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd63400700 1 -- 172.21.15.12:0/3918461773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd641a1e30 con 0x7fdd64106570 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.414+0000 7fdd63400700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd6419cf60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400d980 tx=0x7fdd5400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.415+0000 7fdd60c00700 1 -- 172.21.15.12:0/3918461773 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd54017070 con 0x7fdd64106570 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.415+0000 7fdd60c00700 1 -- 172.21.15.12:0/3918461773 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd54010460 con 0x7fdd64106570 2024-05-27T07:17:54.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.415+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd641a2120 con 0x7fdd64106570 2024-05-27T07:17:54.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.415+0000 7fdd60c00700 1 -- 172.21.15.12:0/3918461773 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5400de60 con 0x7fdd64106570 2024-05-27T07:17:54.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.415+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd641a9a40 con 0x7fdd64106570 2024-05-27T07:17:54.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.419+0000 7fdd60c00700 1 -- 172.21.15.12:0/3918461773 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdd54010610 con 0x7fdd64106570 2024-05-27T07:17:54.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 Processor -- start 2024-05-27T07:17:54.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 -- start start 2024-05-27T07:17:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdc02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdc0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564abcceedf0 con 0x564abcdc0e20 2024-05-27T07:17:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564abccde990 con 0x564abcc41df0 2024-05-27T07:17:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.420+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564abcce29d0 con 0x564abcd2d9a0 2024-05-27T07:17:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd60c00700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd480b2900 0x7fdd480b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd5be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd60c00700 1 -- 172.21.15.12:0/3918461773 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fdd540e20e0 con 0x7fdd64106570 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd63400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdc0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd63400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdc0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:38890/0 (socket says 172.21.15.12:38890) 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd63400700 1 -- 172.21.15.12:0/3501759624 learned_addr learned my addr 172.21.15.12:0/3501759624 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd5be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59146/0 (socket says 172.21.15.12:59146) 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd63e00700 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdc02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.421+0000 7fdd63e00700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd480b2900 0x7fdd480b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd5be00700 1 -- 172.21.15.12:0/3501759624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 msgr2=0x564abcdc02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd5be00700 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdc02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd5be00700 1 -- 172.21.15.12:0/3501759624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 msgr2=0x564abcdc0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd5be00700 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdc0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd5be00700 1 -- 172.21.15.12:0/3501759624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564abcca73d0 con 0x564abcdc0e20 2024-05-27T07:17:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd5be00700 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcd0c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0098e0 tx=0x7fdd4c0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d5d7b79d369cf6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd63e00700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd480b2900 0x7fdd480b4e10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009fd0 tx=0x7fdd5c009050 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.422+0000 7fdd58c00700 1 -- 172.21.15.12:0/3501759624 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c026070 con 0x564abcdc0e20 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.423+0000 7fdd58c00700 1 -- 172.21.15.12:0/3501759624 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd4c003c50 con 0x564abcdc0e20 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.423+0000 7fdd58c00700 1 -- 172.21.15.12:0/3501759624 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c005640 con 0x564abcdc0e20 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.423+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3501759624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 msgr2=0x564abcdcd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.423+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcd0c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0098e0 tx=0x7fdd4c0096a0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3501759624 shutdown_connections 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcd0c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdc02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3501759624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdc0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3501759624 >> 172.21.15.12:0/3501759624 conn(0x564abccdb340 msgr2=0x564abcdbc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:54.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3501759624 shutdown_connections 2024-05-27T07:17:54.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3501759624 wait complete. 2024-05-27T07:17:54.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.424+0000 7fdd78d40b80 1 -- start start 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 1 Processor -- start 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdcab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564abcbe4f50 con 0x564abcdc0e20 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564abcc7f710 con 0x564abcc41df0 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564abcdcdad0 con 0x564abcd2d9a0 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.425+0000 7fdd78d40b80 10 client.0 _ll_register_callbacks cb 0x564abcc88c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59160/0 (socket says 172.21.15.12:59160) 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 -- 172.21.15.12:0/213710731 learned_addr learned my addr 172.21.15.12:0/213710731 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63400700 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd5be00700 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdcab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 -- 172.21.15.12:0/213710731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 msgr2=0x564abcdcab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdcab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 -- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 msgr2=0x564abcdca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 -- 172.21.15.12:0/213710731 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564abcc2e880 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.426+0000 7fdd63e00700 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcb090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c018c50 tx=0x7fdd5c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8653f1ffc91c07af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd51600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd51600700 1 -- 172.21.15.12:0/213710731 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c004bb0 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd4c003c50 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd51600700 1 -- 172.21.15.12:0/213710731 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fdd5c012040 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd51600700 1 -- 172.21.15.12:0/213710731 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fdd5c031050 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd78d40b80 10 client.14721 fetch_fsmap learned FSMap version 17 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x564abccaf010 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.427+0000 7fdd51600700 1 -- 172.21.15.12:0/213710731 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd4c003c50 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.428+0000 7fdd51600700 1 -- 172.21.15.12:0/213710731 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fdd5c026590 con 0x564abcdc0e20 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.428+0000 7fdd78d40b80 10 client.14721 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:17:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.428+0000 7fdd78d40b80 10 client.14721 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:17:54.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.428+0000 7fdd78d40b80 20 client.14721 populate_metadata read hostname 'smithi012' 2024-05-27T07:17:54.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.428+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abcddab40 0x564abcddbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.428+0000 7fdd78d40b80 4 client.14721 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:17:54.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.428+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564abccf0d40 con 0x564abcddab40 2024-05-27T07:17:54.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.429+0000 7fdd63400700 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abcddab40 0x564abcddbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.430+0000 7fdd63400700 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abcddab40 0x564abcddbfb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa98114140cd549e server_cookie=492e3423cc0a8e6c in_seq=0 out_seq=0 2024-05-27T07:17:54.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.430+0000 7fdd51600700 10 client.14721 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:17:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd51600700 1 -- 172.21.15.12:0/213710731 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564abccf0d40 con 0x564abcddab40 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd51600700 10 client.14721 handle_command_reply: tid=0 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd78d40b80 1 client.14721 shutdown 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 msgr2=0x564abcdcb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcb090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c018c50 tx=0x7fdd5c004ad0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 shutdown_connections 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abcddab40 0x564abcddbfb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.432+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.433+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcdcb090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.433+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/213710731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcdcab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.433+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 >> 172.21.15.12:0/213710731 conn(0x564abccdb340 msgr2=0x564abcdc02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.433+0000 7fdd51600700 0 client.14721 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.433+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 shutdown_connections 2024-05-27T07:17:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.433+0000 7fdd78d40b80 1 -- 172.21.15.12:0/213710731 wait complete. 2024-05-27T07:17:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.433+0000 7fdd78d40b80 20 client.14721 trim_cache size 0 max 16384 2024-05-27T07:17:54.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.456+0000 7fdd78d40b80 1 Processor -- start 2024-05-27T07:17:54.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.456+0000 7fdd78d40b80 1 -- start start 2024-05-27T07:17:54.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdbc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcd2d9a0 0x564abcdc5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcdc0e20 0x564abcdd6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564abcdda2f0 con 0x564abcd2d9a0 2024-05-27T07:17:54.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564abcdda9c0 con 0x564abcc41df0 2024-05-27T07:17:54.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564abcddb090 con 0x564abcdc0e20 2024-05-27T07:17:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd5be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcd2d9a0 0x564abcdc5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd5be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcd2d9a0 0x564abcdc5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59176/0 (socket says 172.21.15.12:59176) 2024-05-27T07:17:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.457+0000 7fdd5be00700 1 -- 172.21.15.12:0/2620271812 learned_addr learned my addr 172.21.15.12:0/2620271812 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd63400700 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdbc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd63e00700 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcdc0e20 0x564abcdd6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd5be00700 1 -- 172.21.15.12:0/2620271812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcdc0e20 msgr2=0x564abcdd6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd5be00700 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcdc0e20 0x564abcdd6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd5be00700 1 -- 172.21.15.12:0/2620271812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 msgr2=0x564abcdbc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd5be00700 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdbc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd5be00700 1 -- 172.21.15.12:0/2620271812 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564abcdcb400 con 0x564abcd2d9a0 2024-05-27T07:17:54.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.458+0000 7fdd5be00700 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcd2d9a0 0x564abcdc5de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0098e0 tx=0x7fdd4c005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80836f9ce81da81a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.459+0000 7fdd53e00700 1 -- 172.21.15.12:0/2620271812 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c02f070 con 0x564abcd2d9a0 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.459+0000 7fdd53e00700 1 -- 172.21.15.12:0/2620271812 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd4c0037c0 con 0x564abcd2d9a0 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.459+0000 7fdd53e00700 1 -- 172.21.15.12:0/2620271812 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c026070 con 0x564abcd2d9a0 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 -- 172.21.15.12:0/2620271812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcd2d9a0 msgr2=0x564abcdc5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcd2d9a0 0x564abcdc5de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0098e0 tx=0x7fdd4c005250 comp rx=0 tx=0).stop 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 -- 172.21.15.12:0/2620271812 shutdown_connections 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcdbc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcdc0e20 0x564abcdd6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/2620271812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcd2d9a0 0x564abcdc5de0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 -- 172.21.15.12:0/2620271812 >> 172.21.15.12:0/2620271812 conn(0x564abccdb340 msgr2=0x564abcdbbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 -- 172.21.15.12:0/2620271812 shutdown_connections 2024-05-27T07:17:54.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 -- 172.21.15.12:0/2620271812 wait complete. 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.460+0000 7fdd78d40b80 1 -- start start 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 1 Processor -- start 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcde58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcddebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcddf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564abcddbe30 con 0x564abcdc0e20 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564abcddb760 con 0x564abcc41df0 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564abcddc500 con 0x564abcd2d9a0 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.461+0000 7fdd78d40b80 10 client.0 _ll_register_callbacks cb 0x564abcc88c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcddf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcddf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59180/0 (socket says 172.21.15.12:59180) 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 -- 172.21.15.12:0/3125847736 learned_addr learned my addr 172.21.15.12:0/3125847736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:54.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63400700 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcde58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd5be00700 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcddebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 -- 172.21.15.12:0/3125847736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 msgr2=0x564abcddebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcddebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 -- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 msgr2=0x564abcde58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:54.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcde58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 -- 172.21.15.12:0/3125847736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564abcddf660 con 0x564abcdc0e20 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd63e00700 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcddf120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009450 tx=0x7fdd5c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b272eccc64886e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.462+0000 7fdd51600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.463+0000 7fdd51600700 1 -- 172.21.15.12:0/3125847736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c03a060 con 0x564abcdc0e20 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.463+0000 7fdd51600700 1 -- 172.21.15.12:0/3125847736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fdd5c004d50 con 0x564abcdc0e20 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.463+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd4c0037c0 con 0x564abcdc0e20 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.463+0000 7fdd51600700 1 -- 172.21.15.12:0/3125847736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fdd5c041050 con 0x564abcdc0e20 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.463+0000 7fdd78d40b80 10 client.14727 fetch_fsmap learned FSMap version 17 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.463+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x564abcde4030 con 0x564abcdc0e20 2024-05-27T07:17:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.463+0000 7fdd51600700 1 -- 172.21.15.12:0/3125847736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd4c0037c0 con 0x564abcdc0e20 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd51600700 1 -- 172.21.15.12:0/3125847736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fdd5c005040 con 0x564abcdc0e20 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd78d40b80 10 client.14727 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd78d40b80 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd78d40b80 20 client.14727 populate_metadata read hostname 'smithi012' 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abce48340 0x564abce497b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd78d40b80 4 client.14727 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "status"}) -- 0x564abccea520 con 0x564abce48340 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.464+0000 7fdd63400700 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abce48340 0x564abce497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.465+0000 7fdd63400700 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abce48340 0x564abce497b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d9cfdef455510e3 server_cookie=614d8fc3d0fe4510 in_seq=0 out_seq=0 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.465+0000 7fdd51600700 10 client.14727 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.466+0000 7fdd51600700 1 -- 172.21.15.12:0/3125847736 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x564abccea520 con 0x564abce48340 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.466+0000 7fdd51600700 10 client.14727 handle_command_reply: tid=0 2024-05-27T07:17:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.466+0000 7fdd78d40b80 1 client.14727 shutdown 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.466+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 msgr2=0x564abcddf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.466+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcddf120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009450 tx=0x7fdd5c005d90 comp rx=0 tx=0).stop 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.466+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 shutdown_connections 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.466+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564abce48340 0x564abce497b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abcc41df0 0x564abcde58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564abcdc0e20 0x564abcddf120 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd78d40b80 1 --2- 172.21.15.12:0/3125847736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564abcd2d9a0 0x564abcddebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 >> 172.21.15.12:0/3125847736 conn(0x564abccdb340 msgr2=0x564abcdc0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd51600700 0 client.14727 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 shutdown_connections 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd78d40b80 1 -- 172.21.15.12:0/3125847736 wait complete. 2024-05-27T07:17:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.467+0000 7fdd78d40b80 20 client.14727 trim_cache size 0 max 16384 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 34475, 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 43.679577518000002, 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 17, 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 87, 2024-05-27T07:17:54.462 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 85, 2024-05-27T07:17:54.463 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 63.923432671 2024-05-27T07:17:54.463 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:17:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd480b2900 msgr2=0x7fdd480b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd480b2900 0x7fdd480b4e10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009fd0 tx=0x7fdd5c009050 comp rx=0 tx=0).stop 2024-05-27T07:17:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 msgr2=0x7fdd6419cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd6419cf60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400d980 tx=0x7fdd5400dd40 comp rx=0 tx=0).stop 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 shutdown_connections 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd480b2900 0x7fdd480b4e10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd6410c5b0 0x7fdd641a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd64106570 0x7fdd6419cf60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 --2- 172.21.15.12:0/3918461773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64106ec0 0x7fdd6419d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 >> 172.21.15.12:0/3918461773 conn(0x7fdd64102070 msgr2=0x7fdd64104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 shutdown_connections 2024-05-27T07:17:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:54.468+0000 7fdd6a200700 1 -- 172.21.15.12:0/3918461773 wait complete. 2024-05-27T07:17:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:54 smithi012 ceph-mon[31473]: pgmap v411: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s, 1 objects/s recovering 2024-05-27T07:17:55.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:54 smithi162 ceph-mon[35676]: pgmap v411: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s, 1 objects/s recovering 2024-05-27T07:17:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:54 smithi062 ceph-mon[35304]: pgmap v411: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s, 1 objects/s recovering 2024-05-27T07:17:55.339 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:17:55.339 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:17:55.339 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:17:55.809 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:57.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:56 smithi012 ceph-mon[31473]: pgmap v412: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 219 B/s, 0 objects/s recovering 2024-05-27T07:17:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:56 smithi162 ceph-mon[35676]: pgmap v412: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 219 B/s, 0 objects/s recovering 2024-05-27T07:17:57.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:56 smithi062 ceph-mon[35304]: pgmap v412: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 219 B/s, 0 objects/s recovering 2024-05-27T07:17:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.114+0000 7f043ce00700 1 -- 172.21.15.12:0/2281130854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0438106ea0 msgr2=0x7f0438113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.114+0000 7f043ce00700 1 --2- 172.21.15.12:0/2281130854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0438106ea0 0x7f0438113330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f043000b0a0 tx=0x7f043000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.114+0000 7f043ce00700 1 -- 172.21.15.12:0/2281130854 shutdown_connections 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.114+0000 7f043ce00700 1 --2- 172.21.15.12:0/2281130854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0438106ea0 0x7f0438113330 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.114+0000 7f043ce00700 1 --2- 172.21.15.12:0/2281130854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f043810c590 0x7f043810c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.114+0000 7f043ce00700 1 --2- 172.21.15.12:0/2281130854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0438106550 0x7f0438106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.114+0000 7f043ce00700 1 -- 172.21.15.12:0/2281130854 >> 172.21.15.12:0/2281130854 conn(0x7f0438102050 msgr2=0x7f04381044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.115+0000 7f043ce00700 1 -- 172.21.15.12:0/2281130854 shutdown_connections 2024-05-27T07:17:58.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.115+0000 7f043ce00700 1 -- 172.21.15.12:0/2281130854 wait complete. 2024-05-27T07:17:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 Processor -- start 2024-05-27T07:17:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 -- start start 2024-05-27T07:17:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0438106550 0x7f043819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0438106ea0 0x7f043819d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f043810c590 0x7f04381a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f04381165c0 con 0x7f043810c590 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0438116440 con 0x7f0438106ea0 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.116+0000 7f043ce00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0438116740 con 0x7f0438106550 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f043810c590 0x7f04381a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f043810c590 0x7f04381a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59204/0 (socket says 172.21.15.12:59204) 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 -- 172.21.15.12:0/750863536 learned_addr learned my addr 172.21.15.12:0/750863536 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0435e00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0438106550 0x7f043819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:58.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0435400700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0438106ea0 0x7f043819d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 -- 172.21.15.12:0/750863536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0438106550 msgr2=0x7f043819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0438106550 0x7f043819cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 -- 172.21.15.12:0/750863536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0438106ea0 msgr2=0x7f043819d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0438106ea0 0x7f043819d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.117+0000 7f0436800700 1 -- 172.21.15.12:0/750863536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04381a1d20 con 0x7f043810c590 2024-05-27T07:17:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.118+0000 7f0436800700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f043810c590 0x7f04381a1730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f043000b070 tx=0x7f0430004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.118+0000 7f042ea00700 1 -- 172.21.15.12:0/750863536 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430021070 con 0x7f043810c590 2024-05-27T07:17:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.118+0000 7f042ea00700 1 -- 172.21.15.12:0/750863536 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0430007c00 con 0x7f043810c590 2024-05-27T07:17:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.118+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04381a1fb0 con 0x7f043810c590 2024-05-27T07:17:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.118+0000 7f042ea00700 1 -- 172.21.15.12:0/750863536 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430004780 con 0x7f043810c590 2024-05-27T07:17:58.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.118+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04381a24e0 con 0x7f043810c590 2024-05-27T07:17:58.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.119+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043810e950 con 0x7f043810c590 2024-05-27T07:17:58.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.122+0000 7f042ea00700 1 -- 172.21.15.12:0/750863536 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0430003f20 con 0x7f043810c590 2024-05-27T07:17:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.123+0000 7f042ea00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f041c0b2900 0x7f041c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.123+0000 7f042ea00700 1 -- 172.21.15.12:0/750863536 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f043000f020 con 0x7f043810c590 2024-05-27T07:17:58.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.124+0000 7f0435e00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f041c0b2900 0x7f041c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:58.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.124+0000 7f0435e00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f041c0b2900 0x7f041c0b4e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0428005950 tx=0x7f04280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:58.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.126+0000 7f042ea00700 1 -- 172.21.15.12:0/750863536 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f043002a020 con 0x7f043810c590 2024-05-27T07:17:58.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.420+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f04380512b0 con 0x7f043810c590 2024-05-27T07:17:58.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.421+0000 7f042ea00700 1 -- 172.21.15.12:0/750863536 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f04380512b0 con 0x7f043810c590 2024-05-27T07:17:58.412 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:17:58.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f041c0b2900 msgr2=0x7f041c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:58.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f041c0b2900 0x7f041c0b4e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0428005950 tx=0x7f04280058e0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f043810c590 msgr2=0x7f04381a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:58.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f043810c590 0x7f04381a1730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f043000b070 tx=0x7f0430004e00 comp rx=0 tx=0).stop 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 shutdown_connections 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f041c0b2900 0x7f041c0b4e10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f043810c590 0x7f04381a1730 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0438106550 0x7f043819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 --2- 172.21.15.12:0/750863536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0438106ea0 0x7f043819d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 >> 172.21.15.12:0/750863536 conn(0x7f0438102050 msgr2=0x7f04381048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.424+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 shutdown_connections 2024-05-27T07:17:58.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:58.425+0000 7f043ce00700 1 -- 172.21.15.12:0/750863536 wait complete. 2024-05-27T07:17:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:58 smithi012 ceph-mon[31473]: pgmap v413: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 198 B/s, 0 objects/s recovering 2024-05-27T07:17:59.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:17:58 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/750863536' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:17:59.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:58 smithi162 ceph-mon[35676]: pgmap v413: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 198 B/s, 0 objects/s recovering 2024-05-27T07:17:59.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:17:58 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/750863536' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:17:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:58 smithi062 ceph-mon[35304]: pgmap v413: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 198 B/s, 0 objects/s recovering 2024-05-27T07:17:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:17:58 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/750863536' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:17:59.357 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:17:59.358 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:17:59.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:17:59.359 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-05-27T07:17:59.417 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:17:59.417 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:17:59.417 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-05-27T07:17:59.418 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-05-27T07:17:59.419 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-27T07:17:59.419 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41792sec preferred_lft 41792sec 2024-05-27T07:17:59.419 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-05-27T07:17:59.419 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:17:59.419 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:17:59.419 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-05-27T07:17:59.419 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-05-27T07:17:59.420 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:17:59.420 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:17:59.420 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:17:59.420 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:17:59.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-05-27T07:17:59.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-05-27T07:17:59.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:17:59.421 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:17:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:17:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:17:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:17:59.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:17:59.920 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:17:59.990 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-05-27T07:17:59.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:17:59.995 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-05-27T07:18:00.046 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-05-27T07:18:00.046 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:18:00.046 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-27T07:18:00.046 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-27T07:18:00.046 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:18:00.047 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:18:00.047 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:18:00.047 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-27T07:18:00.047 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-27T07:18:00.047 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-27T07:18:00.047 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:18:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:18:00.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:18:00.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:00.816 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-05-27T07:18:00.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:00.873 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-05-27T07:18:00.927 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:18:00.927 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:18:00.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:18:00.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-27T07:18:00.928 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:18:01.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:18:01.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:00 smithi012 ceph-mon[31473]: pgmap v414: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 182 B/s, 0 objects/s recovering 2024-05-27T07:18:01.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:18:01.122 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-27T07:18:01.122 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:18:01.122 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:18:01.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-27T07:18:01.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:18:01.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-27T07:18:01.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-27T07:18:01.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-27T07:18:01.123 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:18:01.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:00 smithi162 ceph-mon[35676]: pgmap v414: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 182 B/s, 0 objects/s recovering 2024-05-27T07:18:01.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:18:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:00 smithi062 ceph-mon[35304]: pgmap v414: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 182 B/s, 0 objects/s recovering 2024-05-27T07:18:01.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:18:01.423 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:18:01.423 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:18:01.423 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-05-27T07:18:01.423 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:18:01.423 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:18:01.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:18:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:18:01.637 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-27T07:18:01.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:18:01.637 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:18:01.683 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:18:01.683 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:18:01.683 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:18:01.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-05-27T07:18:01.852 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-05-27T07:18:01.907 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:18:01.908 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/7fbf726264480c421aca3218d2013f70788ff034dc144fceebe5ec0cf52bf1ea/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/60aebd048ed37d2028603da24217d71224cd3b5b0b0f883e05ded68b3cc907fc/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/c9e05f2c227477b28c8a4ec8fdd7d2f74da4ac11eb899b318ba9d13c11d9b36f/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/4e3cfc19c14a3fac132bd4f33d5fc2900f75567b100b01f5c8d277aef4cf6142/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/fb7cf68777b803513b45652e1a3fd26be65494cc1d3be62a4780623b2b3ea83f/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b5c7dd98ced0dddde9eefe8d543de5f5d6900d4fe9e6a5de2a13bbc6fd3cdba0/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/86ac1d463a1fee0b386559beedf9fdd66f0fc75c2cb77e088b477352a78b5695/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/453ea3f6e74c1b2ec3db112190abe48827bb5baa90b63bb79c724a315baf8761/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/87eafa0cfd1573804569dce33ed5df40a6693954e5f82037af861b94f0049c6d/merged 2024-05-27T07:18:01.909 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/0df1da80208360021a4377028bf943919a9988a7dfc79ce6406fa6fde4521ec0/merged 2024-05-27T07:18:01.910 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-05-27T07:18:01.910 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:18:01.910 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:18:01.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:01.911 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-05-27T07:18:01.962 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-27T07:18:01.962 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-27T07:18:01.967 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-05-27T07:18:02.036 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:18:02.079 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-05-27T07:18:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/7fbf726264480c421aca3218d2013f70788ff034dc144fceebe5ec0cf52bf1ea/merged 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/60aebd048ed37d2028603da24217d71224cd3b5b0b0f883e05ded68b3cc907fc/merged 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-05-27T07:18:02.081 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/c9e05f2c227477b28c8a4ec8fdd7d2f74da4ac11eb899b318ba9d13c11d9b36f/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/4e3cfc19c14a3fac132bd4f33d5fc2900f75567b100b01f5c8d277aef4cf6142/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/fb7cf68777b803513b45652e1a3fd26be65494cc1d3be62a4780623b2b3ea83f/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b5c7dd98ced0dddde9eefe8d543de5f5d6900d4fe9e6a5de2a13bbc6fd3cdba0/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/86ac1d463a1fee0b386559beedf9fdd66f0fc75c2cb77e088b477352a78b5695/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/453ea3f6e74c1b2ec3db112190abe48827bb5baa90b63bb79c724a315baf8761/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/87eafa0cfd1573804569dce33ed5df40a6693954e5f82037af861b94f0049c6d/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/0df1da80208360021a4377028bf943919a9988a7dfc79ce6406fa6fde4521ec0/merged 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:18:02.082 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:18:02.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:02.083 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-05-27T07:18:02.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:2024-05-27T07:18:02.265+0000 7f2433d20480 -1 init, newargv = 0x7f2418056f20 newargc=15 2024-05-27T07:18:02.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[77712]: starting ceph client 2024-05-27T07:18:02.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[77712]: starting fuse 2024-05-27T07:18:03.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:02 smithi012 ceph-mon[31473]: pgmap v415: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 48 B/s, 0 objects/s recovering 2024-05-27T07:18:03.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:02 smithi162 ceph-mon[35676]: pgmap v415: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 48 B/s, 0 objects/s recovering 2024-05-27T07:18:03.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:02 smithi062 ceph-mon[35304]: pgmap v415: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 48 B/s, 0 objects/s recovering 2024-05-27T07:18:03.277 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-05-27T07:18:03.375 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-05-27T07:18:03.436 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:18:03.437 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/7fbf726264480c421aca3218d2013f70788ff034dc144fceebe5ec0cf52bf1ea/merged 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/60aebd048ed37d2028603da24217d71224cd3b5b0b0f883e05ded68b3cc907fc/merged 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/c9e05f2c227477b28c8a4ec8fdd7d2f74da4ac11eb899b318ba9d13c11d9b36f/merged 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/4e3cfc19c14a3fac132bd4f33d5fc2900f75567b100b01f5c8d277aef4cf6142/merged 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/fb7cf68777b803513b45652e1a3fd26be65494cc1d3be62a4780623b2b3ea83f/merged 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b5c7dd98ced0dddde9eefe8d543de5f5d6900d4fe9e6a5de2a13bbc6fd3cdba0/merged 2024-05-27T07:18:03.438 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/86ac1d463a1fee0b386559beedf9fdd66f0fc75c2cb77e088b477352a78b5695/merged 2024-05-27T07:18:03.439 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/453ea3f6e74c1b2ec3db112190abe48827bb5baa90b63bb79c724a315baf8761/merged 2024-05-27T07:18:03.439 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/87eafa0cfd1573804569dce33ed5df40a6693954e5f82037af861b94f0049c6d/merged 2024-05-27T07:18:03.439 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/0df1da80208360021a4377028bf943919a9988a7dfc79ce6406fa6fde4521ec0/merged 2024-05-27T07:18:03.439 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-05-27T07:18:03.439 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:18:03.439 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:18:03.439 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.admin 2024-05-27T07:18:03.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:03.440 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-05-27T07:18:03.497 INFO:teuthology.orchestra.run.smithi012.stdout:119 2024-05-27T07:18:03.497 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-05-27T07:18:03.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-05-27T07:18:03.497 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-05-27T07:18:03.498 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.admin")) 2024-05-27T07:18:03.499 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-05-27T07:18:03.681 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.admin.77712.asok 2024-05-27T07:18:03.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:18:03.699 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.77712.asok 2024-05-27T07:18:03.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:03.700 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.77712.asok status 2024-05-27T07:18:04.009 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:18:04.009 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-05-27T07:18:04.009 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "38f9d900c83c63799fbdbe61acc9a11b0d3554a6", 2024-05-27T07:18:04.009 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "admin", 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "77712", 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/" 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 14739, 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-05-27T07:18:04.010 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 14739 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 919432593 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 919432593 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.14739 192.168.0.1:0/919432593", 2024-05-27T07:18:04.011 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/919432593", 2024-05-27T07:18:04.012 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 1, 2024-05-27T07:18:04.012 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 17, 2024-05-27T07:18:04.012 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 87, 2024-05-27T07:18:04.012 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-05-27T07:18:04.012 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-05-27T07:18:04.012 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-05-27T07:18:04.012 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:18:04.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:04.031 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-05-27T07:18:04.031 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:18:04.091 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-05-27T07:18:04.092 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:18:04.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:04.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:18:04.183 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-27T07:18:04.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:18:04.343 INFO:teuthology.orchestra.run.smithi012.stdout:# file: . 2024-05-27T07:18:04.343 INFO:teuthology.orchestra.run.smithi012.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-27T07:18:04.343 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:18:04.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:18:04.354 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:18:04.354 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:18:04.412 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:18:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:04 smithi012 ceph-mon[31473]: pgmap v416: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:04 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:04 smithi162 ceph-mon[35676]: pgmap v416: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:04 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:05.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:04 smithi062 ceph-mon[35304]: pgmap v416: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:05.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:04 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:06.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:06 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:06 smithi062 ceph-mon[35304]: pgmap v417: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-05-27T07:18:06.493 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.493 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.493 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.493 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:06 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.493 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:06 smithi162 ceph-mon[35676]: pgmap v417: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-05-27T07:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:06 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:06 smithi012 ceph-mon[31473]: pgmap v417: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-05-27T07:18:07.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 -- 172.21.15.12:0/489758082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038110b90 msgr2=0x7f5038112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:07.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 --2- 172.21.15.12:0/489758082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038110b90 0x7f5038112ff0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f503000b0a0 tx=0x7f503000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 -- 172.21.15.12:0/489758082 shutdown_connections 2024-05-27T07:18:07.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 --2- 172.21.15.12:0/489758082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5038075d20 0x7f5038076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 --2- 172.21.15.12:0/489758082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038110b90 0x7f5038112ff0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 --2- 172.21.15.12:0/489758082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5038076670 0x7f5038110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 -- 172.21.15.12:0/489758082 >> 172.21.15.12:0/489758082 conn(0x7f5038100030 msgr2=0x7f5038102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:07.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 -- 172.21.15.12:0/489758082 shutdown_connections 2024-05-27T07:18:07.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.743+0000 7f503e600700 1 -- 172.21.15.12:0/489758082 wait complete. 2024-05-27T07:18:07.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 Processor -- start 2024-05-27T07:18:07.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 -- start start 2024-05-27T07:18:07.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038075d20 0x7f503819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5038076670 0x7f503819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5038110b90 0x7f50381a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5038116520 con 0x7f5038075d20 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50381163a0 con 0x7f5038110b90 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.745+0000 7f503e600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f50381166a0 con 0x7f5038076670 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5036a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5038076670 0x7f503819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038075d20 0x7f503819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038075d20 0x7f503819ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40854/0 (socket says 172.21.15.12:40854) 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 -- 172.21.15.12:0/3369841861 learned_addr learned my addr 172.21.15.12:0/3369841861 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:07.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037e00700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5038110b90 0x7f50381a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:07.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 -- 172.21.15.12:0/3369841861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5038076670 msgr2=0x7f503819d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:07.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5038076670 0x7f503819d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 -- 172.21.15.12:0/3369841861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5038110b90 msgr2=0x7f50381a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:07.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5038110b90 0x7f50381a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.746+0000 7f5037400700 1 -- 172.21.15.12:0/3369841861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50381a1da0 con 0x7f5038075d20 2024-05-27T07:18:07.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.747+0000 7f5037400700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038075d20 0x7f503819ced0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f502800e9a0 tx=0x7f502800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:07.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.747+0000 7f502fe00700 1 -- 172.21.15.12:0/3369841861 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502800ca10 con 0x7f5038075d20 2024-05-27T07:18:07.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.747+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50381a2030 con 0x7f5038075d20 2024-05-27T07:18:07.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.747+0000 7f502fe00700 1 -- 172.21.15.12:0/3369841861 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f502800cbb0 con 0x7f5038075d20 2024-05-27T07:18:07.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.747+0000 7f502fe00700 1 -- 172.21.15.12:0/3369841861 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028018680 con 0x7f5038075d20 2024-05-27T07:18:07.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.748+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50381041e0 con 0x7f5038075d20 2024-05-27T07:18:07.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.749+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5038077290 con 0x7f5038075d20 2024-05-27T07:18:07.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.755+0000 7f502fe00700 1 -- 172.21.15.12:0/3369841861 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5028010430 con 0x7f5038075d20 2024-05-27T07:18:07.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.757+0000 7f502fe00700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f501c0b2900 0x7f501c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:07.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.757+0000 7f502fe00700 1 -- 172.21.15.12:0/3369841861 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f50280e2b70 con 0x7f5038075d20 2024-05-27T07:18:07.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.758+0000 7f5036a00700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f501c0b2900 0x7f501c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:07.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.758+0000 7f502fe00700 1 -- 172.21.15.12:0/3369841861 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f50280aa4d0 con 0x7f5038075d20 2024-05-27T07:18:07.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:07.759+0000 7f5036a00700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f501c0b2900 0x7f501c0b4e10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50381034c0 tx=0x7f5024009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.027+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f503819e230 con 0x7f5038075d20 2024-05-27T07:18:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.027+0000 7f502fe00700 1 -- 172.21.15.12:0/3369841861 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+589 (secure 0 0 0) 0x7f50280a9dd0 con 0x7f5038075d20 2024-05-27T07:18:08.018 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/4121651169 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1199837030 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/949043977 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1631546627 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1064527569 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:08.019 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3780513737 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.030+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f501c0b2900 msgr2=0x7f501c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.030+0000 7f503e600700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f501c0b2900 0x7f501c0b4e10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50381034c0 tx=0x7f5024009260 comp rx=0 tx=0).stop 2024-05-27T07:18:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.030+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038075d20 msgr2=0x7f503819ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.030+0000 7f503e600700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038075d20 0x7f503819ced0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f502800e9a0 tx=0x7f502800ecb0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 shutdown_connections 2024-05-27T07:18:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f501c0b2900 0x7f501c0b4e10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5038110b90 0x7f50381a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5038075d20 0x7f503819ced0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 --2- 172.21.15.12:0/3369841861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5038076670 0x7f503819d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 >> 172.21.15.12:0/3369841861 conn(0x7f5038100030 msgr2=0x7f5038100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 shutdown_connections 2024-05-27T07:18:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:08.031+0000 7f503e600700 1 -- 172.21.15.12:0/3369841861 wait complete. 2024-05-27T07:18:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-05-27T07:18:08.825 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:18:08.825 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:18:08.883 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:18:09.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:08 smithi062 ceph-mon[35304]: pgmap v418: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-05-27T07:18:09.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:08 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3369841861' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:18:09.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:08 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:08 smithi012 ceph-mon[31473]: pgmap v418: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-05-27T07:18:09.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:08 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3369841861' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:18:09.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:08 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:08 smithi162 ceph-mon[35676]: pgmap v418: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-05-27T07:18:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:08 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3369841861' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:18:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:08 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:09.338 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:10.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:10.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:10.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:10.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:10.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:10.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:11.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:11 smithi012 ceph-mon[31473]: pgmap v419: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:11.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:11.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:11.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:11 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:11.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:11 smithi162 ceph-mon[35676]: pgmap v419: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:11.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:11.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:11.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:11 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:11.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:11 smithi062 ceph-mon[35304]: pgmap v419: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:11.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:11.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:11.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:11 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:18:11.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 -- 172.21.15.12:0/3866490011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078076670 msgr2=0x7fc078110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:11.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 --2- 172.21.15.12:0/3866490011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078076670 0x7fc078110650 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc064009990 tx=0x7fc064009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:11.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 -- 172.21.15.12:0/3866490011 shutdown_connections 2024-05-27T07:18:11.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 --2- 172.21.15.12:0/3866490011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc078075d20 0x7fc078076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:11.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 --2- 172.21.15.12:0/3866490011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc078110b90 0x7fc078112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:11.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 --2- 172.21.15.12:0/3866490011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078076670 0x7fc078110650 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:11.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 -- 172.21.15.12:0/3866490011 >> 172.21.15.12:0/3866490011 conn(0x7fc078100030 msgr2=0x7fc078102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:11.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 -- 172.21.15.12:0/3866490011 shutdown_connections 2024-05-27T07:18:11.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.749+0000 7fc07ea00700 1 -- 172.21.15.12:0/3866490011 wait complete. 2024-05-27T07:18:11.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.751+0000 7fc07ea00700 1 Processor -- start 2024-05-27T07:18:11.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.751+0000 7fc07ea00700 1 -- start start 2024-05-27T07:18:11.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc078075d20 0x7fc07819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc078076670 0x7fc07819d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078110b90 0x7fc0781a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc078116540 con 0x7fc078110b90 2024-05-27T07:18:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781163c0 con 0x7fc078076670 2024-05-27T07:18:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ea00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781166c0 con 0x7fc078075d20 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078110b90 0x7fc0781a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078110b90 0x7fc0781a17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40866/0 (socket says 172.21.15.12:40866) 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc077e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc078075d20 0x7fc07819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc07ca00700 1 -- 172.21.15.12:0/2033791698 learned_addr learned my addr 172.21.15.12:0/2033791698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.752+0000 7fc077400700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc078076670 0x7fc07819d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.753+0000 7fc07ca00700 1 -- 172.21.15.12:0/2033791698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc078075d20 msgr2=0x7fc07819cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.753+0000 7fc07ca00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc078075d20 0x7fc07819cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.753+0000 7fc07ca00700 1 -- 172.21.15.12:0/2033791698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc078076670 msgr2=0x7fc07819d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.753+0000 7fc07ca00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc078076670 0x7fc07819d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.753+0000 7fc07ca00700 1 -- 172.21.15.12:0/2033791698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0781a1dd0 con 0x7fc078110b90 2024-05-27T07:18:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.753+0000 7fc07ca00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078110b90 0x7fc0781a17e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc07000ada0 tx=0x7fc07000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.754+0000 7fc074c00700 1 -- 172.21.15.12:0/2033791698 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070017020 con 0x7fc078110b90 2024-05-27T07:18:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.754+0000 7fc074c00700 1 -- 172.21.15.12:0/2033791698 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc07000cd60 con 0x7fc078110b90 2024-05-27T07:18:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.754+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0781a20c0 con 0x7fc078110b90 2024-05-27T07:18:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.754+0000 7fc074c00700 1 -- 172.21.15.12:0/2033791698 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070005280 con 0x7fc078110b90 2024-05-27T07:18:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.754+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0781a2580 con 0x7fc078110b90 2024-05-27T07:18:11.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.755+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0780512b0 con 0x7fc078110b90 2024-05-27T07:18:11.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.758+0000 7fc074c00700 1 -- 172.21.15.12:0/2033791698 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc07000c8d0 con 0x7fc078110b90 2024-05-27T07:18:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.760+0000 7fc074c00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc05c0b2880 0x7fc05c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.760+0000 7fc074c00700 1 -- 172.21.15.12:0/2033791698 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fc0700e2260 con 0x7fc078110b90 2024-05-27T07:18:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.760+0000 7fc077e00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc05c0b2880 0x7fc05c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.761+0000 7fc077e00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc05c0b2880 0x7fc05c0b4d90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc06800ac50 tx=0x7fc06800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:11.763+0000 7fc074c00700 1 -- 172.21.15.12:0/2033791698 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc070021020 con 0x7fc078110b90 2024-05-27T07:18:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.015+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc07802e5e0 con 0x7fc078110b90 2024-05-27T07:18:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.016+0000 7fc074c00700 1 -- 172.21.15.12:0/2033791698 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+589 (secure 0 0 0) 0x7fc0700a9c50 con 0x7fc078110b90 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/4121651169 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1199837030 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/949043977 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1631546627 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1064527569 2024-05-28T07:06:45.507849+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:12.008 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3780513737 2024-05-28T07:06:09.346476+0000 2024-05-27T07:18:12.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.020+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc05c0b2880 msgr2=0x7fc05c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:12.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.020+0000 7fc07ea00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc05c0b2880 0x7fc05c0b4d90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc06800ac50 tx=0x7fc06800a380 comp rx=0 tx=0).stop 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.020+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078110b90 msgr2=0x7fc0781a17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.020+0000 7fc07ea00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078110b90 0x7fc0781a17e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc07000ada0 tx=0x7fc07000c330 comp rx=0 tx=0).stop 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 shutdown_connections 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc05c0b2880 0x7fc05c0b4d90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc078110b90 0x7fc0781a17e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc078075d20 0x7fc07819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 --2- 172.21.15.12:0/2033791698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc078076670 0x7fc07819d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 >> 172.21.15.12:0/2033791698 conn(0x7fc078100030 msgr2=0x7fc078100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 shutdown_connections 2024-05-27T07:18:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:12.021+0000 7fc07ea00700 1 -- 172.21.15.12:0/2033791698 wait complete. 2024-05-27T07:18:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:listed 10 entries 2024-05-27T07:18:12.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:12 smithi012 ceph-mon[31473]: pgmap v420: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:12.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:12 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2033791698' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:18:12.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:12 smithi162 ceph-mon[35676]: pgmap v420: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:12.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:12 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2033791698' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:18:12.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:12 smithi062 ceph-mon[35304]: pgmap v420: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:12.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:12 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2033791698' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:18:12.836 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-05-27T07:18:12.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:12.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:18:12.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi012.stderr:ceph-fuse[77712]: fuse finished with error 0 and tester_r 0 2024-05-27T07:18:12.934 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:18:15.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:14 smithi012 ceph-mon[31473]: pgmap v421: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:15.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:14 smithi162 ceph-mon[35676]: pgmap v421: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:14 smithi062 ceph-mon[35304]: pgmap v421: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:18:17.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:16 smithi012 ceph-mon[31473]: pgmap v422: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-27T07:18:17.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:16 smithi162 ceph-mon[35676]: pgmap v422: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-27T07:18:17.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:16 smithi062 ceph-mon[35304]: pgmap v422: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-27T07:18:19.036 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:18:19.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:18:19.036 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-27T07:18:19.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:18 smithi012 ceph-mon[31473]: pgmap v423: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:18:19.108 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:18:19.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:18 smithi162 ceph-mon[35676]: pgmap v423: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:18:19.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:18 smithi062 ceph-mon[35304]: pgmap v423: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:18:19.580 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:21.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:20 smithi012 ceph-mon[31473]: pgmap v424: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:18:21.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:20 smithi162 ceph-mon[35676]: pgmap v424: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:18:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:20 smithi062 ceph-mon[35304]: pgmap v424: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:18:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 -- 172.21.15.12:0/2152374595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4105bd0 msgr2=0x7f58d410e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2152374595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4105bd0 0x7f58d410e120 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009990 tx=0x7f58bc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 -- 172.21.15.12:0/2152374595 shutdown_connections 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2152374595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58d410e6f0 0x7f58d4110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2152374595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58d4105280 0x7f58d4105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2152374595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d4105bd0 0x7f58d410e120 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 -- 172.21.15.12:0/2152374595 >> 172.21.15.12:0/2152374595 conn(0x7f58d4078ab0 msgr2=0x7f58d4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.050+0000 7f58d9a00700 1 -- 172.21.15.12:0/2152374595 shutdown_connections 2024-05-27T07:18:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.051+0000 7f58d9a00700 1 -- 172.21.15.12:0/2152374595 wait complete. 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 Processor -- start 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 -- start start 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58d4105280 0x7f58d419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58d4105bd0 0x7f58d419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d410e6f0 0x7f58d41a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f58d4114100 con 0x7f58d410e6f0 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f58d4113f80 con 0x7f58d4105bd0 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.052+0000 7f58d9a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58d4114280 con 0x7f58d4105280 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d410e6f0 0x7f58d41a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d410e6f0 0x7f58d41a16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55270/0 (socket says 172.21.15.12:55270) 2024-05-27T07:18:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 -- 172.21.15.12:0/2255162564 learned_addr learned my addr 172.21.15.12:0/2255162564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d3e00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58d4105280 0x7f58d419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58cb400700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58d4105bd0 0x7f58d419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 -- 172.21.15.12:0/2255162564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58d4105280 msgr2=0x7f58d419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58d4105280 0x7f58d419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 -- 172.21.15.12:0/2255162564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58d4105bd0 msgr2=0x7f58d419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58d4105bd0 0x7f58d419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.053+0000 7f58d9000700 1 -- 172.21.15.12:0/2255162564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d41a1d40 con 0x7f58d410e6f0 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.054+0000 7f58d9000700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d410e6f0 0x7f58d41a16a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f58cc007ee0 tx=0x7f58cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.054+0000 7f58d1600700 1 -- 172.21.15.12:0/2255162564 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc00dbd0 con 0x7f58d410e6f0 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.054+0000 7f58d1600700 1 -- 172.21.15.12:0/2255162564 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58cc00dd70 con 0x7f58d410e6f0 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.054+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d41a2030 con 0x7f58d410e6f0 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.054+0000 7f58d1600700 1 -- 172.21.15.12:0/2255162564 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0058d0 con 0x7f58d410e6f0 2024-05-27T07:18:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.054+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d41a9950 con 0x7f58d410e6f0 2024-05-27T07:18:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.055+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d40512b0 con 0x7f58d410e6f0 2024-05-27T07:18:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.058+0000 7f58d1600700 1 -- 172.21.15.12:0/2255162564 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f58cc01a020 con 0x7f58d410e6f0 2024-05-27T07:18:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.059+0000 7f58d1600700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f58b80b2900 0x7f58b80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.060+0000 7f58d1600700 1 -- 172.21.15.12:0/2255162564 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f58cc0e23a0 con 0x7f58d410e6f0 2024-05-27T07:18:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.060+0000 7f58d3e00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f58b80b2900 0x7f58b80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.061+0000 7f58d3e00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f58b80b2900 0x7f58b80b4e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005fd0 tx=0x7f58c4005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.063+0000 7f58d1600700 1 -- 172.21.15.12:0/2255162564 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f58cc010070 con 0x7f58d410e6f0 2024-05-27T07:18:22.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.303+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58d402e5e0 con 0x7f58d410e6f0 2024-05-27T07:18:22.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.306+0000 7f58d1600700 1 -- 172.21.15.12:0/2255162564 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24618 (secure 0 0 0) 0x7f58cc0a9d00 con 0x7f58d410e6f0 2024-05-27T07:18:22.297 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:18:22.298 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":87,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:17:46.671419+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.309+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f58b80b2900 msgr2=0x7f58b80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.309+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f58b80b2900 0x7f58b80b4e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005fd0 tx=0x7f58c4005f60 comp rx=0 tx=0).stop 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d410e6f0 msgr2=0x7f58d41a16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d410e6f0 0x7f58d41a16a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f58cc007ee0 tx=0x7f58cc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 shutdown_connections 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f58b80b2900 0x7f58b80b4e10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d410e6f0 0x7f58d41a16a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58d4105280 0x7f58d419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 --2- 172.21.15.12:0/2255162564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58d4105bd0 0x7f58d419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:22.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 >> 172.21.15.12:0/2255162564 conn(0x7f58d4078ab0 msgr2=0x7f58d4104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:22.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 shutdown_connections 2024-05-27T07:18:22.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:22.310+0000 7f58d9a00700 1 -- 172.21.15.12:0/2255162564 wait complete. 2024-05-27T07:18:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:22 smithi012 ceph-mon[31473]: pgmap v425: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:23.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:22 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2255162564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:18:23.175 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:18:23.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:22 smithi162 ceph-mon[35676]: pgmap v425: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:22 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2255162564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:18:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:22 smithi062 ceph-mon[35304]: pgmap v425: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:22 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2255162564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:18:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:25.037 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:24 smithi012 ceph-mon[31473]: pgmap v426: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:25.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:24 smithi162 ceph-mon[35676]: pgmap v426: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:25.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:24 smithi062 ceph-mon[35304]: pgmap v426: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:26.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 -- 172.21.15.12:0/3451861198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410b2d0 msgr2=0x7f9834117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 --2- 172.21.15.12:0/3451861198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410b2d0 0x7f9834117800 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f982c00b0a0 tx=0x7f982c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 -- 172.21.15.12:0/3451861198 shutdown_connections 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 --2- 172.21.15.12:0/3451861198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410b2d0 0x7f9834117800 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 --2- 172.21.15.12:0/3451861198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9834110960 0x7f9834110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 --2- 172.21.15.12:0/3451861198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f983410a980 0x7f983410ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 -- 172.21.15.12:0/3451861198 >> 172.21.15.12:0/3451861198 conn(0x7f9834077c70 msgr2=0x7f98340780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 -- 172.21.15.12:0/3451861198 shutdown_connections 2024-05-27T07:18:26.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.075+0000 7f983b800700 1 -- 172.21.15.12:0/3451861198 wait complete. 2024-05-27T07:18:26.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.077+0000 7f983b800700 1 Processor -- start 2024-05-27T07:18:26.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.077+0000 7f983b800700 1 -- start start 2024-05-27T07:18:26.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f983b800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410a980 0x7f98341a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f983b800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f983410b2d0 0x7f98341a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f983b800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9834110960 0x7f98341a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f983b800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f983411a980 con 0x7f983410a980 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f983b800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f983411a800 con 0x7f983410b2d0 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f983b800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f983411ab00 con 0x7f9834110960 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f9838e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410a980 0x7f98341a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f9838e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410a980 0x7f98341a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55280/0 (socket says 172.21.15.12:55280) 2024-05-27T07:18:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f9838e00700 1 -- 172.21.15.12:0/1427866394 learned_addr learned my addr 172.21.15.12:0/1427866394 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f9833e00700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f983410b2d0 0x7f98341a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.078+0000 7f9839800700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9834110960 0x7f98341a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9838e00700 1 -- 172.21.15.12:0/1427866394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9834110960 msgr2=0x7f98341a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9838e00700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9834110960 0x7f98341a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9838e00700 1 -- 172.21.15.12:0/1427866394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f983410b2d0 msgr2=0x7f98341a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9838e00700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f983410b2d0 0x7f98341a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9838e00700 1 -- 172.21.15.12:0/1427866394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98341a6170 con 0x7f983410a980 2024-05-27T07:18:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9833e00700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f983410b2d0 0x7f98341a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9838e00700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410a980 0x7f98341a1350 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f982800c8f0 tx=0x7f982800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.079+0000 7f9831600700 1 -- 172.21.15.12:0/1427866394 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828007b00 con 0x7f983410a980 2024-05-27T07:18:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.080+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98341a6460 con 0x7f983410a980 2024-05-27T07:18:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.080+0000 7f9831600700 1 -- 172.21.15.12:0/1427866394 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9828007ca0 con 0x7f983410a980 2024-05-27T07:18:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.080+0000 7f9831600700 1 -- 172.21.15.12:0/1427866394 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828018660 con 0x7f983410a980 2024-05-27T07:18:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.080+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98341ade30 con 0x7f983410a980 2024-05-27T07:18:26.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.082+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f983410c070 con 0x7f983410a980 2024-05-27T07:18:26.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.084+0000 7f9831600700 1 -- 172.21.15.12:0/1427866394 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9828018800 con 0x7f983410a980 2024-05-27T07:18:26.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.086+0000 7f9831600700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f981c0b2880 0x7f981c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:26.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.086+0000 7f9831600700 1 -- 172.21.15.12:0/1427866394 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f98280e24d0 con 0x7f983410a980 2024-05-27T07:18:26.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.086+0000 7f9833e00700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f981c0b2880 0x7f981c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:26.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.087+0000 7f9833e00700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f981c0b2880 0x7f981c0b4d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9824005fd0 tx=0x7f9824005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:26.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.090+0000 7f9831600700 1 -- 172.21.15.12:0/1427866394 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9828014030 con 0x7f983410a980 2024-05-27T07:18:26.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.328+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98341a26b0 con 0x7f983410a980 2024-05-27T07:18:26.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.331+0000 7f9831600700 1 -- 172.21.15.12:0/1427866394 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24618 (secure 0 0 0) 0x7f98280a9e30 con 0x7f983410a980 2024-05-27T07:18:26.323 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:18:26.323 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":87,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:17:46.671419+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:18:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f981c0b2880 msgr2=0x7f981c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:26.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f981c0b2880 0x7f981c0b4d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9824005fd0 tx=0x7f9824005e30 comp rx=0 tx=0).stop 2024-05-27T07:18:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410a980 msgr2=0x7f98341a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410a980 0x7f98341a1350 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f982800c8f0 tx=0x7f982800ccb0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 shutdown_connections 2024-05-27T07:18:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f981c0b2880 0x7f981c0b4d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9834110960 0x7f98341a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f983410a980 0x7f98341a1350 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 --2- 172.21.15.12:0/1427866394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f983410b2d0 0x7f98341a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 >> 172.21.15.12:0/1427866394 conn(0x7f9834077c70 msgr2=0x7f983407b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 shutdown_connections 2024-05-27T07:18:26.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:26.335+0000 7f983b800700 1 -- 172.21.15.12:0/1427866394 wait complete. 2024-05-27T07:18:27.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:26 smithi012 ceph-mon[31473]: pgmap v427: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:27.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:26 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1427866394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:18:27.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-27T07:18:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:26 smithi162 ceph-mon[35676]: pgmap v427: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:26 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1427866394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:18:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:26 smithi062 ceph-mon[35304]: pgmap v427: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:18:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:26 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1427866394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:18:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:29.013 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:28 smithi012 ceph-mon[31473]: pgmap v428: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:29.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:28 smithi162 ceph-mon[35676]: pgmap v428: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:28 smithi062 ceph-mon[35304]: pgmap v428: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:29.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 -- 172.21.15.12:0/1892243516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81109a0 msgr2=0x7f87f8112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 --2- 172.21.15.12:0/1892243516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81109a0 0x7f87f8112e00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f87f000b0a0 tx=0x7f87f000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 -- 172.21.15.12:0/1892243516 shutdown_connections 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 --2- 172.21.15.12:0/1892243516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f81109a0 0x7f87f8112e00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 --2- 172.21.15.12:0/1892243516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f8075d20 0x7f87f8076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 --2- 172.21.15.12:0/1892243516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87f8076670 0x7f87f8110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 -- 172.21.15.12:0/1892243516 >> 172.21.15.12:0/1892243516 conn(0x7f87f8079d00 msgr2=0x7f87f807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.981+0000 7f87f7200700 1 -- 172.21.15.12:0/1892243516 shutdown_connections 2024-05-27T07:18:29.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.982+0000 7f87f7200700 1 -- 172.21.15.12:0/1892243516 wait complete. 2024-05-27T07:18:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 Processor -- start 2024-05-27T07:18:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 -- start start 2024-05-27T07:18:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87f8075d20 0x7f87f819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f8076670 0x7f87f819d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f81109a0 0x7f87f81a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87f8116380 con 0x7f87f8076670 2024-05-27T07:18:29.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f87f8116200 con 0x7f87f8075d20 2024-05-27T07:18:29.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.983+0000 7f87f7200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87f8116500 con 0x7f87f81109a0 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f8076670 0x7f87f819d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f8076670 0x7f87f819d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55298/0 (socket says 172.21.15.12:55298) 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 -- 172.21.15.12:0/144808841 learned_addr learned my addr 172.21.15.12:0/144808841 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5e00700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87f8075d20 0x7f87f819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f6800700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f81109a0 0x7f87f81a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 -- 172.21.15.12:0/144808841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f81109a0 msgr2=0x7f87f81a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f81109a0 0x7f87f81a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 -- 172.21.15.12:0/144808841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87f8075d20 msgr2=0x7f87f819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87f8075d20 0x7f87f819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f5400700 1 -- 172.21.15.12:0/144808841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f81a1be0 con 0x7f87f8076670 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.984+0000 7f87f6800700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f81109a0 0x7f87f81a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.985+0000 7f87f5400700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f8076670 0x7f87f819d2b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f87e400d950 tx=0x7f87e400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.985+0000 7f87eea00700 1 -- 172.21.15.12:0/144808841 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4016070 con 0x7f87f8076670 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.985+0000 7f87eea00700 1 -- 172.21.15.12:0/144808841 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f87e4010460 con 0x7f87f8076670 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.985+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f81a1e70 con 0x7f87f8076670 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.985+0000 7f87eea00700 1 -- 172.21.15.12:0/144808841 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e400f620 con 0x7f87f8076670 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.985+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f81a97f0 con 0x7f87f8076670 2024-05-27T07:18:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.986+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f80512b0 con 0x7f87f8076670 2024-05-27T07:18:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.990+0000 7f87eea00700 1 -- 172.21.15.12:0/144808841 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f87e4019070 con 0x7f87f8076670 2024-05-27T07:18:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.991+0000 7f87eea00700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87dc0b2900 0x7f87dc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.992+0000 7f87eea00700 1 -- 172.21.15.12:0/144808841 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f87e40e2440 con 0x7f87f8076670 2024-05-27T07:18:29.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.992+0000 7f87f5e00700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87dc0b2900 0x7f87dc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:29.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.993+0000 7f87f5e00700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87dc0b2900 0x7f87dc0b4e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87e8005950 tx=0x7f87e80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:29.993+0000 7f87eea00700 1 -- 172.21.15.12:0/144808841 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f87e40a9da0 con 0x7f87f8076670 2024-05-27T07:18:30.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.284+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f87f819e070 con 0x7f87f8076670 2024-05-27T07:18:30.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.857+0000 7f87eea00700 1 -- 172.21.15.12:0/144808841 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f87e40a96a0 con 0x7f87f8076670 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87dc0b2900 msgr2=0x7f87dc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87dc0b2900 0x7f87dc0b4e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87e8005950 tx=0x7f87e80058e0 comp rx=0 tx=0).stop 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f8076670 msgr2=0x7f87f819d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f8076670 0x7f87f819d2b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f87e400d950 tx=0x7f87e400dd10 comp rx=0 tx=0).stop 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 shutdown_connections 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f87dc0b2900 0x7f87dc0b4e10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87f8075d20 0x7f87f819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f81109a0 0x7f87f81a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 --2- 172.21.15.12:0/144808841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87f8076670 0x7f87f819d2b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.861+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 >> 172.21.15.12:0/144808841 conn(0x7f87f8079d00 msgr2=0x7f87f8100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.862+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 shutdown_connections 2024-05-27T07:18:30.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:30.862+0000 7f87f7200700 1 -- 172.21.15.12:0/144808841 wait complete. 2024-05-27T07:18:31.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:30 smithi162 ceph-mon[35676]: pgmap v429: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:31.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:30 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/144808841' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:18:31.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:30 smithi062 ceph-mon[35304]: pgmap v429: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:31.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:30 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/144808841' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:18:31.310 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:30 smithi012 ceph-mon[31473]: pgmap v429: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:31.310 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:30 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/144808841' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:18:31.680 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-05-27T07:18:31.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-05-27T07:18:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:32.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:31 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/144808841' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:18:32.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:31 smithi162 ceph-mon[35676]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:18:32.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:31 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/144808841' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:18:32.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:31 smithi062 ceph-mon[35304]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:18:32.299 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:31 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/144808841' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:18:32.299 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:31 smithi012 ceph-mon[31473]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:18:33.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:32 smithi012 ceph-mon[31473]: pgmap v430: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:33.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:32 smithi162 ceph-mon[35676]: pgmap v430: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:32 smithi062 ceph-mon[35304]: pgmap v430: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.374+0000 7f9ad6400700 1 -- 172.21.15.12:0/932444452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 msgr2=0x7f9ad0110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.374+0000 7f9ad6400700 1 --2- 172.21.15.12:0/932444452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad0110fd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800b0d0 tx=0x7f9ac800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.374+0000 7f9ad6400700 1 -- 172.21.15.12:0/932444452 shutdown_connections 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.374+0000 7f9ad6400700 1 --2- 172.21.15.12:0/932444452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad01043f0 0x7f9ad0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.374+0000 7f9ad6400700 1 --2- 172.21.15.12:0/932444452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad0110fd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.374+0000 7f9ad6400700 1 --2- 172.21.15.12:0/932444452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad0108d20 0x7f9ad0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.374+0000 7f9ad6400700 1 -- 172.21.15.12:0/932444452 >> 172.21.15.12:0/932444452 conn(0x7f9ad0079d00 msgr2=0x7f9ad007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.375+0000 7f9ad6400700 1 -- 172.21.15.12:0/932444452 shutdown_connections 2024-05-27T07:18:34.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.375+0000 7f9ad6400700 1 -- 172.21.15.12:0/932444452 wait complete. 2024-05-27T07:18:34.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.376+0000 7f9ad6400700 1 Processor -- start 2024-05-27T07:18:34.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.377+0000 7f9ad6400700 1 -- start start 2024-05-27T07:18:34.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.377+0000 7f9ad6400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad01043f0 0x7f9ad01a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:34.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.377+0000 7f9ad6400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad0108d20 0x7f9ad01a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:34.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.377+0000 7f9ad6400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:34.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.377+0000 7f9ad6400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad0114150 con 0x7f9ad0109670 2024-05-27T07:18:34.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.377+0000 7f9ad6400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad0113fd0 con 0x7f9ad0108d20 2024-05-27T07:18:34.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.377+0000 7f9ad6400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad01142d0 con 0x7f9ad01043f0 2024-05-27T07:18:34.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:34.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad019a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39794/0 (socket says 172.21.15.12:39794) 2024-05-27T07:18:34.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 -- 172.21.15.12:0/1631146993 learned_addr learned my addr 172.21.15.12:0/1631146993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:34.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5000700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad01043f0 0x7f9ad01a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:34.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9acfe00700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad0108d20 0x7f9ad01a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:34.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 -- 172.21.15.12:0/1631146993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad01043f0 msgr2=0x7f9ad01a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:34.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad01043f0 0x7f9ad01a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 -- 172.21.15.12:0/1631146993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad0108d20 msgr2=0x7f9ad01a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:34.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad0108d20 0x7f9ad01a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.378+0000 7f9ad5a00700 1 -- 172.21.15.12:0/1631146993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad019afa0 con 0x7f9ad0109670 2024-05-27T07:18:34.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.379+0000 7f9ad5a00700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad019a8f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8007a00 tx=0x7f9ac8007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:34.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.379+0000 7f9acd600700 1 -- 172.21.15.12:0/1631146993 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac801f070 con 0x7f9ad0109670 2024-05-27T07:18:34.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.379+0000 7f9acd600700 1 -- 172.21.15.12:0/1631146993 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9ac8004c50 con 0x7f9ad0109670 2024-05-27T07:18:34.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.379+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad019b230 con 0x7f9ad0109670 2024-05-27T07:18:34.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.379+0000 7f9acd600700 1 -- 172.21.15.12:0/1631146993 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac801b690 con 0x7f9ad0109670 2024-05-27T07:18:34.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.379+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad019b710 con 0x7f9ad0109670 2024-05-27T07:18:34.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.381+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad00512b0 con 0x7f9ad0109670 2024-05-27T07:18:34.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.385+0000 7f9acd600700 1 -- 172.21.15.12:0/1631146993 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9ac8022070 con 0x7f9ad0109670 2024-05-27T07:18:34.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.386+0000 7f9acd600700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ab80b2880 0x7f9ab80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:34.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.387+0000 7f9acd600700 1 -- 172.21.15.12:0/1631146993 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f9ac800f020 con 0x7f9ad0109670 2024-05-27T07:18:34.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.387+0000 7f9ad5000700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ab80b2880 0x7f9ab80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:34.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.388+0000 7f9ad5000700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ab80b2880 0x7f9ab80b4d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9ad019ba30 tx=0x7f9ac4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:34.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.390+0000 7f9acd600700 1 -- 172.21.15.12:0/1631146993 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9ac8029020 con 0x7f9ad0109670 2024-05-27T07:18:34.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.676+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f9ad002e5e0 con 0x7f9ad0109670 2024-05-27T07:18:34.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.895+0000 7f9acd600700 1 -- 172.21.15.12:0/1631146993 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f9ac80ab4d0 con 0x7f9ad0109670 2024-05-27T07:18:34.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.899+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ab80b2880 msgr2=0x7f9ab80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:34.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.899+0000 7f9ad6400700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ab80b2880 0x7f9ab80b4d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9ad019ba30 tx=0x7f9ac4009260 comp rx=0 tx=0).stop 2024-05-27T07:18:34.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 msgr2=0x7f9ad019a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:34.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad019a8f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8007a00 tx=0x7f9ac8007a30 comp rx=0 tx=0).stop 2024-05-27T07:18:34.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 shutdown_connections 2024-05-27T07:18:34.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ab80b2880 0x7f9ab80b4d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ad0109670 0x7f9ad019a8f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad01043f0 0x7f9ad01a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 --2- 172.21.15.12:0/1631146993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad0108d20 0x7f9ad01a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:34.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 >> 172.21.15.12:0/1631146993 conn(0x7f9ad0079d00 msgr2=0x7f9ad0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:34.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 shutdown_connections 2024-05-27T07:18:34.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:34.900+0000 7f9ad6400700 1 -- 172.21.15.12:0/1631146993 wait complete. 2024-05-27T07:18:34.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:34 smithi012 ceph-mon[31473]: pgmap v431: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:34.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:34 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1631146993' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T07:18:35.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:34 smithi162 ceph-mon[35676]: pgmap v431: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:35.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:34 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1631146993' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T07:18:35.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:34 smithi062 ceph-mon[35304]: pgmap v431: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:18:35.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:34 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1631146993' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-27T07:18:35.722 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-27T07:18:36.208 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:35 smithi162 ceph-mon[35676]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:18:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:35 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1631146993' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-27T07:18:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:35 smithi162 ceph-mon[35676]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:18:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:35 smithi162 ceph-mon[35676]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:18:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:35 smithi162 ceph-mon[35676]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:18:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:35 smithi162 ceph-mon[35676]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:18:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:35 smithi062 ceph-mon[35304]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:18:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:35 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1631146993' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-27T07:18:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:35 smithi062 ceph-mon[35304]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:18:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:35 smithi062 ceph-mon[35304]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:18:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:35 smithi062 ceph-mon[35304]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:18:36.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:35 smithi062 ceph-mon[35304]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:18:36.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:35 smithi012 ceph-mon[31473]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:18:36.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:35 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1631146993' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-27T07:18:36.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:35 smithi012 ceph-mon[31473]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:18:36.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:35 smithi012 ceph-mon[31473]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:18:36.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:35 smithi012 ceph-mon[31473]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:18:36.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:35 smithi012 ceph-mon[31473]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: pgmap v432: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] up:active 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: Cluster is now healthy 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-27T07:18:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:36 smithi012 ceph-mon[31473]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-27T07:18:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: pgmap v432: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T07:18:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] up:active 2024-05-27T07:18:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:18:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:18:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: Cluster is now healthy 2024-05-27T07:18:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:18:37.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-27T07:18:37.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:36 smithi162 ceph-mon[35676]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: pgmap v432: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] up:active 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: Cluster is now healthy 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-27T07:18:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:36 smithi062 ceph-mon[35304]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-27T07:18:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:37 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] up:active 2024-05-27T07:18:38.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:37 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:18:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:37 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] up:active 2024-05-27T07:18:38.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:37 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:18:38.285 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:37 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] up:active 2024-05-27T07:18:38.285 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:37 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 -- 172.21.15.12:0/3046015446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10af40 msgr2=0x7fee6c10b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 --2- 172.21.15.12:0/3046015446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10af40 0x7fee6c10b350 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fee540099c0 tx=0x7fee54009cd0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 -- 172.21.15.12:0/3046015446 shutdown_connections 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 --2- 172.21.15.12:0/3046015446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee6c10b890 0x7fee6c1131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 --2- 172.21.15.12:0/3046015446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee6c106610 0x7fee6c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 --2- 172.21.15.12:0/3046015446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10af40 0x7fee6c10b350 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 -- 172.21.15.12:0/3046015446 >> 172.21.15.12:0/3046015446 conn(0x7fee6c102020 msgr2=0x7fee6c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 -- 172.21.15.12:0/3046015446 shutdown_connections 2024-05-27T07:18:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.553+0000 7fee70c00700 1 -- 172.21.15.12:0/3046015446 wait complete. 2024-05-27T07:18:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.555+0000 7fee70c00700 1 Processor -- start 2024-05-27T07:18:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.555+0000 7fee70c00700 1 -- start start 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.555+0000 7fee70c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee6c106610 0x7fee6c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee70c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee6c10af40 0x7fee6c19d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee70c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10b890 0x7fee6c1a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee70c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1163a0 con 0x7fee6c10b890 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee70c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c116220 con 0x7fee6c10af40 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee70c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c116520 con 0x7fee6c106610 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10b890 0x7fee6c1a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10b890 0x7fee6c1a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39812/0 (socket says 172.21.15.12:39812) 2024-05-27T07:18:38.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee6be00700 1 -- 172.21.15.12:0/470431790 learned_addr learned my addr 172.21.15.12:0/470431790 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:38.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee6b400700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee6c106610 0x7fee6c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:38.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.556+0000 7fee63e00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee6c10af40 0x7fee6c19d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:38.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.557+0000 7fee6be00700 1 -- 172.21.15.12:0/470431790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee6c106610 msgr2=0x7fee6c19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:38.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.557+0000 7fee6be00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee6c106610 0x7fee6c19ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.557+0000 7fee6be00700 1 -- 172.21.15.12:0/470431790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee6c10af40 msgr2=0x7fee6c19d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:38.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.557+0000 7fee6be00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee6c10af40 0x7fee6c19d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.557+0000 7fee6be00700 1 -- 172.21.15.12:0/470431790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c1a1d00 con 0x7fee6c10b890 2024-05-27T07:18:38.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.557+0000 7fee6be00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10b890 0x7fee6c1a1660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fee64007ee0 tx=0x7fee6400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:38.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.558+0000 7fee68c00700 1 -- 172.21.15.12:0/470431790 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6400dbd0 con 0x7fee6c10b890 2024-05-27T07:18:38.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.558+0000 7fee68c00700 1 -- 172.21.15.12:0/470431790 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee6400dd70 con 0x7fee6c10b890 2024-05-27T07:18:38.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.558+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6c1a1ff0 con 0x7fee6c10b890 2024-05-27T07:18:38.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.558+0000 7fee68c00700 1 -- 172.21.15.12:0/470431790 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee640058d0 con 0x7fee6c10b890 2024-05-27T07:18:38.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.558+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6c077bf0 con 0x7fee6c10b890 2024-05-27T07:18:38.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.559+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee6c107ac0 con 0x7fee6c10b890 2024-05-27T07:18:38.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.563+0000 7fee68c00700 1 -- 172.21.15.12:0/470431790 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fee6401a020 con 0x7fee6c10b890 2024-05-27T07:18:38.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.563+0000 7fee68c00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee500b2880 0x7fee500b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:38.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.564+0000 7fee68c00700 1 -- 172.21.15.12:0/470431790 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7fee640e2380 con 0x7fee6c10b890 2024-05-27T07:18:38.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.564+0000 7fee6b400700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee500b2880 0x7fee500b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:38.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.565+0000 7fee6b400700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee500b2880 0x7fee500b4d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fee5c005950 tx=0x7fee5c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:38.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.566+0000 7fee68c00700 1 -- 172.21.15.12:0/470431790 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fee64010070 con 0x7fee6c10b890 2024-05-27T07:18:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.858+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fee6c19e190 con 0x7fee6c10b890 2024-05-27T07:18:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.956+0000 7fee68c00700 1 -- 172.21.15.12:0/470431790 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7fee640a9ce0 con 0x7fee6c10b890 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee500b2880 msgr2=0x7fee500b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee500b2880 0x7fee500b4d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fee5c005950 tx=0x7fee5c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10b890 msgr2=0x7fee6c1a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10b890 0x7fee6c1a1660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fee64007ee0 tx=0x7fee6400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 shutdown_connections 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee500b2880 0x7fee500b4d90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee6c10b890 0x7fee6c1a1660 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee6c106610 0x7fee6c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 --2- 172.21.15.12:0/470431790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee6c10af40 0x7fee6c19d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 >> 172.21.15.12:0/470431790 conn(0x7fee6c102020 msgr2=0x7fee6c10c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:38.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 shutdown_connections 2024-05-27T07:18:38.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:38.961+0000 7fee70c00700 1 -- 172.21.15.12:0/470431790 wait complete. 2024-05-27T07:18:39.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:38 smithi162 ceph-mon[35676]: pgmap v433: 129 pgs: 129 active+clean; 582 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 2 op/s 2024-05-27T07:18:39.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:38 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/470431790' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:18:39.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:38 smithi062 ceph-mon[35304]: pgmap v433: 129 pgs: 129 active+clean; 582 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 2 op/s 2024-05-27T07:18:39.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:38 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/470431790' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:18:39.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:38 smithi012 ceph-mon[31473]: pgmap v433: 129 pgs: 129 active+clean; 582 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 2 op/s 2024-05-27T07:18:39.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:38 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/470431790' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:18:39.820 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-27T07:18:40.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:39 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/470431790' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:18:40.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:39 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:18:40.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:39 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:18:40.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:39 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/470431790' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:18:40.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:39 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:18:40.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:39 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:18:40.292 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:40.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:39 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/470431790' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:18:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:39 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:18:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:39 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:18:41.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:40 smithi012 ceph-mon[31473]: pgmap v434: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:18:41.236 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:18:40 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b[66152]: 2024-05-27T07:18:40.832+0000 7f6e0c000700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-05-27T07:18:41.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:40 smithi062 ceph-mon[35304]: pgmap v434: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:18:41.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:40 smithi162 ceph-mon[35676]: pgmap v434: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:18:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 -- 172.21.15.12:0/970835136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7814105470 msgr2=0x7f7814105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 --2- 172.21.15.12:0/970835136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7814105470 0x7f7814105880 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7804009990 tx=0x7f7804009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 -- 172.21.15.12:0/970835136 shutdown_connections 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 --2- 172.21.15.12:0/970835136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f781410e8e0 0x7f7814110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 --2- 172.21.15.12:0/970835136 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7814105470 0x7f7814105880 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 --2- 172.21.15.12:0/970835136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7814105dc0 0x7f781410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 -- 172.21.15.12:0/970835136 >> 172.21.15.12:0/970835136 conn(0x7f78140fef20 msgr2=0x7f7814101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.576+0000 7f781b200700 1 -- 172.21.15.12:0/970835136 shutdown_connections 2024-05-27T07:18:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.577+0000 7f781b200700 1 -- 172.21.15.12:0/970835136 wait complete. 2024-05-27T07:18:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 Processor -- start 2024-05-27T07:18:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 -- start start 2024-05-27T07:18:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7814105470 0x7f781419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7814105dc0 0x7f781419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781410e8e0 0x7f78141a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78141142e0 con 0x7f781410e8e0 2024-05-27T07:18:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7814114160 con 0x7f7814105dc0 2024-05-27T07:18:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.578+0000 7f781b200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7814114460 con 0x7f7814105470 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781410e8e0 0x7f78141a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781410e8e0 0x7f78141a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60826/0 (socket says 172.21.15.12:60826) 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 -- 172.21.15.12:0/2008092637 learned_addr learned my addr 172.21.15.12:0/2008092637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7813e00700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7814105470 0x7f781419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7813400700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7814105dc0 0x7f781419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 -- 172.21.15.12:0/2008092637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7814105470 msgr2=0x7f781419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7814105470 0x7f781419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 -- 172.21.15.12:0/2008092637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7814105dc0 msgr2=0x7f781419d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7814105dc0 0x7f781419d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7819200700 1 -- 172.21.15.12:0/2008092637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78141a1de0 con 0x7f781410e8e0 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.579+0000 7f7813400700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7814105dc0 0x7f781419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.580+0000 7f7819200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781410e8e0 0x7f78141a1740 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f780c007ee0 tx=0x7f780c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.580+0000 7f7810c00700 1 -- 172.21.15.12:0/2008092637 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c00dbd0 con 0x7f781410e8e0 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.580+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78141a20d0 con 0x7f781410e8e0 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.580+0000 7f7810c00700 1 -- 172.21.15.12:0/2008092637 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f780c00dd70 con 0x7f781410e8e0 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.580+0000 7f7810c00700 1 -- 172.21.15.12:0/2008092637 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c0058d0 con 0x7f781410e8e0 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.580+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78141a99f0 con 0x7f781410e8e0 2024-05-27T07:18:42.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.581+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78140512b0 con 0x7f781410e8e0 2024-05-27T07:18:42.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.584+0000 7f7810c00700 1 -- 172.21.15.12:0/2008092637 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f780c01a020 con 0x7f781410e8e0 2024-05-27T07:18:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.587+0000 7f7810c00700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f77f80b2880 0x7f77f80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.587+0000 7f7810c00700 1 -- 172.21.15.12:0/2008092637 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f780c0e2380 con 0x7f781410e8e0 2024-05-27T07:18:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.587+0000 7f7813e00700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f77f80b2880 0x7f77f80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.588+0000 7f7813e00700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f77f80b2880 0x7f77f80b4d90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f780400b5c0 tx=0x7f7804005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.590+0000 7f7810c00700 1 -- 172.21.15.12:0/2008092637 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f780c010070 con 0x7f781410e8e0 2024-05-27T07:18:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.888+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f78141a9bc0 con 0x7f781410e8e0 2024-05-27T07:18:42.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.994+0000 7f7810c00700 1 -- 172.21.15.12:0/2008092637 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v26) ==== 115+0+0 (secure 0 0 0) 0x7f780c0a9ce0 con 0x7f781410e8e0 2024-05-27T07:18:42.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.998+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f77f80b2880 msgr2=0x7f77f80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:42.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.998+0000 7f781b200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f77f80b2880 0x7f77f80b4d90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f780400b5c0 tx=0x7f7804005960 comp rx=0 tx=0).stop 2024-05-27T07:18:42.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781410e8e0 msgr2=0x7f78141a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:42.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781410e8e0 0x7f78141a1740 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f780c007ee0 tx=0x7f780c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 shutdown_connections 2024-05-27T07:18:42.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f77f80b2880 0x7f77f80b4d90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f781410e8e0 0x7f78141a1740 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7814105470 0x7f781419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 --2- 172.21.15.12:0/2008092637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7814105dc0 0x7f781419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:42.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 >> 172.21.15.12:0/2008092637 conn(0x7f78140fef20 msgr2=0x7f7814103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:42.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 shutdown_connections 2024-05-27T07:18:42.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:42.999+0000 7f781b200700 1 -- 172.21.15.12:0/2008092637 wait complete. 2024-05-27T07:18:43.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:43 smithi012 ceph-mon[31473]: pgmap v435: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-05-27T07:18:43.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:43 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2008092637' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:18:43.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:43 smithi162 ceph-mon[35676]: pgmap v435: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-05-27T07:18:43.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:43 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2008092637' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:18:43.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:43 smithi062 ceph-mon[35304]: pgmap v435: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-05-27T07:18:43.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:43 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2008092637' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:18:43.944 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-05-27T07:18:43.945 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-27T07:18:44.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:44 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2008092637' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:18:44.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:44 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:18:44.424 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:44 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2008092637' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:18:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:44 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:18:44.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:44 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2008092637' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:18:44.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:44 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:18:45.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:45 smithi162 ceph-mon[35676]: pgmap v436: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T07:18:45.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:45 smithi012 ceph-mon[31473]: pgmap v436: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T07:18:45.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:45 smithi062 ceph-mon[35304]: pgmap v436: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-05-27T07:18:46.293 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-05-27T07:18:46.293 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-05-27T07:18:46.293 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-05-27T07:18:46.293 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:46 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:18:46.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-05-27T07:18:46.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-05-27T07:18:46.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-05-27T07:18:46.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:46 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:18:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-05-27T07:18:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-05-27T07:18:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-05-27T07:18:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:46 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 -- 172.21.15.12:0/1745473848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988107de0 msgr2=0x7ff9881081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 --2- 172.21.15.12:0/1745473848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988107de0 0x7ff9881081f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff9740099c0 tx=0x7ff974009cd0 comp rx=0 tx=0).stop 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 -- 172.21.15.12:0/1745473848 shutdown_connections 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 --2- 172.21.15.12:0/1745473848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff988108730 0x7ff988110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 --2- 172.21.15.12:0/1745473848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9881034b0 0x7ff9881038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 --2- 172.21.15.12:0/1745473848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988107de0 0x7ff9881081f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 -- 172.21.15.12:0/1745473848 >> 172.21.15.12:0/1745473848 conn(0x7ff9880fedb0 msgr2=0x7ff9881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 -- 172.21.15.12:0/1745473848 shutdown_connections 2024-05-27T07:18:46.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.709+0000 7ff98e200700 1 -- 172.21.15.12:0/1745473848 wait complete. 2024-05-27T07:18:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.710+0000 7ff98e200700 1 Processor -- start 2024-05-27T07:18:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff98e200700 1 -- start start 2024-05-27T07:18:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff98e200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881034b0 0x7ff9880ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff98e200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff988107de0 0x7ff9880ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff98e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff988108730 0x7ff9880ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:46.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff98e200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff988113240 con 0x7ff9881034b0 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff98e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff9881130c0 con 0x7ff988108730 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff98e200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff9881133c0 con 0x7ff988107de0 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.711+0000 7ff987400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881034b0 0x7ff9880ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881034b0 0x7ff9880ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60838/0 (socket says 172.21.15.12:60838) 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987400700 1 -- 172.21.15.12:0/3843638843 learned_addr learned my addr 172.21.15.12:0/3843638843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff986a00700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff988107de0 0x7ff9880ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987e00700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff988108730 0x7ff9880ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987400700 1 -- 172.21.15.12:0/3843638843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff988107de0 msgr2=0x7ff9880ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987400700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff988107de0 0x7ff9880ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987400700 1 -- 172.21.15.12:0/3843638843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff988108730 msgr2=0x7ff9880ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987400700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff988108730 0x7ff9880ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:46.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987400700 1 -- 172.21.15.12:0/3843638843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9881a8fb0 con 0x7ff9881034b0 2024-05-27T07:18:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.712+0000 7ff987e00700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff988108730 0x7ff9880ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:46.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.713+0000 7ff987400700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881034b0 0x7ff9880ff290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c9b0 tx=0x7ff97800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.713+0000 7ff97fe00700 1 -- 172.21.15.12:0/3843638843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978007b00 con 0x7ff9881034b0 2024-05-27T07:18:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.713+0000 7ff97fe00700 1 -- 172.21.15.12:0/3843638843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff978007ca0 con 0x7ff9881034b0 2024-05-27T07:18:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.713+0000 7ff97fe00700 1 -- 172.21.15.12:0/3843638843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978018660 con 0x7ff9881034b0 2024-05-27T07:18:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.713+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9881a92a0 con 0x7ff9881034b0 2024-05-27T07:18:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.713+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9881a9760 con 0x7ff9881034b0 2024-05-27T07:18:46.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.715+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff988104960 con 0x7ff9881034b0 2024-05-27T07:18:46.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.717+0000 7ff97fe00700 1 -- 172.21.15.12:0/3843638843 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff97800f450 con 0x7ff9881034b0 2024-05-27T07:18:46.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.719+0000 7ff97fe00700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff96c0b2900 0x7ff96c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:46.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.719+0000 7ff97fe00700 1 -- 172.21.15.12:0/3843638843 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9208+0+0 (secure 0 0 0) 0x7ff9780e2830 con 0x7ff9881034b0 2024-05-27T07:18:46.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.719+0000 7ff986a00700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff96c0b2900 0x7ff96c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:46.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.720+0000 7ff986a00700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff96c0b2900 0x7ff96c0b4e10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff974009990 tx=0x7ff974005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:46.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.723+0000 7ff97fe00700 1 -- 172.21.15.12:0/3843638843 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff978014030 con 0x7ff9881034b0 2024-05-27T07:18:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:46.999+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff988063bf0 con 0x7ff96c0b2900 2024-05-27T07:18:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.085+0000 7ff97fe00700 1 -- 172.21.15.12:0/3843638843 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff988063bf0 con 0x7ff96c0b2900 2024-05-27T07:18:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.089+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff96c0b2900 msgr2=0x7ff96c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.089+0000 7ff98e200700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff96c0b2900 0x7ff96c0b4e10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff974009990 tx=0x7ff974005fb0 comp rx=0 tx=0).stop 2024-05-27T07:18:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.089+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881034b0 msgr2=0x7ff9880ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.089+0000 7ff98e200700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881034b0 0x7ff9880ff290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c9b0 tx=0x7ff97800ccc0 comp rx=0 tx=0).stop 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 shutdown_connections 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7ff96c0b2900 0x7ff96c0b4e10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff988108730 0x7ff9880ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881034b0 0x7ff9880ff290 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 --2- 172.21.15.12:0/3843638843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff988107de0 0x7ff9880ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 >> 172.21.15.12:0/3843638843 conn(0x7ff9880fedb0 msgr2=0x7ff98806b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 shutdown_connections 2024-05-27T07:18:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:47.090+0000 7ff98e200700 1 -- 172.21.15.12:0/3843638843 wait complete. 2024-05-27T07:18:47.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:47 smithi012 ceph-mon[31473]: pgmap v437: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-27T07:18:47.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:47 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-05-27T07:18:47.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:47 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-05-27T07:18:47.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:47 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-05-27T07:18:47.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:47 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:18:47.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:47 smithi012 ceph-mon[31473]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:18:47.342 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:18:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:18:47.008+0000 7f5749c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:18:47.342 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:18:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:18:47.008+0000 7f5749c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:18:47.342 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:18:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:18:47.008+0000 7f5749c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:18:47.342 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:18:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:18:47.008+0000 7f5749c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:18:47.342 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:18:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:18:47.008+0000 7f5749c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:18:47.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:47 smithi162 ceph-mon[35676]: pgmap v437: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-27T07:18:47.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:47 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-05-27T07:18:47.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:47 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-05-27T07:18:47.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:47 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-05-27T07:18:47.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:47 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:18:47.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:47 smithi162 ceph-mon[35676]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:18:47.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:47 smithi062 ceph-mon[35304]: pgmap v437: 129 pgs: 129 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-27T07:18:47.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:47 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-05-27T07:18:47.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:47 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-05-27T07:18:47.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:47 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-05-27T07:18:47.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:47 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:18:47.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:47 smithi062 ceph-mon[35304]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:18:47.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-27T07:18:48.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:48 smithi012 ceph-mon[31473]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:48.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:48 smithi012 ceph-mon[31473]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:18:48.392 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:48.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:48 smithi162 ceph-mon[35676]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:48.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:48 smithi162 ceph-mon[35676]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:18:48.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:48 smithi062 ceph-mon[35304]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:48.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:48 smithi062 ceph-mon[35304]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:18:49.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:49 smithi012 ceph-mon[31473]: pgmap v440: 129 pgs: 1 peering, 128 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 383 B/s wr, 8 op/s 2024-05-27T07:18:49.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:49 smithi012 ceph-mon[31473]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-27T07:18:49.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:49 smithi012 ceph-mon[31473]: mgrmap e17: x(active, since 12m), standbys: z, y 2024-05-27T07:18:49.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:49 smithi162 ceph-mon[35676]: pgmap v440: 129 pgs: 1 peering, 128 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 383 B/s wr, 8 op/s 2024-05-27T07:18:49.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:49 smithi162 ceph-mon[35676]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-27T07:18:49.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:49 smithi162 ceph-mon[35676]: mgrmap e17: x(active, since 12m), standbys: z, y 2024-05-27T07:18:49.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:49 smithi062 ceph-mon[35304]: pgmap v440: 129 pgs: 1 peering, 128 active+clean; 584 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 383 B/s wr, 8 op/s 2024-05-27T07:18:49.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:49 smithi062 ceph-mon[35304]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-27T07:18:49.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:49 smithi062 ceph-mon[35304]: mgrmap e17: x(active, since 12m), standbys: z, y 2024-05-27T07:18:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.818+0000 7f6c8b800700 1 -- 172.21.15.12:0/1047776505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c84110960 msgr2=0x7f6c84110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.818+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1047776505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c84110960 0x7f6c84110d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009990 tx=0x7f6c78009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.819+0000 7f6c8b800700 1 -- 172.21.15.12:0/1047776505 shutdown_connections 2024-05-27T07:18:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.819+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1047776505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8410b2d0 0x7f6c84117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.819+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1047776505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c84110960 0x7f6c84110d70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.819+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1047776505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c8410a980 0x7f6c8410ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:50.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.819+0000 7f6c8b800700 1 -- 172.21.15.12:0/1047776505 >> 172.21.15.12:0/1047776505 conn(0x7f6c84077c70 msgr2=0x7f6c840780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:50.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.819+0000 7f6c8b800700 1 -- 172.21.15.12:0/1047776505 shutdown_connections 2024-05-27T07:18:50.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.819+0000 7f6c8b800700 1 -- 172.21.15.12:0/1047776505 wait complete. 2024-05-27T07:18:50.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.820+0000 7f6c8b800700 1 Processor -- start 2024-05-27T07:18:50.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c8b800700 1 -- start start 2024-05-27T07:18:50.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c8b800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c8410a980 0x7f6c841a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:50.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c8b800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c8410b2d0 0x7f6c841a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:50.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c8b800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c84110960 0x7f6c841a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:50.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c8b800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8411a920 con 0x7f6c8410a980 2024-05-27T07:18:50.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c8b800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8411a7a0 con 0x7f6c8410b2d0 2024-05-27T07:18:50.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c8b800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8411aaa0 con 0x7f6c84110960 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c88e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c8410a980 0x7f6c841a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.821+0000 7f6c88e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c8410a980 0x7f6c841a1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60882/0 (socket says 172.21.15.12:60882) 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c88e00700 1 -- 172.21.15.12:0/1822060307 learned_addr learned my addr 172.21.15.12:0/1822060307 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c83e00700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c8410b2d0 0x7f6c841a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c88e00700 1 -- 172.21.15.12:0/1822060307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c84110960 msgr2=0x7f6c841a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c88e00700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c84110960 0x7f6c841a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c88e00700 1 -- 172.21.15.12:0/1822060307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c8410b2d0 msgr2=0x7f6c841a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c88e00700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c8410b2d0 0x7f6c841a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c88e00700 1 -- 172.21.15.12:0/1822060307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c841a6240 con 0x7f6c8410a980 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.822+0000 7f6c88e00700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c8410a980 0x7f6c841a1370 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6c78000c00 tx=0x7f6c78009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.823+0000 7f6c81600700 1 -- 172.21.15.12:0/1822060307 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78027070 con 0x7f6c8410a980 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.823+0000 7f6c81600700 1 -- 172.21.15.12:0/1822060307 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c7800fc70 con 0x7f6c8410a980 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.823+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c841a64d0 con 0x7f6c8410a980 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.823+0000 7f6c81600700 1 -- 172.21.15.12:0/1822060307 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78022d30 con 0x7f6c8410a980 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.823+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c841a69b0 con 0x7f6c8410a980 2024-05-27T07:18:50.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.824+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c840512b0 con 0x7f6c8410a980 2024-05-27T07:18:50.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.827+0000 7f6c81600700 1 -- 172.21.15.12:0/1822060307 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6c7802a070 con 0x7f6c8410a980 2024-05-27T07:18:50.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.828+0000 7f6c81600700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6c6c0b2910 0x7f6c6c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:50.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.828+0000 7f6c81600700 1 -- 172.21.15.12:0/1822060307 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f6c780e3a30 con 0x7f6c8410a980 2024-05-27T07:18:50.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.829+0000 7f6c83e00700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6c6c0b2910 0x7f6c6c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:50.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.829+0000 7f6c83e00700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6c6c0b2910 0x7f6c6c0b4e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c70005950 tx=0x7f6c700058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:50.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:50.831+0000 7f6c81600700 1 -- 172.21.15.12:0/1822060307 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6c78031080 con 0x7f6c8410a980 2024-05-27T07:18:51.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.110+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6c84063bf0 con 0x7f6c6c0b2910 2024-05-27T07:18:51.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.186+0000 7f6c81600700 1 -- 172.21.15.12:0/1822060307 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6c84063bf0 con 0x7f6c6c0b2910 2024-05-27T07:18:51.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6c6c0b2910 msgr2=0x7f6c6c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:51.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6c6c0b2910 0x7f6c6c0b4e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c70005950 tx=0x7f6c700058e0 comp rx=0 tx=0).stop 2024-05-27T07:18:51.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c8410a980 msgr2=0x7f6c841a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:51.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c8410a980 0x7f6c841a1370 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6c78000c00 tx=0x7f6c78009f90 comp rx=0 tx=0).stop 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 shutdown_connections 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f6c6c0b2910 0x7f6c6c0b4e20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c84110960 0x7f6c841a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c8410a980 0x7f6c841a1370 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 --2- 172.21.15.12:0/1822060307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c8410b2d0 0x7f6c841a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.190+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 >> 172.21.15.12:0/1822060307 conn(0x7f6c84077c70 msgr2=0x7f6c8407bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.191+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 shutdown_connections 2024-05-27T07:18:51.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:51.191+0000 7f6c8b800700 1 -- 172.21.15.12:0/1822060307 wait complete. 2024-05-27T07:18:51.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:51 smithi012 ceph-mon[31473]: pgmap v441: 129 pgs: 1 peering, 128 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-27T07:18:51.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:51 smithi162 ceph-mon[35676]: pgmap v441: 129 pgs: 1 peering, 128 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-27T07:18:51.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:51 smithi062 ceph-mon[35304]: pgmap v441: 129 pgs: 1 peering, 128 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-27T07:18:52.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-27T07:18:52.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:52 smithi162 ceph-mon[35676]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:52.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:52 smithi162 ceph-mon[35676]: pgmap v442: 129 pgs: 129 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-27T07:18:52.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:52 smithi162 ceph-mon[35676]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-27T07:18:52.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:52 smithi162 ceph-mon[35676]: Cluster is now healthy 2024-05-27T07:18:52.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:52 smithi062 ceph-mon[35304]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:52.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:52 smithi062 ceph-mon[35304]: pgmap v442: 129 pgs: 129 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-27T07:18:52.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:52 smithi062 ceph-mon[35304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-27T07:18:52.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:52 smithi062 ceph-mon[35304]: Cluster is now healthy 2024-05-27T07:18:52.590 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:52.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:52 smithi012 ceph-mon[31473]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:52.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:52 smithi012 ceph-mon[31473]: pgmap v442: 129 pgs: 129 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-27T07:18:52.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:52 smithi012 ceph-mon[31473]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-27T07:18:52.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:52 smithi012 ceph-mon[31473]: Cluster is now healthy 2024-05-27T07:18:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 -- 172.21.15.12:0/3792853074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba010b2d0 msgr2=0x7f9ba0117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3792853074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba010b2d0 0x7f9ba0117800 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800b0a0 tx=0x7f9b9800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 -- 172.21.15.12:0/3792853074 shutdown_connections 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3792853074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba0110960 0x7f9ba0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3792853074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba010b2d0 0x7f9ba0117800 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3792853074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ba010a980 0x7f9ba010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 -- 172.21.15.12:0/3792853074 >> 172.21.15.12:0/3792853074 conn(0x7f9ba0077c70 msgr2=0x7f9ba00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 -- 172.21.15.12:0/3792853074 shutdown_connections 2024-05-27T07:18:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.920+0000 7f9ba5600700 1 -- 172.21.15.12:0/3792853074 wait complete. 2024-05-27T07:18:54.911 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:54 smithi012 ceph-mon[31473]: pgmap v443: 129 pgs: 129 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-27T07:18:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.922+0000 7f9ba5600700 1 Processor -- start 2024-05-27T07:18:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.922+0000 7f9ba5600700 1 -- start start 2024-05-27T07:18:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9ba5600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ba010a980 0x7f9ba01a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9ba5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba010b2d0 0x7f9ba01a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9ba5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba0110960 0x7f9ba01a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9ba5600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba011a980 con 0x7f9ba0110960 2024-05-27T07:18:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9ba5600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba011a800 con 0x7f9ba010b2d0 2024-05-27T07:18:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9ba5600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba011ab00 con 0x7f9ba010a980 2024-05-27T07:18:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9b9f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba0110960 0x7f9ba01a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9b9f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba0110960 0x7f9ba01a5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54464/0 (socket says 172.21.15.12:54464) 2024-05-27T07:18:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9b9f400700 1 -- 172.21.15.12:0/3712902485 learned_addr learned my addr 172.21.15.12:0/3712902485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9b9ea00700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ba010a980 0x7f9ba01a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.923+0000 7f9b9e000700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba010b2d0 0x7f9ba01a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.924+0000 7f9b9f400700 1 -- 172.21.15.12:0/3712902485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ba010a980 msgr2=0x7f9ba01a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.924+0000 7f9b9f400700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ba010a980 0x7f9ba01a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.924+0000 7f9b9f400700 1 -- 172.21.15.12:0/3712902485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba010b2d0 msgr2=0x7f9ba01a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.924+0000 7f9b9f400700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba010b2d0 0x7f9ba01a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.924+0000 7f9b9f400700 1 -- 172.21.15.12:0/3712902485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba01a6200 con 0x7f9ba0110960 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.924+0000 7f9b9f400700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba0110960 0x7f9ba01a5b60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800b070 tx=0x7f9b98004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.924+0000 7f9b97400700 1 -- 172.21.15.12:0/3712902485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98021070 con 0x7f9ba0110960 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.925+0000 7f9b97400700 1 -- 172.21.15.12:0/3712902485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b98007b80 con 0x7f9ba0110960 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.925+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba01a6490 con 0x7f9ba0110960 2024-05-27T07:18:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.925+0000 7f9b97400700 1 -- 172.21.15.12:0/3712902485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98004640 con 0x7f9ba0110960 2024-05-27T07:18:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.925+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba01ade10 con 0x7f9ba0110960 2024-05-27T07:18:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.926+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba010c070 con 0x7f9ba0110960 2024-05-27T07:18:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.929+0000 7f9b97400700 1 -- 172.21.15.12:0/3712902485 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9b98003a00 con 0x7f9ba0110960 2024-05-27T07:18:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.930+0000 7f9b97400700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b840b2990 0x7f9b840b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.930+0000 7f9b97400700 1 -- 172.21.15.12:0/3712902485 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f9b9800f020 con 0x7f9ba0110960 2024-05-27T07:18:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.931+0000 7f9b9ea00700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b840b2990 0x7f9b840b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.931+0000 7f9b9ea00700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b840b2990 0x7f9b840b4ea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9ba007d0e0 tx=0x7f9b900073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:54.936+0000 7f9b97400700 1 -- 172.21.15.12:0/3712902485 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9b9802a020 con 0x7f9ba0110960 2024-05-27T07:18:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.212+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f9ba01a2690 con 0x7f9b840b2990 2024-05-27T07:18:55.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.224+0000 7f9b97400700 1 -- 172.21.15.12:0/3712902485 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9ba01a2690 con 0x7f9b840b2990 2024-05-27T07:18:55.214 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_0/9515fe3d-085a-4399-9da5-21c1d3d75134 2024-05-27T07:18:55.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b840b2990 msgr2=0x7f9b840b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:55.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b840b2990 0x7f9b840b4ea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9ba007d0e0 tx=0x7f9b900073d0 comp rx=0 tx=0).stop 2024-05-27T07:18:55.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba0110960 msgr2=0x7f9ba01a5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:55.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba0110960 0x7f9ba01a5b60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800b070 tx=0x7f9b98004d50 comp rx=0 tx=0).stop 2024-05-27T07:18:55.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 shutdown_connections 2024-05-27T07:18:55.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b840b2990 0x7f9b840b4ea0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:55.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ba0110960 0x7f9ba01a5b60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:55.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ba010a980 0x7f9ba01a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:55.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 --2- 172.21.15.12:0/3712902485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba010b2d0 0x7f9ba01a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:55.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 >> 172.21.15.12:0/3712902485 conn(0x7f9ba0077c70 msgr2=0x7f9ba007b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:55.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 shutdown_connections 2024-05-27T07:18:55.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:55.227+0000 7f9ba5600700 1 -- 172.21.15.12:0/3712902485 wait complete. 2024-05-27T07:18:55.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:54 smithi162 ceph-mon[35676]: pgmap v443: 129 pgs: 129 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-27T07:18:55.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:54 smithi062 ceph-mon[35304]: pgmap v443: 129 pgs: 129 active+clean; 591 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-27T07:18:56.158 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-27T07:18:56.607 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:56 smithi012 ceph-mon[31473]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:57.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:56 smithi012 ceph-mon[31473]: pgmap v444: 129 pgs: 129 active+clean; 605 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-05-27T07:18:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:56 smithi162 ceph-mon[35676]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:57.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:56 smithi162 ceph-mon[35676]: pgmap v444: 129 pgs: 129 active+clean; 605 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-05-27T07:18:57.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:56 smithi062 ceph-mon[35304]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:57.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:56 smithi062 ceph-mon[35304]: pgmap v444: 129 pgs: 129 active+clean; 605 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-05-27T07:18:58.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 -- 172.21.15.12:0/1229853294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 msgr2=0x7f8c4810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/1229853294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c4810ad90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009990 tx=0x7f8c34009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 -- 172.21.15.12:0/1229853294 shutdown_connections 2024-05-27T07:18:58.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/1229853294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c4810b2d0 0x7f8c48117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/1229853294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c48110960 0x7f8c48110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/1229853294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c4810ad90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 -- 172.21.15.12:0/1229853294 >> 172.21.15.12:0/1229853294 conn(0x7f8c48077c70 msgr2=0x7f8c480780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 -- 172.21.15.12:0/1229853294 shutdown_connections 2024-05-27T07:18:58.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.932+0000 7f8c4cc00700 1 -- 172.21.15.12:0/1229853294 wait complete. 2024-05-27T07:18:58.923 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:18:58 smithi012 ceph-mon[31473]: pgmap v445: 129 pgs: 129 active+clean; 605 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-27T07:18:58.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.933+0000 7f8c4cc00700 1 Processor -- start 2024-05-27T07:18:58.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c4cc00700 1 -- start start 2024-05-27T07:18:58.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c4cc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c481a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c4cc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4810b2d0 0x7f8c481a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c4cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c48110960 0x7f8c481a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c4cc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4811a980 con 0x7f8c4810a980 2024-05-27T07:18:58.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c4cc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4811a800 con 0x7f8c48110960 2024-05-27T07:18:58.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c4cc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4811ab00 con 0x7f8c4810b2d0 2024-05-27T07:18:58.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c45e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c481a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c45e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c481a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54492/0 (socket says 172.21.15.12:54492) 2024-05-27T07:18:58.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.934+0000 7f8c45e00700 1 -- 172.21.15.12:0/3671955668 learned_addr learned my addr 172.21.15.12:0/3671955668 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:18:58.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.935+0000 7f8c45400700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4810b2d0 0x7f8c481a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.935+0000 7f8c46800700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c48110960 0x7f8c481a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.935+0000 7f8c45e00700 1 -- 172.21.15.12:0/3671955668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4810b2d0 msgr2=0x7f8c481a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.935+0000 7f8c45e00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4810b2d0 0x7f8c481a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.935+0000 7f8c45e00700 1 -- 172.21.15.12:0/3671955668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c48110960 msgr2=0x7f8c481a5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.935+0000 7f8c45e00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c48110960 0x7f8c481a5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.935+0000 7f8c45e00700 1 -- 172.21.15.12:0/3671955668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c481a6280 con 0x7f8c4810a980 2024-05-27T07:18:58.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.936+0000 7f8c45e00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c481a13b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d950 tx=0x7f8c3800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.936+0000 7f8c3ea00700 1 -- 172.21.15.12:0/3671955668 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38016070 con 0x7f8c4810a980 2024-05-27T07:18:58.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.936+0000 7f8c3ea00700 1 -- 172.21.15.12:0/3671955668 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c3800de30 con 0x7f8c4810a980 2024-05-27T07:18:58.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.936+0000 7f8c3ea00700 1 -- 172.21.15.12:0/3671955668 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38005240 con 0x7f8c4810a980 2024-05-27T07:18:58.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.936+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c481a6570 con 0x7f8c4810a980 2024-05-27T07:18:58.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.937+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c481ade90 con 0x7f8c4810a980 2024-05-27T07:18:58.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.938+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4810c070 con 0x7f8c4810a980 2024-05-27T07:18:58.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.940+0000 7f8c3ea00700 1 -- 172.21.15.12:0/3671955668 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8c380049a0 con 0x7f8c4810a980 2024-05-27T07:18:58.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.942+0000 7f8c3ea00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c2c0b2910 0x7f8c2c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.942+0000 7f8c3ea00700 1 -- 172.21.15.12:0/3671955668 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f8c380e1ff0 con 0x7f8c4810a980 2024-05-27T07:18:58.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.942+0000 7f8c45400700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c2c0b2910 0x7f8c2c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.943+0000 7f8c45400700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c2c0b2910 0x7f8c2c0b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400b5c0 tx=0x7f8c34005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:58.947+0000 7f8c3ea00700 1 -- 172.21.15.12:0/3671955668 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8c3801f070 con 0x7f8c4810a980 2024-05-27T07:18:59.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.222+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8c48063bf0 con 0x7f8c2c0b2910 2024-05-27T07:18:59.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:18:58 smithi162 ceph-mon[35676]: pgmap v445: 129 pgs: 129 active+clean; 605 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-27T07:18:59.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:18:58 smithi062 ceph-mon[35304]: pgmap v445: 129 pgs: 129 active+clean; 605 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-27T07:18:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.295+0000 7f8c3ea00700 1 -- 172.21.15.12:0/3671955668 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8c48063bf0 con 0x7f8c2c0b2910 2024-05-27T07:18:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.298+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c2c0b2910 msgr2=0x7f8c2c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c2c0b2910 0x7f8c2c0b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400b5c0 tx=0x7f8c34005fb0 comp rx=0 tx=0).stop 2024-05-27T07:18:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 msgr2=0x7f8c481a13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c481a13b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d950 tx=0x7f8c3800dd10 comp rx=0 tx=0).stop 2024-05-27T07:18:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 shutdown_connections 2024-05-27T07:18:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f8c2c0b2910 0x7f8c2c0b4e20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c48110960 0x7f8c481a5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c4810a980 0x7f8c481a13b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 --2- 172.21.15.12:0/3671955668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4810b2d0 0x7f8c481a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 >> 172.21.15.12:0/3671955668 conn(0x7f8c48077c70 msgr2=0x7f8c4807b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 shutdown_connections 2024-05-27T07:18:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:18:59.299+0000 7f8c4cc00700 1 -- 172.21.15.12:0/3671955668 wait complete. 2024-05-27T07:19:00.246 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-27T07:19:00.720 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:01.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:00 smithi012 ceph-mon[31473]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:01.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:00 smithi012 ceph-mon[31473]: pgmap v446: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-05-27T07:19:01.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:00 smithi162 ceph-mon[35676]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:00 smithi162 ceph-mon[35676]: pgmap v446: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-05-27T07:19:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:00 smithi062 ceph-mon[35304]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:00 smithi062 ceph-mon[35304]: pgmap v446: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-05-27T07:19:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.972+0000 7fa61fc00700 1 -- 172.21.15.12:0/4038837373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618077780 msgr2=0x7fa618110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.972+0000 7fa61fc00700 1 --2- 172.21.15.12:0/4038837373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618077780 0x7fa618110630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa600009990 tx=0x7fa600009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.973+0000 7fa61fc00700 1 -- 172.21.15.12:0/4038837373 shutdown_connections 2024-05-27T07:19:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.973+0000 7fa61fc00700 1 --2- 172.21.15.12:0/4038837373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618110b70 0x7fa618112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.973+0000 7fa61fc00700 1 --2- 172.21.15.12:0/4038837373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa618076e30 0x7fa618077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.973+0000 7fa61fc00700 1 --2- 172.21.15.12:0/4038837373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618077780 0x7fa618110630 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.973+0000 7fa61fc00700 1 -- 172.21.15.12:0/4038837373 >> 172.21.15.12:0/4038837373 conn(0x7fa6180ffe90 msgr2=0x7fa6181022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.973+0000 7fa61fc00700 1 -- 172.21.15.12:0/4038837373 shutdown_connections 2024-05-27T07:19:02.964 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:02 smithi012 ceph-mon[31473]: pgmap v447: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.973+0000 7fa61fc00700 1 -- 172.21.15.12:0/4038837373 wait complete. 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.974+0000 7fa61fc00700 1 Processor -- start 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61fc00700 1 -- start start 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61fc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618076e30 0x7fa61819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61fc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa618077780 0x7fa61819d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61fc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618110b70 0x7fa6181a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61fc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa618116540 con 0x7fa618110b70 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61fc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa6181163c0 con 0x7fa618076e30 2024-05-27T07:19:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61fc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6181166c0 con 0x7fa618077780 2024-05-27T07:19:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618110b70 0x7fa6181a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.975+0000 7fa61dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618110b70 0x7fa6181a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50724/0 (socket says 172.21.15.12:50724) 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61dc00700 1 -- 172.21.15.12:0/445276558 learned_addr learned my addr 172.21.15.12:0/445276558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61d200700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618076e30 0x7fa61819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa60fe00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa618077780 0x7fa61819d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61dc00700 1 -- 172.21.15.12:0/445276558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa618077780 msgr2=0x7fa61819d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61dc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa618077780 0x7fa61819d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61dc00700 1 -- 172.21.15.12:0/445276558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618076e30 msgr2=0x7fa61819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61dc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618076e30 0x7fa61819cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61dc00700 1 -- 172.21.15.12:0/445276558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6181a1e20 con 0x7fa618110b70 2024-05-27T07:19:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.976+0000 7fa61dc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618110b70 0x7fa6181a1780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa61000e970 tx=0x7fa61000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.977+0000 7fa616000700 1 -- 172.21.15.12:0/445276558 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61000cbd0 con 0x7fa618110b70 2024-05-27T07:19:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.977+0000 7fa616000700 1 -- 172.21.15.12:0/445276558 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa61000cd70 con 0x7fa618110b70 2024-05-27T07:19:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.977+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6181a2110 con 0x7fa618110b70 2024-05-27T07:19:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.977+0000 7fa616000700 1 -- 172.21.15.12:0/445276558 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610005550 con 0x7fa618110b70 2024-05-27T07:19:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.977+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6181a9a30 con 0x7fa618110b70 2024-05-27T07:19:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.978+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6180512b0 con 0x7fa618110b70 2024-05-27T07:19:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.981+0000 7fa616000700 1 -- 172.21.15.12:0/445276558 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa610004750 con 0x7fa618110b70 2024-05-27T07:19:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.982+0000 7fa616000700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5fc0b2990 0x7fa5fc0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.983+0000 7fa616000700 1 -- 172.21.15.12:0/445276558 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fa6100e2ba0 con 0x7fa618110b70 2024-05-27T07:19:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.983+0000 7fa61d200700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5fc0b2990 0x7fa5fc0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.984+0000 7fa61d200700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5fc0b2990 0x7fa5fc0b4ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa608009d10 tx=0x7fa608009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:02.986+0000 7fa616000700 1 -- 172.21.15.12:0/445276558 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa610054df0 con 0x7fa618110b70 2024-05-27T07:19:03.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:02 smithi162 ceph-mon[35676]: pgmap v447: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-05-27T07:19:03.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:02 smithi062 ceph-mon[35304]: pgmap v447: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-05-27T07:19:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.268+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa618063bf0 con 0x7fa5fc0b2990 2024-05-27T07:19:03.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.281+0000 7fa616000700 1 -- 172.21.15.12:0/445276558 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa618063bf0 con 0x7fa5fc0b2990 2024-05-27T07:19:03.271 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_1/8a336912-d5a1-4cbf-b144-ed2f4b17a5f8 2024-05-27T07:19:03.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5fc0b2990 msgr2=0x7fa5fc0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:03.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5fc0b2990 0x7fa5fc0b4ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa608009d10 tx=0x7fa608009040 comp rx=0 tx=0).stop 2024-05-27T07:19:03.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618110b70 msgr2=0x7fa6181a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:03.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618110b70 0x7fa6181a1780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa61000e970 tx=0x7fa61000ed30 comp rx=0 tx=0).stop 2024-05-27T07:19:03.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 shutdown_connections 2024-05-27T07:19:03.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5fc0b2990 0x7fa5fc0b4ea0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:03.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618076e30 0x7fa61819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:03.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa618110b70 0x7fa6181a1780 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:03.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 --2- 172.21.15.12:0/445276558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa618077780 0x7fa61819d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:03.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 >> 172.21.15.12:0/445276558 conn(0x7fa6180ffe90 msgr2=0x7fa6181005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:03.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.284+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 shutdown_connections 2024-05-27T07:19:03.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:03.285+0000 7fa61fc00700 1 -- 172.21.15.12:0/445276558 wait complete. 2024-05-27T07:19:04.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:19:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:04.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:04 smithi012 ceph-mon[31473]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:04.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:04 smithi012 ceph-mon[31473]: pgmap v448: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-27T07:19:05.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:04 smithi162 ceph-mon[35676]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:05.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:04 smithi162 ceph-mon[35676]: pgmap v448: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-27T07:19:05.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:04 smithi062 ceph-mon[35304]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:05.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:04 smithi062 ceph-mon[35304]: pgmap v448: 129 pgs: 129 active+clean; 611 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-27T07:19:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 -- 172.21.15.12:0/3628810832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82141043f0 msgr2=0x7f8214104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 --2- 172.21.15.12:0/3628810832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82141043f0 0x7f8214104800 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f82040099c0 tx=0x7f8204009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 -- 172.21.15.12:0/3628810832 shutdown_connections 2024-05-27T07:19:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 --2- 172.21.15.12:0/3628810832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8214109670 0x7f8214110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 --2- 172.21.15.12:0/3628810832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82141043f0 0x7f8214104800 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 --2- 172.21.15.12:0/3628810832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8214108d20 0x7f8214109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 -- 172.21.15.12:0/3628810832 >> 172.21.15.12:0/3628810832 conn(0x7f8214079d00 msgr2=0x7f821407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 -- 172.21.15.12:0/3628810832 shutdown_connections 2024-05-27T07:19:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.876+0000 7f8213e00700 1 -- 172.21.15.12:0/3628810832 wait complete. 2024-05-27T07:19:06.868 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:06 smithi012 ceph-mon[31473]: pgmap v449: 129 pgs: 129 active+clean; 633 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-05-27T07:19:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.877+0000 7f8213e00700 1 Processor -- start 2024-05-27T07:19:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.878+0000 7f8213e00700 1 -- start start 2024-05-27T07:19:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.878+0000 7f8213e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82141043f0 0x7f821419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.878+0000 7f8213e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8214108d20 0x7f821419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.878+0000 7f8213e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8214109670 0x7f82141a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.878+0000 7f8213e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8214114170 con 0x7f8214109670 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.878+0000 7f8213e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8214113ff0 con 0x7f8214108d20 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.878+0000 7f8213e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f82141142f0 con 0x7f82141043f0 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8214109670 0x7f82141a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8214109670 0x7f82141a1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50734/0 (socket says 172.21.15.12:50734) 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 -- 172.21.15.12:0/831024524 learned_addr learned my addr 172.21.15.12:0/831024524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8212a00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82141043f0 0x7f821419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8212000700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8214108d20 0x7f821419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 -- 172.21.15.12:0/831024524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82141043f0 msgr2=0x7f821419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82141043f0 0x7f821419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 -- 172.21.15.12:0/831024524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8214108d20 msgr2=0x7f821419d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8214108d20 0x7f821419d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 -- 172.21.15.12:0/831024524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82141a1c40 con 0x7f8214109670 2024-05-27T07:19:06.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.879+0000 7f8213400700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8214109670 0x7f82141a1650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f820c007ee0 tx=0x7f820c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.880+0000 7f820b400700 1 -- 172.21.15.12:0/831024524 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c00dbd0 con 0x7f8214109670 2024-05-27T07:19:06.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.880+0000 7f820b400700 1 -- 172.21.15.12:0/831024524 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f820c00dd70 con 0x7f8214109670 2024-05-27T07:19:06.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.880+0000 7f820b400700 1 -- 172.21.15.12:0/831024524 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c0058d0 con 0x7f8214109670 2024-05-27T07:19:06.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.880+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82141a1f30 con 0x7f8214109670 2024-05-27T07:19:06.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.880+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82141a2420 con 0x7f8214109670 2024-05-27T07:19:06.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.881+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82141058a0 con 0x7f8214109670 2024-05-27T07:19:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.884+0000 7f820b400700 1 -- 172.21.15.12:0/831024524 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f820c01a020 con 0x7f8214109670 2024-05-27T07:19:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.885+0000 7f820b400700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f81f80b2990 0x7f81f80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.886+0000 7f820b400700 1 -- 172.21.15.12:0/831024524 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f820c0e1f80 con 0x7f8214109670 2024-05-27T07:19:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.886+0000 7f8212a00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f81f80b2990 0x7f81f80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.886+0000 7f8212a00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f81f80b2990 0x7f81f80b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f82040058e0 tx=0x7f8204005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:06.888+0000 7f820b400700 1 -- 172.21.15.12:0/831024524 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f820c010070 con 0x7f8214109670 2024-05-27T07:19:07.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.176+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82141a9960 con 0x7f8214109670 2024-05-27T07:19:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.179+0000 7f820b400700 1 -- 172.21.15.12:0/831024524 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11867 (secure 0 0 0) 0x7f820c0a9710 con 0x7f8214109670 2024-05-27T07:19:07.170 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:19:07.170 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:18:42.889436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:19:07.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f81f80b2990 msgr2=0x7f81f80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:07.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f81f80b2990 0x7f81f80b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f82040058e0 tx=0x7f8204005940 comp rx=0 tx=0).stop 2024-05-27T07:19:07.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8214109670 msgr2=0x7f82141a1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:07.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8214109670 0x7f82141a1650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f820c007ee0 tx=0x7f820c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:07.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 shutdown_connections 2024-05-27T07:19:07.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f81f80b2990 0x7f81f80b4ea0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:07.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8214109670 0x7f82141a1650 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:07.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82141043f0 0x7f821419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:07.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 --2- 172.21.15.12:0/831024524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8214108d20 0x7f821419d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:07.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 >> 172.21.15.12:0/831024524 conn(0x7f8214079d00 msgr2=0x7f8214101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:07.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 shutdown_connections 2024-05-27T07:19:07.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:07.182+0000 7f8213e00700 1 -- 172.21.15.12:0/831024524 wait complete. 2024-05-27T07:19:07.175 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 26 2024-05-27T07:19:07.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:06 smithi162 ceph-mon[35676]: pgmap v449: 129 pgs: 129 active+clean; 633 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-05-27T07:19:07.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:06 smithi062 ceph-mon[35304]: pgmap v449: 129 pgs: 129 active+clean; 633 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-05-27T07:19:08.139 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:07 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/831024524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:19:08.144 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:19:08.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:07 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/831024524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:19:08.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:07 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/831024524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:19:08.602 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:09.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:08 smithi012 ceph-mon[31473]: pgmap v450: 129 pgs: 129 active+clean; 633 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-05-27T07:19:09.138 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:08 smithi062 ceph-mon[35304]: pgmap v450: 129 pgs: 129 active+clean; 633 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-05-27T07:19:09.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:08 smithi162 ceph-mon[35676]: pgmap v450: 129 pgs: 129 active+clean; 633 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-05-27T07:19:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 -- 172.21.15.12:0/1219797897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105490 msgr2=0x7f604c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 --2- 172.21.15.12:0/1219797897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105490 0x7f604c1058a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f603c009990 tx=0x7f603c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 -- 172.21.15.12:0/1219797897 shutdown_connections 2024-05-27T07:19:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 --2- 172.21.15.12:0/1219797897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604c10e900 0x7f604c110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 --2- 172.21.15.12:0/1219797897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105490 0x7f604c1058a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 --2- 172.21.15.12:0/1219797897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105de0 0x7f604c10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 -- 172.21.15.12:0/1219797897 >> 172.21.15.12:0/1219797897 conn(0x7f604c0fedb0 msgr2=0x7f604c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:10.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 -- 172.21.15.12:0/1219797897 shutdown_connections 2024-05-27T07:19:10.954 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:10 smithi012 ceph-mon[31473]: pgmap v451: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-05-27T07:19:10.954 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:10 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.962+0000 7f6053000700 1 -- 172.21.15.12:0/1219797897 wait complete. 2024-05-27T07:19:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 Processor -- start 2024-05-27T07:19:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 -- start start 2024-05-27T07:19:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105490 0x7f604c103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105de0 0x7f604c1036c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604c10e900 0x7f604c103c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f604c114350 con 0x7f604c105de0 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f604c1141d0 con 0x7f604c105490 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.964+0000 7f6053000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f604c1144d0 con 0x7f604c10e900 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105490 0x7f604c103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105de0 0x7f604c1036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105490 0x7f604c103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:48620/0 (socket says 172.21.15.12:48620) 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 -- 172.21.15.12:0/1425607906 learned_addr learned my addr 172.21.15.12:0/1425607906 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:10.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f6051000700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604c10e900 0x7f604c103c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 -- 172.21.15.12:0/1425607906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604c10e900 msgr2=0x7f604c103c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604c10e900 0x7f604c103c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 -- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105de0 msgr2=0x7f604c1036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105de0 0x7f604c1036c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.965+0000 7f604be00700 1 -- 172.21.15.12:0/1425607906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604c1a9050 con 0x7f604c105490 2024-05-27T07:19:10.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.966+0000 7f604b400700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105de0 0x7f604c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:19:10.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.966+0000 7f604be00700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105490 0x7f604c103180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f603c000c00 tx=0x7f603c0057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.966+0000 7f6048c00700 1 -- 172.21.15.12:0/1425607906 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c027070 con 0x7f604c105490 2024-05-27T07:19:10.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.966+0000 7f6048c00700 1 -- 172.21.15.12:0/1425607906 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f603c00fb00 con 0x7f604c105490 2024-05-27T07:19:10.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.966+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604c1a92e0 con 0x7f604c105490 2024-05-27T07:19:10.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.966+0000 7f6048c00700 1 -- 172.21.15.12:0/1425607906 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c022a90 con 0x7f604c105490 2024-05-27T07:19:10.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.967+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f604c1a9790 con 0x7f604c105490 2024-05-27T07:19:10.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.969+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f604c0512b0 con 0x7f604c105490 2024-05-27T07:19:10.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.972+0000 7f6048c00700 1 -- 172.21.15.12:0/1425607906 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f603c02a070 con 0x7f604c105490 2024-05-27T07:19:10.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.973+0000 7f6048c00700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f60300b2990 0x7f60300b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.973+0000 7f6048c00700 1 -- 172.21.15.12:0/1425607906 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f603c0e3ac0 con 0x7f604c105490 2024-05-27T07:19:10.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.974+0000 7f604b400700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f60300b2990 0x7f60300b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.974+0000 7f604b400700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f60300b2990 0x7f60300b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f604c102450 tx=0x7f60380073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:10.979+0000 7f6048c00700 1 -- 172.21.15.12:0/1425607906 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f603c05fdf0 con 0x7f604c105490 2024-05-27T07:19:11.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.223+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f604c02e5e0 con 0x7f604c105490 2024-05-27T07:19:11.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.228+0000 7f6048c00700 1 -- 172.21.15.12:0/1425607906 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24806 (secure 0 0 0) 0x7f603c013070 con 0x7f604c105490 2024-05-27T07:19:11.220 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:19:11.220 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:18:47.037205+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f60300b2990 msgr2=0x7f60300b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f60300b2990 0x7f60300b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f604c102450 tx=0x7f60380073d0 comp rx=0 tx=0).stop 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105490 msgr2=0x7f604c103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105490 0x7f604c103180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f603c000c00 tx=0x7f603c0057a0 comp rx=0 tx=0).stop 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 shutdown_connections 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f60300b2990 0x7f60300b4ea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c105490 0x7f604c103180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604c10e900 0x7f604c103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 --2- 172.21.15.12:0/1425607906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f604c105de0 0x7f604c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:11.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 >> 172.21.15.12:0/1425607906 conn(0x7f604c0fedb0 msgr2=0x7f604c0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:10 smithi162 ceph-mon[35676]: pgmap v451: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-05-27T07:19:11.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:10 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:11.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 shutdown_connections 2024-05-27T07:19:11.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:11.232+0000 7f6053000700 1 -- 172.21.15.12:0/1425607906 wait complete. 2024-05-27T07:19:11.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:10 smithi062 ceph-mon[35304]: pgmap v451: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-05-27T07:19:11.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:10 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:12.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:11 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1425607906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:12.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:11 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1425607906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:12.209 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:19:12.210 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:19:12.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:11 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1425607906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:13.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:12 smithi012 ceph-mon[31473]: pgmap v452: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-05-27T07:19:13.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:12 smithi162 ceph-mon[35676]: pgmap v452: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-05-27T07:19:13.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:12 smithi062 ceph-mon[35304]: pgmap v452: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-05-27T07:19:13.344 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:14.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:14 smithi012 ceph-mon[31473]: pgmap v453: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-05-27T07:19:14.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:14.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:14.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:14.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:14 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:14 smithi162 ceph-mon[35676]: pgmap v453: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-05-27T07:19:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:14 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:14 smithi062 ceph-mon[35304]: pgmap v453: 129 pgs: 129 active+clean; 638 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-05-27T07:19:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:14 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:15.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 -- 172.21.15.12:0/849452398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 msgr2=0x7f1c481081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:15.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 --2- 172.21.15.12:0/849452398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c481081f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1c340099c0 tx=0x7f1c34009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 -- 172.21.15.12:0/849452398 shutdown_connections 2024-05-27T07:19:15.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 --2- 172.21.15.12:0/849452398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c48108730 0x7f1c48110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 --2- 172.21.15.12:0/849452398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c481034b0 0x7f1c481038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 --2- 172.21.15.12:0/849452398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c481081f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 -- 172.21.15.12:0/849452398 >> 172.21.15.12:0/849452398 conn(0x7f1c480fedb0 msgr2=0x7f1c481011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:15.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 -- 172.21.15.12:0/849452398 shutdown_connections 2024-05-27T07:19:15.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.534+0000 7f1c4f000700 1 -- 172.21.15.12:0/849452398 wait complete. 2024-05-27T07:19:15.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.536+0000 7f1c4f000700 1 Processor -- start 2024-05-27T07:19:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.536+0000 7f1c4f000700 1 -- start start 2024-05-27T07:19:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.536+0000 7f1c4f000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c481034b0 0x7f1c480753f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c4f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c4806e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c4f000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c48108730 0x7f1c4806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:15.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c4f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c481121c0 con 0x7f1c48107de0 2024-05-27T07:19:15.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c4f000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1c48112040 con 0x7f1c48108730 2024-05-27T07:19:15.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c4f000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c48112340 con 0x7f1c481034b0 2024-05-27T07:19:15.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c4806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:15.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c47400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c4806e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43096/0 (socket says 172.21.15.12:43096) 2024-05-27T07:19:15.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c47400700 1 -- 172.21.15.12:0/2891590314 learned_addr learned my addr 172.21.15.12:0/2891590314 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:15.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c47e00700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c481034b0 0x7f1c480753f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.537+0000 7f1c4d000700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c48108730 0x7f1c4806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.538+0000 7f1c47400700 1 -- 172.21.15.12:0/2891590314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c481034b0 msgr2=0x7f1c480753f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.538+0000 7f1c47400700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c481034b0 0x7f1c480753f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.538+0000 7f1c47400700 1 -- 172.21.15.12:0/2891590314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c48108730 msgr2=0x7f1c4806ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:15.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.538+0000 7f1c47400700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c48108730 0x7f1c4806ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.538+0000 7f1c47400700 1 -- 172.21.15.12:0/2891590314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c48072f20 con 0x7f1c48107de0 2024-05-27T07:19:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.538+0000 7f1c47400700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c4806e520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1c340055f0 tx=0x7f1c34005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.539+0000 7f1c44c00700 1 -- 172.21.15.12:0/2891590314 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c34026070 con 0x7f1c48107de0 2024-05-27T07:19:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.539+0000 7f1c44c00700 1 -- 172.21.15.12:0/2891590314 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c3400f460 con 0x7f1c48107de0 2024-05-27T07:19:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.539+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c480731b0 con 0x7f1c48107de0 2024-05-27T07:19:15.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.539+0000 7f1c44c00700 1 -- 172.21.15.12:0/2891590314 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c34009e60 con 0x7f1c48107de0 2024-05-27T07:19:15.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.539+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c48073690 con 0x7f1c48107de0 2024-05-27T07:19:15.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.541+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c48104960 con 0x7f1c48107de0 2024-05-27T07:19:15.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.545+0000 7f1c44c00700 1 -- 172.21.15.12:0/2891590314 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1c34029070 con 0x7f1c48107de0 2024-05-27T07:19:15.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.547+0000 7f1c44c00700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1c2c0b2910 0x7f1c2c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:15.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.547+0000 7f1c44c00700 1 -- 172.21.15.12:0/2891590314 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f1c340e2de0 con 0x7f1c48107de0 2024-05-27T07:19:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.547+0000 7f1c47e00700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1c2c0b2910 0x7f1c2c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:15.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.549+0000 7f1c47e00700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1c2c0b2910 0x7f1c2c0b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c38005950 tx=0x7f1c3800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.553+0000 7f1c44c00700 1 -- 172.21.15.12:0/2891590314 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1c34013070 con 0x7f1c48107de0 2024-05-27T07:19:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.797+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1c480512b0 con 0x7f1c2c0b2910 2024-05-27T07:19:15.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.810+0000 7f1c44c00700 1 -- 172.21.15.12:0/2891590314 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340415 (secure 0 0 0) 0x7f1c480512b0 con 0x7f1c2c0b2910 2024-05-27T07:19:15.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.816+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1c2c0b2910 msgr2=0x7f1c2c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.816+0000 7f1c4f000700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1c2c0b2910 0x7f1c2c0b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c38005950 tx=0x7f1c3800a300 comp rx=0 tx=0).stop 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.816+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 msgr2=0x7f1c4806e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.816+0000 7f1c4f000700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c4806e520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1c340055f0 tx=0x7f1c34005620 comp rx=0 tx=0).stop 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 shutdown_connections 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1c2c0b2910 0x7f1c2c0b4e20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c48108730 0x7f1c4806ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c481034b0 0x7f1c480753f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 --2- 172.21.15.12:0/2891590314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c48107de0 0x7f1c4806e520 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 >> 172.21.15.12:0/2891590314 conn(0x7f1c480fedb0 msgr2=0x7f1c48101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 shutdown_connections 2024-05-27T07:19:15.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:15.817+0000 7f1c4f000700 1 -- 172.21.15.12:0/2891590314 wait complete. 2024-05-27T07:19:15.810 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-05-27T07:19:16.787 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-05-27T07:19:15.780993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657066,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":258,"num_write":295,"num_write_kb":2270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375744,"kb_used_data":14084,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566080,"statfs":{"total":1151940427776,"available":1151555665920,"internally_reserved":0,"allocated":14422016,"data_stored":8129480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31727,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":73,"num_write":19,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007816"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.073972+0000","last_change":"2024-05-27T07:17:06.763882+0000","last_active":"2024-05-27T07:18:47.073972+0000","last_peered":"2024-05-27T07:18:47.073972+0000","last_clean":"2024-05-27T07:18:47.073972+0000","last_became_active":"2024-05-27T07:17:06.759625+0000","last_became_peered":"2024-05-27T07:17:06.759625+0000","last_unstale":"2024-05-27T07:18:47.073972+0000","last_undegraded":"2024-05-27T07:18:47.073972+0000","last_fullsized":"2024-05-27T07:18:47.073972+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:24.105867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127984+0000","last_change":"2024-05-27T07:17:06.755737+0000","last_active":"2024-05-27T07:18:47.127984+0000","last_peered":"2024-05-27T07:18:47.127984+0000","last_clean":"2024-05-27T07:18:47.127984+0000","last_became_active":"2024-05-27T07:17:06.754588+0000","last_became_peered":"2024-05-27T07:17:06.754588+0000","last_unstale":"2024-05-27T07:18:47.127984+0000","last_undegraded":"2024-05-27T07:18:47.127984+0000","last_fullsized":"2024-05-27T07:18:47.127984+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:02.417140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128521+0000","last_change":"2024-05-27T07:17:06.772057+0000","last_active":"2024-05-27T07:18:47.128521+0000","last_peered":"2024-05-27T07:18:47.128521+0000","last_clean":"2024-05-27T07:18:47.128521+0000","last_became_active":"2024-05-27T07:17:06.771636+0000","last_became_peered":"2024-05-27T07:17:06.771636+0000","last_unstale":"2024-05-27T07:18:47.128521+0000","last_undegraded":"2024-05-27T07:18:47.128521+0000","last_fullsized":"2024-05-27T07:18:47.128521+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:45.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082870+0000","last_change":"2024-05-27T07:17:06.763599+0000","last_active":"2024-05-27T07:18:47.082870+0000","last_peered":"2024-05-27T07:18:47.082870+0000","last_clean":"2024-05-27T07:18:47.082870+0000","last_became_active":"2024-05-27T07:17:06.763125+0000","last_became_peered":"2024-05-27T07:17:06.763125+0000","last_unstale":"2024-05-27T07:18:47.082870+0000","last_undegraded":"2024-05-27T07:18:47.082870+0000","last_fullsized":"2024-05-27T07:18:47.082870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:43.359673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125913+0000","last_change":"2024-05-27T07:17:06.755896+0000","last_active":"2024-05-27T07:18:47.125913+0000","last_peered":"2024-05-27T07:18:47.125913+0000","last_clean":"2024-05-27T07:18:47.125913+0000","last_became_active":"2024-05-27T07:17:06.755495+0000","last_became_peered":"2024-05-27T07:17:06.755495+0000","last_unstale":"2024-05-27T07:18:47.125913+0000","last_undegraded":"2024-05-27T07:18:47.125913+0000","last_fullsized":"2024-05-27T07:18:47.125913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:41:37.632012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064604+0000","last_change":"2024-05-27T07:17:46.699881+0000","last_active":"2024-05-27T07:18:47.064604+0000","last_peered":"2024-05-27T07:18:47.064604+0000","last_clean":"2024-05-27T07:18:47.064604+0000","last_became_active":"2024-05-27T07:17:46.699080+0000","last_became_peered":"2024-05-27T07:17:46.699080+0000","last_unstale":"2024-05-27T07:18:47.064604+0000","last_undegraded":"2024-05-27T07:18:47.064604+0000","last_fullsized":"2024-05-27T07:18:47.064604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:43.580762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065531+0000","last_change":"2024-05-27T07:17:06.770468+0000","last_active":"2024-05-27T07:18:47.065531+0000","last_peered":"2024-05-27T07:18:47.065531+0000","last_clean":"2024-05-27T07:18:47.065531+0000","last_became_active":"2024-05-27T07:17:06.768345+0000","last_became_peered":"2024-05-27T07:17:06.768345+0000","last_unstale":"2024-05-27T07:18:47.065531+0000","last_undegraded":"2024-05-27T07:18:47.065531+0000","last_fullsized":"2024-05-27T07:18:47.065531+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:12:27.646654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125170+0000","last_change":"2024-05-27T07:17:06.758769+0000","last_active":"2024-05-27T07:18:47.125170+0000","last_peered":"2024-05-27T07:18:47.125170+0000","last_clean":"2024-05-27T07:18:47.125170+0000","last_became_active":"2024-05-27T07:17:06.758393+0000","last_became_peered":"2024-05-27T07:17:06.758393+0000","last_unstale":"2024-05-27T07:18:47.125170+0000","last_undegraded":"2024-05-27T07:18:47.125170+0000","last_fullsized":"2024-05-27T07:18:47.125170+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:56.450569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.081950+0000","last_change":"2024-05-27T07:17:06.770681+0000","last_active":"2024-05-27T07:18:47.081950+0000","last_peered":"2024-05-27T07:18:47.081950+0000","last_clean":"2024-05-27T07:18:47.081950+0000","last_became_active":"2024-05-27T07:17:06.766520+0000","last_became_peered":"2024-05-27T07:17:06.766520+0000","last_unstale":"2024-05-27T07:18:47.081950+0000","last_undegraded":"2024-05-27T07:18:47.081950+0000","last_fullsized":"2024-05-27T07:18:47.081950+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:56.931743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067266+0000","last_change":"2024-05-27T07:17:06.761462+0000","last_active":"2024-05-27T07:18:47.067266+0000","last_peered":"2024-05-27T07:18:47.067266+0000","last_clean":"2024-05-27T07:18:47.067266+0000","last_became_active":"2024-05-27T07:17:06.759344+0000","last_became_peered":"2024-05-27T07:17:06.759344+0000","last_unstale":"2024-05-27T07:18:47.067266+0000","last_undegraded":"2024-05-27T07:18:47.067266+0000","last_fullsized":"2024-05-27T07:18:47.067266+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:12:22.688477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066831+0000","last_change":"2024-05-27T07:17:06.761947+0000","last_active":"2024-05-27T07:18:47.066831+0000","last_peered":"2024-05-27T07:18:47.066831+0000","last_clean":"2024-05-27T07:18:47.066831+0000","last_became_active":"2024-05-27T07:17:06.761464+0000","last_became_peered":"2024-05-27T07:17:06.761464+0000","last_unstale":"2024-05-27T07:18:47.066831+0000","last_undegraded":"2024-05-27T07:18:47.066831+0000","last_fullsized":"2024-05-27T07:18:47.066831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:08:19.939311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083615+0000","last_change":"2024-05-27T07:17:06.772039+0000","last_active":"2024-05-27T07:18:47.083615+0000","last_peered":"2024-05-27T07:18:47.083615+0000","last_clean":"2024-05-27T07:18:47.083615+0000","last_became_active":"2024-05-27T07:17:06.771303+0000","last_became_peered":"2024-05-27T07:17:06.771303+0000","last_unstale":"2024-05-27T07:18:47.083615+0000","last_undegraded":"2024-05-27T07:18:47.083615+0000","last_fullsized":"2024-05-27T07:18:47.083615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:30.254848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129891+0000","last_change":"2024-05-27T07:17:06.774228+0000","last_active":"2024-05-27T07:18:47.129891+0000","last_peered":"2024-05-27T07:18:47.129891+0000","last_clean":"2024-05-27T07:18:47.129891+0000","last_became_active":"2024-05-27T07:17:06.771436+0000","last_became_peered":"2024-05-27T07:17:06.771436+0000","last_unstale":"2024-05-27T07:18:47.129891+0000","last_undegraded":"2024-05-27T07:18:47.129891+0000","last_fullsized":"2024-05-27T07:18:47.129891+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:25.749143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127433+0000","last_change":"2024-05-27T07:17:06.755558+0000","last_active":"2024-05-27T07:18:47.127433+0000","last_peered":"2024-05-27T07:18:47.127433+0000","last_clean":"2024-05-27T07:18:47.127433+0000","last_became_active":"2024-05-27T07:17:06.754858+0000","last_became_peered":"2024-05-27T07:17:06.754858+0000","last_unstale":"2024-05-27T07:18:47.127433+0000","last_undegraded":"2024-05-27T07:18:47.127433+0000","last_fullsized":"2024-05-27T07:18:47.127433+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:09.853982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126976+0000","last_change":"2024-05-27T07:17:06.770731+0000","last_active":"2024-05-27T07:18:47.126976+0000","last_peered":"2024-05-27T07:18:47.126976+0000","last_clean":"2024-05-27T07:18:47.126976+0000","last_became_active":"2024-05-27T07:17:06.766923+0000","last_became_peered":"2024-05-27T07:17:06.766923+0000","last_unstale":"2024-05-27T07:18:47.126976+0000","last_undegraded":"2024-05-27T07:18:47.126976+0000","last_fullsized":"2024-05-27T07:18:47.126976+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.110121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080436+0000","last_change":"2024-05-27T07:17:06.758928+0000","last_active":"2024-05-27T07:18:47.080436+0000","last_peered":"2024-05-27T07:18:47.080436+0000","last_clean":"2024-05-27T07:18:47.080436+0000","last_became_active":"2024-05-27T07:17:06.758328+0000","last_became_peered":"2024-05-27T07:17:06.758328+0000","last_unstale":"2024-05-27T07:18:47.080436+0000","last_undegraded":"2024-05-27T07:18:47.080436+0000","last_fullsized":"2024-05-27T07:18:47.080436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:36.284395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.073095+0000","last_change":"2024-05-27T07:17:06.760006+0000","last_active":"2024-05-27T07:18:47.073095+0000","last_peered":"2024-05-27T07:18:47.073095+0000","last_clean":"2024-05-27T07:18:47.073095+0000","last_became_active":"2024-05-27T07:17:06.758177+0000","last_became_peered":"2024-05-27T07:17:06.758177+0000","last_unstale":"2024-05-27T07:18:47.073095+0000","last_undegraded":"2024-05-27T07:18:47.073095+0000","last_fullsized":"2024-05-27T07:18:47.073095+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.516289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060373+0000","last_change":"2024-05-27T07:17:06.768484+0000","last_active":"2024-05-27T07:18:47.060373+0000","last_peered":"2024-05-27T07:18:47.060373+0000","last_clean":"2024-05-27T07:18:47.060373+0000","last_became_active":"2024-05-27T07:17:06.768105+0000","last_became_peered":"2024-05-27T07:17:06.768105+0000","last_unstale":"2024-05-27T07:18:47.060373+0000","last_undegraded":"2024-05-27T07:18:47.060373+0000","last_fullsized":"2024-05-27T07:18:47.060373+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:54:48.751678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065152+0000","last_change":"2024-05-27T07:17:06.762782+0000","last_active":"2024-05-27T07:18:47.065152+0000","last_peered":"2024-05-27T07:18:47.065152+0000","last_clean":"2024-05-27T07:18:47.065152+0000","last_became_active":"2024-05-27T07:17:06.760129+0000","last_became_peered":"2024-05-27T07:17:06.760129+0000","last_unstale":"2024-05-27T07:18:47.065152+0000","last_undegraded":"2024-05-27T07:18:47.065152+0000","last_fullsized":"2024-05-27T07:18:47.065152+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:47:19.360662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063483+0000","last_change":"2024-05-27T07:17:21.899772+0000","last_active":"2024-05-27T07:18:47.063483+0000","last_peered":"2024-05-27T07:18:47.063483+0000","last_clean":"2024-05-27T07:18:47.063483+0000","last_became_active":"2024-05-27T07:17:21.899038+0000","last_became_peered":"2024-05-27T07:17:21.899038+0000","last_unstale":"2024-05-27T07:18:47.063483+0000","last_undegraded":"2024-05-27T07:18:47.063483+0000","last_fullsized":"2024-05-27T07:18:47.063483+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:38.784433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059878+0000","last_change":"2024-05-27T07:17:46.708798+0000","last_active":"2024-05-27T07:18:47.059878+0000","last_peered":"2024-05-27T07:18:47.059878+0000","last_clean":"2024-05-27T07:18:47.059878+0000","last_became_active":"2024-05-27T07:17:46.696433+0000","last_became_peered":"2024-05-27T07:17:46.696433+0000","last_unstale":"2024-05-27T07:18:47.059878+0000","last_undegraded":"2024-05-27T07:18:47.059878+0000","last_fullsized":"2024-05-27T07:18:47.059878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:40.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126199+0000","last_change":"2024-05-27T07:17:06.760865+0000","last_active":"2024-05-27T07:18:47.126199+0000","last_peered":"2024-05-27T07:18:47.126199+0000","last_clean":"2024-05-27T07:18:47.126199+0000","last_became_active":"2024-05-27T07:17:06.759491+0000","last_became_peered":"2024-05-27T07:17:06.759491+0000","last_unstale":"2024-05-27T07:18:47.126199+0000","last_undegraded":"2024-05-27T07:18:47.126199+0000","last_fullsized":"2024-05-27T07:18:47.126199+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:31.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082407+0000","last_change":"2024-05-27T07:17:21.898583+0000","last_active":"2024-05-27T07:18:47.082407+0000","last_peered":"2024-05-27T07:18:47.082407+0000","last_clean":"2024-05-27T07:18:47.082407+0000","last_became_active":"2024-05-27T07:17:21.897937+0000","last_became_peered":"2024-05-27T07:17:21.897937+0000","last_unstale":"2024-05-27T07:18:47.082407+0000","last_undegraded":"2024-05-27T07:18:47.082407+0000","last_fullsized":"2024-05-27T07:18:47.082407+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:45:00.412914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126886+0000","last_change":"2024-05-27T07:17:02.432465+0000","last_active":"2024-05-27T07:18:47.126886+0000","last_peered":"2024-05-27T07:18:47.126886+0000","last_clean":"2024-05-27T07:18:47.126886+0000","last_became_active":"2024-05-27T07:17:02.431638+0000","last_became_peered":"2024-05-27T07:17:02.431638+0000","last_unstale":"2024-05-27T07:18:47.126886+0000","last_undegraded":"2024-05-27T07:18:47.126886+0000","last_fullsized":"2024-05-27T07:18:47.126886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:03:13.167952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129569+0000","last_change":"2024-05-27T07:17:06.763133+0000","last_active":"2024-05-27T07:18:47.129569+0000","last_peered":"2024-05-27T07:18:47.129569+0000","last_clean":"2024-05-27T07:18:47.129569+0000","last_became_active":"2024-05-27T07:17:06.762770+0000","last_became_peered":"2024-05-27T07:17:06.762770+0000","last_unstale":"2024-05-27T07:18:47.129569+0000","last_undegraded":"2024-05-27T07:18:47.129569+0000","last_fullsized":"2024-05-27T07:18:47.129569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:33.223722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083366+0000","last_change":"2024-05-27T07:17:21.900369+0000","last_active":"2024-05-27T07:18:47.083366+0000","last_peered":"2024-05-27T07:18:47.083366+0000","last_clean":"2024-05-27T07:18:47.083366+0000","last_became_active":"2024-05-27T07:17:21.899758+0000","last_became_peered":"2024-05-27T07:17:21.899758+0000","last_unstale":"2024-05-27T07:18:47.083366+0000","last_undegraded":"2024-05-27T07:18:47.083366+0000","last_fullsized":"2024-05-27T07:18:47.083366+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:42.116302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127879+0000","last_change":"2024-05-27T07:17:02.347047+0000","last_active":"2024-05-27T07:18:47.127879+0000","last_peered":"2024-05-27T07:18:47.127879+0000","last_clean":"2024-05-27T07:18:47.127879+0000","last_became_active":"2024-05-27T07:17:02.346562+0000","last_became_peered":"2024-05-27T07:17:02.346562+0000","last_unstale":"2024-05-27T07:18:47.127879+0000","last_undegraded":"2024-05-27T07:18:47.127879+0000","last_fullsized":"2024-05-27T07:18:47.127879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:25:50.133089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080814+0000","last_change":"2024-05-27T07:17:06.767930+0000","last_active":"2024-05-27T07:18:47.080814+0000","last_peered":"2024-05-27T07:18:47.080814+0000","last_clean":"2024-05-27T07:18:47.080814+0000","last_became_active":"2024-05-27T07:17:06.767439+0000","last_became_peered":"2024-05-27T07:17:06.767439+0000","last_unstale":"2024-05-27T07:18:47.080814+0000","last_undegraded":"2024-05-27T07:18:47.080814+0000","last_fullsized":"2024-05-27T07:18:47.080814+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:18:47.464602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082783+0000","last_change":"2024-05-27T07:17:21.889199+0000","last_active":"2024-05-27T07:18:47.082783+0000","last_peered":"2024-05-27T07:18:47.082783+0000","last_clean":"2024-05-27T07:18:47.082783+0000","last_became_active":"2024-05-27T07:17:21.888667+0000","last_became_peered":"2024-05-27T07:17:21.888667+0000","last_unstale":"2024-05-27T07:18:47.082783+0000","last_undegraded":"2024-05-27T07:18:47.082783+0000","last_fullsized":"2024-05-27T07:18:47.082783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.029820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126884+0000","last_change":"2024-05-27T07:17:02.339872+0000","last_active":"2024-05-27T07:18:47.126884+0000","last_peered":"2024-05-27T07:18:47.126884+0000","last_clean":"2024-05-27T07:18:47.126884+0000","last_became_active":"2024-05-27T07:17:02.338975+0000","last_became_peered":"2024-05-27T07:17:02.338975+0000","last_unstale":"2024-05-27T07:18:47.126884+0000","last_undegraded":"2024-05-27T07:18:47.126884+0000","last_fullsized":"2024-05-27T07:18:47.126884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:14.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070756+0000","last_change":"2024-05-27T07:17:06.758234+0000","last_active":"2024-05-27T07:18:47.070756+0000","last_peered":"2024-05-27T07:18:47.070756+0000","last_clean":"2024-05-27T07:18:47.070756+0000","last_became_active":"2024-05-27T07:17:06.757827+0000","last_became_peered":"2024-05-27T07:17:06.757827+0000","last_unstale":"2024-05-27T07:18:47.070756+0000","last_undegraded":"2024-05-27T07:18:47.070756+0000","last_fullsized":"2024-05-27T07:18:47.070756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:21.688435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064658+0000","last_change":"2024-05-27T07:17:21.882540+0000","last_active":"2024-05-27T07:18:47.064658+0000","last_peered":"2024-05-27T07:18:47.064658+0000","last_clean":"2024-05-27T07:18:47.064658+0000","last_became_active":"2024-05-27T07:17:21.882087+0000","last_became_peered":"2024-05-27T07:17:21.882087+0000","last_unstale":"2024-05-27T07:18:47.064658+0000","last_undegraded":"2024-05-27T07:18:47.064658+0000","last_fullsized":"2024-05-27T07:18:47.064658+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:53:04.363521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:12.098627+0000","last_change":"2024-05-27T07:17:46.704997+0000","last_active":"2024-05-27T07:19:12.098627+0000","last_peered":"2024-05-27T07:19:12.098627+0000","last_clean":"2024-05-27T07:19:12.098627+0000","last_became_active":"2024-05-27T07:17:46.704355+0000","last_became_peered":"2024-05-27T07:17:46.704355+0000","last_unstale":"2024-05-27T07:19:12.098627+0000","last_undegraded":"2024-05-27T07:19:12.098627+0000","last_fullsized":"2024-05-27T07:19:12.098627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:43:02.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126403+0000","last_change":"2024-05-27T07:17:06.765809+0000","last_active":"2024-05-27T07:18:47.126403+0000","last_peered":"2024-05-27T07:18:47.126403+0000","last_clean":"2024-05-27T07:18:47.126403+0000","last_became_active":"2024-05-27T07:17:06.763845+0000","last_became_peered":"2024-05-27T07:17:06.763845+0000","last_unstale":"2024-05-27T07:18:47.126403+0000","last_undegraded":"2024-05-27T07:18:47.126403+0000","last_fullsized":"2024-05-27T07:18:47.126403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:25.399181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064340+0000","last_change":"2024-05-27T07:17:21.886316+0000","last_active":"2024-05-27T07:18:47.064340+0000","last_peered":"2024-05-27T07:18:47.064340+0000","last_clean":"2024-05-27T07:18:47.064340+0000","last_became_active":"2024-05-27T07:17:21.885129+0000","last_became_peered":"2024-05-27T07:17:21.885129+0000","last_unstale":"2024-05-27T07:18:47.064340+0000","last_undegraded":"2024-05-27T07:18:47.064340+0000","last_fullsized":"2024-05-27T07:18:47.064340+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:10.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064271+0000","last_change":"2024-05-27T07:17:02.338765+0000","last_active":"2024-05-27T07:18:47.064271+0000","last_peered":"2024-05-27T07:18:47.064271+0000","last_clean":"2024-05-27T07:18:47.064271+0000","last_became_active":"2024-05-27T07:17:02.337103+0000","last_became_peered":"2024-05-27T07:17:02.337103+0000","last_unstale":"2024-05-27T07:18:47.064271+0000","last_undegraded":"2024-05-27T07:18:47.064271+0000","last_fullsized":"2024-05-27T07:18:47.064271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:15:07.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069437+0000","last_change":"2024-05-27T07:17:06.766839+0000","last_active":"2024-05-27T07:18:47.069437+0000","last_peered":"2024-05-27T07:18:47.069437+0000","last_clean":"2024-05-27T07:18:47.069437+0000","last_became_active":"2024-05-27T07:17:06.766373+0000","last_became_peered":"2024-05-27T07:17:06.766373+0000","last_unstale":"2024-05-27T07:18:47.069437+0000","last_undegraded":"2024-05-27T07:18:47.069437+0000","last_fullsized":"2024-05-27T07:18:47.069437+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:09:55.064045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133692+0000","last_change":"2024-05-27T07:18:47.133692+0000","last_active":"2024-05-27T07:18:47.133692+0000","last_peered":"2024-05-27T07:18:47.133692+0000","last_clean":"2024-05-27T07:18:47.133692+0000","last_became_active":"2024-05-27T07:18:47.132122+0000","last_became_peered":"2024-05-27T07:18:47.132122+0000","last_unstale":"2024-05-27T07:18:47.133692+0000","last_undegraded":"2024-05-27T07:18:47.133692+0000","last_fullsized":"2024-05-27T07:18:47.133692+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:57.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130197+0000","last_change":"2024-05-27T07:17:02.329809+0000","last_active":"2024-05-27T07:18:47.130197+0000","last_peered":"2024-05-27T07:18:47.130197+0000","last_clean":"2024-05-27T07:18:47.130197+0000","last_became_active":"2024-05-27T07:17:02.328758+0000","last_became_peered":"2024-05-27T07:17:02.328758+0000","last_unstale":"2024-05-27T07:18:47.130197+0000","last_undegraded":"2024-05-27T07:18:47.130197+0000","last_fullsized":"2024-05-27T07:18:47.130197+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:52.479444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125108+0000","last_change":"2024-05-27T07:17:06.749836+0000","last_active":"2024-05-27T07:18:47.125108+0000","last_peered":"2024-05-27T07:18:47.125108+0000","last_clean":"2024-05-27T07:18:47.125108+0000","last_became_active":"2024-05-27T07:17:06.749512+0000","last_became_peered":"2024-05-27T07:17:06.749512+0000","last_unstale":"2024-05-27T07:18:47.125108+0000","last_undegraded":"2024-05-27T07:18:47.125108+0000","last_fullsized":"2024-05-27T07:18:47.125108+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:41:51.000415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127796+0000","last_change":"2024-05-27T07:17:21.888376+0000","last_active":"2024-05-27T07:18:47.127796+0000","last_peered":"2024-05-27T07:18:47.127796+0000","last_clean":"2024-05-27T07:18:47.127796+0000","last_became_active":"2024-05-27T07:17:21.886183+0000","last_became_peered":"2024-05-27T07:17:21.886183+0000","last_unstale":"2024-05-27T07:18:47.127796+0000","last_undegraded":"2024-05-27T07:18:47.127796+0000","last_fullsized":"2024-05-27T07:18:47.127796+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:34.541338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069983+0000","last_change":"2024-05-27T07:17:02.334113+0000","last_active":"2024-05-27T07:18:47.069983+0000","last_peered":"2024-05-27T07:18:47.069983+0000","last_clean":"2024-05-27T07:18:47.069983+0000","last_became_active":"2024-05-27T07:17:02.333201+0000","last_became_peered":"2024-05-27T07:17:02.333201+0000","last_unstale":"2024-05-27T07:18:47.069983+0000","last_undegraded":"2024-05-27T07:18:47.069983+0000","last_fullsized":"2024-05-27T07:18:47.069983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:13.484654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.058837+0000","last_change":"2024-05-27T07:17:06.751676+0000","last_active":"2024-05-27T07:18:47.058837+0000","last_peered":"2024-05-27T07:18:47.058837+0000","last_clean":"2024-05-27T07:18:47.058837+0000","last_became_active":"2024-05-27T07:17:06.751221+0000","last_became_peered":"2024-05-27T07:17:06.751221+0000","last_unstale":"2024-05-27T07:18:47.058837+0000","last_undegraded":"2024-05-27T07:18:47.058837+0000","last_fullsized":"2024-05-27T07:18:47.058837+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:40.012561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125122+0000","last_change":"2024-05-27T07:17:21.897640+0000","last_active":"2024-05-27T07:18:47.125122+0000","last_peered":"2024-05-27T07:18:47.125122+0000","last_clean":"2024-05-27T07:18:47.125122+0000","last_became_active":"2024-05-27T07:17:21.895355+0000","last_became_peered":"2024-05-27T07:17:21.895355+0000","last_unstale":"2024-05-27T07:18:47.125122+0000","last_undegraded":"2024-05-27T07:18:47.125122+0000","last_fullsized":"2024-05-27T07:18:47.125122+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:51:12.030069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:12.097297+0000","last_change":"2024-05-27T07:17:02.339464+0000","last_active":"2024-05-27T07:19:12.097297+0000","last_peered":"2024-05-27T07:19:12.097297+0000","last_clean":"2024-05-27T07:19:12.097297+0000","last_became_active":"2024-05-27T07:17:02.338577+0000","last_became_peered":"2024-05-27T07:17:02.338577+0000","last_unstale":"2024-05-27T07:19:12.097297+0000","last_undegraded":"2024-05-27T07:19:12.097297+0000","last_fullsized":"2024-05-27T07:19:12.097297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:29:59.210157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.081292+0000","last_change":"2024-05-27T07:17:06.766883+0000","last_active":"2024-05-27T07:18:47.081292+0000","last_peered":"2024-05-27T07:18:47.081292+0000","last_clean":"2024-05-27T07:18:47.081292+0000","last_became_active":"2024-05-27T07:17:06.766166+0000","last_became_peered":"2024-05-27T07:17:06.766166+0000","last_unstale":"2024-05-27T07:18:47.081292+0000","last_undegraded":"2024-05-27T07:18:47.081292+0000","last_fullsized":"2024-05-27T07:18:47.081292+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:28.936499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.058804+0000","last_change":"2024-05-27T07:17:21.893930+0000","last_active":"2024-05-27T07:18:47.058804+0000","last_peered":"2024-05-27T07:18:47.058804+0000","last_clean":"2024-05-27T07:18:47.058804+0000","last_became_active":"2024-05-27T07:17:21.893029+0000","last_became_peered":"2024-05-27T07:17:21.893029+0000","last_unstale":"2024-05-27T07:18:47.058804+0000","last_undegraded":"2024-05-27T07:18:47.058804+0000","last_fullsized":"2024-05-27T07:18:47.058804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:32.453078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068295+0000","last_change":"2024-05-27T07:17:02.346531+0000","last_active":"2024-05-27T07:18:47.068295+0000","last_peered":"2024-05-27T07:18:47.068295+0000","last_clean":"2024-05-27T07:18:47.068295+0000","last_became_active":"2024-05-27T07:17:02.345844+0000","last_became_peered":"2024-05-27T07:17:02.345844+0000","last_unstale":"2024-05-27T07:18:47.068295+0000","last_undegraded":"2024-05-27T07:18:47.068295+0000","last_fullsized":"2024-05-27T07:18:47.068295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:28.544616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128038+0000","last_change":"2024-05-27T07:17:06.769845+0000","last_active":"2024-05-27T07:18:47.128038+0000","last_peered":"2024-05-27T07:18:47.128038+0000","last_clean":"2024-05-27T07:18:47.128038+0000","last_became_active":"2024-05-27T07:17:06.768467+0000","last_became_peered":"2024-05-27T07:17:06.768467+0000","last_unstale":"2024-05-27T07:18:47.128038+0000","last_undegraded":"2024-05-27T07:18:47.128038+0000","last_fullsized":"2024-05-27T07:18:47.128038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:30.311824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084012+0000","last_change":"2024-05-27T07:17:21.899885+0000","last_active":"2024-05-27T07:18:47.084012+0000","last_peered":"2024-05-27T07:18:47.084012+0000","last_clean":"2024-05-27T07:18:47.084012+0000","last_became_active":"2024-05-27T07:17:21.899340+0000","last_became_peered":"2024-05-27T07:17:21.899340+0000","last_unstale":"2024-05-27T07:18:47.084012+0000","last_undegraded":"2024-05-27T07:18:47.084012+0000","last_fullsized":"2024-05-27T07:18:47.084012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:12.665409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070628+0000","last_change":"2024-05-27T07:17:02.435770+0000","last_active":"2024-05-27T07:18:47.070628+0000","last_peered":"2024-05-27T07:18:47.070628+0000","last_clean":"2024-05-27T07:18:47.070628+0000","last_became_active":"2024-05-27T07:17:02.434535+0000","last_became_peered":"2024-05-27T07:17:02.434535+0000","last_unstale":"2024-05-27T07:18:47.070628+0000","last_undegraded":"2024-05-27T07:18:47.070628+0000","last_fullsized":"2024-05-27T07:18:47.070628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:53.745583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.081482+0000","last_change":"2024-05-27T07:17:46.704100+0000","last_active":"2024-05-27T07:18:47.081482+0000","last_peered":"2024-05-27T07:18:47.081482+0000","last_clean":"2024-05-27T07:18:47.081482+0000","last_became_active":"2024-05-27T07:17:46.703475+0000","last_became_peered":"2024-05-27T07:17:46.703475+0000","last_unstale":"2024-05-27T07:18:47.081482+0000","last_undegraded":"2024-05-27T07:18:47.081482+0000","last_fullsized":"2024-05-27T07:18:47.081482+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:20:29.173923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133397+0000","last_change":"2024-05-27T07:18:47.131521+0000","last_active":"2024-05-27T07:18:47.133397+0000","last_peered":"2024-05-27T07:18:47.133397+0000","last_clean":"2024-05-27T07:18:47.133397+0000","last_became_active":"2024-05-27T07:18:47.129995+0000","last_became_peered":"2024-05-27T07:18:47.129995+0000","last_unstale":"2024-05-27T07:18:47.133397+0000","last_undegraded":"2024-05-27T07:18:47.133397+0000","last_fullsized":"2024-05-27T07:18:47.133397+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:45:27.268768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":172,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:13.109773+0000","last_change":"2024-05-27T07:17:46.719293+0000","last_active":"2024-05-27T07:19:13.109773+0000","last_peered":"2024-05-27T07:19:13.109773+0000","last_clean":"2024-05-27T07:19:13.109773+0000","last_became_active":"2024-05-27T07:17:46.703496+0000","last_became_peered":"2024-05-27T07:17:46.703496+0000","last_unstale":"2024-05-27T07:19:13.109773+0000","last_undegraded":"2024-05-27T07:19:13.109773+0000","last_fullsized":"2024-05-27T07:19:13.109773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:26.310524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62072,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":79,"num_read_kb":100,"num_write":34,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128691+0000","last_change":"2024-05-27T07:17:06.763141+0000","last_active":"2024-05-27T07:18:47.128691+0000","last_peered":"2024-05-27T07:18:47.128691+0000","last_clean":"2024-05-27T07:18:47.128691+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:18:47.128691+0000","last_undegraded":"2024-05-27T07:18:47.128691+0000","last_fullsized":"2024-05-27T07:18:47.128691+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:47.072552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071302+0000","last_change":"2024-05-27T07:17:21.895342+0000","last_active":"2024-05-27T07:18:47.071302+0000","last_peered":"2024-05-27T07:18:47.071302+0000","last_clean":"2024-05-27T07:18:47.071302+0000","last_became_active":"2024-05-27T07:17:21.894630+0000","last_became_peered":"2024-05-27T07:17:21.894630+0000","last_unstale":"2024-05-27T07:18:47.071302+0000","last_undegraded":"2024-05-27T07:18:47.071302+0000","last_fullsized":"2024-05-27T07:18:47.071302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:50.501683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062761+0000","last_change":"2024-05-27T07:17:02.341848+0000","last_active":"2024-05-27T07:18:47.062761+0000","last_peered":"2024-05-27T07:18:47.062761+0000","last_clean":"2024-05-27T07:18:47.062761+0000","last_became_active":"2024-05-27T07:17:02.339477+0000","last_became_peered":"2024-05-27T07:17:02.339477+0000","last_unstale":"2024-05-27T07:18:47.062761+0000","last_undegraded":"2024-05-27T07:18:47.062761+0000","last_fullsized":"2024-05-27T07:18:47.062761+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:23.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129062+0000","last_change":"2024-05-27T07:17:06.761340+0000","last_active":"2024-05-27T07:18:47.129062+0000","last_peered":"2024-05-27T07:18:47.129062+0000","last_clean":"2024-05-27T07:18:47.129062+0000","last_became_active":"2024-05-27T07:17:06.759103+0000","last_became_peered":"2024-05-27T07:17:06.759103+0000","last_unstale":"2024-05-27T07:18:47.129062+0000","last_undegraded":"2024-05-27T07:18:47.129062+0000","last_fullsized":"2024-05-27T07:18:47.129062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:42.263106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071318+0000","last_change":"2024-05-27T07:17:21.883252+0000","last_active":"2024-05-27T07:18:47.071318+0000","last_peered":"2024-05-27T07:18:47.071318+0000","last_clean":"2024-05-27T07:18:47.071318+0000","last_became_active":"2024-05-27T07:17:21.882468+0000","last_became_peered":"2024-05-27T07:17:21.882468+0000","last_unstale":"2024-05-27T07:18:47.071318+0000","last_undegraded":"2024-05-27T07:18:47.071318+0000","last_fullsized":"2024-05-27T07:18:47.071318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:59.250357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069605+0000","last_change":"2024-05-27T07:17:02.343343+0000","last_active":"2024-05-27T07:18:47.069605+0000","last_peered":"2024-05-27T07:18:47.069605+0000","last_clean":"2024-05-27T07:18:47.069605+0000","last_became_active":"2024-05-27T07:17:02.342741+0000","last_became_peered":"2024-05-27T07:17:02.342741+0000","last_unstale":"2024-05-27T07:18:47.069605+0000","last_undegraded":"2024-05-27T07:18:47.069605+0000","last_fullsized":"2024-05-27T07:18:47.069605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:46.006865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067181+0000","last_change":"2024-05-27T07:17:06.771473+0000","last_active":"2024-05-27T07:18:47.067181+0000","last_peered":"2024-05-27T07:18:47.067181+0000","last_clean":"2024-05-27T07:18:47.067181+0000","last_became_active":"2024-05-27T07:17:06.771082+0000","last_became_peered":"2024-05-27T07:17:06.771082+0000","last_unstale":"2024-05-27T07:18:47.067181+0000","last_undegraded":"2024-05-27T07:18:47.067181+0000","last_fullsized":"2024-05-27T07:18:47.067181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:01.371182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060572+0000","last_change":"2024-05-27T07:17:21.897691+0000","last_active":"2024-05-27T07:18:47.060572+0000","last_peered":"2024-05-27T07:18:47.060572+0000","last_clean":"2024-05-27T07:18:47.060572+0000","last_became_active":"2024-05-27T07:17:21.895221+0000","last_became_peered":"2024-05-27T07:17:21.895221+0000","last_unstale":"2024-05-27T07:18:47.060572+0000","last_undegraded":"2024-05-27T07:18:47.060572+0000","last_fullsized":"2024-05-27T07:18:47.060572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:14.626947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:12.098338+0000","last_change":"2024-05-27T07:17:02.343375+0000","last_active":"2024-05-27T07:19:12.098338+0000","last_peered":"2024-05-27T07:19:12.098338+0000","last_clean":"2024-05-27T07:19:12.098338+0000","last_became_active":"2024-05-27T07:17:02.341315+0000","last_became_peered":"2024-05-27T07:17:02.341315+0000","last_unstale":"2024-05-27T07:19:12.098338+0000","last_undegraded":"2024-05-27T07:19:12.098338+0000","last_fullsized":"2024-05-27T07:19:12.098338+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:41.927265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065851+0000","last_change":"2024-05-27T07:17:06.765862+0000","last_active":"2024-05-27T07:18:47.065851+0000","last_peered":"2024-05-27T07:18:47.065851+0000","last_clean":"2024-05-27T07:18:47.065851+0000","last_became_active":"2024-05-27T07:17:06.765439+0000","last_became_peered":"2024-05-27T07:17:06.765439+0000","last_unstale":"2024-05-27T07:18:47.065851+0000","last_undegraded":"2024-05-27T07:18:47.065851+0000","last_fullsized":"2024-05-27T07:18:47.065851+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:47.586833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129937+0000","last_change":"2024-05-27T07:17:21.891810+0000","last_active":"2024-05-27T07:18:47.129937+0000","last_peered":"2024-05-27T07:18:47.129937+0000","last_clean":"2024-05-27T07:18:47.129937+0000","last_became_active":"2024-05-27T07:17:21.890473+0000","last_became_peered":"2024-05-27T07:17:21.890473+0000","last_unstale":"2024-05-27T07:18:47.129937+0000","last_undegraded":"2024-05-27T07:18:47.129937+0000","last_fullsized":"2024-05-27T07:18:47.129937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:12.731189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065778+0000","last_change":"2024-05-27T07:17:46.715870+0000","last_active":"2024-05-27T07:18:47.065778+0000","last_peered":"2024-05-27T07:18:47.065778+0000","last_clean":"2024-05-27T07:18:47.065778+0000","last_became_active":"2024-05-27T07:17:46.701248+0000","last_became_peered":"2024-05-27T07:17:46.701248+0000","last_unstale":"2024-05-27T07:18:47.065778+0000","last_undegraded":"2024-05-27T07:18:47.065778+0000","last_fullsized":"2024-05-27T07:18:47.065778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:18.693752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068792+0000","last_change":"2024-05-27T07:17:06.759937+0000","last_active":"2024-05-27T07:18:47.068792+0000","last_peered":"2024-05-27T07:18:47.068792+0000","last_clean":"2024-05-27T07:18:47.068792+0000","last_became_active":"2024-05-27T07:17:06.759526+0000","last_became_peered":"2024-05-27T07:17:06.759526+0000","last_unstale":"2024-05-27T07:18:47.068792+0000","last_undegraded":"2024-05-27T07:18:47.068792+0000","last_fullsized":"2024-05-27T07:18:47.068792+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:22.934317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068712+0000","last_change":"2024-05-27T07:17:21.898331+0000","last_active":"2024-05-27T07:18:47.068712+0000","last_peered":"2024-05-27T07:18:47.068712+0000","last_clean":"2024-05-27T07:18:47.068712+0000","last_became_active":"2024-05-27T07:17:21.897965+0000","last_became_peered":"2024-05-27T07:17:21.897965+0000","last_unstale":"2024-05-27T07:18:47.068712+0000","last_undegraded":"2024-05-27T07:18:47.068712+0000","last_fullsized":"2024-05-27T07:18:47.068712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:20.584175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128208+0000","last_change":"2024-05-27T07:17:02.346029+0000","last_active":"2024-05-27T07:18:47.128208+0000","last_peered":"2024-05-27T07:18:47.128208+0000","last_clean":"2024-05-27T07:18:47.128208+0000","last_became_active":"2024-05-27T07:17:02.341292+0000","last_became_peered":"2024-05-27T07:17:02.341292+0000","last_unstale":"2024-05-27T07:18:47.128208+0000","last_undegraded":"2024-05-27T07:18:47.128208+0000","last_fullsized":"2024-05-27T07:18:47.128208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:31.705741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071690+0000","last_change":"2024-05-27T07:17:46.691785+0000","last_active":"2024-05-27T07:18:47.071690+0000","last_peered":"2024-05-27T07:18:47.071690+0000","last_clean":"2024-05-27T07:18:47.071690+0000","last_became_active":"2024-05-27T07:17:46.690445+0000","last_became_peered":"2024-05-27T07:17:46.690445+0000","last_unstale":"2024-05-27T07:18:47.071690+0000","last_undegraded":"2024-05-27T07:18:47.071690+0000","last_fullsized":"2024-05-27T07:18:47.071690+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:59:27.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126733+0000","last_change":"2024-05-27T07:17:21.892434+0000","last_active":"2024-05-27T07:18:47.126733+0000","last_peered":"2024-05-27T07:18:47.126733+0000","last_clean":"2024-05-27T07:18:47.126733+0000","last_became_active":"2024-05-27T07:17:21.891736+0000","last_became_peered":"2024-05-27T07:17:21.891736+0000","last_unstale":"2024-05-27T07:18:47.126733+0000","last_undegraded":"2024-05-27T07:18:47.126733+0000","last_fullsized":"2024-05-27T07:18:47.126733+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:48.458819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059903+0000","last_change":"2024-05-27T07:17:02.436323+0000","last_active":"2024-05-27T07:18:47.059903+0000","last_peered":"2024-05-27T07:18:47.059903+0000","last_clean":"2024-05-27T07:18:47.059903+0000","last_became_active":"2024-05-27T07:17:02.435000+0000","last_became_peered":"2024-05-27T07:17:02.435000+0000","last_unstale":"2024-05-27T07:18:47.059903+0000","last_undegraded":"2024-05-27T07:18:47.059903+0000","last_fullsized":"2024-05-27T07:18:47.059903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:08.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065831+0000","last_change":"2024-05-27T07:17:06.760418+0000","last_active":"2024-05-27T07:18:47.065831+0000","last_peered":"2024-05-27T07:18:47.065831+0000","last_clean":"2024-05-27T07:18:47.065831+0000","last_became_active":"2024-05-27T07:17:06.759875+0000","last_became_peered":"2024-05-27T07:17:06.759875+0000","last_unstale":"2024-05-27T07:18:47.065831+0000","last_undegraded":"2024-05-27T07:18:47.065831+0000","last_fullsized":"2024-05-27T07:18:47.065831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:12:54.324323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130370+0000","last_change":"2024-05-27T07:17:21.888048+0000","last_active":"2024-05-27T07:18:47.130370+0000","last_peered":"2024-05-27T07:18:47.130370+0000","last_clean":"2024-05-27T07:18:47.130370+0000","last_became_active":"2024-05-27T07:17:21.887579+0000","last_became_peered":"2024-05-27T07:17:21.887579+0000","last_unstale":"2024-05-27T07:18:47.130370+0000","last_undegraded":"2024-05-27T07:18:47.130370+0000","last_fullsized":"2024-05-27T07:18:47.130370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:34:44.569959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066518+0000","last_change":"2024-05-27T07:17:02.337655+0000","last_active":"2024-05-27T07:18:47.066518+0000","last_peered":"2024-05-27T07:18:47.066518+0000","last_clean":"2024-05-27T07:18:47.066518+0000","last_became_active":"2024-05-27T07:17:02.337252+0000","last_became_peered":"2024-05-27T07:17:02.337252+0000","last_unstale":"2024-05-27T07:18:47.066518+0000","last_undegraded":"2024-05-27T07:18:47.066518+0000","last_fullsized":"2024-05-27T07:18:47.066518+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:11:25.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063967+0000","last_change":"2024-05-27T07:17:06.762934+0000","last_active":"2024-05-27T07:18:47.063967+0000","last_peered":"2024-05-27T07:18:47.063967+0000","last_clean":"2024-05-27T07:18:47.063967+0000","last_became_active":"2024-05-27T07:17:06.762507+0000","last_became_peered":"2024-05-27T07:17:06.762507+0000","last_unstale":"2024-05-27T07:18:47.063967+0000","last_undegraded":"2024-05-27T07:18:47.063967+0000","last_fullsized":"2024-05-27T07:18:47.063967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:23:35.164892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129885+0000","last_change":"2024-05-27T07:17:21.886991+0000","last_active":"2024-05-27T07:18:47.129885+0000","last_peered":"2024-05-27T07:18:47.129885+0000","last_clean":"2024-05-27T07:18:47.129885+0000","last_became_active":"2024-05-27T07:17:21.885945+0000","last_became_peered":"2024-05-27T07:17:21.885945+0000","last_unstale":"2024-05-27T07:18:47.129885+0000","last_undegraded":"2024-05-27T07:18:47.129885+0000","last_fullsized":"2024-05-27T07:18:47.129885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:23:01.853601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070506+0000","last_change":"2024-05-27T07:17:46.864130+0000","last_active":"2024-05-27T07:18:47.070506+0000","last_peered":"2024-05-27T07:18:47.070506+0000","last_clean":"2024-05-27T07:18:47.070506+0000","last_became_active":"2024-05-27T07:17:46.862997+0000","last_became_peered":"2024-05-27T07:17:46.862997+0000","last_unstale":"2024-05-27T07:18:47.070506+0000","last_undegraded":"2024-05-27T07:18:47.070506+0000","last_fullsized":"2024-05-27T07:18:47.070506+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:10.594627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:11.094245+0000","last_change":"2024-05-27T07:17:02.435463+0000","last_active":"2024-05-27T07:19:11.094245+0000","last_peered":"2024-05-27T07:19:11.094245+0000","last_clean":"2024-05-27T07:19:11.094245+0000","last_became_active":"2024-05-27T07:17:02.432131+0000","last_became_peered":"2024-05-27T07:17:02.432131+0000","last_unstale":"2024-05-27T07:19:11.094245+0000","last_undegraded":"2024-05-27T07:19:11.094245+0000","last_fullsized":"2024-05-27T07:19:11.094245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:41.874577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.135438+0000","last_change":"2024-05-27T07:18:47.134043+0000","last_active":"2024-05-27T07:18:47.135438+0000","last_peered":"2024-05-27T07:18:47.135438+0000","last_clean":"2024-05-27T07:18:47.135438+0000","last_became_active":"2024-05-27T07:18:47.132050+0000","last_became_peered":"2024-05-27T07:18:47.132050+0000","last_unstale":"2024-05-27T07:18:47.135438+0000","last_undegraded":"2024-05-27T07:18:47.135438+0000","last_fullsized":"2024-05-27T07:18:47.135438+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:12.170883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125310+0000","last_change":"2024-05-27T07:17:06.751168+0000","last_active":"2024-05-27T07:18:47.125310+0000","last_peered":"2024-05-27T07:18:47.125310+0000","last_clean":"2024-05-27T07:18:47.125310+0000","last_became_active":"2024-05-27T07:17:06.750816+0000","last_became_peered":"2024-05-27T07:17:06.750816+0000","last_unstale":"2024-05-27T07:18:47.125310+0000","last_undegraded":"2024-05-27T07:18:47.125310+0000","last_fullsized":"2024-05-27T07:18:47.125310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:33:45.607426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083905+0000","last_change":"2024-05-27T07:17:02.337490+0000","last_active":"2024-05-27T07:18:47.083905+0000","last_peered":"2024-05-27T07:18:47.083905+0000","last_clean":"2024-05-27T07:18:47.083905+0000","last_became_active":"2024-05-27T07:17:02.335317+0000","last_became_peered":"2024-05-27T07:17:02.335317+0000","last_unstale":"2024-05-27T07:18:47.083905+0000","last_undegraded":"2024-05-27T07:18:47.083905+0000","last_fullsized":"2024-05-27T07:18:47.083905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:33:46.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065010+0000","last_change":"2024-05-27T07:15:10.825312+0000","last_active":"2024-05-27T07:18:47.065010+0000","last_peered":"2024-05-27T07:18:47.065010+0000","last_clean":"2024-05-27T07:18:47.065010+0000","last_became_active":"2024-05-27T07:15:10.809245+0000","last_became_peered":"2024-05-27T07:15:10.809245+0000","last_unstale":"2024-05-27T07:18:47.065010+0000","last_undegraded":"2024-05-27T07:18:47.065010+0000","last_fullsized":"2024-05-27T07:18:47.065010+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_clean_scrub_stamp":"2024-05-27T07:10:08.008565+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:32.769372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133948+0000","last_change":"2024-05-27T07:18:47.133948+0000","last_active":"2024-05-27T07:18:47.133948+0000","last_peered":"2024-05-27T07:18:47.133948+0000","last_clean":"2024-05-27T07:18:47.133948+0000","last_became_active":"2024-05-27T07:18:47.132566+0000","last_became_peered":"2024-05-27T07:18:47.132566+0000","last_unstale":"2024-05-27T07:18:47.133948+0000","last_undegraded":"2024-05-27T07:18:47.133948+0000","last_fullsized":"2024-05-27T07:18:47.133948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:04.463121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128885+0000","last_change":"2024-05-27T07:17:06.766187+0000","last_active":"2024-05-27T07:18:47.128885+0000","last_peered":"2024-05-27T07:18:47.128885+0000","last_clean":"2024-05-27T07:18:47.128885+0000","last_became_active":"2024-05-27T07:17:06.763365+0000","last_became_peered":"2024-05-27T07:17:06.763365+0000","last_unstale":"2024-05-27T07:18:47.128885+0000","last_undegraded":"2024-05-27T07:18:47.128885+0000","last_fullsized":"2024-05-27T07:18:47.128885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:13.668426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065681+0000","last_change":"2024-05-27T07:17:02.344452+0000","last_active":"2024-05-27T07:18:47.065681+0000","last_peered":"2024-05-27T07:18:47.065681+0000","last_clean":"2024-05-27T07:18:47.065681+0000","last_became_active":"2024-05-27T07:17:02.343885+0000","last_became_peered":"2024-05-27T07:17:02.343885+0000","last_unstale":"2024-05-27T07:18:47.065681+0000","last_undegraded":"2024-05-27T07:18:47.065681+0000","last_fullsized":"2024-05-27T07:18:47.065681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:26.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072458+0000","last_change":"2024-05-27T07:17:21.883024+0000","last_active":"2024-05-27T07:18:47.072458+0000","last_peered":"2024-05-27T07:18:47.072458+0000","last_clean":"2024-05-27T07:18:47.072458+0000","last_became_active":"2024-05-27T07:17:21.882429+0000","last_became_peered":"2024-05-27T07:17:21.882429+0000","last_unstale":"2024-05-27T07:18:47.072458+0000","last_undegraded":"2024-05-27T07:18:47.072458+0000","last_fullsized":"2024-05-27T07:18:47.072458+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:36:40.443537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128649+0000","last_change":"2024-05-27T07:17:02.432768+0000","last_active":"2024-05-27T07:18:47.128649+0000","last_peered":"2024-05-27T07:18:47.128649+0000","last_clean":"2024-05-27T07:18:47.128649+0000","last_became_active":"2024-05-27T07:17:02.431551+0000","last_became_peered":"2024-05-27T07:17:02.431551+0000","last_unstale":"2024-05-27T07:18:47.128649+0000","last_undegraded":"2024-05-27T07:18:47.128649+0000","last_fullsized":"2024-05-27T07:18:47.128649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:39.759512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130801+0000","last_change":"2024-05-27T07:17:06.770992+0000","last_active":"2024-05-27T07:18:47.130801+0000","last_peered":"2024-05-27T07:18:47.130801+0000","last_clean":"2024-05-27T07:18:47.130801+0000","last_became_active":"2024-05-27T07:17:06.770560+0000","last_became_peered":"2024-05-27T07:17:06.770560+0000","last_unstale":"2024-05-27T07:18:47.130801+0000","last_undegraded":"2024-05-27T07:18:47.130801+0000","last_fullsized":"2024-05-27T07:18:47.130801+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:07:34.443551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.124920+0000","last_change":"2024-05-27T07:17:21.895362+0000","last_active":"2024-05-27T07:18:47.124920+0000","last_peered":"2024-05-27T07:18:47.124920+0000","last_clean":"2024-05-27T07:18:47.124920+0000","last_became_active":"2024-05-27T07:17:21.894563+0000","last_became_peered":"2024-05-27T07:17:21.894563+0000","last_unstale":"2024-05-27T07:18:47.124920+0000","last_undegraded":"2024-05-27T07:18:47.124920+0000","last_fullsized":"2024-05-27T07:18:47.124920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:44.394301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:09.826352+0000","last_change":"2024-05-27T07:17:02.338203+0000","last_active":"2024-05-27T07:19:09.826352+0000","last_peered":"2024-05-27T07:19:09.826352+0000","last_clean":"2024-05-27T07:19:09.826352+0000","last_became_active":"2024-05-27T07:17:02.336192+0000","last_became_peered":"2024-05-27T07:17:02.336192+0000","last_unstale":"2024-05-27T07:19:09.826352+0000","last_undegraded":"2024-05-27T07:19:09.826352+0000","last_fullsized":"2024-05-27T07:19:09.826352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:48.187517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126382+0000","last_change":"2024-05-27T07:17:06.759729+0000","last_active":"2024-05-27T07:18:47.126382+0000","last_peered":"2024-05-27T07:18:47.126382+0000","last_clean":"2024-05-27T07:18:47.126382+0000","last_became_active":"2024-05-27T07:17:06.756766+0000","last_became_peered":"2024-05-27T07:17:06.756766+0000","last_unstale":"2024-05-27T07:18:47.126382+0000","last_undegraded":"2024-05-27T07:18:47.126382+0000","last_fullsized":"2024-05-27T07:18:47.126382+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:01.090586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062921+0000","last_change":"2024-05-27T07:17:21.878487+0000","last_active":"2024-05-27T07:18:47.062921+0000","last_peered":"2024-05-27T07:18:47.062921+0000","last_clean":"2024-05-27T07:18:47.062921+0000","last_became_active":"2024-05-27T07:17:21.877623+0000","last_became_peered":"2024-05-27T07:17:21.877623+0000","last_unstale":"2024-05-27T07:18:47.062921+0000","last_undegraded":"2024-05-27T07:18:47.062921+0000","last_fullsized":"2024-05-27T07:18:47.062921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:28:52.618738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.085169+0000","last_change":"2024-05-27T07:17:02.486495+0000","last_active":"2024-05-27T07:18:47.085169+0000","last_peered":"2024-05-27T07:18:47.085169+0000","last_clean":"2024-05-27T07:18:47.085169+0000","last_became_active":"2024-05-27T07:17:02.485493+0000","last_became_peered":"2024-05-27T07:17:02.485493+0000","last_unstale":"2024-05-27T07:18:47.085169+0000","last_undegraded":"2024-05-27T07:18:47.085169+0000","last_fullsized":"2024-05-27T07:18:47.085169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:31.977210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128062+0000","last_change":"2024-05-27T07:17:06.760937+0000","last_active":"2024-05-27T07:18:47.128062+0000","last_peered":"2024-05-27T07:18:47.128062+0000","last_clean":"2024-05-27T07:18:47.128062+0000","last_became_active":"2024-05-27T07:17:06.760254+0000","last_became_peered":"2024-05-27T07:17:06.760254+0000","last_unstale":"2024-05-27T07:18:47.128062+0000","last_undegraded":"2024-05-27T07:18:47.128062+0000","last_fullsized":"2024-05-27T07:18:47.128062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:50.147319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084592+0000","last_change":"2024-05-27T07:17:21.901511+0000","last_active":"2024-05-27T07:18:47.084592+0000","last_peered":"2024-05-27T07:18:47.084592+0000","last_clean":"2024-05-27T07:18:47.084592+0000","last_became_active":"2024-05-27T07:17:21.899847+0000","last_became_peered":"2024-05-27T07:17:21.899847+0000","last_unstale":"2024-05-27T07:18:47.084592+0000","last_undegraded":"2024-05-27T07:18:47.084592+0000","last_fullsized":"2024-05-27T07:18:47.084592+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:28.133509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080859+0000","last_change":"2024-05-27T07:17:02.432520+0000","last_active":"2024-05-27T07:18:47.080859+0000","last_peered":"2024-05-27T07:18:47.080859+0000","last_clean":"2024-05-27T07:18:47.080859+0000","last_became_active":"2024-05-27T07:17:02.431563+0000","last_became_peered":"2024-05-27T07:17:02.431563+0000","last_unstale":"2024-05-27T07:18:47.080859+0000","last_undegraded":"2024-05-27T07:18:47.080859+0000","last_fullsized":"2024-05-27T07:18:47.080859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:17:28.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125610+0000","last_change":"2024-05-27T07:17:06.754408+0000","last_active":"2024-05-27T07:18:47.125610+0000","last_peered":"2024-05-27T07:18:47.125610+0000","last_clean":"2024-05-27T07:18:47.125610+0000","last_became_active":"2024-05-27T07:17:06.753787+0000","last_became_peered":"2024-05-27T07:17:06.753787+0000","last_unstale":"2024-05-27T07:18:47.125610+0000","last_undegraded":"2024-05-27T07:18:47.125610+0000","last_fullsized":"2024-05-27T07:18:47.125610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:21:50.461296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060353+0000","last_change":"2024-05-27T07:17:21.895277+0000","last_active":"2024-05-27T07:18:47.060353+0000","last_peered":"2024-05-27T07:18:47.060353+0000","last_clean":"2024-05-27T07:18:47.060353+0000","last_became_active":"2024-05-27T07:17:21.894518+0000","last_became_peered":"2024-05-27T07:17:21.894518+0000","last_unstale":"2024-05-27T07:18:47.060353+0000","last_undegraded":"2024-05-27T07:18:47.060353+0000","last_fullsized":"2024-05-27T07:18:47.060353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:02.008209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125540+0000","last_change":"2024-05-27T07:17:02.343604+0000","last_active":"2024-05-27T07:18:47.125540+0000","last_peered":"2024-05-27T07:18:47.125540+0000","last_clean":"2024-05-27T07:18:47.125540+0000","last_became_active":"2024-05-27T07:17:02.341890+0000","last_became_peered":"2024-05-27T07:17:02.341890+0000","last_unstale":"2024-05-27T07:18:47.125540+0000","last_undegraded":"2024-05-27T07:18:47.125540+0000","last_fullsized":"2024-05-27T07:18:47.125540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:10:21.670193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067371+0000","last_change":"2024-05-27T07:17:06.754767+0000","last_active":"2024-05-27T07:18:47.067371+0000","last_peered":"2024-05-27T07:18:47.067371+0000","last_clean":"2024-05-27T07:18:47.067371+0000","last_became_active":"2024-05-27T07:17:06.754291+0000","last_became_peered":"2024-05-27T07:17:06.754291+0000","last_unstale":"2024-05-27T07:18:47.067371+0000","last_undegraded":"2024-05-27T07:18:47.067371+0000","last_fullsized":"2024-05-27T07:18:47.067371+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:46:48.145874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084413+0000","last_change":"2024-05-27T07:17:21.892342+0000","last_active":"2024-05-27T07:18:47.084413+0000","last_peered":"2024-05-27T07:18:47.084413+0000","last_clean":"2024-05-27T07:18:47.084413+0000","last_became_active":"2024-05-27T07:17:21.891393+0000","last_became_peered":"2024-05-27T07:17:21.891393+0000","last_unstale":"2024-05-27T07:18:47.084413+0000","last_undegraded":"2024-05-27T07:18:47.084413+0000","last_fullsized":"2024-05-27T07:18:47.084413+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:30.510616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062370+0000","last_change":"2024-05-27T07:17:02.340238+0000","last_active":"2024-05-27T07:18:47.062370+0000","last_peered":"2024-05-27T07:18:47.062370+0000","last_clean":"2024-05-27T07:18:47.062370+0000","last_became_active":"2024-05-27T07:17:02.339561+0000","last_became_peered":"2024-05-27T07:17:02.339561+0000","last_unstale":"2024-05-27T07:18:47.062370+0000","last_undegraded":"2024-05-27T07:18:47.062370+0000","last_fullsized":"2024-05-27T07:18:47.062370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:32.451397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125858+0000","last_change":"2024-05-27T07:17:06.769397+0000","last_active":"2024-05-27T07:18:47.125858+0000","last_peered":"2024-05-27T07:18:47.125858+0000","last_clean":"2024-05-27T07:18:47.125858+0000","last_became_active":"2024-05-27T07:17:06.768936+0000","last_became_peered":"2024-05-27T07:17:06.768936+0000","last_unstale":"2024-05-27T07:18:47.125858+0000","last_undegraded":"2024-05-27T07:18:47.125858+0000","last_fullsized":"2024-05-27T07:18:47.125858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:52.767902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084314+0000","last_change":"2024-05-27T07:17:21.894225+0000","last_active":"2024-05-27T07:18:47.084314+0000","last_peered":"2024-05-27T07:18:47.084314+0000","last_clean":"2024-05-27T07:18:47.084314+0000","last_became_active":"2024-05-27T07:17:21.893499+0000","last_became_peered":"2024-05-27T07:17:21.893499+0000","last_unstale":"2024-05-27T07:18:47.084314+0000","last_undegraded":"2024-05-27T07:18:47.084314+0000","last_fullsized":"2024-05-27T07:18:47.084314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:54.985290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126070+0000","last_change":"2024-05-27T07:17:02.347736+0000","last_active":"2024-05-27T07:18:47.126070+0000","last_peered":"2024-05-27T07:18:47.126070+0000","last_clean":"2024-05-27T07:18:47.126070+0000","last_became_active":"2024-05-27T07:17:02.347237+0000","last_became_peered":"2024-05-27T07:17:02.347237+0000","last_unstale":"2024-05-27T07:18:47.126070+0000","last_undegraded":"2024-05-27T07:18:47.126070+0000","last_fullsized":"2024-05-27T07:18:47.126070+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:36:15.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066402+0000","last_change":"2024-05-27T07:17:06.759348+0000","last_active":"2024-05-27T07:18:47.066402+0000","last_peered":"2024-05-27T07:18:47.066402+0000","last_clean":"2024-05-27T07:18:47.066402+0000","last_became_active":"2024-05-27T07:17:06.758967+0000","last_became_peered":"2024-05-27T07:17:06.758967+0000","last_unstale":"2024-05-27T07:18:47.066402+0000","last_undegraded":"2024-05-27T07:18:47.066402+0000","last_fullsized":"2024-05-27T07:18:47.066402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:45.430719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080436+0000","last_change":"2024-05-27T07:17:21.882919+0000","last_active":"2024-05-27T07:18:47.080436+0000","last_peered":"2024-05-27T07:18:47.080436+0000","last_clean":"2024-05-27T07:18:47.080436+0000","last_became_active":"2024-05-27T07:17:21.882330+0000","last_became_peered":"2024-05-27T07:17:21.882330+0000","last_unstale":"2024-05-27T07:18:47.080436+0000","last_undegraded":"2024-05-27T07:18:47.080436+0000","last_fullsized":"2024-05-27T07:18:47.080436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:51.409079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064706+0000","last_change":"2024-05-27T07:17:02.486099+0000","last_active":"2024-05-27T07:18:47.064706+0000","last_peered":"2024-05-27T07:18:47.064706+0000","last_clean":"2024-05-27T07:18:47.064706+0000","last_became_active":"2024-05-27T07:17:02.485116+0000","last_became_peered":"2024-05-27T07:17:02.485116+0000","last_unstale":"2024-05-27T07:18:47.064706+0000","last_undegraded":"2024-05-27T07:18:47.064706+0000","last_fullsized":"2024-05-27T07:18:47.064706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:36.338287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067048+0000","last_change":"2024-05-27T07:17:06.773586+0000","last_active":"2024-05-27T07:18:47.067048+0000","last_peered":"2024-05-27T07:18:47.067048+0000","last_clean":"2024-05-27T07:18:47.067048+0000","last_became_active":"2024-05-27T07:17:06.773139+0000","last_became_peered":"2024-05-27T07:17:06.773139+0000","last_unstale":"2024-05-27T07:18:47.067048+0000","last_undegraded":"2024-05-27T07:18:47.067048+0000","last_fullsized":"2024-05-27T07:18:47.067048+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:19:36.373091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127421+0000","last_change":"2024-05-27T07:17:21.892351+0000","last_active":"2024-05-27T07:18:47.127421+0000","last_peered":"2024-05-27T07:18:47.127421+0000","last_clean":"2024-05-27T07:18:47.127421+0000","last_became_active":"2024-05-27T07:17:21.891862+0000","last_became_peered":"2024-05-27T07:17:21.891862+0000","last_unstale":"2024-05-27T07:18:47.127421+0000","last_undegraded":"2024-05-27T07:18:47.127421+0000","last_fullsized":"2024-05-27T07:18:47.127421+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:50:18.193149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070981+0000","last_change":"2024-05-27T07:17:02.336917+0000","last_active":"2024-05-27T07:18:47.070981+0000","last_peered":"2024-05-27T07:18:47.070981+0000","last_clean":"2024-05-27T07:18:47.070981+0000","last_became_active":"2024-05-27T07:17:02.335836+0000","last_became_peered":"2024-05-27T07:17:02.335836+0000","last_unstale":"2024-05-27T07:18:47.070981+0000","last_undegraded":"2024-05-27T07:18:47.070981+0000","last_fullsized":"2024-05-27T07:18:47.070981+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:54.109480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067770+0000","last_change":"2024-05-27T07:17:46.705826+0000","last_active":"2024-05-27T07:18:47.067770+0000","last_peered":"2024-05-27T07:18:47.067770+0000","last_clean":"2024-05-27T07:18:47.067770+0000","last_became_active":"2024-05-27T07:17:46.704749+0000","last_became_peered":"2024-05-27T07:17:46.704749+0000","last_unstale":"2024-05-27T07:18:47.067770+0000","last_undegraded":"2024-05-27T07:18:47.067770+0000","last_fullsized":"2024-05-27T07:18:47.067770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:26.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067984+0000","last_change":"2024-05-27T07:17:21.885535+0000","last_active":"2024-05-27T07:18:47.067984+0000","last_peered":"2024-05-27T07:18:47.067984+0000","last_clean":"2024-05-27T07:18:47.067984+0000","last_became_active":"2024-05-27T07:17:21.884118+0000","last_became_peered":"2024-05-27T07:17:21.884118+0000","last_unstale":"2024-05-27T07:18:47.067984+0000","last_undegraded":"2024-05-27T07:18:47.067984+0000","last_fullsized":"2024-05-27T07:18:47.067984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:26.600199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130070+0000","last_change":"2024-05-27T07:17:02.340037+0000","last_active":"2024-05-27T07:18:47.130070+0000","last_peered":"2024-05-27T07:18:47.130070+0000","last_clean":"2024-05-27T07:18:47.130070+0000","last_became_active":"2024-05-27T07:17:02.339378+0000","last_became_peered":"2024-05-27T07:17:02.339378+0000","last_unstale":"2024-05-27T07:18:47.130070+0000","last_undegraded":"2024-05-27T07:18:47.130070+0000","last_fullsized":"2024-05-27T07:18:47.130070+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:33.774917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068712+0000","last_change":"2024-05-27T07:17:06.763277+0000","last_active":"2024-05-27T07:18:47.068712+0000","last_peered":"2024-05-27T07:18:47.068712+0000","last_clean":"2024-05-27T07:18:47.068712+0000","last_became_active":"2024-05-27T07:17:06.762837+0000","last_became_peered":"2024-05-27T07:17:06.762837+0000","last_unstale":"2024-05-27T07:18:47.068712+0000","last_undegraded":"2024-05-27T07:18:47.068712+0000","last_fullsized":"2024-05-27T07:18:47.068712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:52.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066992+0000","last_change":"2024-05-27T07:17:06.752556+0000","last_active":"2024-05-27T07:18:47.066992+0000","last_peered":"2024-05-27T07:18:47.066992+0000","last_clean":"2024-05-27T07:18:47.066992+0000","last_became_active":"2024-05-27T07:17:06.751085+0000","last_became_peered":"2024-05-27T07:17:06.751085+0000","last_unstale":"2024-05-27T07:18:47.066992+0000","last_undegraded":"2024-05-27T07:18:47.066992+0000","last_fullsized":"2024-05-27T07:18:47.066992+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:13:53.883845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064206+0000","last_change":"2024-05-27T07:17:06.753674+0000","last_active":"2024-05-27T07:18:47.064206+0000","last_peered":"2024-05-27T07:18:47.064206+0000","last_clean":"2024-05-27T07:18:47.064206+0000","last_became_active":"2024-05-27T07:17:06.752373+0000","last_became_peered":"2024-05-27T07:17:06.752373+0000","last_unstale":"2024-05-27T07:18:47.064206+0000","last_undegraded":"2024-05-27T07:18:47.064206+0000","last_fullsized":"2024-05-27T07:18:47.064206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:40:17.474462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062007+0000","last_change":"2024-05-27T07:17:06.766796+0000","last_active":"2024-05-27T07:18:47.062007+0000","last_peered":"2024-05-27T07:18:47.062007+0000","last_clean":"2024-05-27T07:18:47.062007+0000","last_became_active":"2024-05-27T07:17:06.766435+0000","last_became_peered":"2024-05-27T07:17:06.766435+0000","last_unstale":"2024-05-27T07:18:47.062007+0000","last_undegraded":"2024-05-27T07:18:47.062007+0000","last_fullsized":"2024-05-27T07:18:47.062007+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:38.747810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128025+0000","last_change":"2024-05-27T07:17:06.762804+0000","last_active":"2024-05-27T07:18:47.128025+0000","last_peered":"2024-05-27T07:18:47.128025+0000","last_clean":"2024-05-27T07:18:47.128025+0000","last_became_active":"2024-05-27T07:17:06.762490+0000","last_became_peered":"2024-05-27T07:17:06.762490+0000","last_unstale":"2024-05-27T07:18:47.128025+0000","last_undegraded":"2024-05-27T07:18:47.128025+0000","last_fullsized":"2024-05-27T07:18:47.128025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:59:33.584848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130704+0000","last_change":"2024-05-27T07:17:06.765640+0000","last_active":"2024-05-27T07:18:47.130704+0000","last_peered":"2024-05-27T07:18:47.130704+0000","last_clean":"2024-05-27T07:18:47.130704+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:18:47.130704+0000","last_undegraded":"2024-05-27T07:18:47.130704+0000","last_fullsized":"2024-05-27T07:18:47.130704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:19.695007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072866+0000","last_change":"2024-05-27T07:17:06.759982+0000","last_active":"2024-05-27T07:18:47.072866+0000","last_peered":"2024-05-27T07:18:47.072866+0000","last_clean":"2024-05-27T07:18:47.072866+0000","last_became_active":"2024-05-27T07:17:06.759367+0000","last_became_peered":"2024-05-27T07:17:06.759367+0000","last_unstale":"2024-05-27T07:18:47.072866+0000","last_undegraded":"2024-05-27T07:18:47.072866+0000","last_fullsized":"2024-05-27T07:18:47.072866+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:35:04.686989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065903+0000","last_change":"2024-05-27T07:17:06.768264+0000","last_active":"2024-05-27T07:18:47.065903+0000","last_peered":"2024-05-27T07:18:47.065903+0000","last_clean":"2024-05-27T07:18:47.065903+0000","last_became_active":"2024-05-27T07:17:06.767952+0000","last_became_peered":"2024-05-27T07:17:06.767952+0000","last_unstale":"2024-05-27T07:18:47.065903+0000","last_undegraded":"2024-05-27T07:18:47.065903+0000","last_fullsized":"2024-05-27T07:18:47.065903+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:34.867113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059291+0000","last_change":"2024-05-27T07:17:46.704705+0000","last_active":"2024-05-27T07:18:47.059291+0000","last_peered":"2024-05-27T07:18:47.059291+0000","last_clean":"2024-05-27T07:18:47.059291+0000","last_became_active":"2024-05-27T07:17:46.703503+0000","last_became_peered":"2024-05-27T07:17:46.703503+0000","last_unstale":"2024-05-27T07:18:47.059291+0000","last_undegraded":"2024-05-27T07:18:47.059291+0000","last_fullsized":"2024-05-27T07:18:47.059291+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:39.975009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068425+0000","last_change":"2024-05-27T07:17:06.752080+0000","last_active":"2024-05-27T07:18:47.068425+0000","last_peered":"2024-05-27T07:18:47.068425+0000","last_clean":"2024-05-27T07:18:47.068425+0000","last_became_active":"2024-05-27T07:17:06.750486+0000","last_became_peered":"2024-05-27T07:17:06.750486+0000","last_unstale":"2024-05-27T07:18:47.068425+0000","last_undegraded":"2024-05-27T07:18:47.068425+0000","last_fullsized":"2024-05-27T07:18:47.068425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:12.461727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125600+0000","last_change":"2024-05-27T07:17:06.759200+0000","last_active":"2024-05-27T07:18:47.125600+0000","last_peered":"2024-05-27T07:18:47.125600+0000","last_clean":"2024-05-27T07:18:47.125600+0000","last_became_active":"2024-05-27T07:17:06.758819+0000","last_became_peered":"2024-05-27T07:17:06.758819+0000","last_unstale":"2024-05-27T07:18:47.125600+0000","last_undegraded":"2024-05-27T07:18:47.125600+0000","last_fullsized":"2024-05-27T07:18:47.125600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:00.274229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080876+0000","last_change":"2024-05-27T07:17:06.760412+0000","last_active":"2024-05-27T07:18:47.080876+0000","last_peered":"2024-05-27T07:18:47.080876+0000","last_clean":"2024-05-27T07:18:47.080876+0000","last_became_active":"2024-05-27T07:17:06.759949+0000","last_became_peered":"2024-05-27T07:17:06.759949+0000","last_unstale":"2024-05-27T07:18:47.080876+0000","last_undegraded":"2024-05-27T07:18:47.080876+0000","last_fullsized":"2024-05-27T07:18:47.080876+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.235488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129710+0000","last_change":"2024-05-27T07:17:06.775371+0000","last_active":"2024-05-27T07:18:47.129710+0000","last_peered":"2024-05-27T07:18:47.129710+0000","last_clean":"2024-05-27T07:18:47.129710+0000","last_became_active":"2024-05-27T07:17:06.774904+0000","last_became_peered":"2024-05-27T07:17:06.774904+0000","last_unstale":"2024-05-27T07:18:47.129710+0000","last_undegraded":"2024-05-27T07:18:47.129710+0000","last_fullsized":"2024-05-27T07:18:47.129710+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:22:13.484733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072043+0000","last_change":"2024-05-27T07:17:06.757066+0000","last_active":"2024-05-27T07:18:47.072043+0000","last_peered":"2024-05-27T07:18:47.072043+0000","last_clean":"2024-05-27T07:18:47.072043+0000","last_became_active":"2024-05-27T07:17:06.756316+0000","last_became_peered":"2024-05-27T07:17:06.756316+0000","last_unstale":"2024-05-27T07:18:47.072043+0000","last_undegraded":"2024-05-27T07:18:47.072043+0000","last_fullsized":"2024-05-27T07:18:47.072043+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:15:12.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66464,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":238,"num_read_kb":189,"num_write":170,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":199392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710765,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":515890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906994,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32848,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712304,"statfs":{"total":95995035648,"available":95961399296,"internally_reserved":0,"allocated":1712128,"data_stored":1164927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070524,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":519528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266755,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":519336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462985,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1630208,"data_stored":1111070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626512,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":510608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":572590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":511996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":511246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182505,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":578734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822745,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1638400,"data_stored":1102753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:19:16.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:19:16.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:16.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:16 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:16.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:16 smithi012 ceph-mon[31473]: pgmap v454: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-05-27T07:19:16.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:16 smithi012 ceph-mon[31473]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:19:16.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:16.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:16 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:16.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:16 smithi162 ceph-mon[35676]: pgmap v454: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-05-27T07:19:16.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:16 smithi162 ceph-mon[35676]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:19:16.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:16.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:16 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:16.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:16 smithi062 ceph-mon[35304]: pgmap v454: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-05-27T07:19:16.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:16 smithi062 ceph-mon[35304]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:19:17.273 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:17.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:17 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:17.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:17 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:17.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:17 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:17.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:17 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:17.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:17 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:17.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:17 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:17.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:17 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:17.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:17 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:17.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:17 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:19:18.934 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:18 smithi012 ceph-mon[31473]: pgmap v455: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-05-27T07:19:18.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:18 smithi162 ceph-mon[35676]: pgmap v455: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-05-27T07:19:18.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:18 smithi062 ceph-mon[35304]: pgmap v455: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.540+0000 7f177ce00700 1 -- 172.21.15.12:0/4156652186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f177810a980 msgr2=0x7f177810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.540+0000 7f177ce00700 1 --2- 172.21.15.12:0/4156652186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f177810a980 0x7f177810ad90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1760009990 tx=0x7f1760009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.541+0000 7f177ce00700 1 -- 172.21.15.12:0/4156652186 shutdown_connections 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.541+0000 7f177ce00700 1 --2- 172.21.15.12:0/4156652186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1778110960 0x7f1778110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.541+0000 7f177ce00700 1 --2- 172.21.15.12:0/4156652186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f177810b2d0 0x7f1778117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.541+0000 7f177ce00700 1 --2- 172.21.15.12:0/4156652186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f177810a980 0x7f177810ad90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.541+0000 7f177ce00700 1 -- 172.21.15.12:0/4156652186 >> 172.21.15.12:0/4156652186 conn(0x7f1778077c70 msgr2=0x7f17780780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.541+0000 7f177ce00700 1 -- 172.21.15.12:0/4156652186 shutdown_connections 2024-05-27T07:19:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.541+0000 7f177ce00700 1 -- 172.21.15.12:0/4156652186 wait complete. 2024-05-27T07:19:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.542+0000 7f177ce00700 1 Processor -- start 2024-05-27T07:19:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.543+0000 7f177ce00700 1 -- start start 2024-05-27T07:19:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.543+0000 7f177ce00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f177810a980 0x7f17781a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.543+0000 7f177ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f177810b2d0 0x7f17781a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.543+0000 7f177ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1778110960 0x7f17781a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.543+0000 7f177ce00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f177811a980 con 0x7f1778110960 2024-05-27T07:19:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.543+0000 7f177ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f177811a800 con 0x7f177810b2d0 2024-05-27T07:19:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.543+0000 7f177ce00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f177811ab00 con 0x7f177810a980 2024-05-27T07:19:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.544+0000 7f1776800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1778110960 0x7f17781a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.544+0000 7f1776800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1778110960 0x7f17781a5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43112/0 (socket says 172.21.15.12:43112) 2024-05-27T07:19:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.544+0000 7f1776800700 1 -- 172.21.15.12:0/1978131695 learned_addr learned my addr 172.21.15.12:0/1978131695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.544+0000 7f176d400700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f177810b2d0 0x7f17781a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.544+0000 7f1775e00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f177810a980 0x7f17781a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f1776800700 1 -- 172.21.15.12:0/1978131695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f177810a980 msgr2=0x7f17781a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f1776800700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f177810a980 0x7f17781a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f1776800700 1 -- 172.21.15.12:0/1978131695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f177810b2d0 msgr2=0x7f17781a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f1776800700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f177810b2d0 0x7f17781a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f1776800700 1 -- 172.21.15.12:0/1978131695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17781a6280 con 0x7f1778110960 2024-05-27T07:19:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f1776800700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1778110960 0x7f17781a5be0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1770007ee0 tx=0x7f177000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f176f400700 1 -- 172.21.15.12:0/1978131695 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770004bb0 con 0x7f1778110960 2024-05-27T07:19:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f176f400700 1 -- 172.21.15.12:0/1978131695 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1770004d50 con 0x7f1778110960 2024-05-27T07:19:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.545+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17781a6570 con 0x7f1778110960 2024-05-27T07:19:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.546+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17781ade90 con 0x7f1778110960 2024-05-27T07:19:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.546+0000 7f176f400700 1 -- 172.21.15.12:0/1978131695 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17700058b0 con 0x7f1778110960 2024-05-27T07:19:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.547+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1754005330 con 0x7f1778110960 2024-05-27T07:19:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.550+0000 7f176f400700 1 -- 172.21.15.12:0/1978131695 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f177001a020 con 0x7f1778110960 2024-05-27T07:19:19.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.551+0000 7f176f400700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f17580b2910 0x7f17580b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.551+0000 7f176f400700 1 -- 172.21.15.12:0/1978131695 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f17700e23e0 con 0x7f1778110960 2024-05-27T07:19:19.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.551+0000 7f1775e00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f17580b2910 0x7f17580b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.552+0000 7f1775e00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f17580b2910 0x7f17580b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1768005950 tx=0x7f176800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.554+0000 7f176f400700 1 -- 172.21.15.12:0/1978131695 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1770010070 con 0x7f1778110960 2024-05-27T07:19:19.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.788+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1754000bc0 con 0x7f17580b2910 2024-05-27T07:19:19.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.807+0000 7f176f400700 1 -- 172.21.15.12:0/1978131695 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340365 (secure 0 0 0) 0x7f1754000bc0 con 0x7f17580b2910 2024-05-27T07:19:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:19:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f17580b2910 msgr2=0x7f17580b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f17580b2910 0x7f17580b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1768005950 tx=0x7f176800b410 comp rx=0 tx=0).stop 2024-05-27T07:19:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1778110960 msgr2=0x7f17781a5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1778110960 0x7f17781a5be0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1770007ee0 tx=0x7f177000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 shutdown_connections 2024-05-27T07:19:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f17580b2910 0x7f17580b4e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1778110960 0x7f17781a5be0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f177810a980 0x7f17781a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 --2- 172.21.15.12:0/1978131695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f177810b2d0 0x7f17781a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.815+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 >> 172.21.15.12:0/1978131695 conn(0x7f1778077c70 msgr2=0x7f177807b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.816+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 shutdown_connections 2024-05-27T07:19:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:19.816+0000 7f177ce00700 1 -- 172.21.15.12:0/1978131695 wait complete. 2024-05-27T07:19:19.809 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-05-27T07:19:20.700 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-05-27T07:19:19.782772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657066,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":349,"num_read_kb":273,"num_write":295,"num_write_kb":2270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375744,"kb_used_data":14084,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566080,"statfs":{"total":1151940427776,"available":1151555665920,"internally_reserved":0,"allocated":14422016,"data_stored":8129480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9083,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":87,"num_read_kb":50,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007076"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.073972+0000","last_change":"2024-05-27T07:17:06.763882+0000","last_active":"2024-05-27T07:18:47.073972+0000","last_peered":"2024-05-27T07:18:47.073972+0000","last_clean":"2024-05-27T07:18:47.073972+0000","last_became_active":"2024-05-27T07:17:06.759625+0000","last_became_peered":"2024-05-27T07:17:06.759625+0000","last_unstale":"2024-05-27T07:18:47.073972+0000","last_undegraded":"2024-05-27T07:18:47.073972+0000","last_fullsized":"2024-05-27T07:18:47.073972+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:24.105867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127984+0000","last_change":"2024-05-27T07:17:06.755737+0000","last_active":"2024-05-27T07:18:47.127984+0000","last_peered":"2024-05-27T07:18:47.127984+0000","last_clean":"2024-05-27T07:18:47.127984+0000","last_became_active":"2024-05-27T07:17:06.754588+0000","last_became_peered":"2024-05-27T07:17:06.754588+0000","last_unstale":"2024-05-27T07:18:47.127984+0000","last_undegraded":"2024-05-27T07:18:47.127984+0000","last_fullsized":"2024-05-27T07:18:47.127984+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:02.417140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128521+0000","last_change":"2024-05-27T07:17:06.772057+0000","last_active":"2024-05-27T07:18:47.128521+0000","last_peered":"2024-05-27T07:18:47.128521+0000","last_clean":"2024-05-27T07:18:47.128521+0000","last_became_active":"2024-05-27T07:17:06.771636+0000","last_became_peered":"2024-05-27T07:17:06.771636+0000","last_unstale":"2024-05-27T07:18:47.128521+0000","last_undegraded":"2024-05-27T07:18:47.128521+0000","last_fullsized":"2024-05-27T07:18:47.128521+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:45.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082870+0000","last_change":"2024-05-27T07:17:06.763599+0000","last_active":"2024-05-27T07:18:47.082870+0000","last_peered":"2024-05-27T07:18:47.082870+0000","last_clean":"2024-05-27T07:18:47.082870+0000","last_became_active":"2024-05-27T07:17:06.763125+0000","last_became_peered":"2024-05-27T07:17:06.763125+0000","last_unstale":"2024-05-27T07:18:47.082870+0000","last_undegraded":"2024-05-27T07:18:47.082870+0000","last_fullsized":"2024-05-27T07:18:47.082870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:43.359673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125913+0000","last_change":"2024-05-27T07:17:06.755896+0000","last_active":"2024-05-27T07:18:47.125913+0000","last_peered":"2024-05-27T07:18:47.125913+0000","last_clean":"2024-05-27T07:18:47.125913+0000","last_became_active":"2024-05-27T07:17:06.755495+0000","last_became_peered":"2024-05-27T07:17:06.755495+0000","last_unstale":"2024-05-27T07:18:47.125913+0000","last_undegraded":"2024-05-27T07:18:47.125913+0000","last_fullsized":"2024-05-27T07:18:47.125913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:41:37.632012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064604+0000","last_change":"2024-05-27T07:17:46.699881+0000","last_active":"2024-05-27T07:18:47.064604+0000","last_peered":"2024-05-27T07:18:47.064604+0000","last_clean":"2024-05-27T07:18:47.064604+0000","last_became_active":"2024-05-27T07:17:46.699080+0000","last_became_peered":"2024-05-27T07:17:46.699080+0000","last_unstale":"2024-05-27T07:18:47.064604+0000","last_undegraded":"2024-05-27T07:18:47.064604+0000","last_fullsized":"2024-05-27T07:18:47.064604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:43.580762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065531+0000","last_change":"2024-05-27T07:17:06.770468+0000","last_active":"2024-05-27T07:18:47.065531+0000","last_peered":"2024-05-27T07:18:47.065531+0000","last_clean":"2024-05-27T07:18:47.065531+0000","last_became_active":"2024-05-27T07:17:06.768345+0000","last_became_peered":"2024-05-27T07:17:06.768345+0000","last_unstale":"2024-05-27T07:18:47.065531+0000","last_undegraded":"2024-05-27T07:18:47.065531+0000","last_fullsized":"2024-05-27T07:18:47.065531+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:12:27.646654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125170+0000","last_change":"2024-05-27T07:17:06.758769+0000","last_active":"2024-05-27T07:18:47.125170+0000","last_peered":"2024-05-27T07:18:47.125170+0000","last_clean":"2024-05-27T07:18:47.125170+0000","last_became_active":"2024-05-27T07:17:06.758393+0000","last_became_peered":"2024-05-27T07:17:06.758393+0000","last_unstale":"2024-05-27T07:18:47.125170+0000","last_undegraded":"2024-05-27T07:18:47.125170+0000","last_fullsized":"2024-05-27T07:18:47.125170+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:56.450569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.081950+0000","last_change":"2024-05-27T07:17:06.770681+0000","last_active":"2024-05-27T07:18:47.081950+0000","last_peered":"2024-05-27T07:18:47.081950+0000","last_clean":"2024-05-27T07:18:47.081950+0000","last_became_active":"2024-05-27T07:17:06.766520+0000","last_became_peered":"2024-05-27T07:17:06.766520+0000","last_unstale":"2024-05-27T07:18:47.081950+0000","last_undegraded":"2024-05-27T07:18:47.081950+0000","last_fullsized":"2024-05-27T07:18:47.081950+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:56.931743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067266+0000","last_change":"2024-05-27T07:17:06.761462+0000","last_active":"2024-05-27T07:18:47.067266+0000","last_peered":"2024-05-27T07:18:47.067266+0000","last_clean":"2024-05-27T07:18:47.067266+0000","last_became_active":"2024-05-27T07:17:06.759344+0000","last_became_peered":"2024-05-27T07:17:06.759344+0000","last_unstale":"2024-05-27T07:18:47.067266+0000","last_undegraded":"2024-05-27T07:18:47.067266+0000","last_fullsized":"2024-05-27T07:18:47.067266+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:12:22.688477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066831+0000","last_change":"2024-05-27T07:17:06.761947+0000","last_active":"2024-05-27T07:18:47.066831+0000","last_peered":"2024-05-27T07:18:47.066831+0000","last_clean":"2024-05-27T07:18:47.066831+0000","last_became_active":"2024-05-27T07:17:06.761464+0000","last_became_peered":"2024-05-27T07:17:06.761464+0000","last_unstale":"2024-05-27T07:18:47.066831+0000","last_undegraded":"2024-05-27T07:18:47.066831+0000","last_fullsized":"2024-05-27T07:18:47.066831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:08:19.939311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083615+0000","last_change":"2024-05-27T07:17:06.772039+0000","last_active":"2024-05-27T07:18:47.083615+0000","last_peered":"2024-05-27T07:18:47.083615+0000","last_clean":"2024-05-27T07:18:47.083615+0000","last_became_active":"2024-05-27T07:17:06.771303+0000","last_became_peered":"2024-05-27T07:17:06.771303+0000","last_unstale":"2024-05-27T07:18:47.083615+0000","last_undegraded":"2024-05-27T07:18:47.083615+0000","last_fullsized":"2024-05-27T07:18:47.083615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:30.254848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129891+0000","last_change":"2024-05-27T07:17:06.774228+0000","last_active":"2024-05-27T07:18:47.129891+0000","last_peered":"2024-05-27T07:18:47.129891+0000","last_clean":"2024-05-27T07:18:47.129891+0000","last_became_active":"2024-05-27T07:17:06.771436+0000","last_became_peered":"2024-05-27T07:17:06.771436+0000","last_unstale":"2024-05-27T07:18:47.129891+0000","last_undegraded":"2024-05-27T07:18:47.129891+0000","last_fullsized":"2024-05-27T07:18:47.129891+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:25.749143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127433+0000","last_change":"2024-05-27T07:17:06.755558+0000","last_active":"2024-05-27T07:18:47.127433+0000","last_peered":"2024-05-27T07:18:47.127433+0000","last_clean":"2024-05-27T07:18:47.127433+0000","last_became_active":"2024-05-27T07:17:06.754858+0000","last_became_peered":"2024-05-27T07:17:06.754858+0000","last_unstale":"2024-05-27T07:18:47.127433+0000","last_undegraded":"2024-05-27T07:18:47.127433+0000","last_fullsized":"2024-05-27T07:18:47.127433+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:09.853982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126976+0000","last_change":"2024-05-27T07:17:06.770731+0000","last_active":"2024-05-27T07:18:47.126976+0000","last_peered":"2024-05-27T07:18:47.126976+0000","last_clean":"2024-05-27T07:18:47.126976+0000","last_became_active":"2024-05-27T07:17:06.766923+0000","last_became_peered":"2024-05-27T07:17:06.766923+0000","last_unstale":"2024-05-27T07:18:47.126976+0000","last_undegraded":"2024-05-27T07:18:47.126976+0000","last_fullsized":"2024-05-27T07:18:47.126976+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.110121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080436+0000","last_change":"2024-05-27T07:17:06.758928+0000","last_active":"2024-05-27T07:18:47.080436+0000","last_peered":"2024-05-27T07:18:47.080436+0000","last_clean":"2024-05-27T07:18:47.080436+0000","last_became_active":"2024-05-27T07:17:06.758328+0000","last_became_peered":"2024-05-27T07:17:06.758328+0000","last_unstale":"2024-05-27T07:18:47.080436+0000","last_undegraded":"2024-05-27T07:18:47.080436+0000","last_fullsized":"2024-05-27T07:18:47.080436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:36.284395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.073095+0000","last_change":"2024-05-27T07:17:06.760006+0000","last_active":"2024-05-27T07:18:47.073095+0000","last_peered":"2024-05-27T07:18:47.073095+0000","last_clean":"2024-05-27T07:18:47.073095+0000","last_became_active":"2024-05-27T07:17:06.758177+0000","last_became_peered":"2024-05-27T07:17:06.758177+0000","last_unstale":"2024-05-27T07:18:47.073095+0000","last_undegraded":"2024-05-27T07:18:47.073095+0000","last_fullsized":"2024-05-27T07:18:47.073095+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.516289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060373+0000","last_change":"2024-05-27T07:17:06.768484+0000","last_active":"2024-05-27T07:18:47.060373+0000","last_peered":"2024-05-27T07:18:47.060373+0000","last_clean":"2024-05-27T07:18:47.060373+0000","last_became_active":"2024-05-27T07:17:06.768105+0000","last_became_peered":"2024-05-27T07:17:06.768105+0000","last_unstale":"2024-05-27T07:18:47.060373+0000","last_undegraded":"2024-05-27T07:18:47.060373+0000","last_fullsized":"2024-05-27T07:18:47.060373+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:54:48.751678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065152+0000","last_change":"2024-05-27T07:17:06.762782+0000","last_active":"2024-05-27T07:18:47.065152+0000","last_peered":"2024-05-27T07:18:47.065152+0000","last_clean":"2024-05-27T07:18:47.065152+0000","last_became_active":"2024-05-27T07:17:06.760129+0000","last_became_peered":"2024-05-27T07:17:06.760129+0000","last_unstale":"2024-05-27T07:18:47.065152+0000","last_undegraded":"2024-05-27T07:18:47.065152+0000","last_fullsized":"2024-05-27T07:18:47.065152+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:47:19.360662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063483+0000","last_change":"2024-05-27T07:17:21.899772+0000","last_active":"2024-05-27T07:18:47.063483+0000","last_peered":"2024-05-27T07:18:47.063483+0000","last_clean":"2024-05-27T07:18:47.063483+0000","last_became_active":"2024-05-27T07:17:21.899038+0000","last_became_peered":"2024-05-27T07:17:21.899038+0000","last_unstale":"2024-05-27T07:18:47.063483+0000","last_undegraded":"2024-05-27T07:18:47.063483+0000","last_fullsized":"2024-05-27T07:18:47.063483+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:38.784433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059878+0000","last_change":"2024-05-27T07:17:46.708798+0000","last_active":"2024-05-27T07:18:47.059878+0000","last_peered":"2024-05-27T07:18:47.059878+0000","last_clean":"2024-05-27T07:18:47.059878+0000","last_became_active":"2024-05-27T07:17:46.696433+0000","last_became_peered":"2024-05-27T07:17:46.696433+0000","last_unstale":"2024-05-27T07:18:47.059878+0000","last_undegraded":"2024-05-27T07:18:47.059878+0000","last_fullsized":"2024-05-27T07:18:47.059878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:40.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126199+0000","last_change":"2024-05-27T07:17:06.760865+0000","last_active":"2024-05-27T07:18:47.126199+0000","last_peered":"2024-05-27T07:18:47.126199+0000","last_clean":"2024-05-27T07:18:47.126199+0000","last_became_active":"2024-05-27T07:17:06.759491+0000","last_became_peered":"2024-05-27T07:17:06.759491+0000","last_unstale":"2024-05-27T07:18:47.126199+0000","last_undegraded":"2024-05-27T07:18:47.126199+0000","last_fullsized":"2024-05-27T07:18:47.126199+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:31.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082407+0000","last_change":"2024-05-27T07:17:21.898583+0000","last_active":"2024-05-27T07:18:47.082407+0000","last_peered":"2024-05-27T07:18:47.082407+0000","last_clean":"2024-05-27T07:18:47.082407+0000","last_became_active":"2024-05-27T07:17:21.897937+0000","last_became_peered":"2024-05-27T07:17:21.897937+0000","last_unstale":"2024-05-27T07:18:47.082407+0000","last_undegraded":"2024-05-27T07:18:47.082407+0000","last_fullsized":"2024-05-27T07:18:47.082407+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:45:00.412914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126886+0000","last_change":"2024-05-27T07:17:02.432465+0000","last_active":"2024-05-27T07:18:47.126886+0000","last_peered":"2024-05-27T07:18:47.126886+0000","last_clean":"2024-05-27T07:18:47.126886+0000","last_became_active":"2024-05-27T07:17:02.431638+0000","last_became_peered":"2024-05-27T07:17:02.431638+0000","last_unstale":"2024-05-27T07:18:47.126886+0000","last_undegraded":"2024-05-27T07:18:47.126886+0000","last_fullsized":"2024-05-27T07:18:47.126886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:03:13.167952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129569+0000","last_change":"2024-05-27T07:17:06.763133+0000","last_active":"2024-05-27T07:18:47.129569+0000","last_peered":"2024-05-27T07:18:47.129569+0000","last_clean":"2024-05-27T07:18:47.129569+0000","last_became_active":"2024-05-27T07:17:06.762770+0000","last_became_peered":"2024-05-27T07:17:06.762770+0000","last_unstale":"2024-05-27T07:18:47.129569+0000","last_undegraded":"2024-05-27T07:18:47.129569+0000","last_fullsized":"2024-05-27T07:18:47.129569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:33.223722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083366+0000","last_change":"2024-05-27T07:17:21.900369+0000","last_active":"2024-05-27T07:18:47.083366+0000","last_peered":"2024-05-27T07:18:47.083366+0000","last_clean":"2024-05-27T07:18:47.083366+0000","last_became_active":"2024-05-27T07:17:21.899758+0000","last_became_peered":"2024-05-27T07:17:21.899758+0000","last_unstale":"2024-05-27T07:18:47.083366+0000","last_undegraded":"2024-05-27T07:18:47.083366+0000","last_fullsized":"2024-05-27T07:18:47.083366+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:42.116302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127879+0000","last_change":"2024-05-27T07:17:02.347047+0000","last_active":"2024-05-27T07:18:47.127879+0000","last_peered":"2024-05-27T07:18:47.127879+0000","last_clean":"2024-05-27T07:18:47.127879+0000","last_became_active":"2024-05-27T07:17:02.346562+0000","last_became_peered":"2024-05-27T07:17:02.346562+0000","last_unstale":"2024-05-27T07:18:47.127879+0000","last_undegraded":"2024-05-27T07:18:47.127879+0000","last_fullsized":"2024-05-27T07:18:47.127879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:25:50.133089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080814+0000","last_change":"2024-05-27T07:17:06.767930+0000","last_active":"2024-05-27T07:18:47.080814+0000","last_peered":"2024-05-27T07:18:47.080814+0000","last_clean":"2024-05-27T07:18:47.080814+0000","last_became_active":"2024-05-27T07:17:06.767439+0000","last_became_peered":"2024-05-27T07:17:06.767439+0000","last_unstale":"2024-05-27T07:18:47.080814+0000","last_undegraded":"2024-05-27T07:18:47.080814+0000","last_fullsized":"2024-05-27T07:18:47.080814+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:18:47.464602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082783+0000","last_change":"2024-05-27T07:17:21.889199+0000","last_active":"2024-05-27T07:18:47.082783+0000","last_peered":"2024-05-27T07:18:47.082783+0000","last_clean":"2024-05-27T07:18:47.082783+0000","last_became_active":"2024-05-27T07:17:21.888667+0000","last_became_peered":"2024-05-27T07:17:21.888667+0000","last_unstale":"2024-05-27T07:18:47.082783+0000","last_undegraded":"2024-05-27T07:18:47.082783+0000","last_fullsized":"2024-05-27T07:18:47.082783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.029820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126884+0000","last_change":"2024-05-27T07:17:02.339872+0000","last_active":"2024-05-27T07:18:47.126884+0000","last_peered":"2024-05-27T07:18:47.126884+0000","last_clean":"2024-05-27T07:18:47.126884+0000","last_became_active":"2024-05-27T07:17:02.338975+0000","last_became_peered":"2024-05-27T07:17:02.338975+0000","last_unstale":"2024-05-27T07:18:47.126884+0000","last_undegraded":"2024-05-27T07:18:47.126884+0000","last_fullsized":"2024-05-27T07:18:47.126884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:14.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070756+0000","last_change":"2024-05-27T07:17:06.758234+0000","last_active":"2024-05-27T07:18:47.070756+0000","last_peered":"2024-05-27T07:18:47.070756+0000","last_clean":"2024-05-27T07:18:47.070756+0000","last_became_active":"2024-05-27T07:17:06.757827+0000","last_became_peered":"2024-05-27T07:17:06.757827+0000","last_unstale":"2024-05-27T07:18:47.070756+0000","last_undegraded":"2024-05-27T07:18:47.070756+0000","last_fullsized":"2024-05-27T07:18:47.070756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:21.688435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064658+0000","last_change":"2024-05-27T07:17:21.882540+0000","last_active":"2024-05-27T07:18:47.064658+0000","last_peered":"2024-05-27T07:18:47.064658+0000","last_clean":"2024-05-27T07:18:47.064658+0000","last_became_active":"2024-05-27T07:17:21.882087+0000","last_became_peered":"2024-05-27T07:17:21.882087+0000","last_unstale":"2024-05-27T07:18:47.064658+0000","last_undegraded":"2024-05-27T07:18:47.064658+0000","last_fullsized":"2024-05-27T07:18:47.064658+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:53:04.363521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:17.112948+0000","last_change":"2024-05-27T07:17:46.704997+0000","last_active":"2024-05-27T07:19:17.112948+0000","last_peered":"2024-05-27T07:19:17.112948+0000","last_clean":"2024-05-27T07:19:17.112948+0000","last_became_active":"2024-05-27T07:17:46.704355+0000","last_became_peered":"2024-05-27T07:17:46.704355+0000","last_unstale":"2024-05-27T07:19:17.112948+0000","last_undegraded":"2024-05-27T07:19:17.112948+0000","last_fullsized":"2024-05-27T07:19:17.112948+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:43:02.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126403+0000","last_change":"2024-05-27T07:17:06.765809+0000","last_active":"2024-05-27T07:18:47.126403+0000","last_peered":"2024-05-27T07:18:47.126403+0000","last_clean":"2024-05-27T07:18:47.126403+0000","last_became_active":"2024-05-27T07:17:06.763845+0000","last_became_peered":"2024-05-27T07:17:06.763845+0000","last_unstale":"2024-05-27T07:18:47.126403+0000","last_undegraded":"2024-05-27T07:18:47.126403+0000","last_fullsized":"2024-05-27T07:18:47.126403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:25.399181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064340+0000","last_change":"2024-05-27T07:17:21.886316+0000","last_active":"2024-05-27T07:18:47.064340+0000","last_peered":"2024-05-27T07:18:47.064340+0000","last_clean":"2024-05-27T07:18:47.064340+0000","last_became_active":"2024-05-27T07:17:21.885129+0000","last_became_peered":"2024-05-27T07:17:21.885129+0000","last_unstale":"2024-05-27T07:18:47.064340+0000","last_undegraded":"2024-05-27T07:18:47.064340+0000","last_fullsized":"2024-05-27T07:18:47.064340+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:10.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064271+0000","last_change":"2024-05-27T07:17:02.338765+0000","last_active":"2024-05-27T07:18:47.064271+0000","last_peered":"2024-05-27T07:18:47.064271+0000","last_clean":"2024-05-27T07:18:47.064271+0000","last_became_active":"2024-05-27T07:17:02.337103+0000","last_became_peered":"2024-05-27T07:17:02.337103+0000","last_unstale":"2024-05-27T07:18:47.064271+0000","last_undegraded":"2024-05-27T07:18:47.064271+0000","last_fullsized":"2024-05-27T07:18:47.064271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:15:07.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069437+0000","last_change":"2024-05-27T07:17:06.766839+0000","last_active":"2024-05-27T07:18:47.069437+0000","last_peered":"2024-05-27T07:18:47.069437+0000","last_clean":"2024-05-27T07:18:47.069437+0000","last_became_active":"2024-05-27T07:17:06.766373+0000","last_became_peered":"2024-05-27T07:17:06.766373+0000","last_unstale":"2024-05-27T07:18:47.069437+0000","last_undegraded":"2024-05-27T07:18:47.069437+0000","last_fullsized":"2024-05-27T07:18:47.069437+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:09:55.064045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133692+0000","last_change":"2024-05-27T07:18:47.133692+0000","last_active":"2024-05-27T07:18:47.133692+0000","last_peered":"2024-05-27T07:18:47.133692+0000","last_clean":"2024-05-27T07:18:47.133692+0000","last_became_active":"2024-05-27T07:18:47.132122+0000","last_became_peered":"2024-05-27T07:18:47.132122+0000","last_unstale":"2024-05-27T07:18:47.133692+0000","last_undegraded":"2024-05-27T07:18:47.133692+0000","last_fullsized":"2024-05-27T07:18:47.133692+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:57.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130197+0000","last_change":"2024-05-27T07:17:02.329809+0000","last_active":"2024-05-27T07:18:47.130197+0000","last_peered":"2024-05-27T07:18:47.130197+0000","last_clean":"2024-05-27T07:18:47.130197+0000","last_became_active":"2024-05-27T07:17:02.328758+0000","last_became_peered":"2024-05-27T07:17:02.328758+0000","last_unstale":"2024-05-27T07:18:47.130197+0000","last_undegraded":"2024-05-27T07:18:47.130197+0000","last_fullsized":"2024-05-27T07:18:47.130197+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:52.479444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125108+0000","last_change":"2024-05-27T07:17:06.749836+0000","last_active":"2024-05-27T07:18:47.125108+0000","last_peered":"2024-05-27T07:18:47.125108+0000","last_clean":"2024-05-27T07:18:47.125108+0000","last_became_active":"2024-05-27T07:17:06.749512+0000","last_became_peered":"2024-05-27T07:17:06.749512+0000","last_unstale":"2024-05-27T07:18:47.125108+0000","last_undegraded":"2024-05-27T07:18:47.125108+0000","last_fullsized":"2024-05-27T07:18:47.125108+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:41:51.000415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127796+0000","last_change":"2024-05-27T07:17:21.888376+0000","last_active":"2024-05-27T07:18:47.127796+0000","last_peered":"2024-05-27T07:18:47.127796+0000","last_clean":"2024-05-27T07:18:47.127796+0000","last_became_active":"2024-05-27T07:17:21.886183+0000","last_became_peered":"2024-05-27T07:17:21.886183+0000","last_unstale":"2024-05-27T07:18:47.127796+0000","last_undegraded":"2024-05-27T07:18:47.127796+0000","last_fullsized":"2024-05-27T07:18:47.127796+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:34.541338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069983+0000","last_change":"2024-05-27T07:17:02.334113+0000","last_active":"2024-05-27T07:18:47.069983+0000","last_peered":"2024-05-27T07:18:47.069983+0000","last_clean":"2024-05-27T07:18:47.069983+0000","last_became_active":"2024-05-27T07:17:02.333201+0000","last_became_peered":"2024-05-27T07:17:02.333201+0000","last_unstale":"2024-05-27T07:18:47.069983+0000","last_undegraded":"2024-05-27T07:18:47.069983+0000","last_fullsized":"2024-05-27T07:18:47.069983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:13.484654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.058837+0000","last_change":"2024-05-27T07:17:06.751676+0000","last_active":"2024-05-27T07:18:47.058837+0000","last_peered":"2024-05-27T07:18:47.058837+0000","last_clean":"2024-05-27T07:18:47.058837+0000","last_became_active":"2024-05-27T07:17:06.751221+0000","last_became_peered":"2024-05-27T07:17:06.751221+0000","last_unstale":"2024-05-27T07:18:47.058837+0000","last_undegraded":"2024-05-27T07:18:47.058837+0000","last_fullsized":"2024-05-27T07:18:47.058837+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:40.012561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125122+0000","last_change":"2024-05-27T07:17:21.897640+0000","last_active":"2024-05-27T07:18:47.125122+0000","last_peered":"2024-05-27T07:18:47.125122+0000","last_clean":"2024-05-27T07:18:47.125122+0000","last_became_active":"2024-05-27T07:17:21.895355+0000","last_became_peered":"2024-05-27T07:17:21.895355+0000","last_unstale":"2024-05-27T07:18:47.125122+0000","last_undegraded":"2024-05-27T07:18:47.125122+0000","last_fullsized":"2024-05-27T07:18:47.125122+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:51:12.030069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:17.111404+0000","last_change":"2024-05-27T07:17:02.339464+0000","last_active":"2024-05-27T07:19:17.111404+0000","last_peered":"2024-05-27T07:19:17.111404+0000","last_clean":"2024-05-27T07:19:17.111404+0000","last_became_active":"2024-05-27T07:17:02.338577+0000","last_became_peered":"2024-05-27T07:17:02.338577+0000","last_unstale":"2024-05-27T07:19:17.111404+0000","last_undegraded":"2024-05-27T07:19:17.111404+0000","last_fullsized":"2024-05-27T07:19:17.111404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:29:59.210157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.081292+0000","last_change":"2024-05-27T07:17:06.766883+0000","last_active":"2024-05-27T07:18:47.081292+0000","last_peered":"2024-05-27T07:18:47.081292+0000","last_clean":"2024-05-27T07:18:47.081292+0000","last_became_active":"2024-05-27T07:17:06.766166+0000","last_became_peered":"2024-05-27T07:17:06.766166+0000","last_unstale":"2024-05-27T07:18:47.081292+0000","last_undegraded":"2024-05-27T07:18:47.081292+0000","last_fullsized":"2024-05-27T07:18:47.081292+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:28.936499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.058804+0000","last_change":"2024-05-27T07:17:21.893930+0000","last_active":"2024-05-27T07:18:47.058804+0000","last_peered":"2024-05-27T07:18:47.058804+0000","last_clean":"2024-05-27T07:18:47.058804+0000","last_became_active":"2024-05-27T07:17:21.893029+0000","last_became_peered":"2024-05-27T07:17:21.893029+0000","last_unstale":"2024-05-27T07:18:47.058804+0000","last_undegraded":"2024-05-27T07:18:47.058804+0000","last_fullsized":"2024-05-27T07:18:47.058804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:32.453078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068295+0000","last_change":"2024-05-27T07:17:02.346531+0000","last_active":"2024-05-27T07:18:47.068295+0000","last_peered":"2024-05-27T07:18:47.068295+0000","last_clean":"2024-05-27T07:18:47.068295+0000","last_became_active":"2024-05-27T07:17:02.345844+0000","last_became_peered":"2024-05-27T07:17:02.345844+0000","last_unstale":"2024-05-27T07:18:47.068295+0000","last_undegraded":"2024-05-27T07:18:47.068295+0000","last_fullsized":"2024-05-27T07:18:47.068295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:28.544616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128038+0000","last_change":"2024-05-27T07:17:06.769845+0000","last_active":"2024-05-27T07:18:47.128038+0000","last_peered":"2024-05-27T07:18:47.128038+0000","last_clean":"2024-05-27T07:18:47.128038+0000","last_became_active":"2024-05-27T07:17:06.768467+0000","last_became_peered":"2024-05-27T07:17:06.768467+0000","last_unstale":"2024-05-27T07:18:47.128038+0000","last_undegraded":"2024-05-27T07:18:47.128038+0000","last_fullsized":"2024-05-27T07:18:47.128038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:30.311824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084012+0000","last_change":"2024-05-27T07:17:21.899885+0000","last_active":"2024-05-27T07:18:47.084012+0000","last_peered":"2024-05-27T07:18:47.084012+0000","last_clean":"2024-05-27T07:18:47.084012+0000","last_became_active":"2024-05-27T07:17:21.899340+0000","last_became_peered":"2024-05-27T07:17:21.899340+0000","last_unstale":"2024-05-27T07:18:47.084012+0000","last_undegraded":"2024-05-27T07:18:47.084012+0000","last_fullsized":"2024-05-27T07:18:47.084012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:12.665409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070628+0000","last_change":"2024-05-27T07:17:02.435770+0000","last_active":"2024-05-27T07:18:47.070628+0000","last_peered":"2024-05-27T07:18:47.070628+0000","last_clean":"2024-05-27T07:18:47.070628+0000","last_became_active":"2024-05-27T07:17:02.434535+0000","last_became_peered":"2024-05-27T07:17:02.434535+0000","last_unstale":"2024-05-27T07:18:47.070628+0000","last_undegraded":"2024-05-27T07:18:47.070628+0000","last_fullsized":"2024-05-27T07:18:47.070628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:53.745583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.081482+0000","last_change":"2024-05-27T07:17:46.704100+0000","last_active":"2024-05-27T07:18:47.081482+0000","last_peered":"2024-05-27T07:18:47.081482+0000","last_clean":"2024-05-27T07:18:47.081482+0000","last_became_active":"2024-05-27T07:17:46.703475+0000","last_became_peered":"2024-05-27T07:17:46.703475+0000","last_unstale":"2024-05-27T07:18:47.081482+0000","last_undegraded":"2024-05-27T07:18:47.081482+0000","last_fullsized":"2024-05-27T07:18:47.081482+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:20:29.173923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133397+0000","last_change":"2024-05-27T07:18:47.131521+0000","last_active":"2024-05-27T07:18:47.133397+0000","last_peered":"2024-05-27T07:18:47.133397+0000","last_clean":"2024-05-27T07:18:47.133397+0000","last_became_active":"2024-05-27T07:18:47.129995+0000","last_became_peered":"2024-05-27T07:18:47.129995+0000","last_unstale":"2024-05-27T07:18:47.133397+0000","last_undegraded":"2024-05-27T07:18:47.133397+0000","last_fullsized":"2024-05-27T07:18:47.133397+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:45:27.268768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":182,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:18.123247+0000","last_change":"2024-05-27T07:17:46.719293+0000","last_active":"2024-05-27T07:19:18.123247+0000","last_peered":"2024-05-27T07:19:18.123247+0000","last_clean":"2024-05-27T07:19:18.123247+0000","last_became_active":"2024-05-27T07:17:46.703496+0000","last_became_peered":"2024-05-27T07:17:46.703496+0000","last_unstale":"2024-05-27T07:19:18.123247+0000","last_undegraded":"2024-05-27T07:19:18.123247+0000","last_fullsized":"2024-05-27T07:19:18.123247+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:26.310524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62072,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":89,"num_read_kb":105,"num_write":34,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128691+0000","last_change":"2024-05-27T07:17:06.763141+0000","last_active":"2024-05-27T07:18:47.128691+0000","last_peered":"2024-05-27T07:18:47.128691+0000","last_clean":"2024-05-27T07:18:47.128691+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:18:47.128691+0000","last_undegraded":"2024-05-27T07:18:47.128691+0000","last_fullsized":"2024-05-27T07:18:47.128691+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:47.072552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071302+0000","last_change":"2024-05-27T07:17:21.895342+0000","last_active":"2024-05-27T07:18:47.071302+0000","last_peered":"2024-05-27T07:18:47.071302+0000","last_clean":"2024-05-27T07:18:47.071302+0000","last_became_active":"2024-05-27T07:17:21.894630+0000","last_became_peered":"2024-05-27T07:17:21.894630+0000","last_unstale":"2024-05-27T07:18:47.071302+0000","last_undegraded":"2024-05-27T07:18:47.071302+0000","last_fullsized":"2024-05-27T07:18:47.071302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:50.501683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062761+0000","last_change":"2024-05-27T07:17:02.341848+0000","last_active":"2024-05-27T07:18:47.062761+0000","last_peered":"2024-05-27T07:18:47.062761+0000","last_clean":"2024-05-27T07:18:47.062761+0000","last_became_active":"2024-05-27T07:17:02.339477+0000","last_became_peered":"2024-05-27T07:17:02.339477+0000","last_unstale":"2024-05-27T07:18:47.062761+0000","last_undegraded":"2024-05-27T07:18:47.062761+0000","last_fullsized":"2024-05-27T07:18:47.062761+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:23.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129062+0000","last_change":"2024-05-27T07:17:06.761340+0000","last_active":"2024-05-27T07:18:47.129062+0000","last_peered":"2024-05-27T07:18:47.129062+0000","last_clean":"2024-05-27T07:18:47.129062+0000","last_became_active":"2024-05-27T07:17:06.759103+0000","last_became_peered":"2024-05-27T07:17:06.759103+0000","last_unstale":"2024-05-27T07:18:47.129062+0000","last_undegraded":"2024-05-27T07:18:47.129062+0000","last_fullsized":"2024-05-27T07:18:47.129062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:42.263106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071318+0000","last_change":"2024-05-27T07:17:21.883252+0000","last_active":"2024-05-27T07:18:47.071318+0000","last_peered":"2024-05-27T07:18:47.071318+0000","last_clean":"2024-05-27T07:18:47.071318+0000","last_became_active":"2024-05-27T07:17:21.882468+0000","last_became_peered":"2024-05-27T07:17:21.882468+0000","last_unstale":"2024-05-27T07:18:47.071318+0000","last_undegraded":"2024-05-27T07:18:47.071318+0000","last_fullsized":"2024-05-27T07:18:47.071318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:59.250357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069605+0000","last_change":"2024-05-27T07:17:02.343343+0000","last_active":"2024-05-27T07:18:47.069605+0000","last_peered":"2024-05-27T07:18:47.069605+0000","last_clean":"2024-05-27T07:18:47.069605+0000","last_became_active":"2024-05-27T07:17:02.342741+0000","last_became_peered":"2024-05-27T07:17:02.342741+0000","last_unstale":"2024-05-27T07:18:47.069605+0000","last_undegraded":"2024-05-27T07:18:47.069605+0000","last_fullsized":"2024-05-27T07:18:47.069605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:46.006865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067181+0000","last_change":"2024-05-27T07:17:06.771473+0000","last_active":"2024-05-27T07:18:47.067181+0000","last_peered":"2024-05-27T07:18:47.067181+0000","last_clean":"2024-05-27T07:18:47.067181+0000","last_became_active":"2024-05-27T07:17:06.771082+0000","last_became_peered":"2024-05-27T07:17:06.771082+0000","last_unstale":"2024-05-27T07:18:47.067181+0000","last_undegraded":"2024-05-27T07:18:47.067181+0000","last_fullsized":"2024-05-27T07:18:47.067181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:01.371182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060572+0000","last_change":"2024-05-27T07:17:21.897691+0000","last_active":"2024-05-27T07:18:47.060572+0000","last_peered":"2024-05-27T07:18:47.060572+0000","last_clean":"2024-05-27T07:18:47.060572+0000","last_became_active":"2024-05-27T07:17:21.895221+0000","last_became_peered":"2024-05-27T07:17:21.895221+0000","last_unstale":"2024-05-27T07:18:47.060572+0000","last_undegraded":"2024-05-27T07:18:47.060572+0000","last_fullsized":"2024-05-27T07:18:47.060572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:14.626947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:17.112006+0000","last_change":"2024-05-27T07:17:02.343375+0000","last_active":"2024-05-27T07:19:17.112006+0000","last_peered":"2024-05-27T07:19:17.112006+0000","last_clean":"2024-05-27T07:19:17.112006+0000","last_became_active":"2024-05-27T07:17:02.341315+0000","last_became_peered":"2024-05-27T07:17:02.341315+0000","last_unstale":"2024-05-27T07:19:17.112006+0000","last_undegraded":"2024-05-27T07:19:17.112006+0000","last_fullsized":"2024-05-27T07:19:17.112006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:41.927265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065851+0000","last_change":"2024-05-27T07:17:06.765862+0000","last_active":"2024-05-27T07:18:47.065851+0000","last_peered":"2024-05-27T07:18:47.065851+0000","last_clean":"2024-05-27T07:18:47.065851+0000","last_became_active":"2024-05-27T07:17:06.765439+0000","last_became_peered":"2024-05-27T07:17:06.765439+0000","last_unstale":"2024-05-27T07:18:47.065851+0000","last_undegraded":"2024-05-27T07:18:47.065851+0000","last_fullsized":"2024-05-27T07:18:47.065851+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:47.586833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129937+0000","last_change":"2024-05-27T07:17:21.891810+0000","last_active":"2024-05-27T07:18:47.129937+0000","last_peered":"2024-05-27T07:18:47.129937+0000","last_clean":"2024-05-27T07:18:47.129937+0000","last_became_active":"2024-05-27T07:17:21.890473+0000","last_became_peered":"2024-05-27T07:17:21.890473+0000","last_unstale":"2024-05-27T07:18:47.129937+0000","last_undegraded":"2024-05-27T07:18:47.129937+0000","last_fullsized":"2024-05-27T07:18:47.129937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:12.731189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065778+0000","last_change":"2024-05-27T07:17:46.715870+0000","last_active":"2024-05-27T07:18:47.065778+0000","last_peered":"2024-05-27T07:18:47.065778+0000","last_clean":"2024-05-27T07:18:47.065778+0000","last_became_active":"2024-05-27T07:17:46.701248+0000","last_became_peered":"2024-05-27T07:17:46.701248+0000","last_unstale":"2024-05-27T07:18:47.065778+0000","last_undegraded":"2024-05-27T07:18:47.065778+0000","last_fullsized":"2024-05-27T07:18:47.065778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:18.693752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068792+0000","last_change":"2024-05-27T07:17:06.759937+0000","last_active":"2024-05-27T07:18:47.068792+0000","last_peered":"2024-05-27T07:18:47.068792+0000","last_clean":"2024-05-27T07:18:47.068792+0000","last_became_active":"2024-05-27T07:17:06.759526+0000","last_became_peered":"2024-05-27T07:17:06.759526+0000","last_unstale":"2024-05-27T07:18:47.068792+0000","last_undegraded":"2024-05-27T07:18:47.068792+0000","last_fullsized":"2024-05-27T07:18:47.068792+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:22.934317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068712+0000","last_change":"2024-05-27T07:17:21.898331+0000","last_active":"2024-05-27T07:18:47.068712+0000","last_peered":"2024-05-27T07:18:47.068712+0000","last_clean":"2024-05-27T07:18:47.068712+0000","last_became_active":"2024-05-27T07:17:21.897965+0000","last_became_peered":"2024-05-27T07:17:21.897965+0000","last_unstale":"2024-05-27T07:18:47.068712+0000","last_undegraded":"2024-05-27T07:18:47.068712+0000","last_fullsized":"2024-05-27T07:18:47.068712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:20.584175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128389+0000","last_change":"2024-05-27T07:17:02.346029+0000","last_active":"2024-05-27T07:18:47.128389+0000","last_peered":"2024-05-27T07:18:47.128389+0000","last_clean":"2024-05-27T07:18:47.128389+0000","last_became_active":"2024-05-27T07:17:02.341292+0000","last_became_peered":"2024-05-27T07:17:02.341292+0000","last_unstale":"2024-05-27T07:18:47.128389+0000","last_undegraded":"2024-05-27T07:18:47.128389+0000","last_fullsized":"2024-05-27T07:18:47.128389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:31.705741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071690+0000","last_change":"2024-05-27T07:17:46.691785+0000","last_active":"2024-05-27T07:18:47.071690+0000","last_peered":"2024-05-27T07:18:47.071690+0000","last_clean":"2024-05-27T07:18:47.071690+0000","last_became_active":"2024-05-27T07:17:46.690445+0000","last_became_peered":"2024-05-27T07:17:46.690445+0000","last_unstale":"2024-05-27T07:18:47.071690+0000","last_undegraded":"2024-05-27T07:18:47.071690+0000","last_fullsized":"2024-05-27T07:18:47.071690+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:59:27.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126733+0000","last_change":"2024-05-27T07:17:21.892434+0000","last_active":"2024-05-27T07:18:47.126733+0000","last_peered":"2024-05-27T07:18:47.126733+0000","last_clean":"2024-05-27T07:18:47.126733+0000","last_became_active":"2024-05-27T07:17:21.891736+0000","last_became_peered":"2024-05-27T07:17:21.891736+0000","last_unstale":"2024-05-27T07:18:47.126733+0000","last_undegraded":"2024-05-27T07:18:47.126733+0000","last_fullsized":"2024-05-27T07:18:47.126733+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:48.458819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059903+0000","last_change":"2024-05-27T07:17:02.436323+0000","last_active":"2024-05-27T07:18:47.059903+0000","last_peered":"2024-05-27T07:18:47.059903+0000","last_clean":"2024-05-27T07:18:47.059903+0000","last_became_active":"2024-05-27T07:17:02.435000+0000","last_became_peered":"2024-05-27T07:17:02.435000+0000","last_unstale":"2024-05-27T07:18:47.059903+0000","last_undegraded":"2024-05-27T07:18:47.059903+0000","last_fullsized":"2024-05-27T07:18:47.059903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:08.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065831+0000","last_change":"2024-05-27T07:17:06.760418+0000","last_active":"2024-05-27T07:18:47.065831+0000","last_peered":"2024-05-27T07:18:47.065831+0000","last_clean":"2024-05-27T07:18:47.065831+0000","last_became_active":"2024-05-27T07:17:06.759875+0000","last_became_peered":"2024-05-27T07:17:06.759875+0000","last_unstale":"2024-05-27T07:18:47.065831+0000","last_undegraded":"2024-05-27T07:18:47.065831+0000","last_fullsized":"2024-05-27T07:18:47.065831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:12:54.324323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130370+0000","last_change":"2024-05-27T07:17:21.888048+0000","last_active":"2024-05-27T07:18:47.130370+0000","last_peered":"2024-05-27T07:18:47.130370+0000","last_clean":"2024-05-27T07:18:47.130370+0000","last_became_active":"2024-05-27T07:17:21.887579+0000","last_became_peered":"2024-05-27T07:17:21.887579+0000","last_unstale":"2024-05-27T07:18:47.130370+0000","last_undegraded":"2024-05-27T07:18:47.130370+0000","last_fullsized":"2024-05-27T07:18:47.130370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:34:44.569959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066518+0000","last_change":"2024-05-27T07:17:02.337655+0000","last_active":"2024-05-27T07:18:47.066518+0000","last_peered":"2024-05-27T07:18:47.066518+0000","last_clean":"2024-05-27T07:18:47.066518+0000","last_became_active":"2024-05-27T07:17:02.337252+0000","last_became_peered":"2024-05-27T07:17:02.337252+0000","last_unstale":"2024-05-27T07:18:47.066518+0000","last_undegraded":"2024-05-27T07:18:47.066518+0000","last_fullsized":"2024-05-27T07:18:47.066518+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:11:25.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063967+0000","last_change":"2024-05-27T07:17:06.762934+0000","last_active":"2024-05-27T07:18:47.063967+0000","last_peered":"2024-05-27T07:18:47.063967+0000","last_clean":"2024-05-27T07:18:47.063967+0000","last_became_active":"2024-05-27T07:17:06.762507+0000","last_became_peered":"2024-05-27T07:17:06.762507+0000","last_unstale":"2024-05-27T07:18:47.063967+0000","last_undegraded":"2024-05-27T07:18:47.063967+0000","last_fullsized":"2024-05-27T07:18:47.063967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:23:35.164892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129885+0000","last_change":"2024-05-27T07:17:21.886991+0000","last_active":"2024-05-27T07:18:47.129885+0000","last_peered":"2024-05-27T07:18:47.129885+0000","last_clean":"2024-05-27T07:18:47.129885+0000","last_became_active":"2024-05-27T07:17:21.885945+0000","last_became_peered":"2024-05-27T07:17:21.885945+0000","last_unstale":"2024-05-27T07:18:47.129885+0000","last_undegraded":"2024-05-27T07:18:47.129885+0000","last_fullsized":"2024-05-27T07:18:47.129885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:23:01.853601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070506+0000","last_change":"2024-05-27T07:17:46.864130+0000","last_active":"2024-05-27T07:18:47.070506+0000","last_peered":"2024-05-27T07:18:47.070506+0000","last_clean":"2024-05-27T07:18:47.070506+0000","last_became_active":"2024-05-27T07:17:46.862997+0000","last_became_peered":"2024-05-27T07:17:46.862997+0000","last_unstale":"2024-05-27T07:18:47.070506+0000","last_undegraded":"2024-05-27T07:18:47.070506+0000","last_fullsized":"2024-05-27T07:18:47.070506+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:10.594627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:16.107765+0000","last_change":"2024-05-27T07:17:02.435463+0000","last_active":"2024-05-27T07:19:16.107765+0000","last_peered":"2024-05-27T07:19:16.107765+0000","last_clean":"2024-05-27T07:19:16.107765+0000","last_became_active":"2024-05-27T07:17:02.432131+0000","last_became_peered":"2024-05-27T07:17:02.432131+0000","last_unstale":"2024-05-27T07:19:16.107765+0000","last_undegraded":"2024-05-27T07:19:16.107765+0000","last_fullsized":"2024-05-27T07:19:16.107765+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:41.874577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.135438+0000","last_change":"2024-05-27T07:18:47.134043+0000","last_active":"2024-05-27T07:18:47.135438+0000","last_peered":"2024-05-27T07:18:47.135438+0000","last_clean":"2024-05-27T07:18:47.135438+0000","last_became_active":"2024-05-27T07:18:47.132050+0000","last_became_peered":"2024-05-27T07:18:47.132050+0000","last_unstale":"2024-05-27T07:18:47.135438+0000","last_undegraded":"2024-05-27T07:18:47.135438+0000","last_fullsized":"2024-05-27T07:18:47.135438+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:12.170883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125310+0000","last_change":"2024-05-27T07:17:06.751168+0000","last_active":"2024-05-27T07:18:47.125310+0000","last_peered":"2024-05-27T07:18:47.125310+0000","last_clean":"2024-05-27T07:18:47.125310+0000","last_became_active":"2024-05-27T07:17:06.750816+0000","last_became_peered":"2024-05-27T07:17:06.750816+0000","last_unstale":"2024-05-27T07:18:47.125310+0000","last_undegraded":"2024-05-27T07:18:47.125310+0000","last_fullsized":"2024-05-27T07:18:47.125310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:33:45.607426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083905+0000","last_change":"2024-05-27T07:17:02.337490+0000","last_active":"2024-05-27T07:18:47.083905+0000","last_peered":"2024-05-27T07:18:47.083905+0000","last_clean":"2024-05-27T07:18:47.083905+0000","last_became_active":"2024-05-27T07:17:02.335317+0000","last_became_peered":"2024-05-27T07:17:02.335317+0000","last_unstale":"2024-05-27T07:18:47.083905+0000","last_undegraded":"2024-05-27T07:18:47.083905+0000","last_fullsized":"2024-05-27T07:18:47.083905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:33:46.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065010+0000","last_change":"2024-05-27T07:15:10.825312+0000","last_active":"2024-05-27T07:18:47.065010+0000","last_peered":"2024-05-27T07:18:47.065010+0000","last_clean":"2024-05-27T07:18:47.065010+0000","last_became_active":"2024-05-27T07:15:10.809245+0000","last_became_peered":"2024-05-27T07:15:10.809245+0000","last_unstale":"2024-05-27T07:18:47.065010+0000","last_undegraded":"2024-05-27T07:18:47.065010+0000","last_fullsized":"2024-05-27T07:18:47.065010+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_clean_scrub_stamp":"2024-05-27T07:10:08.008565+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:32.769372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133948+0000","last_change":"2024-05-27T07:18:47.133948+0000","last_active":"2024-05-27T07:18:47.133948+0000","last_peered":"2024-05-27T07:18:47.133948+0000","last_clean":"2024-05-27T07:18:47.133948+0000","last_became_active":"2024-05-27T07:18:47.132566+0000","last_became_peered":"2024-05-27T07:18:47.132566+0000","last_unstale":"2024-05-27T07:18:47.133948+0000","last_undegraded":"2024-05-27T07:18:47.133948+0000","last_fullsized":"2024-05-27T07:18:47.133948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:04.463121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128885+0000","last_change":"2024-05-27T07:17:06.766187+0000","last_active":"2024-05-27T07:18:47.128885+0000","last_peered":"2024-05-27T07:18:47.128885+0000","last_clean":"2024-05-27T07:18:47.128885+0000","last_became_active":"2024-05-27T07:17:06.763365+0000","last_became_peered":"2024-05-27T07:17:06.763365+0000","last_unstale":"2024-05-27T07:18:47.128885+0000","last_undegraded":"2024-05-27T07:18:47.128885+0000","last_fullsized":"2024-05-27T07:18:47.128885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:13.668426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065681+0000","last_change":"2024-05-27T07:17:02.344452+0000","last_active":"2024-05-27T07:18:47.065681+0000","last_peered":"2024-05-27T07:18:47.065681+0000","last_clean":"2024-05-27T07:18:47.065681+0000","last_became_active":"2024-05-27T07:17:02.343885+0000","last_became_peered":"2024-05-27T07:17:02.343885+0000","last_unstale":"2024-05-27T07:18:47.065681+0000","last_undegraded":"2024-05-27T07:18:47.065681+0000","last_fullsized":"2024-05-27T07:18:47.065681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:26.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072458+0000","last_change":"2024-05-27T07:17:21.883024+0000","last_active":"2024-05-27T07:18:47.072458+0000","last_peered":"2024-05-27T07:18:47.072458+0000","last_clean":"2024-05-27T07:18:47.072458+0000","last_became_active":"2024-05-27T07:17:21.882429+0000","last_became_peered":"2024-05-27T07:17:21.882429+0000","last_unstale":"2024-05-27T07:18:47.072458+0000","last_undegraded":"2024-05-27T07:18:47.072458+0000","last_fullsized":"2024-05-27T07:18:47.072458+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:36:40.443537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128649+0000","last_change":"2024-05-27T07:17:02.432768+0000","last_active":"2024-05-27T07:18:47.128649+0000","last_peered":"2024-05-27T07:18:47.128649+0000","last_clean":"2024-05-27T07:18:47.128649+0000","last_became_active":"2024-05-27T07:17:02.431551+0000","last_became_peered":"2024-05-27T07:17:02.431551+0000","last_unstale":"2024-05-27T07:18:47.128649+0000","last_undegraded":"2024-05-27T07:18:47.128649+0000","last_fullsized":"2024-05-27T07:18:47.128649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:39.759512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130801+0000","last_change":"2024-05-27T07:17:06.770992+0000","last_active":"2024-05-27T07:18:47.130801+0000","last_peered":"2024-05-27T07:18:47.130801+0000","last_clean":"2024-05-27T07:18:47.130801+0000","last_became_active":"2024-05-27T07:17:06.770560+0000","last_became_peered":"2024-05-27T07:17:06.770560+0000","last_unstale":"2024-05-27T07:18:47.130801+0000","last_undegraded":"2024-05-27T07:18:47.130801+0000","last_fullsized":"2024-05-27T07:18:47.130801+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:07:34.443551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.124920+0000","last_change":"2024-05-27T07:17:21.895362+0000","last_active":"2024-05-27T07:18:47.124920+0000","last_peered":"2024-05-27T07:18:47.124920+0000","last_clean":"2024-05-27T07:18:47.124920+0000","last_became_active":"2024-05-27T07:17:21.894563+0000","last_became_peered":"2024-05-27T07:17:21.894563+0000","last_unstale":"2024-05-27T07:18:47.124920+0000","last_undegraded":"2024-05-27T07:18:47.124920+0000","last_fullsized":"2024-05-27T07:18:47.124920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:44.394301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:09.826352+0000","last_change":"2024-05-27T07:17:02.338203+0000","last_active":"2024-05-27T07:19:09.826352+0000","last_peered":"2024-05-27T07:19:09.826352+0000","last_clean":"2024-05-27T07:19:09.826352+0000","last_became_active":"2024-05-27T07:17:02.336192+0000","last_became_peered":"2024-05-27T07:17:02.336192+0000","last_unstale":"2024-05-27T07:19:09.826352+0000","last_undegraded":"2024-05-27T07:19:09.826352+0000","last_fullsized":"2024-05-27T07:19:09.826352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:48.187517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126382+0000","last_change":"2024-05-27T07:17:06.759729+0000","last_active":"2024-05-27T07:18:47.126382+0000","last_peered":"2024-05-27T07:18:47.126382+0000","last_clean":"2024-05-27T07:18:47.126382+0000","last_became_active":"2024-05-27T07:17:06.756766+0000","last_became_peered":"2024-05-27T07:17:06.756766+0000","last_unstale":"2024-05-27T07:18:47.126382+0000","last_undegraded":"2024-05-27T07:18:47.126382+0000","last_fullsized":"2024-05-27T07:18:47.126382+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:01.090586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062921+0000","last_change":"2024-05-27T07:17:21.878487+0000","last_active":"2024-05-27T07:18:47.062921+0000","last_peered":"2024-05-27T07:18:47.062921+0000","last_clean":"2024-05-27T07:18:47.062921+0000","last_became_active":"2024-05-27T07:17:21.877623+0000","last_became_peered":"2024-05-27T07:17:21.877623+0000","last_unstale":"2024-05-27T07:18:47.062921+0000","last_undegraded":"2024-05-27T07:18:47.062921+0000","last_fullsized":"2024-05-27T07:18:47.062921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:28:52.618738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.085169+0000","last_change":"2024-05-27T07:17:02.486495+0000","last_active":"2024-05-27T07:18:47.085169+0000","last_peered":"2024-05-27T07:18:47.085169+0000","last_clean":"2024-05-27T07:18:47.085169+0000","last_became_active":"2024-05-27T07:17:02.485493+0000","last_became_peered":"2024-05-27T07:17:02.485493+0000","last_unstale":"2024-05-27T07:18:47.085169+0000","last_undegraded":"2024-05-27T07:18:47.085169+0000","last_fullsized":"2024-05-27T07:18:47.085169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:31.977210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128123+0000","last_change":"2024-05-27T07:17:06.760937+0000","last_active":"2024-05-27T07:18:47.128123+0000","last_peered":"2024-05-27T07:18:47.128123+0000","last_clean":"2024-05-27T07:18:47.128123+0000","last_became_active":"2024-05-27T07:17:06.760254+0000","last_became_peered":"2024-05-27T07:17:06.760254+0000","last_unstale":"2024-05-27T07:18:47.128123+0000","last_undegraded":"2024-05-27T07:18:47.128123+0000","last_fullsized":"2024-05-27T07:18:47.128123+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:50.147319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084592+0000","last_change":"2024-05-27T07:17:21.901511+0000","last_active":"2024-05-27T07:18:47.084592+0000","last_peered":"2024-05-27T07:18:47.084592+0000","last_clean":"2024-05-27T07:18:47.084592+0000","last_became_active":"2024-05-27T07:17:21.899847+0000","last_became_peered":"2024-05-27T07:17:21.899847+0000","last_unstale":"2024-05-27T07:18:47.084592+0000","last_undegraded":"2024-05-27T07:18:47.084592+0000","last_fullsized":"2024-05-27T07:18:47.084592+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:28.133509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080859+0000","last_change":"2024-05-27T07:17:02.432520+0000","last_active":"2024-05-27T07:18:47.080859+0000","last_peered":"2024-05-27T07:18:47.080859+0000","last_clean":"2024-05-27T07:18:47.080859+0000","last_became_active":"2024-05-27T07:17:02.431563+0000","last_became_peered":"2024-05-27T07:17:02.431563+0000","last_unstale":"2024-05-27T07:18:47.080859+0000","last_undegraded":"2024-05-27T07:18:47.080859+0000","last_fullsized":"2024-05-27T07:18:47.080859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:17:28.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125793+0000","last_change":"2024-05-27T07:17:06.754408+0000","last_active":"2024-05-27T07:18:47.125793+0000","last_peered":"2024-05-27T07:18:47.125793+0000","last_clean":"2024-05-27T07:18:47.125793+0000","last_became_active":"2024-05-27T07:17:06.753787+0000","last_became_peered":"2024-05-27T07:17:06.753787+0000","last_unstale":"2024-05-27T07:18:47.125793+0000","last_undegraded":"2024-05-27T07:18:47.125793+0000","last_fullsized":"2024-05-27T07:18:47.125793+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:21:50.461296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060353+0000","last_change":"2024-05-27T07:17:21.895277+0000","last_active":"2024-05-27T07:18:47.060353+0000","last_peered":"2024-05-27T07:18:47.060353+0000","last_clean":"2024-05-27T07:18:47.060353+0000","last_became_active":"2024-05-27T07:17:21.894518+0000","last_became_peered":"2024-05-27T07:17:21.894518+0000","last_unstale":"2024-05-27T07:18:47.060353+0000","last_undegraded":"2024-05-27T07:18:47.060353+0000","last_fullsized":"2024-05-27T07:18:47.060353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:02.008209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125540+0000","last_change":"2024-05-27T07:17:02.343604+0000","last_active":"2024-05-27T07:18:47.125540+0000","last_peered":"2024-05-27T07:18:47.125540+0000","last_clean":"2024-05-27T07:18:47.125540+0000","last_became_active":"2024-05-27T07:17:02.341890+0000","last_became_peered":"2024-05-27T07:17:02.341890+0000","last_unstale":"2024-05-27T07:18:47.125540+0000","last_undegraded":"2024-05-27T07:18:47.125540+0000","last_fullsized":"2024-05-27T07:18:47.125540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:10:21.670193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067371+0000","last_change":"2024-05-27T07:17:06.754767+0000","last_active":"2024-05-27T07:18:47.067371+0000","last_peered":"2024-05-27T07:18:47.067371+0000","last_clean":"2024-05-27T07:18:47.067371+0000","last_became_active":"2024-05-27T07:17:06.754291+0000","last_became_peered":"2024-05-27T07:17:06.754291+0000","last_unstale":"2024-05-27T07:18:47.067371+0000","last_undegraded":"2024-05-27T07:18:47.067371+0000","last_fullsized":"2024-05-27T07:18:47.067371+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:46:48.145874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084413+0000","last_change":"2024-05-27T07:17:21.892342+0000","last_active":"2024-05-27T07:18:47.084413+0000","last_peered":"2024-05-27T07:18:47.084413+0000","last_clean":"2024-05-27T07:18:47.084413+0000","last_became_active":"2024-05-27T07:17:21.891393+0000","last_became_peered":"2024-05-27T07:17:21.891393+0000","last_unstale":"2024-05-27T07:18:47.084413+0000","last_undegraded":"2024-05-27T07:18:47.084413+0000","last_fullsized":"2024-05-27T07:18:47.084413+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:30.510616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062370+0000","last_change":"2024-05-27T07:17:02.340238+0000","last_active":"2024-05-27T07:18:47.062370+0000","last_peered":"2024-05-27T07:18:47.062370+0000","last_clean":"2024-05-27T07:18:47.062370+0000","last_became_active":"2024-05-27T07:17:02.339561+0000","last_became_peered":"2024-05-27T07:17:02.339561+0000","last_unstale":"2024-05-27T07:18:47.062370+0000","last_undegraded":"2024-05-27T07:18:47.062370+0000","last_fullsized":"2024-05-27T07:18:47.062370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:32.451397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125858+0000","last_change":"2024-05-27T07:17:06.769397+0000","last_active":"2024-05-27T07:18:47.125858+0000","last_peered":"2024-05-27T07:18:47.125858+0000","last_clean":"2024-05-27T07:18:47.125858+0000","last_became_active":"2024-05-27T07:17:06.768936+0000","last_became_peered":"2024-05-27T07:17:06.768936+0000","last_unstale":"2024-05-27T07:18:47.125858+0000","last_undegraded":"2024-05-27T07:18:47.125858+0000","last_fullsized":"2024-05-27T07:18:47.125858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:52.767902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084314+0000","last_change":"2024-05-27T07:17:21.894225+0000","last_active":"2024-05-27T07:18:47.084314+0000","last_peered":"2024-05-27T07:18:47.084314+0000","last_clean":"2024-05-27T07:18:47.084314+0000","last_became_active":"2024-05-27T07:17:21.893499+0000","last_became_peered":"2024-05-27T07:17:21.893499+0000","last_unstale":"2024-05-27T07:18:47.084314+0000","last_undegraded":"2024-05-27T07:18:47.084314+0000","last_fullsized":"2024-05-27T07:18:47.084314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:54.985290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126297+0000","last_change":"2024-05-27T07:17:02.347736+0000","last_active":"2024-05-27T07:18:47.126297+0000","last_peered":"2024-05-27T07:18:47.126297+0000","last_clean":"2024-05-27T07:18:47.126297+0000","last_became_active":"2024-05-27T07:17:02.347237+0000","last_became_peered":"2024-05-27T07:17:02.347237+0000","last_unstale":"2024-05-27T07:18:47.126297+0000","last_undegraded":"2024-05-27T07:18:47.126297+0000","last_fullsized":"2024-05-27T07:18:47.126297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:36:15.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066531+0000","last_change":"2024-05-27T07:17:06.759348+0000","last_active":"2024-05-27T07:18:47.066531+0000","last_peered":"2024-05-27T07:18:47.066531+0000","last_clean":"2024-05-27T07:18:47.066531+0000","last_became_active":"2024-05-27T07:17:06.758967+0000","last_became_peered":"2024-05-27T07:17:06.758967+0000","last_unstale":"2024-05-27T07:18:47.066531+0000","last_undegraded":"2024-05-27T07:18:47.066531+0000","last_fullsized":"2024-05-27T07:18:47.066531+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:45.430719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080436+0000","last_change":"2024-05-27T07:17:21.882919+0000","last_active":"2024-05-27T07:18:47.080436+0000","last_peered":"2024-05-27T07:18:47.080436+0000","last_clean":"2024-05-27T07:18:47.080436+0000","last_became_active":"2024-05-27T07:17:21.882330+0000","last_became_peered":"2024-05-27T07:17:21.882330+0000","last_unstale":"2024-05-27T07:18:47.080436+0000","last_undegraded":"2024-05-27T07:18:47.080436+0000","last_fullsized":"2024-05-27T07:18:47.080436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:51.409079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064865+0000","last_change":"2024-05-27T07:17:02.486099+0000","last_active":"2024-05-27T07:18:47.064865+0000","last_peered":"2024-05-27T07:18:47.064865+0000","last_clean":"2024-05-27T07:18:47.064865+0000","last_became_active":"2024-05-27T07:17:02.485116+0000","last_became_peered":"2024-05-27T07:17:02.485116+0000","last_unstale":"2024-05-27T07:18:47.064865+0000","last_undegraded":"2024-05-27T07:18:47.064865+0000","last_fullsized":"2024-05-27T07:18:47.064865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:36.338287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067048+0000","last_change":"2024-05-27T07:17:06.773586+0000","last_active":"2024-05-27T07:18:47.067048+0000","last_peered":"2024-05-27T07:18:47.067048+0000","last_clean":"2024-05-27T07:18:47.067048+0000","last_became_active":"2024-05-27T07:17:06.773139+0000","last_became_peered":"2024-05-27T07:17:06.773139+0000","last_unstale":"2024-05-27T07:18:47.067048+0000","last_undegraded":"2024-05-27T07:18:47.067048+0000","last_fullsized":"2024-05-27T07:18:47.067048+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:19:36.373091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127421+0000","last_change":"2024-05-27T07:17:21.892351+0000","last_active":"2024-05-27T07:18:47.127421+0000","last_peered":"2024-05-27T07:18:47.127421+0000","last_clean":"2024-05-27T07:18:47.127421+0000","last_became_active":"2024-05-27T07:17:21.891862+0000","last_became_peered":"2024-05-27T07:17:21.891862+0000","last_unstale":"2024-05-27T07:18:47.127421+0000","last_undegraded":"2024-05-27T07:18:47.127421+0000","last_fullsized":"2024-05-27T07:18:47.127421+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:50:18.193149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071128+0000","last_change":"2024-05-27T07:17:02.336917+0000","last_active":"2024-05-27T07:18:47.071128+0000","last_peered":"2024-05-27T07:18:47.071128+0000","last_clean":"2024-05-27T07:18:47.071128+0000","last_became_active":"2024-05-27T07:17:02.335836+0000","last_became_peered":"2024-05-27T07:17:02.335836+0000","last_unstale":"2024-05-27T07:18:47.071128+0000","last_undegraded":"2024-05-27T07:18:47.071128+0000","last_fullsized":"2024-05-27T07:18:47.071128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:54.109480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067770+0000","last_change":"2024-05-27T07:17:46.705826+0000","last_active":"2024-05-27T07:18:47.067770+0000","last_peered":"2024-05-27T07:18:47.067770+0000","last_clean":"2024-05-27T07:18:47.067770+0000","last_became_active":"2024-05-27T07:17:46.704749+0000","last_became_peered":"2024-05-27T07:17:46.704749+0000","last_unstale":"2024-05-27T07:18:47.067770+0000","last_undegraded":"2024-05-27T07:18:47.067770+0000","last_fullsized":"2024-05-27T07:18:47.067770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:26.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.067984+0000","last_change":"2024-05-27T07:17:21.885535+0000","last_active":"2024-05-27T07:18:47.067984+0000","last_peered":"2024-05-27T07:18:47.067984+0000","last_clean":"2024-05-27T07:18:47.067984+0000","last_became_active":"2024-05-27T07:17:21.884118+0000","last_became_peered":"2024-05-27T07:17:21.884118+0000","last_unstale":"2024-05-27T07:18:47.067984+0000","last_undegraded":"2024-05-27T07:18:47.067984+0000","last_fullsized":"2024-05-27T07:18:47.067984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:26.600199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130129+0000","last_change":"2024-05-27T07:17:02.340037+0000","last_active":"2024-05-27T07:18:47.130129+0000","last_peered":"2024-05-27T07:18:47.130129+0000","last_clean":"2024-05-27T07:18:47.130129+0000","last_became_active":"2024-05-27T07:17:02.339378+0000","last_became_peered":"2024-05-27T07:17:02.339378+0000","last_unstale":"2024-05-27T07:18:47.130129+0000","last_undegraded":"2024-05-27T07:18:47.130129+0000","last_fullsized":"2024-05-27T07:18:47.130129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:33.774917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068712+0000","last_change":"2024-05-27T07:17:06.763277+0000","last_active":"2024-05-27T07:18:47.068712+0000","last_peered":"2024-05-27T07:18:47.068712+0000","last_clean":"2024-05-27T07:18:47.068712+0000","last_became_active":"2024-05-27T07:17:06.762837+0000","last_became_peered":"2024-05-27T07:17:06.762837+0000","last_unstale":"2024-05-27T07:18:47.068712+0000","last_undegraded":"2024-05-27T07:18:47.068712+0000","last_fullsized":"2024-05-27T07:18:47.068712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:52.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066992+0000","last_change":"2024-05-27T07:17:06.752556+0000","last_active":"2024-05-27T07:18:47.066992+0000","last_peered":"2024-05-27T07:18:47.066992+0000","last_clean":"2024-05-27T07:18:47.066992+0000","last_became_active":"2024-05-27T07:17:06.751085+0000","last_became_peered":"2024-05-27T07:17:06.751085+0000","last_unstale":"2024-05-27T07:18:47.066992+0000","last_undegraded":"2024-05-27T07:18:47.066992+0000","last_fullsized":"2024-05-27T07:18:47.066992+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:13:53.883845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064206+0000","last_change":"2024-05-27T07:17:06.753674+0000","last_active":"2024-05-27T07:18:47.064206+0000","last_peered":"2024-05-27T07:18:47.064206+0000","last_clean":"2024-05-27T07:18:47.064206+0000","last_became_active":"2024-05-27T07:17:06.752373+0000","last_became_peered":"2024-05-27T07:17:06.752373+0000","last_unstale":"2024-05-27T07:18:47.064206+0000","last_undegraded":"2024-05-27T07:18:47.064206+0000","last_fullsized":"2024-05-27T07:18:47.064206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:40:17.474462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062007+0000","last_change":"2024-05-27T07:17:06.766796+0000","last_active":"2024-05-27T07:18:47.062007+0000","last_peered":"2024-05-27T07:18:47.062007+0000","last_clean":"2024-05-27T07:18:47.062007+0000","last_became_active":"2024-05-27T07:17:06.766435+0000","last_became_peered":"2024-05-27T07:17:06.766435+0000","last_unstale":"2024-05-27T07:18:47.062007+0000","last_undegraded":"2024-05-27T07:18:47.062007+0000","last_fullsized":"2024-05-27T07:18:47.062007+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:38.747810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128025+0000","last_change":"2024-05-27T07:17:06.762804+0000","last_active":"2024-05-27T07:18:47.128025+0000","last_peered":"2024-05-27T07:18:47.128025+0000","last_clean":"2024-05-27T07:18:47.128025+0000","last_became_active":"2024-05-27T07:17:06.762490+0000","last_became_peered":"2024-05-27T07:17:06.762490+0000","last_unstale":"2024-05-27T07:18:47.128025+0000","last_undegraded":"2024-05-27T07:18:47.128025+0000","last_fullsized":"2024-05-27T07:18:47.128025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:59:33.584848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130804+0000","last_change":"2024-05-27T07:17:06.765640+0000","last_active":"2024-05-27T07:18:47.130804+0000","last_peered":"2024-05-27T07:18:47.130804+0000","last_clean":"2024-05-27T07:18:47.130804+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:18:47.130804+0000","last_undegraded":"2024-05-27T07:18:47.130804+0000","last_fullsized":"2024-05-27T07:18:47.130804+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:19.695007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072866+0000","last_change":"2024-05-27T07:17:06.759982+0000","last_active":"2024-05-27T07:18:47.072866+0000","last_peered":"2024-05-27T07:18:47.072866+0000","last_clean":"2024-05-27T07:18:47.072866+0000","last_became_active":"2024-05-27T07:17:06.759367+0000","last_became_peered":"2024-05-27T07:17:06.759367+0000","last_unstale":"2024-05-27T07:18:47.072866+0000","last_undegraded":"2024-05-27T07:18:47.072866+0000","last_fullsized":"2024-05-27T07:18:47.072866+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:35:04.686989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066038+0000","last_change":"2024-05-27T07:17:06.768264+0000","last_active":"2024-05-27T07:18:47.066038+0000","last_peered":"2024-05-27T07:18:47.066038+0000","last_clean":"2024-05-27T07:18:47.066038+0000","last_became_active":"2024-05-27T07:17:06.767952+0000","last_became_peered":"2024-05-27T07:17:06.767952+0000","last_unstale":"2024-05-27T07:18:47.066038+0000","last_undegraded":"2024-05-27T07:18:47.066038+0000","last_fullsized":"2024-05-27T07:18:47.066038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:34.867113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059291+0000","last_change":"2024-05-27T07:17:46.704705+0000","last_active":"2024-05-27T07:18:47.059291+0000","last_peered":"2024-05-27T07:18:47.059291+0000","last_clean":"2024-05-27T07:18:47.059291+0000","last_became_active":"2024-05-27T07:17:46.703503+0000","last_became_peered":"2024-05-27T07:17:46.703503+0000","last_unstale":"2024-05-27T07:18:47.059291+0000","last_undegraded":"2024-05-27T07:18:47.059291+0000","last_fullsized":"2024-05-27T07:18:47.059291+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:39.975009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069503+0000","last_change":"2024-05-27T07:17:06.752080+0000","last_active":"2024-05-27T07:18:47.069503+0000","last_peered":"2024-05-27T07:18:47.069503+0000","last_clean":"2024-05-27T07:18:47.069503+0000","last_became_active":"2024-05-27T07:17:06.750486+0000","last_became_peered":"2024-05-27T07:17:06.750486+0000","last_unstale":"2024-05-27T07:18:47.069503+0000","last_undegraded":"2024-05-27T07:18:47.069503+0000","last_fullsized":"2024-05-27T07:18:47.069503+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:12.461727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125600+0000","last_change":"2024-05-27T07:17:06.759200+0000","last_active":"2024-05-27T07:18:47.125600+0000","last_peered":"2024-05-27T07:18:47.125600+0000","last_clean":"2024-05-27T07:18:47.125600+0000","last_became_active":"2024-05-27T07:17:06.758819+0000","last_became_peered":"2024-05-27T07:17:06.758819+0000","last_unstale":"2024-05-27T07:18:47.125600+0000","last_undegraded":"2024-05-27T07:18:47.125600+0000","last_fullsized":"2024-05-27T07:18:47.125600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:00.274229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080876+0000","last_change":"2024-05-27T07:17:06.760412+0000","last_active":"2024-05-27T07:18:47.080876+0000","last_peered":"2024-05-27T07:18:47.080876+0000","last_clean":"2024-05-27T07:18:47.080876+0000","last_became_active":"2024-05-27T07:17:06.759949+0000","last_became_peered":"2024-05-27T07:17:06.759949+0000","last_unstale":"2024-05-27T07:18:47.080876+0000","last_undegraded":"2024-05-27T07:18:47.080876+0000","last_fullsized":"2024-05-27T07:18:47.080876+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.235488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129710+0000","last_change":"2024-05-27T07:17:06.775371+0000","last_active":"2024-05-27T07:18:47.129710+0000","last_peered":"2024-05-27T07:18:47.129710+0000","last_clean":"2024-05-27T07:18:47.129710+0000","last_became_active":"2024-05-27T07:17:06.774904+0000","last_became_peered":"2024-05-27T07:17:06.774904+0000","last_unstale":"2024-05-27T07:18:47.129710+0000","last_undegraded":"2024-05-27T07:18:47.129710+0000","last_fullsized":"2024-05-27T07:18:47.129710+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:22:13.484733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072043+0000","last_change":"2024-05-27T07:17:06.757066+0000","last_active":"2024-05-27T07:18:47.072043+0000","last_peered":"2024-05-27T07:18:47.072043+0000","last_clean":"2024-05-27T07:18:47.072043+0000","last_became_active":"2024-05-27T07:17:06.756316+0000","last_became_peered":"2024-05-27T07:17:06.756316+0000","last_unstale":"2024-05-27T07:18:47.072043+0000","last_undegraded":"2024-05-27T07:18:47.072043+0000","last_fullsized":"2024-05-27T07:18:47.072043+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:15:12.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66464,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":268,"num_read_kb":204,"num_write":170,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":199392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710765,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":515890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906995,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32848,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712304,"statfs":{"total":95995035648,"available":95961399296,"internally_reserved":0,"allocated":1712128,"data_stored":1164927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070524,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":519528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266756,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":519336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462986,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1630208,"data_stored":1111070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626513,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":510608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":572590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":511996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":511246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182505,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986273,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":578734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822746,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1638400,"data_stored":1102753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:19:20.705 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:19:20.706 INFO:tasks.cephadm:Setting up client nodes... 2024-05-27T07:19:20.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-27T07:19:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:20 smithi012 ceph-mon[31473]: pgmap v456: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-05-27T07:19:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:20 smithi012 ceph-mon[31473]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:19:21.200 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:21.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:20 smithi162 ceph-mon[35676]: pgmap v456: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-05-27T07:19:21.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:20 smithi162 ceph-mon[35676]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:19:21.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:20 smithi062 ceph-mon[35304]: pgmap v456: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-05-27T07:19:21.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:20 smithi062 ceph-mon[35304]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:19:23.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:22 smithi012 ceph-mon[31473]: pgmap v457: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T07:19:23.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:22 smithi162 ceph-mon[35676]: pgmap v457: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T07:19:23.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:22 smithi062 ceph-mon[35304]: pgmap v457: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-27T07:19:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 -- 172.21.15.12:0/3626335769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ba70 msgr2=0x7f414810be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 --2- 172.21.15.12:0/3626335769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ba70 0x7f414810be80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4134009990 tx=0x7f4134009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 -- 172.21.15.12:0/3626335769 shutdown_connections 2024-05-27T07:19:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 --2- 172.21.15.12:0/3626335769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414810ab00 0x7f414810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 --2- 172.21.15.12:0/3626335769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f414810c410 0x7f4148112cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 --2- 172.21.15.12:0/3626335769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ba70 0x7f414810be80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 -- 172.21.15.12:0/3626335769 >> 172.21.15.12:0/3626335769 conn(0x7f41480799b0 msgr2=0x7f414807be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 -- 172.21.15.12:0/3626335769 shutdown_connections 2024-05-27T07:19:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.495+0000 7f4150600700 1 -- 172.21.15.12:0/3626335769 wait complete. 2024-05-27T07:19:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.497+0000 7f4150600700 1 Processor -- start 2024-05-27T07:19:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.497+0000 7f4150600700 1 -- start start 2024-05-27T07:19:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f4150600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ab00 0x7f41481a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f4150600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414810ba70 0x7f41481a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f4150600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f414810c410 0x7f41481a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f4150600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4148116260 con 0x7f414810ab00 2024-05-27T07:19:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f4150600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f41481160e0 con 0x7f414810ba70 2024-05-27T07:19:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f4150600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41481163e0 con 0x7f414810c410 2024-05-27T07:19:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f414dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ab00 0x7f41481a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f414dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ab00 0x7f41481a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46634/0 (socket says 172.21.15.12:46634) 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f414dc00700 1 -- 172.21.15.12:0/3316024153 learned_addr learned my addr 172.21.15.12:0/3316024153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f414d200700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414810ba70 0x7f41481a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.498+0000 7f414e600700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f414810c410 0x7f41481a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f414dc00700 1 -- 172.21.15.12:0/3316024153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f414810c410 msgr2=0x7f41481a82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f414dc00700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f414810c410 0x7f41481a82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f414dc00700 1 -- 172.21.15.12:0/3316024153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414810ba70 msgr2=0x7f41481a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f414dc00700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414810ba70 0x7f41481a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f414dc00700 1 -- 172.21.15.12:0/3316024153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41481a8960 con 0x7f414810ab00 2024-05-27T07:19:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f414dc00700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ab00 0x7f41481a0e00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f413c00d950 tx=0x7f413c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f4146000700 1 -- 172.21.15.12:0/3316024153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c017070 con 0x7f414810ab00 2024-05-27T07:19:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.499+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41481a8c50 con 0x7f414810ab00 2024-05-27T07:19:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.500+0000 7f4146000700 1 -- 172.21.15.12:0/3316024153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f413c00de30 con 0x7f414810ab00 2024-05-27T07:19:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.500+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41481a9190 con 0x7f414810ab00 2024-05-27T07:19:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.500+0000 7f4146000700 1 -- 172.21.15.12:0/3316024153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c003a40 con 0x7f414810ab00 2024-05-27T07:19:23.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.501+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41480512b0 con 0x7f414810ab00 2024-05-27T07:19:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.503+0000 7f4146000700 1 -- 172.21.15.12:0/3316024153 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f413c003be0 con 0x7f414810ab00 2024-05-27T07:19:23.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.505+0000 7f4146000700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f41300b2910 0x7f41300b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:23.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.505+0000 7f4146000700 1 -- 172.21.15.12:0/3316024153 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f413c0e2110 con 0x7f414810ab00 2024-05-27T07:19:23.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.505+0000 7f414d200700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f41300b2910 0x7f41300b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.506+0000 7f414d200700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f41300b2910 0x7f41300b4e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f413400b5c0 tx=0x7f413401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.510+0000 7f4146000700 1 -- 172.21.15.12:0/3316024153 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f413c020030 con 0x7f414810ab00 2024-05-27T07:19:23.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.842+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f41480643c0 con 0x7f414810ab00 2024-05-27T07:19:23.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.849+0000 7f4146000700 1 -- 172.21.15.12:0/3316024153 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v34) ==== 170+0+59 (secure 0 0 0) 0x7f413c009940 con 0x7f414810ab00 2024-05-27T07:19:23.840 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-05-27T07:19:23.840 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQD7M1Rm1clbMhAA9m3kvYRTgN/1ALieoJ0LXw== 2024-05-27T07:19:23.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f41300b2910 msgr2=0x7f41300b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:23.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f41300b2910 0x7f41300b4e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f413400b5c0 tx=0x7f413401a040 comp rx=0 tx=0).stop 2024-05-27T07:19:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ab00 msgr2=0x7f41481a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ab00 0x7f41481a0e00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f413c00d950 tx=0x7f413c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:19:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 shutdown_connections 2024-05-27T07:19:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f41300b2910 0x7f41300b4e20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f414810c410 0x7f41481a82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f414810ab00 0x7f41481a0e00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 --2- 172.21.15.12:0/3316024153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414810ba70 0x7f41481a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:23.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.853+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 >> 172.21.15.12:0/3316024153 conn(0x7f41480799b0 msgr2=0x7f414810eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:23.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.854+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 shutdown_connections 2024-05-27T07:19:23.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:23.854+0000 7f4150600700 1 -- 172.21.15.12:0/3316024153 wait complete. 2024-05-27T07:19:24.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:23 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3316024153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:19:24.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:23 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3316024153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:19:24.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:23 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3316024153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:19:24.730 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:19:24.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-27T07:19:24.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-27T07:19:24.879 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-27T07:19:24.880 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-27T07:19:24.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-05-27T07:19:25.002 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:24 smithi012 ceph-mon[31473]: pgmap v458: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T07:19:25.002 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:24 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3316024153' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:19:25.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:24 smithi162 ceph-mon[35676]: pgmap v458: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T07:19:25.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:24 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3316024153' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:19:25.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:24 smithi062 ceph-mon[35304]: pgmap v458: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T07:19:25.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:24 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3316024153' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:19:25.410 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:27.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:26 smithi162 ceph-mon[35676]: pgmap v459: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T07:19:27.232 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:26 smithi012 ceph-mon[31473]: pgmap v459: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T07:19:27.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:26 smithi062 ceph-mon[35304]: pgmap v459: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-27T07:19:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 -- 172.21.15.12:0/3515468357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c110b70 msgr2=0x7f682c112fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 --2- 172.21.15.12:0/3515468357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c110b70 0x7f682c112fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f682400b0a0 tx=0x7f682400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 -- 172.21.15.12:0/3515468357 shutdown_connections 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 --2- 172.21.15.12:0/3515468357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f682c076e30 0x7f682c077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 --2- 172.21.15.12:0/3515468357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c110b70 0x7f682c112fd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 --2- 172.21.15.12:0/3515468357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f682c077780 0x7f682c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 -- 172.21.15.12:0/3515468357 >> 172.21.15.12:0/3515468357 conn(0x7f682c0ffe90 msgr2=0x7f682c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 -- 172.21.15.12:0/3515468357 shutdown_connections 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.813+0000 7f6830c00700 1 -- 172.21.15.12:0/3515468357 wait complete. 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.814+0000 7f6830c00700 1 Processor -- start 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.814+0000 7f6830c00700 1 -- start start 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6830c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c076e30 0x7f682c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6830c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f682c077780 0x7f682c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6830c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f682c110b70 0x7f682c1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6830c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f682c116550 con 0x7f682c076e30 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6830c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f682c1163d0 con 0x7f682c077780 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6830c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f682c1166d0 con 0x7f682c110b70 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6829e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c076e30 0x7f682c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6829e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c076e30 0x7f682c19cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46650/0 (socket says 172.21.15.12:46650) 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6829e00700 1 -- 172.21.15.12:0/2513599021 learned_addr learned my addr 172.21.15.12:0/2513599021 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.815+0000 7f6823e00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f682c077780 0x7f682c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.816+0000 7f682a800700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f682c110b70 0x7f682c1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.816+0000 7f6829e00700 1 -- 172.21.15.12:0/2513599021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f682c110b70 msgr2=0x7f682c1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.816+0000 7f6829e00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f682c110b70 0x7f682c1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.816+0000 7f6829e00700 1 -- 172.21.15.12:0/2513599021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f682c077780 msgr2=0x7f682c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.816+0000 7f6829e00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f682c077780 0x7f682c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.816+0000 7f6829e00700 1 -- 172.21.15.12:0/2513599021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c1a1df0 con 0x7f682c076e30 2024-05-27T07:19:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.816+0000 7f6829e00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c076e30 0x7f682c19cf20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f681c00d950 tx=0x7f681c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.817+0000 7f6822a00700 1 -- 172.21.15.12:0/2513599021 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c016070 con 0x7f682c076e30 2024-05-27T07:19:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.817+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682c1a20e0 con 0x7f682c076e30 2024-05-27T07:19:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.817+0000 7f6822a00700 1 -- 172.21.15.12:0/2513599021 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f681c009dd0 con 0x7f682c076e30 2024-05-27T07:19:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.817+0000 7f6822a00700 1 -- 172.21.15.12:0/2513599021 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c00f3c0 con 0x7f682c076e30 2024-05-27T07:19:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.817+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f682c1a9a00 con 0x7f682c076e30 2024-05-27T07:19:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.818+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682c0512b0 con 0x7f682c076e30 2024-05-27T07:19:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.823+0000 7f6822a00700 1 -- 172.21.15.12:0/2513599021 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f681c019070 con 0x7f682c076e30 2024-05-27T07:19:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.824+0000 7f6822a00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f68100b2910 0x7f68100b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.825+0000 7f6822a00700 1 -- 172.21.15.12:0/2513599021 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f681c0e2400 con 0x7f682c076e30 2024-05-27T07:19:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.825+0000 7f6823e00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f68100b2910 0x7f68100b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.826+0000 7f6823e00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f68100b2910 0x7f68100b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f682c1033d0 tx=0x7f681400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:27.828+0000 7f6822a00700 1 -- 172.21.15.12:0/2513599021 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f681c0a9b90 con 0x7f682c076e30 2024-05-27T07:19:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.142+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f682c02e5e0 con 0x7f682c076e30 2024-05-27T07:19:28.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.151+0000 7f6822a00700 1 -- 172.21.15.12:0/2513599021 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288644 (secure 0 0 0) 0x7f681c0a9490 con 0x7f682c076e30 2024-05-27T07:19:28.144 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:19:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.157+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f68100b2910 msgr2=0x7f68100b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.157+0000 7f6830c00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f68100b2910 0x7f68100b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f682c1033d0 tx=0x7f681400b410 comp rx=0 tx=0).stop 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.157+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c076e30 msgr2=0x7f682c19cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.157+0000 7f6830c00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c076e30 0x7f682c19cf20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f681c00d950 tx=0x7f681c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 shutdown_connections 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f68100b2910 0x7f68100b4e20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f682c110b70 0x7f682c1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f682c076e30 0x7f682c19cf20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 --2- 172.21.15.12:0/2513599021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f682c077780 0x7f682c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 >> 172.21.15.12:0/2513599021 conn(0x7f682c0ffe90 msgr2=0x7f682c100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 shutdown_connections 2024-05-27T07:19:28.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:28.158+0000 7f6830c00700 1 -- 172.21.15.12:0/2513599021 wait complete. 2024-05-27T07:19:29.075 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2856894494},{"type":"v1","addr":"172.21.15.12:6801","nonce":2856894494}]},"active_addr":"172.21.15.12:6801/2856894494","active_change":"2024-05-27T07:06:45.508144+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":34109,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.12:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":435630590}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3259206461}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":725627344}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1619685219}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":307787456}]}]} 2024-05-27T07:19:29.081 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-27T07:19:29.081 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-27T07:19:29.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:19:29.167 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:28 smithi012 ceph-mon[31473]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:29.167 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:28 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2513599021' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:19:29.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:28 smithi162 ceph-mon[35676]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:28 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2513599021' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:19:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:28 smithi062 ceph-mon[35304]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:28 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2513599021' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:19:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:31.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:30 smithi012 ceph-mon[31473]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:19:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:30 smithi162 ceph-mon[35676]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:19:31.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:30 smithi062 ceph-mon[35304]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:19:31.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 -- 172.21.15.12:0/1066725360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010b2d0 msgr2=0x7f02f0117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 --2- 172.21.15.12:0/1066725360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010b2d0 0x7f02f0117800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f02e800b0a0 tx=0x7f02e800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 -- 172.21.15.12:0/1066725360 shutdown_connections 2024-05-27T07:19:31.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 --2- 172.21.15.12:0/1066725360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010b2d0 0x7f02f0117800 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 --2- 172.21.15.12:0/1066725360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f0110960 0x7f02f0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 --2- 172.21.15.12:0/1066725360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02f010a980 0x7f02f010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 -- 172.21.15.12:0/1066725360 >> 172.21.15.12:0/1066725360 conn(0x7f02f0077c70 msgr2=0x7f02f00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:31.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 -- 172.21.15.12:0/1066725360 shutdown_connections 2024-05-27T07:19:31.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.995+0000 7f02f6200700 1 -- 172.21.15.12:0/1066725360 wait complete. 2024-05-27T07:19:31.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 Processor -- start 2024-05-27T07:19:31.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 -- start start 2024-05-27T07:19:31.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010a980 0x7f02f01a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02f010b2d0 0x7f02f01a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f0110960 0x7f02f01a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f02f011a980 con 0x7f02f010a980 2024-05-27T07:19:31.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f02f011a800 con 0x7f02f010b2d0 2024-05-27T07:19:31.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.997+0000 7f02f6200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02f011ab00 con 0x7f02f0110960 2024-05-27T07:19:31.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010a980 0x7f02f01a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010a980 0x7f02f01a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41148/0 (socket says 172.21.15.12:41148) 2024-05-27T07:19:31.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 -- 172.21.15.12:0/3919591814 learned_addr learned my addr 172.21.15.12:0/3919591814 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:31.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02efe00700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f0110960 0x7f02f01a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02e7e00700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02f010b2d0 0x7f02f01a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 -- 172.21.15.12:0/3919591814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f0110960 msgr2=0x7f02f01a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f0110960 0x7f02f01a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 -- 172.21.15.12:0/3919591814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02f010b2d0 msgr2=0x7f02f01a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02f010b2d0 0x7f02f01a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.998+0000 7f02ef400700 1 -- 172.21.15.12:0/3919591814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f01a6220 con 0x7f02f010a980 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.999+0000 7f02ef400700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010a980 0x7f02f01a1350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f02e000d980 tx=0x7f02e000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.999+0000 7f02ecc00700 1 -- 172.21.15.12:0/3919591814 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0017070 con 0x7f02f010a980 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.999+0000 7f02ecc00700 1 -- 172.21.15.12:0/3919591814 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02e0010460 con 0x7f02f010a980 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.999+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f01a6510 con 0x7f02f010a980 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.999+0000 7f02ecc00700 1 -- 172.21.15.12:0/3919591814 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e000de60 con 0x7f02f010a980 2024-05-27T07:19:31.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:31.999+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f01ade30 con 0x7f02f010a980 2024-05-27T07:19:31.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.001+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f00512b0 con 0x7f02f010a980 2024-05-27T07:19:31.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.005+0000 7f02ecc00700 1 -- 172.21.15.12:0/3919591814 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f02e0010610 con 0x7f02f010a980 2024-05-27T07:19:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.007+0000 7f02ecc00700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f02d40b2990 0x7f02d40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.007+0000 7f02ecc00700 1 -- 172.21.15.12:0/3919591814 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f02e00e2200 con 0x7f02f010a980 2024-05-27T07:19:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.007+0000 7f02e7e00700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f02d40b2990 0x7f02d40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.008+0000 7f02e7e00700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f02d40b2990 0x7f02d40b4ea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f02d8005950 tx=0x7f02d80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.009+0000 7f02ecc00700 1 -- 172.21.15.12:0/3919591814 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f02e00a9a10 con 0x7f02f010a980 2024-05-27T07:19:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.249+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02f002e5e0 con 0x7f02f010a980 2024-05-27T07:19:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.255+0000 7f02ecc00700 1 -- 172.21.15.12:0/3919591814 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24806 (secure 0 0 0) 0x7f02e00a9310 con 0x7f02f010a980 2024-05-27T07:19:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:19:32.247 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:18:47.037205+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:19:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f02d40b2990 msgr2=0x7f02d40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f02d40b2990 0x7f02d40b4ea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f02d8005950 tx=0x7f02d80058e0 comp rx=0 tx=0).stop 2024-05-27T07:19:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010a980 msgr2=0x7f02f01a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010a980 0x7f02f01a1350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f02e000d980 tx=0x7f02e000dd40 comp rx=0 tx=0).stop 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 shutdown_connections 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f02d40b2990 0x7f02d40b4ea0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f0110960 0x7f02f01a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f02f010a980 0x7f02f01a1350 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 --2- 172.21.15.12:0/3919591814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02f010b2d0 0x7f02f01a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 >> 172.21.15.12:0/3919591814 conn(0x7f02f0077c70 msgr2=0x7f02f007b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 shutdown_connections 2024-05-27T07:19:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:32.259+0000 7f02f6200700 1 -- 172.21.15.12:0/3919591814 wait complete. 2024-05-27T07:19:33.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:32 smithi012 ceph-mon[31473]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:33.176 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:32 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3919591814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:33.181 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-27T07:19:33.182 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:19:33.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:32 smithi162 ceph-mon[35676]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:33.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:32 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3919591814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:32 smithi062 ceph-mon[35304]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:32 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3919591814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:35.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:34 smithi012 ceph-mon[31473]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:35.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:34 smithi162 ceph-mon[35676]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:35.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:34 smithi062 ceph-mon[35304]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.798+0000 7fd5ea400700 1 -- 172.21.15.12:0/2261615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 msgr2=0x7fd5e410c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.798+0000 7fd5ea400700 1 --2- 172.21.15.12:0/2261615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e410c9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009990 tx=0x7fd5d4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.799+0000 7fd5ea400700 1 -- 172.21.15.12:0/2261615426 shutdown_connections 2024-05-27T07:19:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.799+0000 7fd5ea400700 1 --2- 172.21.15.12:0/2261615426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5e4106ec0 0x7fd5e4113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.799+0000 7fd5ea400700 1 --2- 172.21.15.12:0/2261615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e410c9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.799+0000 7fd5ea400700 1 --2- 172.21.15.12:0/2261615426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5e4106570 0x7fd5e4106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.799+0000 7fd5ea400700 1 -- 172.21.15.12:0/2261615426 >> 172.21.15.12:0/2261615426 conn(0x7fd5e4102070 msgr2=0x7fd5e41044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.799+0000 7fd5ea400700 1 -- 172.21.15.12:0/2261615426 shutdown_connections 2024-05-27T07:19:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.799+0000 7fd5ea400700 1 -- 172.21.15.12:0/2261615426 wait complete. 2024-05-27T07:19:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.800+0000 7fd5ea400700 1 Processor -- start 2024-05-27T07:19:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.801+0000 7fd5ea400700 1 -- start start 2024-05-27T07:19:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.801+0000 7fd5ea400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5e4106570 0x7fd5e419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.801+0000 7fd5ea400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5e4106ec0 0x7fd5e419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.801+0000 7fd5ea400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e41a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.801+0000 7fd5ea400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41165c0 con 0x7fd5e410c5b0 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.801+0000 7fd5ea400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4116440 con 0x7fd5e4106570 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.801+0000 7fd5ea400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4116740 con 0x7fd5e4106ec0 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e41a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e41a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41164/0 (socket says 172.21.15.12:41164) 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 -- 172.21.15.12:0/285135764 learned_addr learned my addr 172.21.15.12:0/285135764 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5e4106570 0x7fd5e419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e2a00700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5e4106ec0 0x7fd5e419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 -- 172.21.15.12:0/285135764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5e4106ec0 msgr2=0x7fd5e419d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5e4106ec0 0x7fd5e419d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 -- 172.21.15.12:0/285135764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5e4106570 msgr2=0x7fd5e419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5e4106570 0x7fd5e419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.802+0000 7fd5e3e00700 1 -- 172.21.15.12:0/285135764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e41a1da0 con 0x7fd5e410c5b0 2024-05-27T07:19:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.803+0000 7fd5e3e00700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e41a17b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc007ee0 tx=0x7fd5dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.803+0000 7fd5dbe00700 1 -- 172.21.15.12:0/285135764 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc00dbd0 con 0x7fd5e410c5b0 2024-05-27T07:19:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.803+0000 7fd5dbe00700 1 -- 172.21.15.12:0/285135764 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5dc00dd70 con 0x7fd5e410c5b0 2024-05-27T07:19:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.803+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e41a2090 con 0x7fd5e410c5b0 2024-05-27T07:19:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.803+0000 7fd5dbe00700 1 -- 172.21.15.12:0/285135764 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0058d0 con 0x7fd5e410c5b0 2024-05-27T07:19:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.803+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e41a25d0 con 0x7fd5e410c5b0 2024-05-27T07:19:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.805+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e40512b0 con 0x7fd5e410c5b0 2024-05-27T07:19:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.807+0000 7fd5dbe00700 1 -- 172.21.15.12:0/285135764 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd5dc01a020 con 0x7fd5e410c5b0 2024-05-27T07:19:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.809+0000 7fd5dbe00700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd5c80b2990 0x7fd5c80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.809+0000 7fd5dbe00700 1 -- 172.21.15.12:0/285135764 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fd5dc0e25c0 con 0x7fd5e410c5b0 2024-05-27T07:19:35.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.809+0000 7fd5e3400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd5c80b2990 0x7fd5c80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:35.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.810+0000 7fd5e3400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd5c80b2990 0x7fd5c80b4ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4006010 tx=0x7fd5d400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:35.812+0000 7fd5dbe00700 1 -- 172.21.15.12:0/285135764 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd5dc010070 con 0x7fd5e410c5b0 2024-05-27T07:19:36.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.056+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5e402e5e0 con 0x7fd5e410c5b0 2024-05-27T07:19:36.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.059+0000 7fd5dbe00700 1 -- 172.21.15.12:0/285135764 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24806 (secure 0 0 0) 0x7fd5dc0a9d50 con 0x7fd5e410c5b0 2024-05-27T07:19:36.051 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:19:36.051 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:18:47.037205+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:19:36.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd5c80b2990 msgr2=0x7fd5c80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:36.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd5c80b2990 0x7fd5c80b4ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4006010 tx=0x7fd5d400b540 comp rx=0 tx=0).stop 2024-05-27T07:19:36.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 msgr2=0x7fd5e41a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:36.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e41a17b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc007ee0 tx=0x7fd5dc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:36.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 shutdown_connections 2024-05-27T07:19:36.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd5c80b2990 0x7fd5c80b4ea0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:36.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5e4106570 0x7fd5e419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:36.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e410c5b0 0x7fd5e41a17b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:36.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 --2- 172.21.15.12:0/285135764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5e4106ec0 0x7fd5e419d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:36.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.063+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 >> 172.21.15.12:0/285135764 conn(0x7fd5e4102070 msgr2=0x7fd5e41049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:36.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.064+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 shutdown_connections 2024-05-27T07:19:36.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:36.064+0000 7fd5ea400700 1 -- 172.21.15.12:0/285135764 wait complete. 2024-05-27T07:19:37.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-27T07:19:37.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-27T07:19:37.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-27T07:19:37.053 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-27T07:19:37.053 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-27T07:19:37.054 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-27T07:19:37.054 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-27T07:19:37.055 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-27T07:19:37.055 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-27T07:19:37.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-27T07:19:37.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-27T07:19:37.057 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-27T07:19:37.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:36 smithi012 ceph-mon[31473]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:37.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:36 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/285135764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:37.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:36 smithi162 ceph-mon[35676]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:37.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:36 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/285135764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:37.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:36 smithi062 ceph-mon[35304]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:37.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:36 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/285135764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:19:38.050 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:38.094 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:38.526 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:38.538 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:38.538 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:38.539 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:39.221 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:38 smithi012 ceph-mon[31473]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:39.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:38 smithi162 ceph-mon[35676]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:39.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:38 smithi062 ceph-mon[35304]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:41.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:40 smithi162 ceph-mon[35676]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:41.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:40 smithi062 ceph-mon[35304]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:41.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:40 smithi012 ceph-mon[31473]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:41.854 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:42.691 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:42.693 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:43.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:42 smithi012 ceph-mon[31473]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:43.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:42 smithi162 ceph-mon[35676]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:43.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:42 smithi062 ceph-mon[35304]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:44.564 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:45.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:44 smithi162 ceph-mon[35676]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:45.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:44 smithi062 ceph-mon[35304]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:45.332 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:44 smithi012 ceph-mon[31473]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 -- 172.21.15.12:0/277428977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748109860 msgr2=0x7f07481111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 --2- 172.21.15.12:0/277428977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748109860 0x7f07481111c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f074000b0d0 tx=0x7f074000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:19:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 -- 172.21.15.12:0/277428977 shutdown_connections 2024-05-27T07:19:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 --2- 172.21.15.12:0/277428977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748109860 0x7f07481111c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 --2- 172.21.15.12:0/277428977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07481045e0 0x7f07481049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 --2- 172.21.15.12:0/277428977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0748108f10 0x7f0748109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 -- 172.21.15.12:0/277428977 >> 172.21.15.12:0/277428977 conn(0x7f07480fffd0 msgr2=0x7f07481023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.733+0000 7f074d000700 1 -- 172.21.15.12:0/277428977 shutdown_connections 2024-05-27T07:19:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.734+0000 7f074d000700 1 -- 172.21.15.12:0/277428977 wait complete. 2024-05-27T07:19:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.735+0000 7f074d000700 1 Processor -- start 2024-05-27T07:19:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.735+0000 7f074d000700 1 -- start start 2024-05-27T07:19:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f074d000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07481045e0 0x7f074819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f074d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748108f10 0x7f074819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f074d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0748109860 0x7f074819f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f074d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0748114410 con 0x7f0748108f10 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f074d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0748114290 con 0x7f0748109860 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f074d000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0748114590 con 0x7f07481045e0 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f0745400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748108f10 0x7f074819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f0745400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748108f10 0x7f074819b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54958/0 (socket says 172.21.15.12:54958) 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f0745400700 1 -- 172.21.15.12:0/3713974787 learned_addr learned my addr 172.21.15.12:0/3713974787 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f0745e00700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07481045e0 0x7f074819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.736+0000 7f0746800700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0748109860 0x7f074819f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.737+0000 7f0745400700 1 -- 172.21.15.12:0/3713974787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07481045e0 msgr2=0x7f074819ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.737+0000 7f0745400700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07481045e0 0x7f074819ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.737+0000 7f0745400700 1 -- 172.21.15.12:0/3713974787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0748109860 msgr2=0x7f074819f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.737+0000 7f0745400700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0748109860 0x7f074819f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.737+0000 7f0745400700 1 -- 172.21.15.12:0/3713974787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074819fca0 con 0x7f0748108f10 2024-05-27T07:19:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.737+0000 7f0745400700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748108f10 0x7f074819b3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f073400e970 tx=0x7f073400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.738+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073400cbd0 con 0x7f0748108f10 2024-05-27T07:19:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.738+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f073400cd70 con 0x7f0748108f10 2024-05-27T07:19:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.738+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f074819ff90 con 0x7f0748108f10 2024-05-27T07:19:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.738+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07340055f0 con 0x7f0748108f10 2024-05-27T07:19:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.738+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07480759e0 con 0x7f0748108f10 2024-05-27T07:19:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.739+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0748063f40 con 0x7f0748108f10 2024-05-27T07:19:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.743+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0734004750 con 0x7f0748108f10 2024-05-27T07:19:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.744+0000 7f073ea00700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f072c0b2990 0x7f072c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.744+0000 7f0745e00700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f072c0b2990 0x7f072c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.745+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f07340e3150 con 0x7f0748108f10 2024-05-27T07:19:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.745+0000 7f073ea00700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] conn(0x7f072c0bdd40 0x7f072c0c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.745+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 --> [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f073401aec0 con 0x7f072c0bdd40 2024-05-27T07:19:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.745+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f07340e3500 con 0x7f0748108f10 2024-05-27T07:19:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.745+0000 7f0746800700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] conn(0x7f072c0bdd40 0x7f072c0c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.746+0000 7f0745e00700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f072c0b2990 0x7f072c0b4ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0738009a10 tx=0x7f0738009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.746+0000 7f0746800700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] conn(0x7f072c0bdd40 0x7f072c0c0230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.747+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== osd.0 v2:172.21.15.12:6802/667573268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f073401aec0 con 0x7f072c0bdd40 2024-05-27T07:19:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.766+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 --> [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0748064370 con 0x7f072c0bdd40 2024-05-27T07:19:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.766+0000 7f073ea00700 1 -- 172.21.15.12:0/3713974787 <== osd.0 v2:172.21.15.12:6802/667573268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0748064370 con 0x7f072c0bdd40 2024-05-27T07:19:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] conn(0x7f072c0bdd40 msgr2=0x7f072c0c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] conn(0x7f072c0bdd40 0x7f072c0c0230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f072c0b2990 msgr2=0x7f072c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f072c0b2990 0x7f072c0b4ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0738009a10 tx=0x7f0738009190 comp rx=0 tx=0).stop 2024-05-27T07:19:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748108f10 msgr2=0x7f074819b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 --2- 172.21.15.12:0/3713974787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0748108f10 0x7f074819b3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f073400e970 tx=0x7f073400ec80 comp rx=0 tx=0).stop 2024-05-27T07:19:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f0746800700 1 -- 172.21.15.12:0/3713974787 reap_dead start 2024-05-27T07:19:45.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 shutdown_connections 2024-05-27T07:19:45.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.767+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 >> 172.21.15.12:0/3713974787 conn(0x7f07480fffd0 msgr2=0x7f0748107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:45.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.768+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 shutdown_connections 2024-05-27T07:19:45.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:45.768+0000 7f074d000700 1 -- 172.21.15.12:0/3713974787 wait complete. 2024-05-27T07:19:46.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:46 smithi012 ceph-mon[31473]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:47.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:46 smithi162 ceph-mon[35676]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:47.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:46 smithi062 ceph-mon[35304]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:19:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 -- 172.21.15.12:0/2846024427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac106610 msgr2=0x7fbbac106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 --2- 172.21.15.12:0/2846024427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac106610 0x7fbbac106a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0099c0 tx=0x7fbb9c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 -- 172.21.15.12:0/2846024427 shutdown_connections 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 --2- 172.21.15.12:0/2846024427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbac10b890 0x7fbbac1131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 --2- 172.21.15.12:0/2846024427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac106610 0x7fbbac106a20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 --2- 172.21.15.12:0/2846024427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbac10af40 0x7fbbac10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 -- 172.21.15.12:0/2846024427 >> 172.21.15.12:0/2846024427 conn(0x7fbbac102020 msgr2=0x7fbbac104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 -- 172.21.15.12:0/2846024427 shutdown_connections 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.152+0000 7fbbabe00700 1 -- 172.21.15.12:0/2846024427 wait complete. 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.153+0000 7fbbabe00700 1 Processor -- start 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.154+0000 7fbbabe00700 1 -- start start 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.154+0000 7fbbabe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbac106610 0x7fbbac19ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.154+0000 7fbbabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac10af40 0x7fbbac19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.154+0000 7fbbabe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbac10b890 0x7fbbac1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.154+0000 7fbbabe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac1163c0 con 0x7fbbac10af40 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.154+0000 7fbbabe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac116240 con 0x7fbbac10b890 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.154+0000 7fbbabe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac116540 con 0x7fbbac106610 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac10af40 0x7fbbac19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac10af40 0x7fbbac19d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54974/0 (socket says 172.21.15.12:54974) 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 -- 172.21.15.12:0/1871432908 learned_addr learned my addr 172.21.15.12:0/1871432908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbab400700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbac10b890 0x7fbbac1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaaa00700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbac106610 0x7fbbac19ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 -- 172.21.15.12:0/1871432908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbac106610 msgr2=0x7fbbac19ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbac106610 0x7fbbac19ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 -- 172.21.15.12:0/1871432908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbac10b890 msgr2=0x7fbbac1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbac10b890 0x7fbbac1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbaa000700 1 -- 172.21.15.12:0/1871432908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac1a1d20 con 0x7fbbac10af40 2024-05-27T07:19:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.155+0000 7fbbab400700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbac10b890 0x7fbbac1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.156+0000 7fbbaa000700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac10af40 0x7fbbac19d3f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800d980 tx=0x7fbb9800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.156+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98017070 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.156+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb98010460 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.156+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbac1a2010 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.156+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9800de60 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.156+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbac1a9930 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.157+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbbac063ec0 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.161+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fbb98010610 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.162+0000 7fbba3400700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbb900b2990 0x7fbb900b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.163+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fbb980e26c0 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.163+0000 7fbbaaa00700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbb900b2990 0x7fbb900b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.163+0000 7fbba3400700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] conn(0x7fbb900bdd40 0x7fbb900c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.163+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 --> [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb98016ec0 con 0x7fbb900bdd40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.163+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fbb980e2a70 con 0x7fbbac10af40 2024-05-27T07:19:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.163+0000 7fbbab400700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] conn(0x7fbb900bdd40 0x7fbb900c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.164+0000 7fbbaaa00700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbb900b2990 0x7fbb900b4ea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c000c00 tx=0x7fbb9c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.164+0000 7fbbab400700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] conn(0x7fbb900bdd40 0x7fbb900c0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.165+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== osd.9 v2:172.21.15.12:6826/667220154 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbb98016ec0 con 0x7fbb900bdd40 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.184+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 --> [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbbac0642c0 con 0x7fbb900bdd40 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.184+0000 7fbba3400700 1 -- 172.21.15.12:0/1871432908 <== osd.9 v2:172.21.15.12:6826/667220154 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbbac0642c0 con 0x7fbb900bdd40 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] conn(0x7fbb900bdd40 msgr2=0x7fbb900c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] conn(0x7fbb900bdd40 0x7fbb900c0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbb900b2990 msgr2=0x7fbb900b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbb900b2990 0x7fbb900b4ea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c000c00 tx=0x7fbb9c005fb0 comp rx=0 tx=0).stop 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac10af40 msgr2=0x7fbbac19d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 --2- 172.21.15.12:0/1871432908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbac10af40 0x7fbbac19d3f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800d980 tx=0x7fbb9800dd40 comp rx=0 tx=0).stop 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbab400700 1 -- 172.21.15.12:0/1871432908 reap_dead start 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 shutdown_connections 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 >> 172.21.15.12:0/1871432908 conn(0x7fbbac102020 msgr2=0x7fbbac10a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 shutdown_connections 2024-05-27T07:19:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:48.185+0000 7fbbabe00700 1 -- 172.21.15.12:0/1871432908 wait complete. 2024-05-27T07:19:49.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:48 smithi012 ceph-mon[31473]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:49.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:48 smithi162 ceph-mon[35676]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:49.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:48 smithi062 ceph-mon[35304]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:50.382 INFO:teuthology.orchestra.run.smithi012.stdout:34359738500 2024-05-27T07:19:50.383 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-27T07:19:51.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:50 smithi162 ceph-mon[35676]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:19:51.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:50 smithi062 ceph-mon[35304]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:19:51.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:50 smithi012 ceph-mon[31473]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:19:52.080 INFO:teuthology.orchestra.run.smithi012.stdout:253403070531 2024-05-27T07:19:52.080 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-27T07:19:52.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 -- 172.21.15.12:0/2519785531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 msgr2=0x7f13c0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 --2- 172.21.15.12:0/2519785531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 0x7f13c0109130 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0099c0 tx=0x7f13ac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 -- 172.21.15.12:0/2519785531 shutdown_connections 2024-05-27T07:19:52.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 --2- 172.21.15.12:0/2519785531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13c0109670 0x7f13c0110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 --2- 172.21.15.12:0/2519785531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13c01043f0 0x7f13c0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 --2- 172.21.15.12:0/2519785531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 0x7f13c0109130 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 -- 172.21.15.12:0/2519785531 >> 172.21.15.12:0/2519785531 conn(0x7f13c0079d00 msgr2=0x7f13c007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 -- 172.21.15.12:0/2519785531 shutdown_connections 2024-05-27T07:19:52.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.560+0000 7f13bfe00700 1 -- 172.21.15.12:0/2519785531 wait complete. 2024-05-27T07:19:52.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.561+0000 7f13bfe00700 1 Processor -- start 2024-05-27T07:19:52.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.561+0000 7f13bfe00700 1 -- start start 2024-05-27T07:19:52.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13bfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13c01043f0 0x7f13c019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 0x7f13c019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13bfe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13c0109670 0x7f13c01a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13bfe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f13c0114180 con 0x7f13c0108d20 2024-05-27T07:19:52.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13bfe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13c0114000 con 0x7f13c01043f0 2024-05-27T07:19:52.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13bfe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f13c0114300 con 0x7f13c0109670 2024-05-27T07:19:52.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 0x7f13c019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 0x7f13c019d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47896/0 (socket says 172.21.15.12:47896) 2024-05-27T07:19:52.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.562+0000 7f13be000700 1 -- 172.21.15.12:0/776500611 learned_addr learned my addr 172.21.15.12:0/776500611 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:52.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13bea00700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13c01043f0 0x7f13c019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13bf400700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13c0109670 0x7f13c01a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13be000700 1 -- 172.21.15.12:0/776500611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13c0109670 msgr2=0x7f13c01a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13be000700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13c0109670 0x7f13c01a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13be000700 1 -- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13c01043f0 msgr2=0x7f13c019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13be000700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13c01043f0 0x7f13c019cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13be000700 1 -- 172.21.15.12:0/776500611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c01a1ca0 con 0x7f13c0108d20 2024-05-27T07:19:52.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13bf400700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13c0109670 0x7f13c01a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:19:52.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13bea00700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13c01043f0 0x7f13c019cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:19:52.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.563+0000 7f13be000700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 0x7f13c019d370 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0098e0 tx=0x7f13ac00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:52.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.564+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac027070 con 0x7f13c0108d20 2024-05-27T07:19:52.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.564+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13ac00fdd0 con 0x7f13c0108d20 2024-05-27T07:19:52.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.564+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c01a1f30 con 0x7f13c0108d20 2024-05-27T07:19:52.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.564+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac022e30 con 0x7f13c0108d20 2024-05-27T07:19:52.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.564+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c01a2410 con 0x7f13c0108d20 2024-05-27T07:19:52.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.565+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13c010fdb0 con 0x7f13c0108d20 2024-05-27T07:19:52.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.569+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f13ac02a070 con 0x7f13c0108d20 2024-05-27T07:19:52.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.570+0000 7f13b7400700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f13a40b2910 0x7f13a40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.570+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f13ac0e3c80 con 0x7f13c0108d20 2024-05-27T07:19:52.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.570+0000 7f13bea00700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f13a40b2910 0x7f13a40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.570+0000 7f13b7400700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] conn(0x7f13a40bdcc0 0x7f13a40c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.570+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 --> [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13a40c0880 con 0x7f13a40bdcc0 2024-05-27T07:19:52.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.570+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f13ac0e4030 con 0x7f13c0108d20 2024-05-27T07:19:52.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.571+0000 7f13bea00700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f13a40b2910 0x7f13a40b4e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f13b0005950 tx=0x7f13b00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:52.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.571+0000 7f13bf400700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] conn(0x7f13a40bdcc0 0x7f13a40c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.573+0000 7f13bf400700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] conn(0x7f13a40bdcc0 0x7f13a40c01b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:52.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.575+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== osd.1 v2:172.21.15.62:6800/1268848674 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13a40c0880 con 0x7f13a40bdcc0 2024-05-27T07:19:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.594+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 --> [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13c01058a0 con 0x7f13a40bdcc0 2024-05-27T07:19:52.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.595+0000 7f13b7400700 1 -- 172.21.15.12:0/776500611 <== osd.1 v2:172.21.15.62:6800/1268848674 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f13c01058a0 con 0x7f13a40bdcc0 2024-05-27T07:19:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.595+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] conn(0x7f13a40bdcc0 msgr2=0x7f13a40c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.595+0000 7f13bfe00700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] conn(0x7f13a40bdcc0 0x7f13a40c01b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f13a40b2910 msgr2=0x7f13a40b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f13a40b2910 0x7f13a40b4e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f13b0005950 tx=0x7f13b00058e0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 msgr2=0x7f13c019d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 --2- 172.21.15.12:0/776500611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13c0108d20 0x7f13c019d370 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0098e0 tx=0x7f13ac00f510 comp rx=0 tx=0).stop 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bf400700 1 -- 172.21.15.12:0/776500611 reap_dead start 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 shutdown_connections 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 >> 172.21.15.12:0/776500611 conn(0x7f13c0079d00 msgr2=0x7f13c010f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 shutdown_connections 2024-05-27T07:19:52.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.596+0000 7f13bfe00700 1 -- 172.21.15.12:0/776500611 wait complete. 2024-05-27T07:19:52.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.163+0000 7f9ba3000700 1 -- 172.21.15.12:0/3486476311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 msgr2=0x7f9b9c1130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.163+0000 7f9ba3000700 1 --2- 172.21.15.12:0/3486476311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 0x7f9b9c1130c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400b0a0 tx=0x7f9b9400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.164+0000 7f9ba3000700 1 -- 172.21.15.12:0/3486476311 shutdown_connections 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.164+0000 7f9ba3000700 1 --2- 172.21.15.12:0/3486476311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 0x7f9b9c1130c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.164+0000 7f9ba3000700 1 --2- 172.21.15.12:0/3486476311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c1030f0 0x7f9b9c103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.164+0000 7f9ba3000700 1 --2- 172.21.15.12:0/3486476311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9c103a40 0x7f9b9c110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.164+0000 7f9ba3000700 1 -- 172.21.15.12:0/3486476311 >> 172.21.15.12:0/3486476311 conn(0x7f9b9c0fedb0 msgr2=0x7f9b9c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.164+0000 7f9ba3000700 1 -- 172.21.15.12:0/3486476311 shutdown_connections 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.164+0000 7f9ba3000700 1 -- 172.21.15.12:0/3486476311 wait complete. 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.165+0000 7f9ba3000700 1 Processor -- start 2024-05-27T07:19:52.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.165+0000 7f9ba3000700 1 -- start start 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba3000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9c1030f0 0x7f9b9c19cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba3000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c103a40 0x7f9b9c19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba3000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 0x7f9b9c1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba3000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c116530 con 0x7f9b9c110ca0 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba3000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1163b0 con 0x7f9b9c1030f0 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba3000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1166b0 con 0x7f9b9c103a40 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 0x7f9b9c1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 0x7f9b9c1a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47884/0 (socket says 172.21.15.12:47884) 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba1000700 1 -- 172.21.15.12:0/2740370778 learned_addr learned my addr 172.21.15.12:0/2740370778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9b9be00700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9c1030f0 0x7f9b9c19cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9b9b400700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c103a40 0x7f9b9c19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba1000700 1 -- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c103a40 msgr2=0x7f9b9c19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.166+0000 7f9ba1000700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c103a40 0x7f9b9c19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9ba1000700 1 -- 172.21.15.12:0/2740370778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9c1030f0 msgr2=0x7f9b9c19cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9ba1000700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9c1030f0 0x7f9b9c19cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9ba1000700 1 -- 172.21.15.12:0/2740370778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c1a1dd0 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9ba1000700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 0x7f9b9c1a1730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400b830 tx=0x7f9b94004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94021070 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9c1a2060 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b94007c80 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.167+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94005260 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.168+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b9c1a2540 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.169+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b9c063f40 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.172+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9b94024070 con 0x7f9b9c110ca0 2024-05-27T07:19:52.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.173+0000 7f9b98c00700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b800b2910 0x7f9b800b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.173+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f9b9400f020 con 0x7f9b9c110ca0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.174+0000 7f9b9be00700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b800b2910 0x7f9b800b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.174+0000 7f9b98c00700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] conn(0x7f9b800bdcc0 0x7f9b800c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.174+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 --> [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b94027e50 con 0x7f9b800bdcc0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.174+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f9b940e4440 con 0x7f9b9c110ca0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.174+0000 7f9b9be00700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b800b2910 0x7f9b800b4e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c005950 tx=0x7f9b8c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.175+0000 7f9b9b400700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] conn(0x7f9b800bdcc0 0x7f9b800c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.176+0000 7f9b9b400700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] conn(0x7f9b800bdcc0 0x7f9b800c01b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.179+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== osd.11 v2:172.21.15.162:6824/1037138310 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b94027e50 con 0x7f9b800bdcc0 2024-05-27T07:19:52.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.198+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 --> [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b9c0643c0 con 0x7f9b800bdcc0 2024-05-27T07:19:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.199+0000 7f9b98c00700 1 -- 172.21.15.12:0/2740370778 <== osd.11 v2:172.21.15.162:6824/1037138310 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9b9c0643c0 con 0x7f9b800bdcc0 2024-05-27T07:19:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] conn(0x7f9b800bdcc0 msgr2=0x7f9b800c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] conn(0x7f9b800bdcc0 0x7f9b800c01b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b800b2910 msgr2=0x7f9b800b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9b800b2910 0x7f9b800b4e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c005950 tx=0x7f9b8c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 msgr2=0x7f9b9c1a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 --2- 172.21.15.12:0/2740370778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b9c110ca0 0x7f9b9c1a1730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400b830 tx=0x7f9b94004db0 comp rx=0 tx=0).stop 2024-05-27T07:19:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba1000700 1 -- 172.21.15.12:0/2740370778 reap_dead start 2024-05-27T07:19:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 shutdown_connections 2024-05-27T07:19:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 >> 172.21.15.12:0/2740370778 conn(0x7f9b9c0fedb0 msgr2=0x7f9b9c112820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 shutdown_connections 2024-05-27T07:19:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:52.200+0000 7f9ba3000700 1 -- 172.21.15.12:0/2740370778 wait complete. 2024-05-27T07:19:53.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:52 smithi162 ceph-mon[35676]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:53.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:52 smithi062 ceph-mon[35304]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:53.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:52 smithi012 ceph-mon[31473]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:19:54.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.018+0000 7f53adc00700 1 -- 172.21.15.12:0/184362039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a81045e0 msgr2=0x7f53a81049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.018+0000 7f53adc00700 1 --2- 172.21.15.12:0/184362039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a81045e0 0x7f53a81049f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f53980099c0 tx=0x7f5398009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.018+0000 7f53adc00700 1 -- 172.21.15.12:0/184362039 shutdown_connections 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.018+0000 7f53adc00700 1 --2- 172.21.15.12:0/184362039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8109860 0x7f53a81111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.018+0000 7f53adc00700 1 --2- 172.21.15.12:0/184362039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a81045e0 0x7f53a81049f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.018+0000 7f53adc00700 1 --2- 172.21.15.12:0/184362039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8108f10 0x7f53a8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.018+0000 7f53adc00700 1 -- 172.21.15.12:0/184362039 >> 172.21.15.12:0/184362039 conn(0x7f53a8100030 msgr2=0x7f53a8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.019+0000 7f53adc00700 1 -- 172.21.15.12:0/184362039 shutdown_connections 2024-05-27T07:19:54.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.019+0000 7f53adc00700 1 -- 172.21.15.12:0/184362039 wait complete. 2024-05-27T07:19:54.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.020+0000 7f53adc00700 1 Processor -- start 2024-05-27T07:19:54.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.020+0000 7f53adc00700 1 -- start start 2024-05-27T07:19:54.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f53adc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a81045e0 0x7f53a8075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:54.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f53adc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a8108f10 0x7f53a8075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f53adc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8109860 0x7f53a8076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f53adc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8114390 con 0x7f53a8108f10 2024-05-27T07:19:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f53adc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8114210 con 0x7f53a81045e0 2024-05-27T07:19:54.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f53adc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8114510 con 0x7f53a8109860 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f539e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a8108f10 0x7f53a8075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f539e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a8108f10 0x7f53a8075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47908/0 (socket says 172.21.15.12:47908) 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f539e000700 1 -- 172.21.15.12:0/2793132705 learned_addr learned my addr 172.21.15.12:0/2793132705 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.021+0000 7f53a6a00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a81045e0 0x7f53a8075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.022+0000 7f53a7400700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8109860 0x7f53a8076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.022+0000 7f539e000700 1 -- 172.21.15.12:0/2793132705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8109860 msgr2=0x7f53a8076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.022+0000 7f539e000700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53a8109860 0x7f53a8076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.022+0000 7f539e000700 1 -- 172.21.15.12:0/2793132705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a81045e0 msgr2=0x7f53a8075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.022+0000 7f539e000700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a81045e0 0x7f53a8075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.022+0000 7f539e000700 1 -- 172.21.15.12:0/2793132705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a81a9140 con 0x7f53a8108f10 2024-05-27T07:19:54.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.022+0000 7f539e000700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a8108f10 0x7f53a8075ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f539000d980 tx=0x7f539000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:54.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.023+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390017070 con 0x7f53a8108f10 2024-05-27T07:19:54.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.023+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5390009dc0 con 0x7f53a8108f10 2024-05-27T07:19:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.023+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539000f3c0 con 0x7f53a8108f10 2024-05-27T07:19:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.023+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a81a9430 con 0x7f53a8108f10 2024-05-27T07:19:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.023+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a81a9920 con 0x7f53a8108f10 2024-05-27T07:19:54.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.024+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f53a8063ec0 con 0x7f53a8108f10 2024-05-27T07:19:54.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.027+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5390009930 con 0x7f53a8108f10 2024-05-27T07:19:54.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.029+0000 7f539fe00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f538c0b2990 0x7f538c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:54.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.029+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f53900e25e0 con 0x7f53a8108f10 2024-05-27T07:19:54.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.029+0000 7f53a6a00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f538c0b2990 0x7f538c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:54.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.029+0000 7f539fe00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] conn(0x7f538c0bdd40 0x7f538c0c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:54.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.030+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 --> [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5390011e50 con 0x7f538c0bdd40 2024-05-27T07:19:54.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.030+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f53900e2a30 con 0x7f53a8108f10 2024-05-27T07:19:54.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.030+0000 7f53a7400700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] conn(0x7f538c0bdd40 0x7f538c0c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:54.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.030+0000 7f53a6a00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f538c0b2990 0x7f538c0b4ea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5398000c00 tx=0x7f5398005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:54.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.031+0000 7f53a7400700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] conn(0x7f538c0bdd40 0x7f538c0c0230 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:54.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.033+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== osd.6 v2:172.21.15.12:6818/3556220791 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5390011e50 con 0x7f538c0bdd40 2024-05-27T07:19:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.052+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 --> [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f53a8064310 con 0x7f538c0bdd40 2024-05-27T07:19:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.052+0000 7f539fe00700 1 -- 172.21.15.12:0/2793132705 <== osd.6 v2:172.21.15.12:6818/3556220791 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f53a8064310 con 0x7f538c0bdd40 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] conn(0x7f538c0bdd40 msgr2=0x7f538c0c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] conn(0x7f538c0bdd40 0x7f538c0c0230 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f538c0b2990 msgr2=0x7f538c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f538c0b2990 0x7f538c0b4ea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5398000c00 tx=0x7f5398005fb0 comp rx=0 tx=0).stop 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a8108f10 msgr2=0x7f53a8075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 --2- 172.21.15.12:0/2793132705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f53a8108f10 0x7f53a8075ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f539000d980 tx=0x7f539000dc90 comp rx=0 tx=0).stop 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53a7400700 1 -- 172.21.15.12:0/2793132705 reap_dead start 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 shutdown_connections 2024-05-27T07:19:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 >> 172.21.15.12:0/2793132705 conn(0x7f53a8100030 msgr2=0x7f53a810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 shutdown_connections 2024-05-27T07:19:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:54.053+0000 7f53adc00700 1 -- 172.21.15.12:0/2793132705 wait complete. 2024-05-27T07:19:55.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:54 smithi012 ceph-mon[31473]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:55.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:54 smithi162 ceph-mon[35676]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:55.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:54 smithi062 ceph-mon[35304]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:19:55.241 INFO:teuthology.orchestra.run.smithi012.stdout:55834574974 2024-05-27T07:19:55.241 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-27T07:19:57.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:56 smithi012 ceph-mon[31473]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-27T07:19:57.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:56 smithi162 ceph-mon[35676]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-27T07:19:57.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:56 smithi062 ceph-mon[35304]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-27T07:19:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 -- 172.21.15.12:0/547354541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5634106610 msgr2=0x7f5634106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 --2- 172.21.15.12:0/547354541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5634106610 0x7f5634106a20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f56280099c0 tx=0x7f5628009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 -- 172.21.15.12:0/547354541 shutdown_connections 2024-05-27T07:19:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 --2- 172.21.15.12:0/547354541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f563410b890 0x7f56341131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 --2- 172.21.15.12:0/547354541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5634106610 0x7f5634106a20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 --2- 172.21.15.12:0/547354541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f563410af40 0x7f563410b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 -- 172.21.15.12:0/547354541 >> 172.21.15.12:0/547354541 conn(0x7f5634102020 msgr2=0x7f5634104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 -- 172.21.15.12:0/547354541 shutdown_connections 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.718+0000 7f5639e00700 1 -- 172.21.15.12:0/547354541 wait complete. 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.719+0000 7f5639e00700 1 Processor -- start 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.719+0000 7f5639e00700 1 -- start start 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5639e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634106610 0x7f563419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5639e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f563410af40 0x7f563419d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5639e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f563410b890 0x7f56341a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5639e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f56341163c0 con 0x7f563410af40 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5639e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5634116240 con 0x7f5634106610 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5639e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5634116540 con 0x7f563410b890 2024-05-27T07:19:57.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5633e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f563410af40 0x7f563419d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5633e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f563410af40 0x7f563419d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47940/0 (socket says 172.21.15.12:47940) 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5633e00700 1 -- 172.21.15.12:0/1789146587 learned_addr learned my addr 172.21.15.12:0/1789146587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5638a00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634106610 0x7f563419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.720+0000 7f5639400700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f563410b890 0x7f56341a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5633e00700 1 -- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f563410b890 msgr2=0x7f56341a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5633e00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f563410b890 0x7f56341a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5633e00700 1 -- 172.21.15.12:0/1789146587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634106610 msgr2=0x7f563419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5633e00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634106610 0x7f563419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5633e00700 1 -- 172.21.15.12:0/1789146587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56341a1c80 con 0x7f563410af40 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5639400700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f563410b890 0x7f56341a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5633e00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f563410af40 0x7f563419d350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f562000d980 tx=0x7f562000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620017070 con 0x7f563410af40 2024-05-27T07:19:57.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56341a1f70 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5620010460 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562000de60 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.721+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56341a9890 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.722+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5634063f40 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.725+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5620010610 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.727+0000 7f5631600700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f561c0b2910 0x7f561c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.727+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f56200e2750 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.727+0000 7f5638a00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f561c0b2910 0x7f561c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.727+0000 7f5631600700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] conn(0x7f561c0bdcc0 0x7f561c0c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.727+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 --> [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5620016ec0 con 0x7f561c0bdcc0 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.727+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f56200e2b00 con 0x7f563410af40 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.728+0000 7f5638a00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f561c0b2910 0x7f561c0b4e20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5628000c00 tx=0x7f5628005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:57.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.728+0000 7f5639400700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] conn(0x7f561c0bdcc0 0x7f561c0c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.729+0000 7f5639400700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] conn(0x7f561c0bdcc0 0x7f561c0c01b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.732+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== osd.5 v2:172.21.15.162:6808/1419812101 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5620016ec0 con 0x7f561c0bdcc0 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.750+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 --> [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f56340643c0 con 0x7f561c0bdcc0 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.751+0000 7f5631600700 1 -- 172.21.15.12:0/1789146587 <== osd.5 v2:172.21.15.162:6808/1419812101 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f56340643c0 con 0x7f561c0bdcc0 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] conn(0x7f561c0bdcc0 msgr2=0x7f561c0c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] conn(0x7f561c0bdcc0 0x7f561c0c01b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f561c0b2910 msgr2=0x7f561c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f561c0b2910 0x7f561c0b4e20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5628000c00 tx=0x7f5628005fb0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f563410af40 msgr2=0x7f563419d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 --2- 172.21.15.12:0/1789146587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f563410af40 0x7f563419d350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f562000d980 tx=0x7f562000dd40 comp rx=0 tx=0).stop 2024-05-27T07:19:57.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639400700 1 -- 172.21.15.12:0/1789146587 reap_dead start 2024-05-27T07:19:57.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 shutdown_connections 2024-05-27T07:19:57.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 >> 172.21.15.12:0/1789146587 conn(0x7f5634102020 msgr2=0x7f563410a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:57.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.752+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 shutdown_connections 2024-05-27T07:19:57.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.753+0000 7f5639e00700 1 -- 172.21.15.12:0/1789146587 wait complete. 2024-05-27T07:19:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.232+0000 7fdf05400700 1 -- 172.21.15.12:0/312233434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00075d20 msgr2=0x7fdf00076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.232+0000 7fdf05400700 1 --2- 172.21.15.12:0/312233434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00075d20 0x7fdf00076130 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009990 tx=0x7fdef0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.233+0000 7fdf05400700 1 -- 172.21.15.12:0/312233434 shutdown_connections 2024-05-27T07:19:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.233+0000 7fdf05400700 1 --2- 172.21.15.12:0/312233434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf00110b90 0x7fdf00112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.233+0000 7fdf05400700 1 --2- 172.21.15.12:0/312233434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00075d20 0x7fdf00076130 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.233+0000 7fdf05400700 1 --2- 172.21.15.12:0/312233434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf00076670 0x7fdf00110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.233+0000 7fdf05400700 1 -- 172.21.15.12:0/312233434 >> 172.21.15.12:0/312233434 conn(0x7fdf00100030 msgr2=0x7fdf00102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.233+0000 7fdf05400700 1 -- 172.21.15.12:0/312233434 shutdown_connections 2024-05-27T07:19:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.233+0000 7fdf05400700 1 -- 172.21.15.12:0/312233434 wait complete. 2024-05-27T07:19:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.234+0000 7fdf05400700 1 Processor -- start 2024-05-27T07:19:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.234+0000 7fdf05400700 1 -- start start 2024-05-27T07:19:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdf05400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf00075d20 0x7fdf0019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdf05400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00076670 0x7fdf0019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdf05400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf00110b90 0x7fdf001a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdf05400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00116580 con 0x7fdf00076670 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdf05400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00116400 con 0x7fdf00075d20 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdf05400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00116700 con 0x7fdf00110b90 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00076670 0x7fdf0019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00076670 0x7fdf0019d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47936/0 (socket says 172.21.15.12:47936) 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 -- 172.21.15.12:0/3740973046 learned_addr learned my addr 172.21.15.12:0/3740973046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdefe800700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf00075d20 0x7fdf0019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdeff200700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf00110b90 0x7fdf001a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 -- 172.21.15.12:0/3740973046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf00110b90 msgr2=0x7fdf001a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf00110b90 0x7fdf001a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 -- 172.21.15.12:0/3740973046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf00075d20 msgr2=0x7fdf0019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf00075d20 0x7fdf0019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.235+0000 7fdef7e00700 1 -- 172.21.15.12:0/3740973046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf001a1ea0 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.236+0000 7fdef7e00700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00076670 0x7fdf0019d570 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdee800d950 tx=0x7fdee800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.236+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee8016070 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.236+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdee8010460 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.236+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf001a2190 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.236+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee800f620 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.236+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf001a9ab0 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.237+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdf00063f40 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.240+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fdee8019070 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.241+0000 7fdef7400700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdee40b2990 0x7fdee40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.242+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdee80e28a0 con 0x7fdf00076670 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.242+0000 7fdefe800700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdee40b2990 0x7fdee40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.242+0000 7fdef7400700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] conn(0x7fdee40bdd40 0x7fdee40c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.242+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 --> [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdee40c0900 con 0x7fdee40bdd40 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.242+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fdee80e2c50 con 0x7fdf00076670 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.242+0000 7fdeff200700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] conn(0x7fdee40bdd40 0x7fdee40c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.243+0000 7fdefe800700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdee40b2990 0x7fdee40b4ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdef000b5c0 tx=0x7fdef0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.243+0000 7fdeff200700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] conn(0x7fdee40bdd40 0x7fdee40c0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.244+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== osd.3 v2:172.21.15.12:6810/3716699952 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdee40c0900 con 0x7fdee40bdd40 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.265+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 --> [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdf00064340 con 0x7fdee40bdd40 2024-05-27T07:19:57.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.266+0000 7fdef7400700 1 -- 172.21.15.12:0/3740973046 <== osd.3 v2:172.21.15.12:6810/3716699952 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdf00064340 con 0x7fdee40bdd40 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] conn(0x7fdee40bdd40 msgr2=0x7fdee40c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] conn(0x7fdee40bdd40 0x7fdee40c0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdee40b2990 msgr2=0x7fdee40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdee40b2990 0x7fdee40b4ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdef000b5c0 tx=0x7fdef0005fb0 comp rx=0 tx=0).stop 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00076670 msgr2=0x7fdf0019d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 --2- 172.21.15.12:0/3740973046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf00076670 0x7fdf0019d570 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdee800d950 tx=0x7fdee800dd10 comp rx=0 tx=0).stop 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdeff200700 1 -- 172.21.15.12:0/3740973046 reap_dead start 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 shutdown_connections 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 >> 172.21.15.12:0/3740973046 conn(0x7fdf00100030 msgr2=0x7fdf00100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:57.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 shutdown_connections 2024-05-27T07:19:57.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:56.267+0000 7fdf05400700 1 -- 172.21.15.12:0/3740973046 wait complete. 2024-05-27T07:19:58.691 INFO:teuthology.orchestra.run.smithi012.stdout:107374182514 2024-05-27T07:19:58.692 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-27T07:19:58.947 INFO:teuthology.orchestra.run.smithi012.stdout:300647710773 2024-05-27T07:19:58.947 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-27T07:19:59.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:19:58 smithi162 ceph-mon[35676]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:59.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:19:58 smithi062 ceph-mon[35304]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:59.255 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:19:58 smithi012 ceph-mon[31473]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:19:59.467 INFO:teuthology.orchestra.run.smithi012.stdout:180388626521 2024-05-27T07:19:59.467 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.908+0000 7f04da000700 1 -- 172.21.15.12:0/3937438702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105bd0 msgr2=0x7f04d410e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.908+0000 7f04da000700 1 --2- 172.21.15.12:0/3937438702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105bd0 0x7f04d410e120 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009990 tx=0x7f04c0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.909+0000 7f04da000700 1 -- 172.21.15.12:0/3937438702 shutdown_connections 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.909+0000 7f04da000700 1 --2- 172.21.15.12:0/3937438702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d4105280 0x7f04d4105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.909+0000 7f04da000700 1 --2- 172.21.15.12:0/3937438702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d410e6f0 0x7f04d4110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.909+0000 7f04da000700 1 --2- 172.21.15.12:0/3937438702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105bd0 0x7f04d410e120 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.909+0000 7f04da000700 1 -- 172.21.15.12:0/3937438702 >> 172.21.15.12:0/3937438702 conn(0x7f04d4078b40 msgr2=0x7f04d4078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.909+0000 7f04da000700 1 -- 172.21.15.12:0/3937438702 shutdown_connections 2024-05-27T07:19:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.909+0000 7f04da000700 1 -- 172.21.15.12:0/3937438702 wait complete. 2024-05-27T07:19:59.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.910+0000 7f04da000700 1 Processor -- start 2024-05-27T07:19:59.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.910+0000 7f04da000700 1 -- start start 2024-05-27T07:19:59.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.910+0000 7f04da000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105280 0x7f04d4102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04da000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4105bd0 0x7f04d41034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04da000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d410e6f0 0x7f04d41039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04da000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4114170 con 0x7f04d4105280 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04da000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4113ff0 con 0x7f04d410e6f0 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04da000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41142f0 con 0x7f04d4105bd0 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105280 0x7f04d4102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105280 0x7f04d4102f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47962/0 (socket says 172.21.15.12:47962) 2024-05-27T07:19:59.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 -- 172.21.15.12:0/3182913330 learned_addr learned my addr 172.21.15.12:0/3182913330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:19:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d3e00700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4105bd0 0x7f04d41034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 -- 172.21.15.12:0/3182913330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4105bd0 msgr2=0x7f04d41034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4105bd0 0x7f04d41034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 -- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d410e6f0 msgr2=0x7f04d41039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d9600700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d410e6f0 0x7f04d41039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d410e6f0 0x7f04d41039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d8c00700 1 -- 172.21.15.12:0/3182913330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d41a8fa0 con 0x7f04d4105280 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.911+0000 7f04d9600700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d410e6f0 0x7f04d41039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.912+0000 7f04d8c00700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105280 0x7f04d4102f70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f04c800d980 tx=0x7f04c800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.912+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8017070 con 0x7f04d4105280 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.912+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f04c8004830 con 0x7f04d4105280 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.912+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c80053a0 con 0x7f04d4105280 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.912+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d41a9290 con 0x7f04d4105280 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.912+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d41a97d0 con 0x7f04d4105280 2024-05-27T07:19:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.913+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f04d4063ec0 con 0x7f04d4105280 2024-05-27T07:19:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.916+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f04c80049e0 con 0x7f04d4105280 2024-05-27T07:19:59.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.917+0000 7f04d1600700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f04bc0b2990 0x7f04bc0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:59.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.918+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f04c80e2c20 con 0x7f04d4105280 2024-05-27T07:19:59.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.918+0000 7f04d3e00700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f04bc0b2990 0x7f04bc0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:59.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.918+0000 7f04d1600700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] conn(0x7f04bc0bdd40 0x7f04bc0c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:59.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.918+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 --> [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04c801ae70 con 0x7f04bc0bdd40 2024-05-27T07:19:59.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.918+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f04c80e2fd0 con 0x7f04d4105280 2024-05-27T07:19:59.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.919+0000 7f04d3e00700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f04bc0b2990 0x7f04bc0b4ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f04c000b5c0 tx=0x7f04c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:59.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.919+0000 7f04d9600700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] conn(0x7f04bc0bdd40 0x7f04bc0c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.920+0000 7f04d9600700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] conn(0x7f04bc0bdd40 0x7f04bc0c0230 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.922+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== osd.10 v2:172.21.15.62:6824/3686530646 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04c801ae70 con 0x7f04bc0bdd40 2024-05-27T07:19:59.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.941+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 --> [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f04d40642c0 con 0x7f04bc0bdd40 2024-05-27T07:19:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.942+0000 7f04d1600700 1 -- 172.21.15.12:0/3182913330 <== osd.10 v2:172.21.15.62:6824/3686530646 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f04d40642c0 con 0x7f04bc0bdd40 2024-05-27T07:19:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.942+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] conn(0x7f04bc0bdd40 msgr2=0x7f04bc0c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.942+0000 7f04da000700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] conn(0x7f04bc0bdd40 0x7f04bc0c0230 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.942+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f04bc0b2990 msgr2=0x7f04bc0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.942+0000 7f04da000700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f04bc0b2990 0x7f04bc0b4ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f04c000b5c0 tx=0x7f04c0005fb0 comp rx=0 tx=0).stop 2024-05-27T07:19:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.943+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105280 msgr2=0x7f04d4102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.943+0000 7f04da000700 1 --2- 172.21.15.12:0/3182913330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04d4105280 0x7f04d4102f70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f04c800d980 tx=0x7f04c800dd40 comp rx=0 tx=0).stop 2024-05-27T07:19:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.943+0000 7f04d9600700 1 -- 172.21.15.12:0/3182913330 reap_dead start 2024-05-27T07:19:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.943+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 shutdown_connections 2024-05-27T07:19:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.943+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 >> 172.21.15.12:0/3182913330 conn(0x7f04d4078b40 msgr2=0x7f04d41010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:59.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.943+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 shutdown_connections 2024-05-27T07:19:59.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:19:59.943+0000 7f04da000700 1 -- 172.21.15.12:0/3182913330 wait complete. 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.509+0000 7fb55ea00700 1 -- 172.21.15.12:0/3845012319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558110b90 msgr2=0x7fb558112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.509+0000 7fb55ea00700 1 --2- 172.21.15.12:0/3845012319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558110b90 0x7fb558112ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb550009990 tx=0x7fb550009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.510+0000 7fb55ea00700 1 -- 172.21.15.12:0/3845012319 shutdown_connections 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.510+0000 7fb55ea00700 1 --2- 172.21.15.12:0/3845012319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558110b90 0x7fb558112ff0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.510+0000 7fb55ea00700 1 --2- 172.21.15.12:0/3845012319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558075d20 0x7fb558076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.510+0000 7fb55ea00700 1 --2- 172.21.15.12:0/3845012319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb558076670 0x7fb558110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.510+0000 7fb55ea00700 1 -- 172.21.15.12:0/3845012319 >> 172.21.15.12:0/3845012319 conn(0x7fb558100030 msgr2=0x7fb558102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.510+0000 7fb55ea00700 1 -- 172.21.15.12:0/3845012319 shutdown_connections 2024-05-27T07:20:00.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.510+0000 7fb55ea00700 1 -- 172.21.15.12:0/3845012319 wait complete. 2024-05-27T07:20:00.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 Processor -- start 2024-05-27T07:20:00.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 -- start start 2024-05-27T07:20:00.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558075d20 0x7fb55819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:00.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558076670 0x7fb55819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb558110b90 0x7fb5581a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb558116520 con 0x7fb558076670 2024-05-27T07:20:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581163a0 con 0x7fb558110b90 2024-05-27T07:20:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.511+0000 7fb55ea00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581166a0 con 0x7fb558075d20 2024-05-27T07:20:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558076670 0x7fb55819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558076670 0x7fb55819d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47984/0 (socket says 172.21.15.12:47984) 2024-05-27T07:20:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 -- 172.21.15.12:0/1209875909 learned_addr learned my addr 172.21.15.12:0/1209875909 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb557e00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558075d20 0x7fb55819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb55ca00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb558110b90 0x7fb5581a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 -- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558075d20 msgr2=0x7fb55819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558075d20 0x7fb55819ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 -- 172.21.15.12:0/1209875909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb558110b90 msgr2=0x7fb5581a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb558110b90 0x7fb5581a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.512+0000 7fb54f400700 1 -- 172.21.15.12:0/1209875909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5581a1cd0 con 0x7fb558076670 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.513+0000 7fb54f400700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558076670 0x7fb55819d450 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c9b0 tx=0x7fb54000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.513+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540007b00 con 0x7fb558076670 2024-05-27T07:20:00.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.513+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb540007ca0 con 0x7fb558076670 2024-05-27T07:20:00.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.513+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5581a1fc0 con 0x7fb558076670 2024-05-27T07:20:00.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.513+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540018660 con 0x7fb558076670 2024-05-27T07:20:00.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.513+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5581a9990 con 0x7fb558076670 2024-05-27T07:20:00.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.514+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb534001060 con 0x7fb558076670 2024-05-27T07:20:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.517+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb5400188c0 con 0x7fb558076670 2024-05-27T07:20:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.519+0000 7fb555600700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb53c0b2990 0x7fb53c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.519+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fb5400e2cd0 con 0x7fb558076670 2024-05-27T07:20:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.519+0000 7fb557e00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb53c0b2990 0x7fb53c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.519+0000 7fb555600700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] conn(0x7fb53c0bdd40 0x7fb53c0c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.519+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 --> [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb540014ec0 con 0x7fb53c0bdd40 2024-05-27T07:20:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.519+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb5400e3120 con 0x7fb558076670 2024-05-27T07:20:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.520+0000 7fb557e00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb53c0b2990 0x7fb53c0b4ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb5480060b0 tx=0x7fb548007b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:00.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.520+0000 7fb55ca00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] conn(0x7fb53c0bdd40 0x7fb53c0c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:00.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.521+0000 7fb55ca00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] conn(0x7fb53c0bdd40 0x7fb53c0c0230 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:00.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.524+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== osd.2 v2:172.21.15.162:6800/2540739935 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb540014ec0 con 0x7fb53c0bdd40 2024-05-27T07:20:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.542+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 --> [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb534002e90 con 0x7fb53c0bdd40 2024-05-27T07:20:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.543+0000 7fb555600700 1 -- 172.21.15.12:0/1209875909 <== osd.2 v2:172.21.15.162:6800/2540739935 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb534002e90 con 0x7fb53c0bdd40 2024-05-27T07:20:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] conn(0x7fb53c0bdd40 msgr2=0x7fb53c0c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] conn(0x7fb53c0bdd40 0x7fb53c0c0230 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb53c0b2990 msgr2=0x7fb53c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb53c0b2990 0x7fb53c0b4ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb5480060b0 tx=0x7fb548007b80 comp rx=0 tx=0).stop 2024-05-27T07:20:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558076670 msgr2=0x7fb55819d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 --2- 172.21.15.12:0/1209875909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb558076670 0x7fb55819d450 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c9b0 tx=0x7fb54000cd70 comp rx=0 tx=0).stop 2024-05-27T07:20:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ca00700 1 -- 172.21.15.12:0/1209875909 reap_dead start 2024-05-27T07:20:00.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 shutdown_connections 2024-05-27T07:20:00.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.544+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 >> 172.21.15.12:0/1209875909 conn(0x7fb558100030 msgr2=0x7fb558100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:00.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.545+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 shutdown_connections 2024-05-27T07:20:00.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:00.545+0000 7fb55ea00700 1 -- 172.21.15.12:0/1209875909 wait complete. 2024-05-27T07:20:01.070 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:00 smithi062 ceph-mon[35304]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:00 smithi062 ceph-mon[35304]: overall HEALTH_OK 2024-05-27T07:20:01.284 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:00 smithi012 ceph-mon[31473]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:01.284 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:00 smithi012 ceph-mon[31473]: overall HEALTH_OK 2024-05-27T07:20:01.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:00 smithi162 ceph-mon[35676]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:01.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:00 smithi162 ceph-mon[35676]: overall HEALTH_OK 2024-05-27T07:20:01.715 INFO:teuthology.orchestra.run.smithi012.stdout:77309411449 2024-05-27T07:20:01.715 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-27T07:20:02.365 INFO:teuthology.orchestra.run.smithi012.stdout:154618822754 2024-05-27T07:20:02.366 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-27T07:20:02.683 INFO:teuthology.orchestra.run.smithi012.stdout:274877907004 2024-05-27T07:20:02.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-27T07:20:02.979 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:02 smithi012 ceph-mon[31473]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:20:03.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:02 smithi062 ceph-mon[35304]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:20:03.276 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:02 smithi162 ceph-mon[35676]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:20:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 -- 172.21.15.12:0/2143393701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00075d20 msgr2=0x7f9b00076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 --2- 172.21.15.12:0/2143393701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00075d20 0x7f9b00076130 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009990 tx=0x7f9af0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 -- 172.21.15.12:0/2143393701 shutdown_connections 2024-05-27T07:20:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 --2- 172.21.15.12:0/2143393701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b001109a0 0x7f9b00112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 --2- 172.21.15.12:0/2143393701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00075d20 0x7f9b00076130 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 --2- 172.21.15.12:0/2143393701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b00076670 0x7f9b00110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 -- 172.21.15.12:0/2143393701 >> 172.21.15.12:0/2143393701 conn(0x7f9b00079d00 msgr2=0x7f9b0007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.957+0000 7f9aff200700 1 -- 172.21.15.12:0/2143393701 shutdown_connections 2024-05-27T07:20:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.958+0000 7f9aff200700 1 -- 172.21.15.12:0/2143393701 wait complete. 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.959+0000 7f9aff200700 1 Processor -- start 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.959+0000 7f9aff200700 1 -- start start 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.959+0000 7f9aff200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b00075d20 0x7f9b0019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.959+0000 7f9aff200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00076670 0x7f9b0019d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.959+0000 7f9aff200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b001109a0 0x7f9b001a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9aff200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00116340 con 0x7f9b00076670 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9aff200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001161c0 con 0x7f9b00075d20 2024-05-27T07:20:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9aff200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001164c0 con 0x7f9b001109a0 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00076670 0x7f9b0019d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00076670 0x7f9b0019d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36138/0 (socket says 172.21.15.12:36138) 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 -- 172.21.15.12:0/3796952418 learned_addr learned my addr 172.21.15.12:0/3796952418 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afde00700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b00075d20 0x7f9b0019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afe800700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b001109a0 0x7f9b001a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 -- 172.21.15.12:0/3796952418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b001109a0 msgr2=0x7f9b001a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b001109a0 0x7f9b001a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 -- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b00075d20 msgr2=0x7f9b0019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b00075d20 0x7f9b0019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afd400700 1 -- 172.21.15.12:0/3796952418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b001a1c00 con 0x7f9b00076670 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.960+0000 7f9afde00700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b00075d20 0x7f9b0019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.961+0000 7f9afd400700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00076670 0x7f9b0019d2d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00d950 tx=0x7f9aec00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.961+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec016070 con 0x7f9b00076670 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.961+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9aec010460 con 0x7f9b00076670 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.961+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b001a1ef0 con 0x7f9b00076670 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.961+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00f620 con 0x7f9b00076670 2024-05-27T07:20:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.961+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b001a9810 con 0x7f9b00076670 2024-05-27T07:20:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.962+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b00063f40 con 0x7f9b00076670 2024-05-27T07:20:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.966+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9aec019070 con 0x7f9b00076670 2024-05-27T07:20:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.968+0000 7f9af6a00700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ae40b2910 0x7f9ae40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.968+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f9aec0e28a0 con 0x7f9b00076670 2024-05-27T07:20:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.968+0000 7f9afde00700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ae40b2910 0x7f9ae40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.968+0000 7f9af6a00700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] conn(0x7f9ae40bdcc0 0x7f9ae40c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.968+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 --> [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ae40c0880 con 0x7f9ae40bdcc0 2024-05-27T07:20:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.968+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f9aec0e2cf0 con 0x7f9b00076670 2024-05-27T07:20:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.969+0000 7f9afde00700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ae40b2910 0x7f9ae40b4e20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9af00098e0 tx=0x7f9af000b580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.969+0000 7f9afe800700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] conn(0x7f9ae40bdcc0 0x7f9ae40c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.970+0000 7f9afe800700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] conn(0x7f9ae40bdcc0 0x7f9ae40c01b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.972+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== osd.4 v2:172.21.15.62:6808/2898003339 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ae40c0880 con 0x7f9ae40bdcc0 2024-05-27T07:20:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.991+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 --> [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b000643c0 con 0x7f9ae40bdcc0 2024-05-27T07:20:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.992+0000 7f9af6a00700 1 -- 172.21.15.12:0/3796952418 <== osd.4 v2:172.21.15.62:6808/2898003339 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9b000643c0 con 0x7f9ae40bdcc0 2024-05-27T07:20:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] conn(0x7f9ae40bdcc0 msgr2=0x7f9ae40c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] conn(0x7f9ae40bdcc0 0x7f9ae40c01b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ae40b2910 msgr2=0x7f9ae40b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9ae40b2910 0x7f9ae40b4e20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9af00098e0 tx=0x7f9af000b580 comp rx=0 tx=0).stop 2024-05-27T07:20:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00076670 msgr2=0x7f9b0019d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 --2- 172.21.15.12:0/3796952418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b00076670 0x7f9b0019d2d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00d950 tx=0x7f9aec00dd10 comp rx=0 tx=0).stop 2024-05-27T07:20:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9afe800700 1 -- 172.21.15.12:0/3796952418 reap_dead start 2024-05-27T07:20:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 shutdown_connections 2024-05-27T07:20:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.993+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 >> 172.21.15.12:0/3796952418 conn(0x7f9b00079d00 msgr2=0x7f9b00100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.994+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 shutdown_connections 2024-05-27T07:20:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.994+0000 7f9aff200700 1 -- 172.21.15.12:0/3796952418 wait complete. 2024-05-27T07:20:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.759+0000 7fc048600700 1 -- 172.21.15.12:0/2163836432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010b2d0 msgr2=0x7fc040117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.759+0000 7fc048600700 1 --2- 172.21.15.12:0/2163836432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010b2d0 0x7fc040117800 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc03800b0a0 tx=0x7fc03800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.760+0000 7fc048600700 1 -- 172.21.15.12:0/2163836432 shutdown_connections 2024-05-27T07:20:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.760+0000 7fc048600700 1 --2- 172.21.15.12:0/2163836432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010b2d0 0x7fc040117800 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.760+0000 7fc048600700 1 --2- 172.21.15.12:0/2163836432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc040110960 0x7fc040110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:04.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.760+0000 7fc048600700 1 --2- 172.21.15.12:0/2163836432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04010a980 0x7fc04010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.760+0000 7fc048600700 1 -- 172.21.15.12:0/2163836432 >> 172.21.15.12:0/2163836432 conn(0x7fc040077c70 msgr2=0x7fc0400780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.761+0000 7fc048600700 1 -- 172.21.15.12:0/2163836432 shutdown_connections 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.761+0000 7fc048600700 1 -- 172.21.15.12:0/2163836432 wait complete. 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.762+0000 7fc048600700 1 Processor -- start 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.763+0000 7fc048600700 1 -- start start 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.763+0000 7fc048600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010a980 0x7fc0401a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.763+0000 7fc048600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04010b2d0 0x7fc0401a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.763+0000 7fc048600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc040110960 0x7fc0401a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.763+0000 7fc048600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc04011a930 con 0x7fc04010a980 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.763+0000 7fc048600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc04011a7b0 con 0x7fc04010b2d0 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.763+0000 7fc048600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc04011aab0 con 0x7fc040110960 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.764+0000 7fc045c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010a980 0x7fc0401a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:04.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.764+0000 7fc045c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010a980 0x7fc0401a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36124/0 (socket says 172.21.15.12:36124) 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.764+0000 7fc045c00700 1 -- 172.21.15.12:0/1814793708 learned_addr learned my addr 172.21.15.12:0/1814793708 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.764+0000 7fc045200700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04010b2d0 0x7fc0401a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.764+0000 7fc046600700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc040110960 0x7fc0401a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.765+0000 7fc045c00700 1 -- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc040110960 msgr2=0x7fc0401a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.765+0000 7fc045c00700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc040110960 0x7fc0401a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.765+0000 7fc045c00700 1 -- 172.21.15.12:0/1814793708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04010b2d0 msgr2=0x7fc0401a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.765+0000 7fc045c00700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04010b2d0 0x7fc0401a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.765+0000 7fc045c00700 1 -- 172.21.15.12:0/1814793708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0401a61f0 con 0x7fc04010a980 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.765+0000 7fc045c00700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010a980 0x7fc0401a1320 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc03400d980 tx=0x7fc03400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.766+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034017070 con 0x7fc04010a980 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.766+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc03400fd20 con 0x7fc04010a980 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.766+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03400e3b0 con 0x7fc04010a980 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.767+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0401a6480 con 0x7fc04010a980 2024-05-27T07:20:04.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.767+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0401a6970 con 0x7fc04010a980 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.768+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc040063ec0 con 0x7fc04010a980 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.771+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc03400f890 con 0x7fc04010a980 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.772+0000 7fc03e000700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0280b2a60 0x7fc0280b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.773+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fc0340e24c0 con 0x7fc04010a980 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.773+0000 7fc03e000700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] conn(0x7fc0280bde10 0x7fc0280c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.773+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 --> [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc034022e70 con 0x7fc0280bde10 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.773+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fc0340e2870 con 0x7fc04010a980 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.774+0000 7fc045200700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0280b2a60 0x7fc0280b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.774+0000 7fc046600700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] conn(0x7fc0280bde10 0x7fc0280c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.774+0000 7fc045200700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0280b2a60 0x7fc0280b4f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc040109940 tx=0x7fc02c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.775+0000 7fc046600700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] conn(0x7fc0280bde10 0x7fc0280c0300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.777+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== osd.8 v2:172.21.15.162:6816/1474732705 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc034022e70 con 0x7fc0280bde10 2024-05-27T07:20:04.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.796+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 --> [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc040064310 con 0x7fc0280bde10 2024-05-27T07:20:04.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc03e000700 1 -- 172.21.15.12:0/1814793708 <== osd.8 v2:172.21.15.162:6816/1474732705 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc040064310 con 0x7fc0280bde10 2024-05-27T07:20:04.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] conn(0x7fc0280bde10 msgr2=0x7fc0280c0300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:04.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] conn(0x7fc0280bde10 0x7fc0280c0300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0280b2a60 msgr2=0x7fc0280b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc0280b2a60 0x7fc0280b4f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc040109940 tx=0x7fc02c00b410 comp rx=0 tx=0).stop 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010a980 msgr2=0x7fc0401a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 --2- 172.21.15.12:0/1814793708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc04010a980 0x7fc0401a1320 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc03400d980 tx=0x7fc03400dc90 comp rx=0 tx=0).stop 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc046600700 1 -- 172.21.15.12:0/1814793708 reap_dead start 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 shutdown_connections 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.797+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 >> 172.21.15.12:0/1814793708 conn(0x7fc040077c70 msgr2=0x7fc04007b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:04.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.798+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 shutdown_connections 2024-05-27T07:20:04.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:03.798+0000 7fc048600700 1 -- 172.21.15.12:0/1814793708 wait complete. 2024-05-27T07:20:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:05.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:04 smithi062 ceph-mon[35304]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-27T07:20:05.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:04 smithi012 ceph-mon[31473]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-27T07:20:05.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:04 smithi162 ceph-mon[35676]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-27T07:20:05.737 INFO:teuthology.orchestra.run.smithi012.stdout:133143986283 2024-05-27T07:20:05.738 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-27T07:20:07.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:07 smithi012 ceph-mon[31473]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:07.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:07 smithi162 ceph-mon[35676]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:07.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:07 smithi062 ceph-mon[35304]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:08.064 INFO:teuthology.orchestra.run.smithi012.stdout:227633266766 2024-05-27T07:20:08.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-27T07:20:08.685 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:08.686 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:08.686 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:08.931 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:08.932 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:08.934 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:09.229 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:09 smithi012 ceph-mon[31473]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T07:20:09.288 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:09 smithi062 ceph-mon[35304]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T07:20:09.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:09 smithi162 ceph-mon[35676]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T07:20:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 -- 172.21.15.12:0/3629775200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105bd0 msgr2=0x7f194810e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 --2- 172.21.15.12:0/3629775200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105bd0 0x7f194810e120 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1934009990 tx=0x7f1934009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 -- 172.21.15.12:0/3629775200 shutdown_connections 2024-05-27T07:20:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 --2- 172.21.15.12:0/3629775200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810e6f0 0x7f1948110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 --2- 172.21.15.12:0/3629775200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1948105280 0x7f1948105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 --2- 172.21.15.12:0/3629775200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105bd0 0x7f194810e120 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 -- 172.21.15.12:0/3629775200 >> 172.21.15.12:0/3629775200 conn(0x7f1948078b40 msgr2=0x7f1948078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 -- 172.21.15.12:0/3629775200 shutdown_connections 2024-05-27T07:20:09.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.592+0000 7f194e400700 1 -- 172.21.15.12:0/3629775200 wait complete. 2024-05-27T07:20:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 Processor -- start 2024-05-27T07:20:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 -- start start 2024-05-27T07:20:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105280 0x7f19481a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1948105bd0 0x7f19481a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810e6f0 0x7f194819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1948114150 con 0x7f1948105280 2024-05-27T07:20:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1948113fd0 con 0x7f194810e6f0 2024-05-27T07:20:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.594+0000 7f194e400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19481142d0 con 0x7f1948105bd0 2024-05-27T07:20:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105280 0x7f19481a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105280 0x7f19481a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36154/0 (socket says 172.21.15.12:36154) 2024-05-27T07:20:09.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 -- 172.21.15.12:0/482593712 learned_addr learned my addr 172.21.15.12:0/482593712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f1947e00700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1948105bd0 0x7f19481a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 -- 172.21.15.12:0/482593712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1948105bd0 msgr2=0x7f19481a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1948105bd0 0x7f19481a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 -- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810e6f0 msgr2=0x7f194819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194da00700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810e6f0 0x7f194819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810e6f0 0x7f194819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194d000700 1 -- 172.21.15.12:0/482593712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194819aff0 con 0x7f1948105280 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.595+0000 7f194da00700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194810e6f0 0x7f194819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.596+0000 7f194d000700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105280 0x7f19481a0790 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f193c00c920 tx=0x7f193c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.596+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c007b00 con 0x7f1948105280 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.596+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f193c004cc0 con 0x7f1948105280 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.596+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c0054f0 con 0x7f1948105280 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.596+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194819b2e0 con 0x7f1948105280 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.596+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194819b7d0 con 0x7f1948105280 2024-05-27T07:20:09.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.597+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1948063f40 con 0x7f1948105280 2024-05-27T07:20:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.600+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f193c007ca0 con 0x7f1948105280 2024-05-27T07:20:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.602+0000 7f1945600700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f19300b2990 0x7f19300b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.602+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f193c0e2780 con 0x7f1948105280 2024-05-27T07:20:09.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.602+0000 7f1947e00700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f19300b2990 0x7f19300b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.602+0000 7f1945600700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] conn(0x7f19300bdd40 0x7f19300c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.603+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 --> [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f193c004b10 con 0x7f19300bdd40 2024-05-27T07:20:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.603+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f193c0e2b30 con 0x7f1948105280 2024-05-27T07:20:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.603+0000 7f1947e00700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f19300b2990 0x7f19300b4ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1934003760 tx=0x7f1934005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:09.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.603+0000 7f194da00700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] conn(0x7f19300bdd40 0x7f19300c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.604+0000 7f194da00700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] conn(0x7f19300bdd40 0x7f19300c0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:09.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.607+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== osd.7 v2:172.21.15.62:6816/3439452424 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f193c004b10 con 0x7f19300bdd40 2024-05-27T07:20:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.626+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 --> [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1948064410 con 0x7f19300bdd40 2024-05-27T07:20:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f1945600700 1 -- 172.21.15.12:0/482593712 <== osd.7 v2:172.21.15.62:6816/3439452424 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1948064410 con 0x7f19300bdd40 2024-05-27T07:20:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] conn(0x7f19300bdd40 msgr2=0x7f19300c0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f194e400700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] conn(0x7f19300bdd40 0x7f19300c0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f19300b2990 msgr2=0x7f19300b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f194e400700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f19300b2990 0x7f19300b4ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1934003760 tx=0x7f1934005fb0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105280 msgr2=0x7f19481a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f194e400700 1 --2- 172.21.15.12:0/482593712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1948105280 0x7f19481a0790 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f193c00c920 tx=0x7f193c00cce0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.628+0000 7f194da00700 1 -- 172.21.15.12:0/482593712 reap_dead start 2024-05-27T07:20:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.629+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 shutdown_connections 2024-05-27T07:20:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.629+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 >> 172.21.15.12:0/482593712 conn(0x7f1948078b40 msgr2=0x7f19480fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.629+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 shutdown_connections 2024-05-27T07:20:09.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:09.629+0000 7f194e400700 1 -- 172.21.15.12:0/482593712 wait complete. 2024-05-27T07:20:11.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:11 smithi012 ceph-mon[31473]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:11.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:11 smithi162 ceph-mon[35676]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:11.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:11 smithi062 ceph-mon[35304]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:13.215 INFO:teuthology.orchestra.run.smithi012.stdout:201863462997 2024-05-27T07:20:13.216 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-27T07:20:13.285 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:13 smithi012 ceph-mon[31473]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:13.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:13 smithi162 ceph-mon[35676]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:13.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:13 smithi062 ceph-mon[35304]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:14.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.782+0000 7fd846800700 1 -- 172.21.15.12:0/4278531264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd84010af40 msgr2=0x7fd84010b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:14.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.782+0000 7fd846800700 1 --2- 172.21.15.12:0/4278531264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd84010af40 0x7fd84010b350 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0099c0 tx=0x7fd82c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.782+0000 7fd846800700 1 -- 172.21.15.12:0/4278531264 shutdown_connections 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.783+0000 7fd846800700 1 --2- 172.21.15.12:0/4278531264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106610 0x7fd840106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.783+0000 7fd846800700 1 --2- 172.21.15.12:0/4278531264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd84010b890 0x7fd8401131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.783+0000 7fd846800700 1 --2- 172.21.15.12:0/4278531264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd84010af40 0x7fd84010b350 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.783+0000 7fd846800700 1 -- 172.21.15.12:0/4278531264 >> 172.21.15.12:0/4278531264 conn(0x7fd840102020 msgr2=0x7fd840104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.783+0000 7fd846800700 1 -- 172.21.15.12:0/4278531264 shutdown_connections 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.783+0000 7fd846800700 1 -- 172.21.15.12:0/4278531264 wait complete. 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.784+0000 7fd846800700 1 Processor -- start 2024-05-27T07:20:14.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.785+0000 7fd846800700 1 -- start start 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.785+0000 7fd846800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd840106610 0x7fd84019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.785+0000 7fd846800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd84010af40 0x7fd84019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.785+0000 7fd846800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010b890 0x7fd8401a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.785+0000 7fd846800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd8401163a0 con 0x7fd840106610 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.785+0000 7fd846800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd840116220 con 0x7fd84010b890 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.785+0000 7fd846800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd840116520 con 0x7fd84010af40 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd840106610 0x7fd84019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd840106610 0x7fd84019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48694/0 (socket says 172.21.15.12:48694) 2024-05-27T07:20:14.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 -- 172.21.15.12:0/2337282030 learned_addr learned my addr 172.21.15.12:0/2337282030 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:14.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd844a00700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd84010af40 0x7fd84019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:14.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845e00700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010b890 0x7fd8401a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 -- 172.21.15.12:0/2337282030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd84010af40 msgr2=0x7fd84019d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd84010af40 0x7fd84019d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 -- 172.21.15.12:0/2337282030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010b890 msgr2=0x7fd8401a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010b890 0x7fd8401a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845400700 1 -- 172.21.15.12:0/2337282030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8401a1c70 con 0x7fd840106610 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.786+0000 7fd845e00700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010b890 0x7fd8401a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.787+0000 7fd845400700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd840106610 0x7fd84019ce50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd83400d950 tx=0x7fd83400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.787+0000 7fd83e000700 1 -- 172.21.15.12:0/2337282030 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd834016070 con 0x7fd840106610 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.787+0000 7fd83e000700 1 -- 172.21.15.12:0/2337282030 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd834010460 con 0x7fd840106610 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.787+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8401a1f60 con 0x7fd840106610 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.787+0000 7fd83e000700 1 -- 172.21.15.12:0/2337282030 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83400f620 con 0x7fd840106610 2024-05-27T07:20:14.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.787+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8401a24c0 con 0x7fd840106610 2024-05-27T07:20:14.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.789+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8400512b0 con 0x7fd840106610 2024-05-27T07:20:14.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.792+0000 7fd83e000700 1 -- 172.21.15.12:0/2337282030 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd834019070 con 0x7fd840106610 2024-05-27T07:20:14.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.794+0000 7fd83e000700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8280b2990 0x7fd8280b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:14.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.795+0000 7fd83e000700 1 -- 172.21.15.12:0/2337282030 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fd8340e2330 con 0x7fd840106610 2024-05-27T07:20:14.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.795+0000 7fd844a00700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8280b2990 0x7fd8280b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:14.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.795+0000 7fd844a00700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8280b2990 0x7fd8280b4ea0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009990 tx=0x7fd82c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:14.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:14.798+0000 7fd83e000700 1 -- 172.21.15.12:0/2337282030 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd8340a9ac0 con 0x7fd840106610 2024-05-27T07:20:15.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.042+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd84002e5e0 con 0x7fd840106610 2024-05-27T07:20:15.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:15 smithi012 ceph-mon[31473]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:20:15.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.043+0000 7fd83e000700 1 -- 172.21.15.12:0/2337282030 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd8340a93c0 con 0x7fd840106610 2024-05-27T07:20:15.034 INFO:teuthology.orchestra.run.smithi012.stdout:34359738506 2024-05-27T07:20:15.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8280b2990 msgr2=0x7fd8280b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:15.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8280b2990 0x7fd8280b4ea0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009990 tx=0x7fd82c005fb0 comp rx=0 tx=0).stop 2024-05-27T07:20:15.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd840106610 msgr2=0x7fd84019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:15.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd840106610 0x7fd84019ce50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd83400d950 tx=0x7fd83400dd10 comp rx=0 tx=0).stop 2024-05-27T07:20:15.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 shutdown_connections 2024-05-27T07:20:15.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8280b2990 0x7fd8280b4ea0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:15.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010b890 0x7fd8401a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:15.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd840106610 0x7fd84019ce50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:15.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 --2- 172.21.15.12:0/2337282030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd84010af40 0x7fd84019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:15.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.047+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 >> 172.21.15.12:0/2337282030 conn(0x7fd840102020 msgr2=0x7fd84010c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:15.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.048+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 shutdown_connections 2024-05-27T07:20:15.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:15.048+0000 7fd846800700 1 -- 172.21.15.12:0/2337282030 wait complete. 2024-05-27T07:20:15.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:15 smithi162 ceph-mon[35676]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:20:15.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:15 smithi062 ceph-mon[35304]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:20:16.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:16 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2337282030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:20:16.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:16 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2337282030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:20:16.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:16 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2337282030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.721+0000 7f4bd2c00700 1 -- 172.21.15.12:0/313102981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 msgr2=0x7f4bcc1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.721+0000 7f4bd2c00700 1 --2- 172.21.15.12:0/313102981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc1038c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0099c0 tx=0x7f4bbc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.721+0000 7f4bd2c00700 1 -- 172.21.15.12:0/313102981 shutdown_connections 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.721+0000 7f4bd2c00700 1 --2- 172.21.15.12:0/313102981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bcc108730 0x7f4bcc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.721+0000 7f4bd2c00700 1 --2- 172.21.15.12:0/313102981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc1038c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.721+0000 7f4bd2c00700 1 --2- 172.21.15.12:0/313102981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107de0 0x7f4bcc1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.721+0000 7f4bd2c00700 1 -- 172.21.15.12:0/313102981 >> 172.21.15.12:0/313102981 conn(0x7f4bcc0fedb0 msgr2=0x7f4bcc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.722+0000 7f4bd2c00700 1 -- 172.21.15.12:0/313102981 shutdown_connections 2024-05-27T07:20:16.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.722+0000 7f4bd2c00700 1 -- 172.21.15.12:0/313102981 wait complete. 2024-05-27T07:20:16.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.723+0000 7f4bd2c00700 1 Processor -- start 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.723+0000 7f4bd2c00700 1 -- start start 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.723+0000 7f4bd2c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.723+0000 7f4bd2c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bcc107de0 0x7f4bcc199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bd2c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc108730 0x7f4bcc19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bd2c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc113260 con 0x7f4bcc1034b0 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bd2c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1130e0 con 0x7f4bcc107de0 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bd2c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1133e0 con 0x7f4bcc108730 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bcbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:16.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bcbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48716/0 (socket says 172.21.15.12:48716) 2024-05-27T07:20:16.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bcbe00700 1 -- 172.21.15.12:0/607044553 learned_addr learned my addr 172.21.15.12:0/607044553 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:16.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bc3400700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bcc107de0 0x7f4bcc199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:16.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.724+0000 7f4bd0c00700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc108730 0x7f4bcc19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:16.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.725+0000 7f4bcbe00700 1 -- 172.21.15.12:0/607044553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc108730 msgr2=0x7f4bcc19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.725+0000 7f4bcbe00700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc108730 0x7f4bcc19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.725+0000 7f4bcbe00700 1 -- 172.21.15.12:0/607044553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bcc107de0 msgr2=0x7f4bcc199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.725+0000 7f4bcbe00700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bcc107de0 0x7f4bcc199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.725+0000 7f4bcbe00700 1 -- 172.21.15.12:0/607044553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc19a700 con 0x7f4bcc1034b0 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.725+0000 7f4bcbe00700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc199910 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00b580 tx=0x7f4bbc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.725+0000 7f4bc9600700 1 -- 172.21.15.12:0/607044553 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc027070 con 0x7f4bcc1034b0 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.726+0000 7f4bc9600700 1 -- 172.21.15.12:0/607044553 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4bbc00fd30 con 0x7f4bcc1034b0 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.726+0000 7f4bd2c00700 1 -- 172.21.15.12:0/607044553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bcc19ef60 con 0x7f4bcc1034b0 2024-05-27T07:20:16.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.726+0000 7f4bd2c00700 1 -- 172.21.15.12:0/607044553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bcc19f440 con 0x7f4bcc1034b0 2024-05-27T07:20:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.726+0000 7f4bc9600700 1 -- 172.21.15.12:0/607044553 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc022e50 con 0x7f4bcc1034b0 2024-05-27T07:20:16.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.727+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba4005330 con 0x7f4bcc1034b0 2024-05-27T07:20:16.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.730+0000 7f4bc9600700 1 -- 172.21.15.12:0/607044553 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4bbc02a070 con 0x7f4bcc1034b0 2024-05-27T07:20:16.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.734+0000 7f4bc9600700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4bb00b2b30 0x7f4bb00b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:16.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.734+0000 7f4bc9600700 1 -- 172.21.15.12:0/607044553 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f4bbc0e33d0 con 0x7f4bcc1034b0 2024-05-27T07:20:16.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.735+0000 7f4bc3400700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4bb00b2b30 0x7f4bb00b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:16.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.736+0000 7f4bc3400700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4bb00b2b30 0x7f4bb00b5040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc1069f0 tx=0x7f4bb400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:16.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:16.739+0000 7f4bc9600700 1 -- 172.21.15.12:0/607044553 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4bbc013070 con 0x7f4bcc1034b0 2024-05-27T07:20:16.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.006+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f4ba40058f0 con 0x7f4bcc1034b0 2024-05-27T07:20:16.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.007+0000 7f4bc9600700 1 -- 172.21.15.12:0/607044553 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4bbc0aac50 con 0x7f4bcc1034b0 2024-05-27T07:20:16.998 INFO:teuthology.orchestra.run.smithi012.stdout:55834574979 2024-05-27T07:20:17.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4bb00b2b30 msgr2=0x7f4bb00b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4bb00b2b30 0x7f4bb00b5040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc1069f0 tx=0x7f4bb400b410 comp rx=0 tx=0).stop 2024-05-27T07:20:17.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 msgr2=0x7f4bcc199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc199910 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00b580 tx=0x7f4bbc0047b0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 shutdown_connections 2024-05-27T07:20:17.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f4bb00b2b30 0x7f4bb00b5040 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc108730 0x7f4bcc19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1034b0 0x7f4bcc199910 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 --2- 172.21.15.12:0/607044553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bcc107de0 0x7f4bcc199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 >> 172.21.15.12:0/607044553 conn(0x7f4bcc0fedb0 msgr2=0x7f4bcc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 shutdown_connections 2024-05-27T07:20:17.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:17.010+0000 7f4bc2000700 1 -- 172.21.15.12:0/607044553 wait complete. 2024-05-27T07:20:17.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:17 smithi012 ceph-mon[31473]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T07:20:17.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:17 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/607044553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:20:17.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:17 smithi162 ceph-mon[35676]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T07:20:17.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:17 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/607044553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:20:17.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:17 smithi062 ceph-mon[35304]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-27T07:20:17.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:17 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/607044553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:20:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:18.055 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:18.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:18 smithi012 ceph-mon[31473]: mgrmap e18: x(active, since 13m), standbys: z, y 2024-05-27T07:20:18.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:18 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:18.385 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:18.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:18 smithi162 ceph-mon[35676]: mgrmap e18: x(active, since 13m), standbys: z, y 2024-05-27T07:20:18.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:18 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:18.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:18 smithi062 ceph-mon[35304]: mgrmap e18: x(active, since 13m), standbys: z, y 2024-05-27T07:20:18.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:18 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 -- 172.21.15.12:0/3320115389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4110ad0 msgr2=0x7f73f4112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 --2- 172.21.15.12:0/3320115389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4110ad0 0x7f73f4112ef0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00b0a0 tx=0x7f73ec00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 -- 172.21.15.12:0/3320115389 shutdown_connections 2024-05-27T07:20:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 --2- 172.21.15.12:0/3320115389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73f4102f20 0x7f73f4103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 --2- 172.21.15.12:0/3320115389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4110ad0 0x7f73f4112ef0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 --2- 172.21.15.12:0/3320115389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4103870 0x7f73f4110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 -- 172.21.15.12:0/3320115389 >> 172.21.15.12:0/3320115389 conn(0x7f73f4078ab0 msgr2=0x7f73f4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 -- 172.21.15.12:0/3320115389 shutdown_connections 2024-05-27T07:20:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.619+0000 7f73f3200700 1 -- 172.21.15.12:0/3320115389 wait complete. 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.620+0000 7f73f3200700 1 Processor -- start 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f3200700 1 -- start start 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f3200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4102f20 0x7f73f419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4103870 0x7f73f419d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73f4110ad0 0x7f73f41a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f3200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41163a0 con 0x7f73f4103870 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4116220 con 0x7f73f4110ad0 2024-05-27T07:20:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f3200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4116520 con 0x7f73f4102f20 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.621+0000 7f73f1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4103870 0x7f73f419d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4102f20 0x7f73f419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4103870 0x7f73f419d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48740/0 (socket says 172.21.15.12:48740) 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1400700 1 -- 172.21.15.12:0/4215976648 learned_addr learned my addr 172.21.15.12:0/4215976648 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f2800700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73f4110ad0 0x7f73f41a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1400700 1 -- 172.21.15.12:0/4215976648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4102f20 msgr2=0x7f73f419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1400700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4102f20 0x7f73f419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1400700 1 -- 172.21.15.12:0/4215976648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73f4110ad0 msgr2=0x7f73f41a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1400700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73f4110ad0 0x7f73f41a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f1400700 1 -- 172.21.15.12:0/4215976648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f41a1c00 con 0x7f73f4103870 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.622+0000 7f73f2800700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73f4110ad0 0x7f73f41a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.623+0000 7f73f1400700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4103870 0x7f73f419d2d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f73e000d950 tx=0x7f73e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.623+0000 7f73eaa00700 1 -- 172.21.15.12:0/4215976648 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0017070 con 0x7f73f4103870 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.623+0000 7f73eaa00700 1 -- 172.21.15.12:0/4215976648 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73e0009730 con 0x7f73f4103870 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.623+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f41a1ef0 con 0x7f73f4103870 2024-05-27T07:20:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.623+0000 7f73eaa00700 1 -- 172.21.15.12:0/4215976648 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0003a00 con 0x7f73f4103870 2024-05-27T07:20:18.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.623+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f41a9810 con 0x7f73f4103870 2024-05-27T07:20:18.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.624+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f40512b0 con 0x7f73f4103870 2024-05-27T07:20:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.629+0000 7f73eaa00700 1 -- 172.21.15.12:0/4215976648 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f73e0003ba0 con 0x7f73f4103870 2024-05-27T07:20:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.632+0000 7f73eaa00700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f73d80b2880 0x7f73d80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.632+0000 7f73eaa00700 1 -- 172.21.15.12:0/4215976648 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f73e00e20e0 con 0x7f73f4103870 2024-05-27T07:20:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.632+0000 7f73f1e00700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f73d80b2880 0x7f73d80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.633+0000 7f73f1e00700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f73d80b2880 0x7f73d80b4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f73e4005950 tx=0x7f73e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:18.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.636+0000 7f73eaa00700 1 -- 172.21.15.12:0/4215976648 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73e0020030 con 0x7f73f4103870 2024-05-27T07:20:18.870 INFO:teuthology.orchestra.run.smithi012.stdout:77309411453 2024-05-27T07:20:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.878+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f73f402e5e0 con 0x7f73f4103870 2024-05-27T07:20:18.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.879+0000 7f73eaa00700 1 -- 172.21.15.12:0/4215976648 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f73e00a99d0 con 0x7f73f4103870 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f73d80b2880 msgr2=0x7f73d80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f73d80b2880 0x7f73d80b4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f73e4005950 tx=0x7f73e400b410 comp rx=0 tx=0).stop 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4103870 msgr2=0x7f73f419d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4103870 0x7f73f419d2d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f73e000d950 tx=0x7f73e000dc60 comp rx=0 tx=0).stop 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 shutdown_connections 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f73d80b2880 0x7f73d80b4d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73f4110ad0 0x7f73f41a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4102f20 0x7f73f419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 --2- 172.21.15.12:0/4215976648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f73f4103870 0x7f73f419d2d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:18.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 >> 172.21.15.12:0/4215976648 conn(0x7f73f4078ab0 msgr2=0x7f73f4111e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:18.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 shutdown_connections 2024-05-27T07:20:18.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:18.884+0000 7f73f3200700 1 -- 172.21.15.12:0/4215976648 wait complete. 2024-05-27T07:20:19.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:19 smithi012 ceph-mon[31473]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:20:19.172 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:19 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/4215976648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:20:19.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574974 got 55834574979 for osd.1 2024-05-27T07:20:19.233 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:19.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:19 smithi162 ceph-mon[35676]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:20:19.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:19 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/4215976648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:20:19.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:19 smithi062 ceph-mon[35304]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:20:19.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:19 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/4215976648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:20:20.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 -- 172.21.15.12:0/855016944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd184075b50 msgr2=0x7fd184075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 --2- 172.21.15.12:0/855016944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd184075b50 0x7fd184075f60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd174009990 tx=0x7fd174009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 -- 172.21.15.12:0/855016944 shutdown_connections 2024-05-27T07:20:20.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 --2- 172.21.15.12:0/855016944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd184115360 0x7fd184117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 --2- 172.21.15.12:0/855016944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd184075b50 0x7fd184075f60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 --2- 172.21.15.12:0/855016944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1840764a0 0x7fd184114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 -- 172.21.15.12:0/855016944 >> 172.21.15.12:0/855016944 conn(0x7fd184100030 msgr2=0x7fd184102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:20.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 -- 172.21.15.12:0/855016944 shutdown_connections 2024-05-27T07:20:20.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.141+0000 7fd18b600700 1 -- 172.21.15.12:0/855016944 wait complete. 2024-05-27T07:20:20.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.142+0000 7fd18b600700 1 Processor -- start 2024-05-27T07:20:20.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd18b600700 1 -- start start 2024-05-27T07:20:20.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd18b600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd184075b50 0x7fd1841a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd18b600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1840764a0 0x7fd1841a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd18b600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd184115360 0x7fd1841a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd18b600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd18411a960 con 0x7fd1840764a0 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd18b600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd18411a7e0 con 0x7fd184075b50 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd18b600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd18411aae0 con 0x7fd184115360 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd17be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1840764a0 0x7fd1841a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd17be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1840764a0 0x7fd1841a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48774/0 (socket says 172.21.15.12:48774) 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.143+0000 7fd17be00700 1 -- 172.21.15.12:0/1514180165 learned_addr learned my addr 172.21.15.12:0/1514180165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.144+0000 7fd188c00700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd184075b50 0x7fd1841a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.144+0000 7fd189600700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd184115360 0x7fd1841a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.144+0000 7fd17be00700 1 -- 172.21.15.12:0/1514180165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd184115360 msgr2=0x7fd1841a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.144+0000 7fd17be00700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd184115360 0x7fd1841a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.144+0000 7fd17be00700 1 -- 172.21.15.12:0/1514180165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd184075b50 msgr2=0x7fd1841a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.144+0000 7fd17be00700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd184075b50 0x7fd1841a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.144+0000 7fd17be00700 1 -- 172.21.15.12:0/1514180165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1841a62c0 con 0x7fd1840764a0 2024-05-27T07:20:20.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.145+0000 7fd17be00700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1840764a0 0x7fd1841a1990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00d980 tx=0x7fd16c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:20.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.145+0000 7fd182000700 1 -- 172.21.15.12:0/1514180165 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c016070 con 0x7fd1840764a0 2024-05-27T07:20:20.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.145+0000 7fd182000700 1 -- 172.21.15.12:0/1514180165 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd16c004830 con 0x7fd1840764a0 2024-05-27T07:20:20.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.145+0000 7fd182000700 1 -- 172.21.15.12:0/1514180165 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c005440 con 0x7fd1840764a0 2024-05-27T07:20:20.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.145+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1841a65b0 con 0x7fd1840764a0 2024-05-27T07:20:20.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.146+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1841a6a30 con 0x7fd1840764a0 2024-05-27T07:20:20.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.148+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1840512b0 con 0x7fd1840764a0 2024-05-27T07:20:20.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.150+0000 7fd182000700 1 -- 172.21.15.12:0/1514180165 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd16c0055e0 con 0x7fd1840764a0 2024-05-27T07:20:20.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.153+0000 7fd182000700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd1680b2900 0x7fd1680b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.153+0000 7fd182000700 1 -- 172.21.15.12:0/1514180165 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fd16c0e53b0 con 0x7fd1840764a0 2024-05-27T07:20:20.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.153+0000 7fd188c00700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd1680b2900 0x7fd1680b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.154+0000 7fd188c00700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd1680b2900 0x7fd1680b4e10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd17400b5c0 tx=0x7fd174005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:20.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.157+0000 7fd182000700 1 -- 172.21.15.12:0/1514180165 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd16c04ddf0 con 0x7fd1840764a0 2024-05-27T07:20:20.362 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:20 smithi012 ceph-mon[31473]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:20.362 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:20 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/252510355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:20:20.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.419+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd18402e5e0 con 0x7fd1840764a0 2024-05-27T07:20:20.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.420+0000 7fd182000700 1 -- 172.21.15.12:0/1514180165 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd16c0a8f60 con 0x7fd1840764a0 2024-05-27T07:20:20.410 INFO:teuthology.orchestra.run.smithi012.stdout:107374182518 2024-05-27T07:20:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd1680b2900 msgr2=0x7fd1680b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd1680b2900 0x7fd1680b4e10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd17400b5c0 tx=0x7fd174005fd0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1840764a0 msgr2=0x7fd1841a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1840764a0 0x7fd1841a1990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00d980 tx=0x7fd16c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 shutdown_connections 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd1680b2900 0x7fd1680b4e10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd184075b50 0x7fd1841a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd184115360 0x7fd1841a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 --2- 172.21.15.12:0/1514180165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd1840764a0 0x7fd1841a1990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.422+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 >> 172.21.15.12:0/1514180165 conn(0x7fd184100030 msgr2=0x7fd184100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.423+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 shutdown_connections 2024-05-27T07:20:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:20.423+0000 7fd18b600700 1 -- 172.21.15.12:0/1514180165 wait complete. 2024-05-27T07:20:20.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:20 smithi162 ceph-mon[35676]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:20.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:20 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/252510355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:20:20.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:20 smithi062 ceph-mon[35304]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:20.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:20 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/252510355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:20:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.566+0000 7fee82400700 1 -- 172.21.15.12:0/2778773264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c108f10 msgr2=0x7fee7c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.566+0000 7fee82400700 1 --2- 172.21.15.12:0/2778773264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c108f10 0x7fee7c109320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fee680099c0 tx=0x7fee68009cd0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.566+0000 7fee82400700 1 -- 172.21.15.12:0/2778773264 shutdown_connections 2024-05-27T07:20:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.566+0000 7fee82400700 1 --2- 172.21.15.12:0/2778773264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee7c1045e0 0x7fee7c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.566+0000 7fee82400700 1 --2- 172.21.15.12:0/2778773264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee7c109860 0x7fee7c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.566+0000 7fee82400700 1 --2- 172.21.15.12:0/2778773264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c108f10 0x7fee7c109320 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.566+0000 7fee82400700 1 -- 172.21.15.12:0/2778773264 >> 172.21.15.12:0/2778773264 conn(0x7fee7c100030 msgr2=0x7fee7c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.567+0000 7fee82400700 1 -- 172.21.15.12:0/2778773264 shutdown_connections 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.567+0000 7fee82400700 1 -- 172.21.15.12:0/2778773264 wait complete. 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 Processor -- start 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 -- start start 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee7c1045e0 0x7fee7c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee7c108f10 0x7fee7c19d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c109860 0x7fee7c1a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c114370 con 0x7fee7c109860 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c1141f0 con 0x7fee7c1045e0 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.568+0000 7fee82400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c1144f0 con 0x7fee7c108f10 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c109860 0x7fee7c1a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c109860 0x7fee7c1a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48760/0 (socket says 172.21.15.12:48760) 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 -- 172.21.15.12:0/252510355 learned_addr learned my addr 172.21.15.12:0/252510355 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7b400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee7c1045e0 0x7fee7c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7aa00700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee7c108f10 0x7fee7c19d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 -- 172.21.15.12:0/252510355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee7c108f10 msgr2=0x7fee7c19d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee7c108f10 0x7fee7c19d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 -- 172.21.15.12:0/252510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee7c1045e0 msgr2=0x7fee7c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee7c1045e0 0x7fee7c19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.569+0000 7fee7be00700 1 -- 172.21.15.12:0/252510355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c1a1e30 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.570+0000 7fee7be00700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c109860 0x7fee7c1a1790 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fee74007ee0 tx=0x7fee7400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.570+0000 7fee73e00700 1 -- 172.21.15.12:0/252510355 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7400dbd0 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.570+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7c1a2120 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.570+0000 7fee73e00700 1 -- 172.21.15.12:0/252510355 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee7400dd70 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.570+0000 7fee73e00700 1 -- 172.21.15.12:0/252510355 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee740058d0 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.570+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7c1a9a40 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.572+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee7c0512b0 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.574+0000 7fee73e00700 1 -- 172.21.15.12:0/252510355 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fee7401a020 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.575+0000 7fee73e00700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee600b2900 0x7fee600b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.576+0000 7fee73e00700 1 -- 172.21.15.12:0/252510355 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fee740e1dd0 con 0x7fee7c109860 2024-05-27T07:20:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.576+0000 7fee7b400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee600b2900 0x7fee600b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:20.853 INFO:teuthology.orchestra.run.smithi012.stdout:253403070537 2024-05-27T07:20:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.577+0000 7fee7b400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee600b2900 0x7fee600b4e10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee6c005950 tx=0x7fee6c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.581+0000 7fee73e00700 1 -- 172.21.15.12:0/252510355 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fee74010070 con 0x7fee7c109860 2024-05-27T07:20:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.821+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fee7c1a9c10 con 0x7fee7c109860 2024-05-27T07:20:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.822+0000 7fee73e00700 1 -- 172.21.15.12:0/252510355 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fee740a96c0 con 0x7fee7c109860 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee600b2900 msgr2=0x7fee600b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee600b2900 0x7fee600b4e10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee6c005950 tx=0x7fee6c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c109860 msgr2=0x7fee7c1a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c109860 0x7fee7c1a1790 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fee74007ee0 tx=0x7fee7400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 shutdown_connections 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee600b2900 0x7fee600b4e10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee7c1045e0 0x7fee7c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee7c109860 0x7fee7c1a1790 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 --2- 172.21.15.12:0/252510355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee7c108f10 0x7fee7c19d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 >> 172.21.15.12:0/252510355 conn(0x7fee7c100030 msgr2=0x7fee7c10a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 shutdown_connections 2024-05-27T07:20:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:19.826+0000 7fee82400700 1 -- 172.21.15.12:0/252510355 wait complete. 2024-05-27T07:20:21.371 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:21 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1514180165' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:20:21.371 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:21.371 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:21.371 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:20:21.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:21 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1514180165' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:20:21.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:21.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:21.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:20:21.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:21 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1514180165' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:20:21.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:21.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:21.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:20:22.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:22 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1013267363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:20:22.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:22 smithi012 ceph-mon[31473]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-27T07:20:22.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:22 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1013267363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:20:22.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:22 smithi162 ceph-mon[35676]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-27T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:22 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1013267363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:20:22.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:22 smithi062 ceph-mon[35304]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-27T07:20:22.675 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738500 got 34359738506 for osd.0 2024-05-27T07:20:22.675 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 -- 172.21.15.12:0/638893966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103240 msgr2=0x7fee20103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 --2- 172.21.15.12:0/638893966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103240 0x7fee20103650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fee10009990 tx=0x7fee10009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 -- 172.21.15.12:0/638893966 shutdown_connections 2024-05-27T07:20:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 --2- 172.21.15.12:0/638893966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee20108690 0x7fee20108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 --2- 172.21.15.12:0/638893966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103240 0x7fee20103650 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 --2- 172.21.15.12:0/638893966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20103b90 0x7fee201080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 -- 172.21.15.12:0/638893966 >> 172.21.15.12:0/638893966 conn(0x7fee200fedb0 msgr2=0x7fee201011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 -- 172.21.15.12:0/638893966 shutdown_connections 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.441+0000 7fee25200700 1 -- 172.21.15.12:0/638893966 wait complete. 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.442+0000 7fee25200700 1 Processor -- start 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.443+0000 7fee25200700 1 -- start start 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.443+0000 7fee25200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee20103240 0x7fee201a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.443+0000 7fee25200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103b90 0x7fee201a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.443+0000 7fee25200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20108690 0x7fee201a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.443+0000 7fee25200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fee2011a910 con 0x7fee20103b90 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.443+0000 7fee25200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fee2011a790 con 0x7fee20103240 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.443+0000 7fee25200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee2011aa90 con 0x7fee20108690 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103b90 0x7fee201a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103b90 0x7fee201a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48804/0 (socket says 172.21.15.12:48804) 2024-05-27T07:20:22.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 -- 172.21.15.12:0/1013267363 learned_addr learned my addr 172.21.15.12:0/1013267363 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee1de00700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee20103240 0x7fee201a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee1e800700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20108690 0x7fee201a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 -- 172.21.15.12:0/1013267363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20108690 msgr2=0x7fee201a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20108690 0x7fee201a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 -- 172.21.15.12:0/1013267363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee20103240 msgr2=0x7fee201a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee20103240 0x7fee201a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.444+0000 7fee17e00700 1 -- 172.21.15.12:0/1013267363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee201a61f0 con 0x7fee20103b90 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.445+0000 7fee17e00700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103b90 0x7fee201a18c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fee0800d980 tx=0x7fee0800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.445+0000 7fee16a00700 1 -- 172.21.15.12:0/1013267363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08016070 con 0x7fee20103b90 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.445+0000 7fee16a00700 1 -- 172.21.15.12:0/1013267363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee08004830 con 0x7fee20103b90 2024-05-27T07:20:22.852 INFO:teuthology.orchestra.run.smithi012.stdout:300647710778 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.445+0000 7fee16a00700 1 -- 172.21.15.12:0/1013267363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08005440 con 0x7fee20103b90 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.445+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee201a64e0 con 0x7fee20103b90 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.445+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee201ade00 con 0x7fee20103b90 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.446+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee200512b0 con 0x7fee20103b90 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.450+0000 7fee16a00700 1 -- 172.21.15.12:0/1013267363 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fee080055e0 con 0x7fee20103b90 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.451+0000 7fee16a00700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee040b2900 0x7fee040b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.451+0000 7fee16a00700 1 -- 172.21.15.12:0/1013267363 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fee080e28b0 con 0x7fee20103b90 2024-05-27T07:20:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.452+0000 7fee1de00700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee040b2900 0x7fee040b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.453+0000 7fee1de00700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee040b2900 0x7fee040b4e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fee1000b2e0 tx=0x7fee10005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.453+0000 7fee16a00700 1 -- 172.21.15.12:0/1013267363 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fee0801f020 con 0x7fee20103b90 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.700+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fee201adfd0 con 0x7fee20103b90 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.702+0000 7fee16a00700 1 -- 172.21.15.12:0/1013267363 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fee080aa1a0 con 0x7fee20103b90 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee040b2900 msgr2=0x7fee040b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee040b2900 0x7fee040b4e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fee1000b2e0 tx=0x7fee10005fb0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103b90 msgr2=0x7fee201a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103b90 0x7fee201a18c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fee0800d980 tx=0x7fee0800dc90 comp rx=0 tx=0).stop 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 shutdown_connections 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fee040b2900 0x7fee040b4e10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee20103240 0x7fee201a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee20108690 0x7fee201a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 --2- 172.21.15.12:0/1013267363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fee20103b90 0x7fee201a18c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 >> 172.21.15.12:0/1013267363 conn(0x7fee200fedb0 msgr2=0x7fee20106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 shutdown_connections 2024-05-27T07:20:22.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:21.705+0000 7fee25200700 1 -- 172.21.15.12:0/1013267363 wait complete. 2024-05-27T07:20:23.621 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070531 got 253403070537 for osd.9 2024-05-27T07:20:23.621 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:25.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:24 smithi012 ceph-mon[31473]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:25.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:24 smithi162 ceph-mon[35676]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:25.235 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411449 got 77309411453 for osd.2 2024-05-27T07:20:25.236 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:25.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:24 smithi062 ceph-mon[35304]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.577+0000 7fef5fc00700 1 -- 172.21.15.12:0/1538908959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 msgr2=0x7fef581111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.577+0000 7fef5fc00700 1 --2- 172.21.15.12:0/1538908959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef581111e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fef5000b0d0 tx=0x7fef5000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.578+0000 7fef5fc00700 1 -- 172.21.15.12:0/1538908959 shutdown_connections 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.578+0000 7fef5fc00700 1 --2- 172.21.15.12:0/1538908959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef58104600 0x7fef58104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.578+0000 7fef5fc00700 1 --2- 172.21.15.12:0/1538908959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef581111e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.578+0000 7fef5fc00700 1 --2- 172.21.15.12:0/1538908959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef58108f30 0x7fef58109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.578+0000 7fef5fc00700 1 -- 172.21.15.12:0/1538908959 >> 172.21.15.12:0/1538908959 conn(0x7fef5807af60 msgr2=0x7fef5807b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.578+0000 7fef5fc00700 1 -- 172.21.15.12:0/1538908959 shutdown_connections 2024-05-27T07:20:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.578+0000 7fef5fc00700 1 -- 172.21.15.12:0/1538908959 wait complete. 2024-05-27T07:20:26.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.579+0000 7fef5fc00700 1 Processor -- start 2024-05-27T07:20:26.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.579+0000 7fef5fc00700 1 -- start start 2024-05-27T07:20:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5fc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef58104600 0x7fef5819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5fc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef58108f30 0x7fef5819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5fc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef5819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5fc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef581143b0 con 0x7fef58109880 2024-05-27T07:20:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5fc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef58114230 con 0x7fef58108f30 2024-05-27T07:20:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5fc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef58114530 con 0x7fef58104600 2024-05-27T07:20:26.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef5819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef5819f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57154/0 (socket says 172.21.15.12:57154) 2024-05-27T07:20:26.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5dc00700 1 -- 172.21.15.12:0/2754808218 learned_addr learned my addr 172.21.15.12:0/2754808218 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.580+0000 7fef5d200700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef58104600 0x7fef5819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.581+0000 7fef5dc00700 1 -- 172.21.15.12:0/2754808218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef58104600 msgr2=0x7fef5819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.581+0000 7fef5dc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef58104600 0x7fef5819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.581+0000 7fef5dc00700 1 -- 172.21.15.12:0/2754808218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef58108f30 msgr2=0x7fef5819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.581+0000 7fef5dc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef58108f30 0x7fef5819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.581+0000 7fef5dc00700 1 -- 172.21.15.12:0/2754808218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5819fc20 con 0x7fef58109880 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.581+0000 7fef5dc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef5819f580 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fef50007d10 tx=0x7fef50007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.581+0000 7fef56000700 1 -- 172.21.15.12:0/2754808218 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5001f070 con 0x7fef58109880 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.582+0000 7fef56000700 1 -- 172.21.15.12:0/2754808218 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef50003e10 con 0x7fef58109880 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.582+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5819feb0 con 0x7fef58109880 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.582+0000 7fef56000700 1 -- 172.21.15.12:0/2754808218 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5001b740 con 0x7fef58109880 2024-05-27T07:20:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.582+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef581a7830 con 0x7fef58109880 2024-05-27T07:20:26.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.583+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef580512b0 con 0x7fef58109880 2024-05-27T07:20:26.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.586+0000 7fef56000700 1 -- 172.21.15.12:0/2754808218 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fef50022070 con 0x7fef58109880 2024-05-27T07:20:26.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.587+0000 7fef56000700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fef3c0b2900 0x7fef3c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.588+0000 7fef56000700 1 -- 172.21.15.12:0/2754808218 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fef5000f020 con 0x7fef58109880 2024-05-27T07:20:26.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.588+0000 7fef5d200700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fef3c0b2900 0x7fef3c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.589+0000 7fef5d200700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fef3c0b2900 0x7fef3c0b4e10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef48005950 tx=0x7fef4800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.591+0000 7fef56000700 1 -- 172.21.15.12:0/2754808218 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fef50029020 con 0x7fef58109880 2024-05-27T07:20:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.844+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fef5802e5e0 con 0x7fef58109880 2024-05-27T07:20:26.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.845+0000 7fef56000700 1 -- 172.21.15.12:0/2754808218 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fef500ab8a0 con 0x7fef58109880 2024-05-27T07:20:26.835 INFO:teuthology.orchestra.run.smithi012.stdout:180388626527 2024-05-27T07:20:26.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fef3c0b2900 msgr2=0x7fef3c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fef3c0b2900 0x7fef3c0b4e10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef48005950 tx=0x7fef4800b410 comp rx=0 tx=0).stop 2024-05-27T07:20:26.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 msgr2=0x7fef5819f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef5819f580 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fef50007d10 tx=0x7fef50007d40 comp rx=0 tx=0).stop 2024-05-27T07:20:26.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 shutdown_connections 2024-05-27T07:20:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fef3c0b2900 0x7fef3c0b4e10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef58109880 0x7fef5819f580 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef58104600 0x7fef5819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 --2- 172.21.15.12:0/2754808218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef58108f30 0x7fef5819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 >> 172.21.15.12:0/2754808218 conn(0x7fef5807af60 msgr2=0x7fef581019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.848+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 shutdown_connections 2024-05-27T07:20:26.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:26.849+0000 7fef5fc00700 1 -- 172.21.15.12:0/2754808218 wait complete. 2024-05-27T07:20:26.933 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182514 got 107374182518 for osd.3 2024-05-27T07:20:26.933 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:27.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:26 smithi012 ceph-mon[31473]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:27.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:26 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2754808218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:20:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:26 smithi162 ceph-mon[35676]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:27.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:26 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2754808218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:20:27.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:26 smithi062 ceph-mon[35304]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-27T07:20:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:26 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2754808218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 -- 172.21.15.12:0/3026929181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64081029b0 msgr2=0x7f6408102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 --2- 172.21.15.12:0/3026929181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64081029b0 0x7f6408102dc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009990 tx=0x7f63f4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 -- 172.21.15.12:0/3026929181 shutdown_connections 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 --2- 172.21.15.12:0/3026929181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6408103300 0x7f640810f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 --2- 172.21.15.12:0/3026929181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64081089e0 0x7f6408108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 --2- 172.21.15.12:0/3026929181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64081029b0 0x7f6408102dc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 -- 172.21.15.12:0/3026929181 >> 172.21.15.12:0/3026929181 conn(0x7f64080fe4f0 msgr2=0x7f6408100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 -- 172.21.15.12:0/3026929181 shutdown_connections 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.717+0000 7f640fe00700 1 -- 172.21.15.12:0/3026929181 wait complete. 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.718+0000 7f640fe00700 1 Processor -- start 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.718+0000 7f640fe00700 1 -- start start 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64081029b0 0x7f64080fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6408103300 0x7f64080fef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64081089e0 0x7f64080ff450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f64081121e0 con 0x7f6408103300 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6408112060 con 0x7f64081089e0 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640fe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6408112360 con 0x7f64081029b0 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6408103300 0x7f64080fef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6408103300 0x7f64080fef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57180/0 (socket says 172.21.15.12:57180) 2024-05-27T07:20:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640ca00700 1 -- 172.21.15.12:0/1154103060 learned_addr learned my addr 172.21.15.12:0/1154103060 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640d400700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64081029b0 0x7f64080fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.719+0000 7f640de00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64081089e0 0x7f64080ff450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.720+0000 7f640ca00700 1 -- 172.21.15.12:0/1154103060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64081029b0 msgr2=0x7f64080fe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.720+0000 7f640ca00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64081029b0 0x7f64080fe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.720+0000 7f640ca00700 1 -- 172.21.15.12:0/1154103060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64081089e0 msgr2=0x7f64080ff450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.720+0000 7f640ca00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64081089e0 0x7f64080ff450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.720+0000 7f640ca00700 1 -- 172.21.15.12:0/1154103060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64081a91f0 con 0x7f6408103300 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.720+0000 7f640ca00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6408103300 0x7f64080fef10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f63f4000c00 tx=0x7f63f40057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.721+0000 7f6406000700 1 -- 172.21.15.12:0/1154103060 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4027070 con 0x7f6408103300 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.721+0000 7f6406000700 1 -- 172.21.15.12:0/1154103060 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63f400fb00 con 0x7f6408103300 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.721+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64081a9480 con 0x7f6408103300 2024-05-27T07:20:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.721+0000 7f6406000700 1 -- 172.21.15.12:0/1154103060 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022a00 con 0x7f6408103300 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.721+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64081a98d0 con 0x7f6408103300 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.722+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64080512b0 con 0x7f6408103300 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.724+0000 7f6406000700 1 -- 172.21.15.12:0/1154103060 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f63f402a070 con 0x7f6408103300 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.726+0000 7f6406000700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f63f00b2880 0x7f63f00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.726+0000 7f6406000700 1 -- 172.21.15.12:0/1154103060 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f63f40e3470 con 0x7f6408103300 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.726+0000 7f640d400700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f63f00b2880 0x7f63f00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.727+0000 7f640d400700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f63f00b2880 0x7f63f00b4d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63fc005950 tx=0x7f63fc00b510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.729+0000 7f6406000700 1 -- 172.21.15.12:0/1154103060 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f63f4013070 con 0x7f6408103300 2024-05-27T07:20:28.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.816+0000 7f0662400700 1 -- 172.21.15.12:0/3413704772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 msgr2=0x7f065c110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.816+0000 7f0662400700 1 --2- 172.21.15.12:0/3413704772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c110fd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f065400b0d0 tx=0x7f065400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.817+0000 7f0662400700 1 -- 172.21.15.12:0/3413704772 shutdown_connections 2024-05-27T07:20:28.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.817+0000 7f0662400700 1 --2- 172.21.15.12:0/3413704772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c110fd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.817+0000 7f0662400700 1 --2- 172.21.15.12:0/3413704772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f065c1043f0 0x7f065c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.817+0000 7f0662400700 1 --2- 172.21.15.12:0/3413704772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065c108d20 0x7f065c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.817+0000 7f0662400700 1 -- 172.21.15.12:0/3413704772 >> 172.21.15.12:0/3413704772 conn(0x7f065c079d00 msgr2=0x7f065c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.817+0000 7f0662400700 1 -- 172.21.15.12:0/3413704772 shutdown_connections 2024-05-27T07:20:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.817+0000 7f0662400700 1 -- 172.21.15.12:0/3413704772 wait complete. 2024-05-27T07:20:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.818+0000 7f0662400700 1 Processor -- start 2024-05-27T07:20:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.818+0000 7f0662400700 1 -- start start 2024-05-27T07:20:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.818+0000 7f0662400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f065c1043f0 0x7f065c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.818+0000 7f0662400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065c108d20 0x7f065c19d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.818+0000 7f0662400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c1a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0662400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f065c1141d0 con 0x7f065c109670 2024-05-27T07:20:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0662400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f065c114050 con 0x7f065c108d20 2024-05-27T07:20:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0662400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f065c114350 con 0x7f065c1043f0 2024-05-27T07:20:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c1a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c1a15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57204/0 (socket says 172.21.15.12:57204) 2024-05-27T07:20:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 -- 172.21.15.12:0/3895801409 learned_addr learned my addr 172.21.15.12:0/3895801409 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:28.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661000700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f065c1043f0 0x7f065c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 -- 172.21.15.12:0/3895801409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f065c1043f0 msgr2=0x7f065c19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f065c1043f0 0x7f065c19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 -- 172.21.15.12:0/3895801409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065c108d20 msgr2=0x7f065c19d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065c108d20 0x7f065c19d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.819+0000 7f0661a00700 1 -- 172.21.15.12:0/3895801409 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065c1a1c50 con 0x7f065c109670 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.820+0000 7f0661a00700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c1a15b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0654009850 tx=0x7f0654007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.820+0000 7f0659600700 1 -- 172.21.15.12:0/3895801409 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065401f030 con 0x7f065c109670 2024-05-27T07:20:28.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.820+0000 7f0659600700 1 -- 172.21.15.12:0/3895801409 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f065401bad0 con 0x7f065c109670 2024-05-27T07:20:28.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.820+0000 7f0659600700 1 -- 172.21.15.12:0/3895801409 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654012bb0 con 0x7f065c109670 2024-05-27T07:20:28.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.820+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065c1a1ee0 con 0x7f065c109670 2024-05-27T07:20:28.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.820+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f065c1a9860 con 0x7f065c109670 2024-05-27T07:20:28.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.821+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f065c1058a0 con 0x7f065c109670 2024-05-27T07:20:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.825+0000 7f0659600700 1 -- 172.21.15.12:0/3895801409 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0654022030 con 0x7f065c109670 2024-05-27T07:20:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.826+0000 7f0659600700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f06440b2900 0x7f06440b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.826+0000 7f0659600700 1 -- 172.21.15.12:0/3895801409 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f065400f020 con 0x7f065c109670 2024-05-27T07:20:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.827+0000 7f0661000700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f06440b2900 0x7f06440b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:28.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.827+0000 7f0661000700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f06440b2900 0x7f06440b4e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0650006fd0 tx=0x7f0650008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:28.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.829+0000 7f0659600700 1 -- 172.21.15.12:0/3895801409 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0654029020 con 0x7f065c109670 2024-05-27T07:20:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.978+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f640802e5e0 con 0x7f6408103300 2024-05-27T07:20:28.969 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:28 smithi012 ceph-mon[31473]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T07:20:28.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.979+0000 7f6406000700 1 -- 172.21.15.12:0/1154103060 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f63f40aae50 con 0x7f6408103300 2024-05-27T07:20:28.970 INFO:teuthology.orchestra.run.smithi012.stdout:154618822760 2024-05-27T07:20:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f63f00b2880 msgr2=0x7f63f00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f63f00b2880 0x7f63f00b4d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63fc005950 tx=0x7f63fc00b510 comp rx=0 tx=0).stop 2024-05-27T07:20:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6408103300 msgr2=0x7f64080fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6408103300 0x7f64080fef10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f63f4000c00 tx=0x7f63f40057a0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 shutdown_connections 2024-05-27T07:20:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f63f00b2880 0x7f63f00b4d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64081089e0 0x7f64080ff450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64081029b0 0x7f64080fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 --2- 172.21.15.12:0/1154103060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6408103300 0x7f64080fef10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.982+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 >> 172.21.15.12:0/1154103060 conn(0x7f64080fe4f0 msgr2=0x7f6408105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.983+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 shutdown_connections 2024-05-27T07:20:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:28.983+0000 7f640fe00700 1 -- 172.21.15.12:0/1154103060 wait complete. 2024-05-27T07:20:29.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.077+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f065c1a9a30 con 0x7f065c109670 2024-05-27T07:20:29.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.078+0000 7f0659600700 1 -- 172.21.15.12:0/3895801409 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f06540ab8a0 con 0x7f065c109670 2024-05-27T07:20:29.069 INFO:teuthology.orchestra.run.smithi012.stdout:227633266770 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f06440b2900 msgr2=0x7f06440b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f06440b2900 0x7f06440b4e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0650006fd0 tx=0x7f0650008040 comp rx=0 tx=0).stop 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 msgr2=0x7f065c1a15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c1a15b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0654009850 tx=0x7f0654007e40 comp rx=0 tx=0).stop 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 shutdown_connections 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f06440b2900 0x7f06440b4e10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f065c109670 0x7f065c1a15b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f065c1043f0 0x7f065c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 --2- 172.21.15.12:0/3895801409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065c108d20 0x7f065c19d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 >> 172.21.15.12:0/3895801409 conn(0x7f065c079d00 msgr2=0x7f065c10f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:29.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 shutdown_connections 2024-05-27T07:20:29.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:29.081+0000 7f0662400700 1 -- 172.21.15.12:0/3895801409 wait complete. 2024-05-27T07:20:29.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:28 smithi162 ceph-mon[35676]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T07:20:29.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:28 smithi062 ceph-mon[35304]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-27T07:20:29.654 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710773 got 300647710778 for osd.11 2024-05-27T07:20:29.654 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:30.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:29 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1154103060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:20:30.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:29 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3895801409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:20:30.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:29 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1154103060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:20:30.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:29 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3895801409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:20:30.276 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:29 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1154103060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:20:30.277 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:29 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3895801409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:20:31.010 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822754 got 154618822760 for osd.5 2024-05-27T07:20:31.011 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:30 smithi162 ceph-mon[35676]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-27T07:20:31.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:30 smithi062 ceph-mon[35304]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-27T07:20:31.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:30 smithi012 ceph-mon[31473]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-27T07:20:31.899 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626521 got 180388626527 for osd.6 2024-05-27T07:20:31.899 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:32.509 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266766 got 227633266770 for osd.8 2024-05-27T07:20:32.509 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:33.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:32 smithi012 ceph-mon[31473]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-27T07:20:33.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 -- 172.21.15.12:0/2000119184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8384105dc0 msgr2=0x7f838410e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 --2- 172.21.15.12:0/2000119184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8384105dc0 0x7f838410e310 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8370009990 tx=0x7f8370009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 -- 172.21.15.12:0/2000119184 shutdown_connections 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 --2- 172.21.15.12:0/2000119184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8384105470 0x7f8384105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 --2- 172.21.15.12:0/2000119184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838410e8e0 0x7f8384110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 --2- 172.21.15.12:0/2000119184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8384105dc0 0x7f838410e310 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 -- 172.21.15.12:0/2000119184 >> 172.21.15.12:0/2000119184 conn(0x7f83840fef20 msgr2=0x7f8384101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.169+0000 7f838ba00700 1 -- 172.21.15.12:0/2000119184 shutdown_connections 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.170+0000 7f838ba00700 1 -- 172.21.15.12:0/2000119184 wait complete. 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.171+0000 7f838ba00700 1 Processor -- start 2024-05-27T07:20:33.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.171+0000 7f838ba00700 1 -- start start 2024-05-27T07:20:33.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.171+0000 7f838ba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8384105470 0x7f83841a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.171+0000 7f838ba00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384105dc0 0x7f83841a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f838ba00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f838410e8e0 0x7f838419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f838ba00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8384114340 con 0x7f838410e8e0 2024-05-27T07:20:33.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f838ba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f83841141c0 con 0x7f8384105470 2024-05-27T07:20:33.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f838ba00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f83841144c0 con 0x7f8384105dc0 2024-05-27T07:20:33.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f8389a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f838410e8e0 0x7f838419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:33.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f8389a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f838410e8e0 0x7f838419ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40276/0 (socket says 172.21.15.12:40276) 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f8389a00700 1 -- 172.21.15.12:0/2502998906 learned_addr learned my addr 172.21.15.12:0/2502998906 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f8383e00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384105dc0 0x7f83841a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.172+0000 7f8389000700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8384105470 0x7f83841a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8389a00700 1 -- 172.21.15.12:0/2502998906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384105dc0 msgr2=0x7f83841a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8389a00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384105dc0 0x7f83841a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8389a00700 1 -- 172.21.15.12:0/2502998906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8384105470 msgr2=0x7f83841a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8389a00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8384105470 0x7f83841a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8389a00700 1 -- 172.21.15.12:0/2502998906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838419b270 con 0x7f838410e8e0 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8389000700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8384105470 0x7f83841a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:33.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8389a00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f838410e8e0 0x7f838419ab40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f837c00ef30 tx=0x7f837c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:33.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8381600700 1 -- 172.21.15.12:0/2502998906 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c0092e0 con 0x7f838410e8e0 2024-05-27T07:20:33.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8381600700 1 -- 172.21.15.12:0/2502998906 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f837c004bb0 con 0x7f838410e8e0 2024-05-27T07:20:33.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f8381600700 1 -- 172.21.15.12:0/2502998906 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c004020 con 0x7f838410e8e0 2024-05-27T07:20:33.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.173+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f838419b500 con 0x7f838410e8e0 2024-05-27T07:20:33.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.174+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83841a7760 con 0x7f838410e8e0 2024-05-27T07:20:33.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.175+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f838406c910 con 0x7f838410e8e0 2024-05-27T07:20:33.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.180+0000 7f8381600700 1 -- 172.21.15.12:0/2502998906 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f837c01b030 con 0x7f838410e8e0 2024-05-27T07:20:33.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.181+0000 7f8381600700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f836c0b2900 0x7f836c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.182+0000 7f8381600700 1 -- 172.21.15.12:0/2502998906 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f837c0e2ac0 con 0x7f838410e8e0 2024-05-27T07:20:33.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.182+0000 7f8389000700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f836c0b2900 0x7f836c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:33.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.183+0000 7f8389000700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f836c0b2900 0x7f836c0b4e10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8378009cb0 tx=0x7f8378009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:33.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.184+0000 7f8381600700 1 -- 172.21.15.12:0/2502998906 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f837c022020 con 0x7f838410e8e0 2024-05-27T07:20:33.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:32 smithi162 ceph-mon[35676]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-27T07:20:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:32 smithi062 ceph-mon[35304]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-27T07:20:33.275 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 -- 172.21.15.12:0/494067396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd900109860 msgr2=0x7fd9001111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 --2- 172.21.15.12:0/494067396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd900109860 0x7fd9001111c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800b0d0 tx=0x7fd8f800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 -- 172.21.15.12:0/494067396 shutdown_connections 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 --2- 172.21.15.12:0/494067396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9001045e0 0x7fd9001049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 --2- 172.21.15.12:0/494067396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd900109860 0x7fd9001111c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 --2- 172.21.15.12:0/494067396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd900108f10 0x7fd900109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 -- 172.21.15.12:0/494067396 >> 172.21.15.12:0/494067396 conn(0x7fd900100030 msgr2=0x7fd900102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.299+0000 7fd904a00700 1 -- 172.21.15.12:0/494067396 shutdown_connections 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.300+0000 7fd904a00700 1 -- 172.21.15.12:0/494067396 wait complete. 2024-05-27T07:20:33.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.301+0000 7fd904a00700 1 Processor -- start 2024-05-27T07:20:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.301+0000 7fd904a00700 1 -- start start 2024-05-27T07:20:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd904a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9001045e0 0x7fd900075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd904a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd900108f10 0x7fd900075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd904a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd900109860 0x7fd900076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd904a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd900114390 con 0x7fd9001045e0 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd904a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd900114210 con 0x7fd900109860 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd904a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd900114510 con 0x7fd900108f10 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd8fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9001045e0 0x7fd900075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd8fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9001045e0 0x7fd900075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40298/0 (socket says 172.21.15.12:40298) 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd8fde00700 1 -- 172.21.15.12:0/349211904 learned_addr learned my addr 172.21.15.12:0/349211904 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.302+0000 7fd8fe800700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd900109860 0x7fd900076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.303+0000 7fd8fde00700 1 -- 172.21.15.12:0/349211904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd900108f10 msgr2=0x7fd900075ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.303+0000 7fd8fde00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd900108f10 0x7fd900075ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.303+0000 7fd8fde00700 1 -- 172.21.15.12:0/349211904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd900109860 msgr2=0x7fd900076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.303+0000 7fd8fde00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd900109860 0x7fd900076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.303+0000 7fd8fde00700 1 -- 172.21.15.12:0/349211904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9001a9100 con 0x7fd9001045e0 2024-05-27T07:20:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.303+0000 7fd8fe800700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd900109860 0x7fd900076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.303+0000 7fd8fde00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9001045e0 0x7fd900075ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000e9a0 tx=0x7fd8f000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.304+0000 7fd8f6a00700 1 -- 172.21.15.12:0/349211904 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f000cbd0 con 0x7fd9001045e0 2024-05-27T07:20:33.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.304+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9001a93f0 con 0x7fd9001045e0 2024-05-27T07:20:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.304+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9001a9980 con 0x7fd9001045e0 2024-05-27T07:20:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.304+0000 7fd8f6a00700 1 -- 172.21.15.12:0/349211904 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8f000cd70 con 0x7fd9001045e0 2024-05-27T07:20:33.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.304+0000 7fd8f6a00700 1 -- 172.21.15.12:0/349211904 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0018680 con 0x7fd9001045e0 2024-05-27T07:20:33.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.306+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e0005330 con 0x7fd9001045e0 2024-05-27T07:20:33.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.310+0000 7fd8f6a00700 1 -- 172.21.15.12:0/349211904 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd8f0018820 con 0x7fd9001045e0 2024-05-27T07:20:33.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.311+0000 7fd8f6a00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8e40b2900 0x7fd8e40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:33.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.312+0000 7fd8f6a00700 1 -- 172.21.15.12:0/349211904 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fd8f00e2a20 con 0x7fd9001045e0 2024-05-27T07:20:33.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.312+0000 7fd8fd400700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8e40b2900 0x7fd8e40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:33.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.313+0000 7fd8fd400700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8e40b2900 0x7fd8e40b4e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd900107950 tx=0x7fd8ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:33.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.318+0000 7fd8f6a00700 1 -- 172.21.15.12:0/349211904 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd8f00aa390 con 0x7fd9001045e0 2024-05-27T07:20:33.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.450+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f838419bfe0 con 0x7f838410e8e0 2024-05-27T07:20:33.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.451+0000 7f8381600700 1 -- 172.21.15.12:0/2502998906 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f837c010040 con 0x7f838410e8e0 2024-05-27T07:20:33.443 INFO:teuthology.orchestra.run.smithi012.stdout:274877907010 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f836c0b2900 msgr2=0x7f836c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f836c0b2900 0x7f836c0b4e10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8378009cb0 tx=0x7f8378009260 comp rx=0 tx=0).stop 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f838410e8e0 msgr2=0x7f838419ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f838410e8e0 0x7f838419ab40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f837c00ef30 tx=0x7f837c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 shutdown_connections 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f836c0b2900 0x7f836c0b4e10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8384105470 0x7f83841a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f838410e8e0 0x7f838419ab40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 --2- 172.21.15.12:0/2502998906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384105dc0 0x7f83841a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 >> 172.21.15.12:0/2502998906 conn(0x7f83840fef20 msgr2=0x7f83840ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 shutdown_connections 2024-05-27T07:20:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.454+0000 7f838ba00700 1 -- 172.21.15.12:0/2502998906 wait complete. 2024-05-27T07:20:33.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.567+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd8e00058f0 con 0x7fd9001045e0 2024-05-27T07:20:33.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.568+0000 7fd8f6a00700 1 -- 172.21.15.12:0/349211904 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd8f00b2410 con 0x7fd9001045e0 2024-05-27T07:20:33.559 INFO:teuthology.orchestra.run.smithi012.stdout:133143986288 2024-05-27T07:20:33.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.572+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8e40b2900 msgr2=0x7fd8e40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.572+0000 7fd904a00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8e40b2900 0x7fd8e40b4e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd900107950 tx=0x7fd8ec009260 comp rx=0 tx=0).stop 2024-05-27T07:20:33.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.572+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9001045e0 msgr2=0x7fd900075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.572+0000 7fd904a00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9001045e0 0x7fd900075ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000e9a0 tx=0x7fd8f000ed60 comp rx=0 tx=0).stop 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 shutdown_connections 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fd8e40b2900 0x7fd8e40b4e10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd900109860 0x7fd900076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9001045e0 0x7fd900075ab0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 --2- 172.21.15.12:0/349211904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd900108f10 0x7fd900075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 >> 172.21.15.12:0/349211904 conn(0x7fd900100030 msgr2=0x7fd90010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 shutdown_connections 2024-05-27T07:20:33.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:33.573+0000 7fd904a00700 1 -- 172.21.15.12:0/349211904 wait complete. 2024-05-27T07:20:34.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:33 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2502998906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:20:34.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:33 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/349211904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:20:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:33 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2502998906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:20:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:33 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/349211904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:20:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:33 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2502998906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:20:34.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:33 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/349211904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:20:34.783 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877907004 got 274877907010 for osd.10 2024-05-27T07:20:34.783 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:35.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:34 smithi012 ceph-mon[31473]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:35.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:34 smithi162 ceph-mon[35676]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:35.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:34 smithi062 ceph-mon[35304]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:35.605 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986283 got 133143986288 for osd.4 2024-05-27T07:20:35.605 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.801+0000 7fdd9e800700 1 -- 172.21.15.12:0/3368367303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98108ef0 msgr2=0x7fdd98109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.801+0000 7fdd9e800700 1 --2- 172.21.15.12:0/3368367303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98108ef0 0x7fdd98109300 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000b0d0 tx=0x7fdd8000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.801+0000 7fdd9e800700 1 -- 172.21.15.12:0/3368367303 shutdown_connections 2024-05-27T07:20:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.801+0000 7fdd9e800700 1 --2- 172.21.15.12:0/3368367303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd981045c0 0x7fdd981049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:36.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.801+0000 7fdd9e800700 1 --2- 172.21.15.12:0/3368367303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd98109840 0x7fdd981111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.801+0000 7fdd9e800700 1 --2- 172.21.15.12:0/3368367303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98108ef0 0x7fdd98109300 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.801+0000 7fdd9e800700 1 -- 172.21.15.12:0/3368367303 >> 172.21.15.12:0/3368367303 conn(0x7fdd980ffe90 msgr2=0x7fdd981022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.802+0000 7fdd9e800700 1 -- 172.21.15.12:0/3368367303 shutdown_connections 2024-05-27T07:20:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.802+0000 7fdd9e800700 1 -- 172.21.15.12:0/3368367303 wait complete. 2024-05-27T07:20:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.804+0000 7fdd9e800700 1 Processor -- start 2024-05-27T07:20:36.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.804+0000 7fdd9e800700 1 -- start start 2024-05-27T07:20:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd9e800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd981045c0 0x7fdd9819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd9e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd98108ef0 0x7fdd9819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:36.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd9e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98109840 0x7fdd9819f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd9e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98114300 con 0x7fdd98109840 2024-05-27T07:20:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd9e800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98114180 con 0x7fdd98108ef0 2024-05-27T07:20:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd9e800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98114480 con 0x7fdd981045c0 2024-05-27T07:20:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98109840 0x7fdd9819f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:36.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd97e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98109840 0x7fdd9819f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40312/0 (socket says 172.21.15.12:40312) 2024-05-27T07:20:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd97e00700 1 -- 172.21.15.12:0/2501485501 learned_addr learned my addr 172.21.15.12:0/2501485501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd8ea00700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd98108ef0 0x7fdd9819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.805+0000 7fdd97400700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd981045c0 0x7fdd9819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd97e00700 1 -- 172.21.15.12:0/2501485501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd981045c0 msgr2=0x7fdd9819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd97e00700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd981045c0 0x7fdd9819ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:36.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd97e00700 1 -- 172.21.15.12:0/2501485501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd98108ef0 msgr2=0x7fdd9819b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd97e00700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd98108ef0 0x7fdd9819b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd97e00700 1 -- 172.21.15.12:0/2501485501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9819fc00 con 0x7fdd98109840 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd97e00700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98109840 0x7fdd9819f560 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000d980 tx=0x7fdd9000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd94c00700 1 -- 172.21.15.12:0/2501485501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90017070 con 0x7fdd98109840 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd94c00700 1 -- 172.21.15.12:0/2501485501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd90005020 con 0x7fdd98109840 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.806+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9819fef0 con 0x7fdd98109840 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.807+0000 7fdd94c00700 1 -- 172.21.15.12:0/2501485501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90009990 con 0x7fdd98109840 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.807+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd981a7810 con 0x7fdd98109840 2024-05-27T07:20:36.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.808+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd980512b0 con 0x7fdd98109840 2024-05-27T07:20:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.812+0000 7fdd94c00700 1 -- 172.21.15.12:0/2501485501 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fdd90004a30 con 0x7fdd98109840 2024-05-27T07:20:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.814+0000 7fdd94c00700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd7c0b2900 0x7fdd7c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.814+0000 7fdd94c00700 1 -- 172.21.15.12:0/2501485501 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdd900e26e0 con 0x7fdd98109840 2024-05-27T07:20:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.814+0000 7fdd97400700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd7c0b2900 0x7fdd7c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.815+0000 7fdd97400700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd7c0b2900 0x7fdd7c0b4e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009bf0 tx=0x7fdd88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:36.816+0000 7fdd94c00700 1 -- 172.21.15.12:0/2501485501 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdd90020020 con 0x7fdd98109840 2024-05-27T07:20:37.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.062+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fdd9802e5e0 con 0x7fdd98109840 2024-05-27T07:20:37.052 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:36 smithi012 ceph-mon[31473]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:37.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.063+0000 7fdd94c00700 1 -- 172.21.15.12:0/2501485501 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdd900aa050 con 0x7fdd98109840 2024-05-27T07:20:37.053 INFO:teuthology.orchestra.run.smithi012.stdout:201863463002 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd7c0b2900 msgr2=0x7fdd7c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd7c0b2900 0x7fdd7c0b4e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009bf0 tx=0x7fdd88009260 comp rx=0 tx=0).stop 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98109840 msgr2=0x7fdd9819f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98109840 0x7fdd9819f560 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000d980 tx=0x7fdd9000dc90 comp rx=0 tx=0).stop 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 shutdown_connections 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdd7c0b2900 0x7fdd7c0b4e10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd98109840 0x7fdd9819f560 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd981045c0 0x7fdd9819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:37.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 --2- 172.21.15.12:0/2501485501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd98108ef0 0x7fdd9819b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:37.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.066+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 >> 172.21.15.12:0/2501485501 conn(0x7fdd980ffe90 msgr2=0x7fdd981022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:37.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.067+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 shutdown_connections 2024-05-27T07:20:37.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:37.067+0000 7fdd9e800700 1 -- 172.21.15.12:0/2501485501 wait complete. 2024-05-27T07:20:37.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:36 smithi162 ceph-mon[35676]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:37.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:36 smithi062 ceph-mon[35304]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:37.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462997 got 201863463002 for osd.7 2024-05-27T07:20:37.830 DEBUG:teuthology.parallel:result is None 2024-05-27T07:20:37.830 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:20:37.830 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:20:38.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:37 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2501485501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:20:38.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:37 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2501485501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:20:38.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:37 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2501485501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:20:38.301 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:39.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:38 smithi012 ceph-mon[31473]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:39.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:38 smithi162 ceph-mon[35676]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:39.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:38 smithi062 ceph-mon[35304]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:40.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 -- 172.21.15.12:0/2568420427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d6810c590 msgr2=0x7f2d6810c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:40.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 --2- 172.21.15.12:0/2568420427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d6810c590 0x7f2d6810c9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009990 tx=0x7f2d58009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:40.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 -- 172.21.15.12:0/2568420427 shutdown_connections 2024-05-27T07:20:40.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 --2- 172.21.15.12:0/2568420427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d68106ea0 0x7f2d68113330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:40.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 --2- 172.21.15.12:0/2568420427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d6810c590 0x7f2d6810c9a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:40.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 --2- 172.21.15.12:0/2568420427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68106550 0x7f2d68106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:40.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 -- 172.21.15.12:0/2568420427 >> 172.21.15.12:0/2568420427 conn(0x7f2d68102050 msgr2=0x7f2d681044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:40.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 -- 172.21.15.12:0/2568420427 shutdown_connections 2024-05-27T07:20:40.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.730+0000 7f2d6e600700 1 -- 172.21.15.12:0/2568420427 wait complete. 2024-05-27T07:20:40.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.731+0000 7f2d6e600700 1 Processor -- start 2024-05-27T07:20:40.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.732+0000 7f2d6e600700 1 -- start start 2024-05-27T07:20:40.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.732+0000 7f2d6e600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d68106550 0x7f2d6819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.732+0000 7f2d6e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d68106ea0 0x7f2d6819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.732+0000 7f2d6e600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d6810c590 0x7f2d681a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.732+0000 7f2d6e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681165a0 con 0x7f2d68106ea0 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.732+0000 7f2d6e600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68116420 con 0x7f2d68106550 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.732+0000 7f2d6e600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68116720 con 0x7f2d6810c590 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d68106ea0 0x7f2d6819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d68106ea0 0x7f2d6819d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40332/0 (socket says 172.21.15.12:40332) 2024-05-27T07:20:40.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 -- 172.21.15.12:0/1641599165 learned_addr learned my addr 172.21.15.12:0/1641599165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d67e00700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d6810c590 0x7f2d681a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d67400700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d68106550 0x7f2d6819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 -- 172.21.15.12:0/1641599165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d6810c590 msgr2=0x7f2d681a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d6810c590 0x7f2d681a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 -- 172.21.15.12:0/1641599165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d68106550 msgr2=0x7f2d6819cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d68106550 0x7f2d6819cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.733+0000 7f2d66a00700 1 -- 172.21.15.12:0/1641599165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d681a1e40 con 0x7f2d68106ea0 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.734+0000 7f2d66a00700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d68106ea0 0x7f2d6819d510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400d950 tx=0x7f2d5400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:40.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.734+0000 7f2d5fe00700 1 -- 172.21.15.12:0/1641599165 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54016070 con 0x7f2d68106ea0 2024-05-27T07:20:40.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.734+0000 7f2d5fe00700 1 -- 172.21.15.12:0/1641599165 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d54004830 con 0x7f2d68106ea0 2024-05-27T07:20:40.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.734+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d681a2130 con 0x7f2d68106ea0 2024-05-27T07:20:40.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.734+0000 7f2d5fe00700 1 -- 172.21.15.12:0/1641599165 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54005450 con 0x7f2d68106ea0 2024-05-27T07:20:40.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.734+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d681a9a50 con 0x7f2d68106ea0 2024-05-27T07:20:40.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.736+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6810e950 con 0x7f2d68106ea0 2024-05-27T07:20:40.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.739+0000 7f2d5fe00700 1 -- 172.21.15.12:0/1641599165 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2d540055f0 con 0x7f2d68106ea0 2024-05-27T07:20:40.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.740+0000 7f2d5fe00700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2d4c0b2900 0x7f2d4c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:40.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.740+0000 7f2d5fe00700 1 -- 172.21.15.12:0/1641599165 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f2d540e22f0 con 0x7f2d68106ea0 2024-05-27T07:20:40.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.741+0000 7f2d67400700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2d4c0b2900 0x7f2d4c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:40.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.742+0000 7f2d67400700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2d4c0b2900 0x7f2d4c0b4e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2d58006010 tx=0x7f2d5800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.746+0000 7f2d5fe00700 1 -- 172.21.15.12:0/1641599165 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2d5401f020 con 0x7f2d68106ea0 2024-05-27T07:20:40.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:40.979+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2d68063bf0 con 0x7f2d4c0b2900 2024-05-27T07:20:41.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.011+0000 7f2d5fe00700 1 -- 172.21.15.12:0/1641599165 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340480 (secure 0 0 0) 0x7f2d5807b210 con 0x7f2d4c0b2900 2024-05-27T07:20:41.001 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:40 smithi012 ceph-mon[31473]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:41.010 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:20:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2d4c0b2900 msgr2=0x7f2d4c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2d4c0b2900 0x7f2d4c0b4e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2d58006010 tx=0x7f2d5800b540 comp rx=0 tx=0).stop 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d68106ea0 msgr2=0x7f2d6819d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d68106ea0 0x7f2d6819d510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400d950 tx=0x7f2d5400dc60 comp rx=0 tx=0).stop 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 shutdown_connections 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2d4c0b2900 0x7f2d4c0b4e10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d68106550 0x7f2d6819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d6810c590 0x7f2d681a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 --2- 172.21.15.12:0/1641599165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d68106ea0 0x7f2d6819d510 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:41.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.018+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 >> 172.21.15.12:0/1641599165 conn(0x7f2d68102050 msgr2=0x7f2d681048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:41.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.019+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 shutdown_connections 2024-05-27T07:20:41.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:41.019+0000 7f2d6e600700 1 -- 172.21.15.12:0/1641599165 wait complete. 2024-05-27T07:20:41.016 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-05-27T07:20:41.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:40 smithi162 ceph-mon[35676]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:41.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:40 smithi062 ceph-mon[35304]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:41.823 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-05-27T07:20:39.827668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657597,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":831,"num_read_kb":516,"num_write":304,"num_write_kb":2277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375728,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566096,"statfs":{"total":1151940427776,"available":1151555682304,"internally_reserved":0,"allocated":14434304,"data_stored":8131073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007721"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.726652+0000","last_change":"2024-05-27T07:17:06.763882+0000","last_active":"2024-05-27T07:20:15.726652+0000","last_peered":"2024-05-27T07:20:15.726652+0000","last_clean":"2024-05-27T07:20:15.726652+0000","last_became_active":"2024-05-27T07:17:06.759625+0000","last_became_peered":"2024-05-27T07:17:06.759625+0000","last_unstale":"2024-05-27T07:20:15.726652+0000","last_undegraded":"2024-05-27T07:20:15.726652+0000","last_fullsized":"2024-05-27T07:20:15.726652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:24.105867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.699117+0000","last_change":"2024-05-27T07:17:06.755737+0000","last_active":"2024-05-27T07:20:15.699117+0000","last_peered":"2024-05-27T07:20:15.699117+0000","last_clean":"2024-05-27T07:20:15.699117+0000","last_became_active":"2024-05-27T07:17:06.754588+0000","last_became_peered":"2024-05-27T07:17:06.754588+0000","last_unstale":"2024-05-27T07:20:15.699117+0000","last_undegraded":"2024-05-27T07:20:15.699117+0000","last_fullsized":"2024-05-27T07:20:15.699117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:02.417140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.730543+0000","last_change":"2024-05-27T07:17:06.772057+0000","last_active":"2024-05-27T07:20:15.730543+0000","last_peered":"2024-05-27T07:20:15.730543+0000","last_clean":"2024-05-27T07:20:15.730543+0000","last_became_active":"2024-05-27T07:17:06.771636+0000","last_became_peered":"2024-05-27T07:17:06.771636+0000","last_unstale":"2024-05-27T07:20:15.730543+0000","last_undegraded":"2024-05-27T07:20:15.730543+0000","last_fullsized":"2024-05-27T07:20:15.730543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:45.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714732+0000","last_change":"2024-05-27T07:17:06.763599+0000","last_active":"2024-05-27T07:20:15.714732+0000","last_peered":"2024-05-27T07:20:15.714732+0000","last_clean":"2024-05-27T07:20:15.714732+0000","last_became_active":"2024-05-27T07:17:06.763125+0000","last_became_peered":"2024-05-27T07:17:06.763125+0000","last_unstale":"2024-05-27T07:20:15.714732+0000","last_undegraded":"2024-05-27T07:20:15.714732+0000","last_fullsized":"2024-05-27T07:20:15.714732+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:43.359673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717624+0000","last_change":"2024-05-27T07:17:06.755896+0000","last_active":"2024-05-27T07:20:15.717624+0000","last_peered":"2024-05-27T07:20:15.717624+0000","last_clean":"2024-05-27T07:20:15.717624+0000","last_became_active":"2024-05-27T07:17:06.755495+0000","last_became_peered":"2024-05-27T07:17:06.755495+0000","last_unstale":"2024-05-27T07:20:15.717624+0000","last_undegraded":"2024-05-27T07:20:15.717624+0000","last_fullsized":"2024-05-27T07:20:15.717624+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:41:37.632012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706194+0000","last_change":"2024-05-27T07:17:46.699881+0000","last_active":"2024-05-27T07:20:15.706194+0000","last_peered":"2024-05-27T07:20:15.706194+0000","last_clean":"2024-05-27T07:20:15.706194+0000","last_became_active":"2024-05-27T07:17:46.699080+0000","last_became_peered":"2024-05-27T07:17:46.699080+0000","last_unstale":"2024-05-27T07:20:15.706194+0000","last_undegraded":"2024-05-27T07:20:15.706194+0000","last_fullsized":"2024-05-27T07:20:15.706194+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:43.580762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714004+0000","last_change":"2024-05-27T07:17:06.770468+0000","last_active":"2024-05-27T07:20:15.714004+0000","last_peered":"2024-05-27T07:20:15.714004+0000","last_clean":"2024-05-27T07:20:15.714004+0000","last_became_active":"2024-05-27T07:17:06.768345+0000","last_became_peered":"2024-05-27T07:17:06.768345+0000","last_unstale":"2024-05-27T07:20:15.714004+0000","last_undegraded":"2024-05-27T07:20:15.714004+0000","last_fullsized":"2024-05-27T07:20:15.714004+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:12:27.646654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717873+0000","last_change":"2024-05-27T07:17:06.758769+0000","last_active":"2024-05-27T07:20:15.717873+0000","last_peered":"2024-05-27T07:20:15.717873+0000","last_clean":"2024-05-27T07:20:15.717873+0000","last_became_active":"2024-05-27T07:17:06.758393+0000","last_became_peered":"2024-05-27T07:17:06.758393+0000","last_unstale":"2024-05-27T07:20:15.717873+0000","last_undegraded":"2024-05-27T07:20:15.717873+0000","last_fullsized":"2024-05-27T07:20:15.717873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:56.450569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.725941+0000","last_change":"2024-05-27T07:17:06.770681+0000","last_active":"2024-05-27T07:20:15.725941+0000","last_peered":"2024-05-27T07:20:15.725941+0000","last_clean":"2024-05-27T07:20:15.725941+0000","last_became_active":"2024-05-27T07:17:06.766520+0000","last_became_peered":"2024-05-27T07:17:06.766520+0000","last_unstale":"2024-05-27T07:20:15.725941+0000","last_undegraded":"2024-05-27T07:20:15.725941+0000","last_fullsized":"2024-05-27T07:20:15.725941+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:56.931743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.713503+0000","last_change":"2024-05-27T07:17:06.761462+0000","last_active":"2024-05-27T07:20:15.713503+0000","last_peered":"2024-05-27T07:20:15.713503+0000","last_clean":"2024-05-27T07:20:15.713503+0000","last_became_active":"2024-05-27T07:17:06.759344+0000","last_became_peered":"2024-05-27T07:17:06.759344+0000","last_unstale":"2024-05-27T07:20:15.713503+0000","last_undegraded":"2024-05-27T07:20:15.713503+0000","last_fullsized":"2024-05-27T07:20:15.713503+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:12:22.688477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.731716+0000","last_change":"2024-05-27T07:17:06.761947+0000","last_active":"2024-05-27T07:20:15.731716+0000","last_peered":"2024-05-27T07:20:15.731716+0000","last_clean":"2024-05-27T07:20:15.731716+0000","last_became_active":"2024-05-27T07:17:06.761464+0000","last_became_peered":"2024-05-27T07:17:06.761464+0000","last_unstale":"2024-05-27T07:20:15.731716+0000","last_undegraded":"2024-05-27T07:20:15.731716+0000","last_fullsized":"2024-05-27T07:20:15.731716+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:08:19.939311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715044+0000","last_change":"2024-05-27T07:17:06.772039+0000","last_active":"2024-05-27T07:20:15.715044+0000","last_peered":"2024-05-27T07:20:15.715044+0000","last_clean":"2024-05-27T07:20:15.715044+0000","last_became_active":"2024-05-27T07:17:06.771303+0000","last_became_peered":"2024-05-27T07:17:06.771303+0000","last_unstale":"2024-05-27T07:20:15.715044+0000","last_undegraded":"2024-05-27T07:20:15.715044+0000","last_fullsized":"2024-05-27T07:20:15.715044+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:30.254848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.719705+0000","last_change":"2024-05-27T07:17:06.774228+0000","last_active":"2024-05-27T07:20:15.719705+0000","last_peered":"2024-05-27T07:20:15.719705+0000","last_clean":"2024-05-27T07:20:15.719705+0000","last_became_active":"2024-05-27T07:17:06.771436+0000","last_became_peered":"2024-05-27T07:17:06.771436+0000","last_unstale":"2024-05-27T07:20:15.719705+0000","last_undegraded":"2024-05-27T07:20:15.719705+0000","last_fullsized":"2024-05-27T07:20:15.719705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:25.749143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706852+0000","last_change":"2024-05-27T07:17:06.755558+0000","last_active":"2024-05-27T07:20:15.706852+0000","last_peered":"2024-05-27T07:20:15.706852+0000","last_clean":"2024-05-27T07:20:15.706852+0000","last_became_active":"2024-05-27T07:17:06.754858+0000","last_became_peered":"2024-05-27T07:17:06.754858+0000","last_unstale":"2024-05-27T07:20:15.706852+0000","last_undegraded":"2024-05-27T07:20:15.706852+0000","last_fullsized":"2024-05-27T07:20:15.706852+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:09.853982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.709868+0000","last_change":"2024-05-27T07:17:06.770731+0000","last_active":"2024-05-27T07:20:15.709868+0000","last_peered":"2024-05-27T07:20:15.709868+0000","last_clean":"2024-05-27T07:20:15.709868+0000","last_became_active":"2024-05-27T07:17:06.766923+0000","last_became_peered":"2024-05-27T07:17:06.766923+0000","last_unstale":"2024-05-27T07:20:15.709868+0000","last_undegraded":"2024-05-27T07:20:15.709868+0000","last_fullsized":"2024-05-27T07:20:15.709868+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.110121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.731693+0000","last_change":"2024-05-27T07:17:06.758928+0000","last_active":"2024-05-27T07:20:15.731693+0000","last_peered":"2024-05-27T07:20:15.731693+0000","last_clean":"2024-05-27T07:20:15.731693+0000","last_became_active":"2024-05-27T07:17:06.758328+0000","last_became_peered":"2024-05-27T07:17:06.758328+0000","last_unstale":"2024-05-27T07:20:15.731693+0000","last_undegraded":"2024-05-27T07:20:15.731693+0000","last_fullsized":"2024-05-27T07:20:15.731693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:36.284395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.726854+0000","last_change":"2024-05-27T07:17:06.760006+0000","last_active":"2024-05-27T07:20:15.726854+0000","last_peered":"2024-05-27T07:20:15.726854+0000","last_clean":"2024-05-27T07:20:15.726854+0000","last_became_active":"2024-05-27T07:17:06.758177+0000","last_became_peered":"2024-05-27T07:17:06.758177+0000","last_unstale":"2024-05-27T07:20:15.726854+0000","last_undegraded":"2024-05-27T07:20:15.726854+0000","last_fullsized":"2024-05-27T07:20:15.726854+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.516289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.729927+0000","last_change":"2024-05-27T07:17:06.768484+0000","last_active":"2024-05-27T07:20:15.729927+0000","last_peered":"2024-05-27T07:20:15.729927+0000","last_clean":"2024-05-27T07:20:15.729927+0000","last_became_active":"2024-05-27T07:17:06.768105+0000","last_became_peered":"2024-05-27T07:17:06.768105+0000","last_unstale":"2024-05-27T07:20:15.729927+0000","last_undegraded":"2024-05-27T07:20:15.729927+0000","last_fullsized":"2024-05-27T07:20:15.729927+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:54:48.751678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717458+0000","last_change":"2024-05-27T07:17:06.762782+0000","last_active":"2024-05-27T07:20:15.717458+0000","last_peered":"2024-05-27T07:20:15.717458+0000","last_clean":"2024-05-27T07:20:15.717458+0000","last_became_active":"2024-05-27T07:17:06.760129+0000","last_became_peered":"2024-05-27T07:17:06.760129+0000","last_unstale":"2024-05-27T07:20:15.717458+0000","last_undegraded":"2024-05-27T07:20:15.717458+0000","last_fullsized":"2024-05-27T07:20:15.717458+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:47:19.360662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063670+0000","last_change":"2024-05-27T07:17:21.899772+0000","last_active":"2024-05-27T07:18:47.063670+0000","last_peered":"2024-05-27T07:18:47.063670+0000","last_clean":"2024-05-27T07:18:47.063670+0000","last_became_active":"2024-05-27T07:17:21.899038+0000","last_became_peered":"2024-05-27T07:17:21.899038+0000","last_unstale":"2024-05-27T07:18:47.063670+0000","last_undegraded":"2024-05-27T07:18:47.063670+0000","last_fullsized":"2024-05-27T07:18:47.063670+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:38.784433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059878+0000","last_change":"2024-05-27T07:17:46.708798+0000","last_active":"2024-05-27T07:18:47.059878+0000","last_peered":"2024-05-27T07:18:47.059878+0000","last_clean":"2024-05-27T07:18:47.059878+0000","last_became_active":"2024-05-27T07:17:46.696433+0000","last_became_peered":"2024-05-27T07:17:46.696433+0000","last_unstale":"2024-05-27T07:18:47.059878+0000","last_undegraded":"2024-05-27T07:18:47.059878+0000","last_fullsized":"2024-05-27T07:18:47.059878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:40.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717457+0000","last_change":"2024-05-27T07:17:06.760865+0000","last_active":"2024-05-27T07:20:15.717457+0000","last_peered":"2024-05-27T07:20:15.717457+0000","last_clean":"2024-05-27T07:20:15.717457+0000","last_became_active":"2024-05-27T07:17:06.759491+0000","last_became_peered":"2024-05-27T07:17:06.759491+0000","last_unstale":"2024-05-27T07:20:15.717457+0000","last_undegraded":"2024-05-27T07:20:15.717457+0000","last_fullsized":"2024-05-27T07:20:15.717457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:31.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082530+0000","last_change":"2024-05-27T07:17:21.898583+0000","last_active":"2024-05-27T07:18:47.082530+0000","last_peered":"2024-05-27T07:18:47.082530+0000","last_clean":"2024-05-27T07:18:47.082530+0000","last_became_active":"2024-05-27T07:17:21.897937+0000","last_became_peered":"2024-05-27T07:17:21.897937+0000","last_unstale":"2024-05-27T07:18:47.082530+0000","last_undegraded":"2024-05-27T07:18:47.082530+0000","last_fullsized":"2024-05-27T07:18:47.082530+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:45:00.412914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126886+0000","last_change":"2024-05-27T07:17:02.432465+0000","last_active":"2024-05-27T07:18:47.126886+0000","last_peered":"2024-05-27T07:18:47.126886+0000","last_clean":"2024-05-27T07:18:47.126886+0000","last_became_active":"2024-05-27T07:17:02.431638+0000","last_became_peered":"2024-05-27T07:17:02.431638+0000","last_unstale":"2024-05-27T07:18:47.126886+0000","last_undegraded":"2024-05-27T07:18:47.126886+0000","last_fullsized":"2024-05-27T07:18:47.126886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:03:13.167952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.730513+0000","last_change":"2024-05-27T07:17:06.763133+0000","last_active":"2024-05-27T07:20:15.730513+0000","last_peered":"2024-05-27T07:20:15.730513+0000","last_clean":"2024-05-27T07:20:15.730513+0000","last_became_active":"2024-05-27T07:17:06.762770+0000","last_became_peered":"2024-05-27T07:17:06.762770+0000","last_unstale":"2024-05-27T07:20:15.730513+0000","last_undegraded":"2024-05-27T07:20:15.730513+0000","last_fullsized":"2024-05-27T07:20:15.730513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:33.223722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083574+0000","last_change":"2024-05-27T07:17:21.900369+0000","last_active":"2024-05-27T07:18:47.083574+0000","last_peered":"2024-05-27T07:18:47.083574+0000","last_clean":"2024-05-27T07:18:47.083574+0000","last_became_active":"2024-05-27T07:17:21.899758+0000","last_became_peered":"2024-05-27T07:17:21.899758+0000","last_unstale":"2024-05-27T07:18:47.083574+0000","last_undegraded":"2024-05-27T07:18:47.083574+0000","last_fullsized":"2024-05-27T07:18:47.083574+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:42.116302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128098+0000","last_change":"2024-05-27T07:17:02.347047+0000","last_active":"2024-05-27T07:18:47.128098+0000","last_peered":"2024-05-27T07:18:47.128098+0000","last_clean":"2024-05-27T07:18:47.128098+0000","last_became_active":"2024-05-27T07:17:02.346562+0000","last_became_peered":"2024-05-27T07:17:02.346562+0000","last_unstale":"2024-05-27T07:18:47.128098+0000","last_undegraded":"2024-05-27T07:18:47.128098+0000","last_fullsized":"2024-05-27T07:18:47.128098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:25:50.133089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732803+0000","last_change":"2024-05-27T07:17:06.767930+0000","last_active":"2024-05-27T07:20:15.732803+0000","last_peered":"2024-05-27T07:20:15.732803+0000","last_clean":"2024-05-27T07:20:15.732803+0000","last_became_active":"2024-05-27T07:17:06.767439+0000","last_became_peered":"2024-05-27T07:17:06.767439+0000","last_unstale":"2024-05-27T07:20:15.732803+0000","last_undegraded":"2024-05-27T07:20:15.732803+0000","last_fullsized":"2024-05-27T07:20:15.732803+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:18:47.464602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082887+0000","last_change":"2024-05-27T07:17:21.889199+0000","last_active":"2024-05-27T07:18:47.082887+0000","last_peered":"2024-05-27T07:18:47.082887+0000","last_clean":"2024-05-27T07:18:47.082887+0000","last_became_active":"2024-05-27T07:17:21.888667+0000","last_became_peered":"2024-05-27T07:17:21.888667+0000","last_unstale":"2024-05-27T07:18:47.082887+0000","last_undegraded":"2024-05-27T07:18:47.082887+0000","last_fullsized":"2024-05-27T07:18:47.082887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.029820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127096+0000","last_change":"2024-05-27T07:17:02.339872+0000","last_active":"2024-05-27T07:18:47.127096+0000","last_peered":"2024-05-27T07:18:47.127096+0000","last_clean":"2024-05-27T07:18:47.127096+0000","last_became_active":"2024-05-27T07:17:02.338975+0000","last_became_peered":"2024-05-27T07:17:02.338975+0000","last_unstale":"2024-05-27T07:18:47.127096+0000","last_undegraded":"2024-05-27T07:18:47.127096+0000","last_fullsized":"2024-05-27T07:18:47.127096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:14.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.723806+0000","last_change":"2024-05-27T07:17:06.758234+0000","last_active":"2024-05-27T07:20:15.723806+0000","last_peered":"2024-05-27T07:20:15.723806+0000","last_clean":"2024-05-27T07:20:15.723806+0000","last_became_active":"2024-05-27T07:17:06.757827+0000","last_became_peered":"2024-05-27T07:17:06.757827+0000","last_unstale":"2024-05-27T07:20:15.723806+0000","last_undegraded":"2024-05-27T07:20:15.723806+0000","last_fullsized":"2024-05-27T07:20:15.723806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:21.688435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064804+0000","last_change":"2024-05-27T07:17:21.882540+0000","last_active":"2024-05-27T07:18:47.064804+0000","last_peered":"2024-05-27T07:18:47.064804+0000","last_clean":"2024-05-27T07:18:47.064804+0000","last_became_active":"2024-05-27T07:17:21.882087+0000","last_became_peered":"2024-05-27T07:17:21.882087+0000","last_unstale":"2024-05-27T07:18:47.064804+0000","last_undegraded":"2024-05-27T07:18:47.064804+0000","last_fullsized":"2024-05-27T07:18:47.064804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:53:04.363521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'7","reported_seq":181,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:36.365476+0000","last_change":"2024-05-27T07:17:46.704997+0000","last_active":"2024-05-27T07:20:36.365476+0000","last_peered":"2024-05-27T07:20:36.365476+0000","last_clean":"2024-05-27T07:20:36.365476+0000","last_became_active":"2024-05-27T07:17:46.704355+0000","last_became_peered":"2024-05-27T07:17:46.704355+0000","last_unstale":"2024-05-27T07:20:36.365476+0000","last_undegraded":"2024-05-27T07:20:36.365476+0000","last_fullsized":"2024-05-27T07:20:36.365476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:43:02.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.705254+0000","last_change":"2024-05-27T07:17:06.765809+0000","last_active":"2024-05-27T07:20:15.705254+0000","last_peered":"2024-05-27T07:20:15.705254+0000","last_clean":"2024-05-27T07:20:15.705254+0000","last_became_active":"2024-05-27T07:17:06.763845+0000","last_became_peered":"2024-05-27T07:17:06.763845+0000","last_unstale":"2024-05-27T07:20:15.705254+0000","last_undegraded":"2024-05-27T07:20:15.705254+0000","last_fullsized":"2024-05-27T07:20:15.705254+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:25.399181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064484+0000","last_change":"2024-05-27T07:17:21.886316+0000","last_active":"2024-05-27T07:18:47.064484+0000","last_peered":"2024-05-27T07:18:47.064484+0000","last_clean":"2024-05-27T07:18:47.064484+0000","last_became_active":"2024-05-27T07:17:21.885129+0000","last_became_peered":"2024-05-27T07:17:21.885129+0000","last_unstale":"2024-05-27T07:18:47.064484+0000","last_undegraded":"2024-05-27T07:18:47.064484+0000","last_fullsized":"2024-05-27T07:18:47.064484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:10.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064424+0000","last_change":"2024-05-27T07:17:02.338765+0000","last_active":"2024-05-27T07:18:47.064424+0000","last_peered":"2024-05-27T07:18:47.064424+0000","last_clean":"2024-05-27T07:18:47.064424+0000","last_became_active":"2024-05-27T07:17:02.337103+0000","last_became_peered":"2024-05-27T07:17:02.337103+0000","last_unstale":"2024-05-27T07:18:47.064424+0000","last_undegraded":"2024-05-27T07:18:47.064424+0000","last_fullsized":"2024-05-27T07:18:47.064424+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:15:07.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.713789+0000","last_change":"2024-05-27T07:17:06.766839+0000","last_active":"2024-05-27T07:20:15.713789+0000","last_peered":"2024-05-27T07:20:15.713789+0000","last_clean":"2024-05-27T07:20:15.713789+0000","last_became_active":"2024-05-27T07:17:06.766373+0000","last_became_peered":"2024-05-27T07:17:06.766373+0000","last_unstale":"2024-05-27T07:20:15.713789+0000","last_undegraded":"2024-05-27T07:20:15.713789+0000","last_fullsized":"2024-05-27T07:20:15.713789+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:09:55.064045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.138290+0000","last_change":"2024-05-27T07:18:47.133692+0000","last_active":"2024-05-27T07:18:47.138290+0000","last_peered":"2024-05-27T07:18:47.138290+0000","last_clean":"2024-05-27T07:18:47.138290+0000","last_became_active":"2024-05-27T07:18:47.132122+0000","last_became_peered":"2024-05-27T07:18:47.132122+0000","last_unstale":"2024-05-27T07:18:47.138290+0000","last_undegraded":"2024-05-27T07:18:47.138290+0000","last_fullsized":"2024-05-27T07:18:47.138290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:57.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130263+0000","last_change":"2024-05-27T07:17:02.329809+0000","last_active":"2024-05-27T07:18:47.130263+0000","last_peered":"2024-05-27T07:18:47.130263+0000","last_clean":"2024-05-27T07:18:47.130263+0000","last_became_active":"2024-05-27T07:17:02.328758+0000","last_became_peered":"2024-05-27T07:17:02.328758+0000","last_unstale":"2024-05-27T07:18:47.130263+0000","last_undegraded":"2024-05-27T07:18:47.130263+0000","last_fullsized":"2024-05-27T07:18:47.130263+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:52.479444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716195+0000","last_change":"2024-05-27T07:17:06.749836+0000","last_active":"2024-05-27T07:20:15.716195+0000","last_peered":"2024-05-27T07:20:15.716195+0000","last_clean":"2024-05-27T07:20:15.716195+0000","last_became_active":"2024-05-27T07:17:06.749512+0000","last_became_peered":"2024-05-27T07:17:06.749512+0000","last_unstale":"2024-05-27T07:20:15.716195+0000","last_undegraded":"2024-05-27T07:20:15.716195+0000","last_fullsized":"2024-05-27T07:20:15.716195+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:41:51.000415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127857+0000","last_change":"2024-05-27T07:17:21.888376+0000","last_active":"2024-05-27T07:18:47.127857+0000","last_peered":"2024-05-27T07:18:47.127857+0000","last_clean":"2024-05-27T07:18:47.127857+0000","last_became_active":"2024-05-27T07:17:21.886183+0000","last_became_peered":"2024-05-27T07:17:21.886183+0000","last_unstale":"2024-05-27T07:18:47.127857+0000","last_undegraded":"2024-05-27T07:18:47.127857+0000","last_fullsized":"2024-05-27T07:18:47.127857+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:34.541338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070176+0000","last_change":"2024-05-27T07:17:02.334113+0000","last_active":"2024-05-27T07:18:47.070176+0000","last_peered":"2024-05-27T07:18:47.070176+0000","last_clean":"2024-05-27T07:18:47.070176+0000","last_became_active":"2024-05-27T07:17:02.333201+0000","last_became_peered":"2024-05-27T07:17:02.333201+0000","last_unstale":"2024-05-27T07:18:47.070176+0000","last_undegraded":"2024-05-27T07:18:47.070176+0000","last_fullsized":"2024-05-27T07:18:47.070176+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:13.484654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716170+0000","last_change":"2024-05-27T07:17:06.751676+0000","last_active":"2024-05-27T07:20:15.716170+0000","last_peered":"2024-05-27T07:20:15.716170+0000","last_clean":"2024-05-27T07:20:15.716170+0000","last_became_active":"2024-05-27T07:17:06.751221+0000","last_became_peered":"2024-05-27T07:17:06.751221+0000","last_unstale":"2024-05-27T07:20:15.716170+0000","last_undegraded":"2024-05-27T07:20:15.716170+0000","last_fullsized":"2024-05-27T07:20:15.716170+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:40.012561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125183+0000","last_change":"2024-05-27T07:17:21.897640+0000","last_active":"2024-05-27T07:18:47.125183+0000","last_peered":"2024-05-27T07:18:47.125183+0000","last_clean":"2024-05-27T07:18:47.125183+0000","last_became_active":"2024-05-27T07:17:21.895355+0000","last_became_peered":"2024-05-27T07:17:21.895355+0000","last_unstale":"2024-05-27T07:18:47.125183+0000","last_undegraded":"2024-05-27T07:18:47.125183+0000","last_fullsized":"2024-05-27T07:18:47.125183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:51:12.030069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:37.367376+0000","last_change":"2024-05-27T07:17:02.339464+0000","last_active":"2024-05-27T07:20:37.367376+0000","last_peered":"2024-05-27T07:20:37.367376+0000","last_clean":"2024-05-27T07:20:37.367376+0000","last_became_active":"2024-05-27T07:17:02.338577+0000","last_became_peered":"2024-05-27T07:17:02.338577+0000","last_unstale":"2024-05-27T07:20:37.367376+0000","last_undegraded":"2024-05-27T07:20:37.367376+0000","last_fullsized":"2024-05-27T07:20:37.367376+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:29:59.210157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.729317+0000","last_change":"2024-05-27T07:17:06.766883+0000","last_active":"2024-05-27T07:20:15.729317+0000","last_peered":"2024-05-27T07:20:15.729317+0000","last_clean":"2024-05-27T07:20:15.729317+0000","last_became_active":"2024-05-27T07:17:06.766166+0000","last_became_peered":"2024-05-27T07:17:06.766166+0000","last_unstale":"2024-05-27T07:20:15.729317+0000","last_undegraded":"2024-05-27T07:20:15.729317+0000","last_fullsized":"2024-05-27T07:20:15.729317+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:28.936499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.058915+0000","last_change":"2024-05-27T07:17:21.893930+0000","last_active":"2024-05-27T07:18:47.058915+0000","last_peered":"2024-05-27T07:18:47.058915+0000","last_clean":"2024-05-27T07:18:47.058915+0000","last_became_active":"2024-05-27T07:17:21.893029+0000","last_became_peered":"2024-05-27T07:17:21.893029+0000","last_unstale":"2024-05-27T07:18:47.058915+0000","last_undegraded":"2024-05-27T07:18:47.058915+0000","last_fullsized":"2024-05-27T07:18:47.058915+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:32.453078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068465+0000","last_change":"2024-05-27T07:17:02.346531+0000","last_active":"2024-05-27T07:18:47.068465+0000","last_peered":"2024-05-27T07:18:47.068465+0000","last_clean":"2024-05-27T07:18:47.068465+0000","last_became_active":"2024-05-27T07:17:02.345844+0000","last_became_peered":"2024-05-27T07:17:02.345844+0000","last_unstale":"2024-05-27T07:18:47.068465+0000","last_undegraded":"2024-05-27T07:18:47.068465+0000","last_fullsized":"2024-05-27T07:18:47.068465+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:28.544616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.712857+0000","last_change":"2024-05-27T07:17:06.769845+0000","last_active":"2024-05-27T07:20:15.712857+0000","last_peered":"2024-05-27T07:20:15.712857+0000","last_clean":"2024-05-27T07:20:15.712857+0000","last_became_active":"2024-05-27T07:17:06.768467+0000","last_became_peered":"2024-05-27T07:17:06.768467+0000","last_unstale":"2024-05-27T07:20:15.712857+0000","last_undegraded":"2024-05-27T07:20:15.712857+0000","last_fullsized":"2024-05-27T07:20:15.712857+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:30.311824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084110+0000","last_change":"2024-05-27T07:17:21.899885+0000","last_active":"2024-05-27T07:18:47.084110+0000","last_peered":"2024-05-27T07:18:47.084110+0000","last_clean":"2024-05-27T07:18:47.084110+0000","last_became_active":"2024-05-27T07:17:21.899340+0000","last_became_peered":"2024-05-27T07:17:21.899340+0000","last_unstale":"2024-05-27T07:18:47.084110+0000","last_undegraded":"2024-05-27T07:18:47.084110+0000","last_fullsized":"2024-05-27T07:18:47.084110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:12.665409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070793+0000","last_change":"2024-05-27T07:17:02.435770+0000","last_active":"2024-05-27T07:18:47.070793+0000","last_peered":"2024-05-27T07:18:47.070793+0000","last_clean":"2024-05-27T07:18:47.070793+0000","last_became_active":"2024-05-27T07:17:02.434535+0000","last_became_peered":"2024-05-27T07:17:02.434535+0000","last_unstale":"2024-05-27T07:18:47.070793+0000","last_undegraded":"2024-05-27T07:18:47.070793+0000","last_fullsized":"2024-05-27T07:18:47.070793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:53.745583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.733492+0000","last_change":"2024-05-27T07:17:46.704100+0000","last_active":"2024-05-27T07:20:15.733492+0000","last_peered":"2024-05-27T07:20:15.733492+0000","last_clean":"2024-05-27T07:20:15.733492+0000","last_became_active":"2024-05-27T07:17:46.703475+0000","last_became_peered":"2024-05-27T07:17:46.703475+0000","last_unstale":"2024-05-27T07:20:15.733492+0000","last_undegraded":"2024-05-27T07:20:15.733492+0000","last_fullsized":"2024-05-27T07:20:15.733492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:20:29.173923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133397+0000","last_change":"2024-05-27T07:18:47.131521+0000","last_active":"2024-05-27T07:18:47.133397+0000","last_peered":"2024-05-27T07:18:47.133397+0000","last_clean":"2024-05-27T07:18:47.133397+0000","last_became_active":"2024-05-27T07:18:47.129995+0000","last_became_peered":"2024-05-27T07:18:47.129995+0000","last_unstale":"2024-05-27T07:18:47.133397+0000","last_undegraded":"2024-05-27T07:18:47.133397+0000","last_fullsized":"2024-05-27T07:18:47.133397+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:45:27.268768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'36","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:38.367751+0000","last_change":"2024-05-27T07:17:46.719293+0000","last_active":"2024-05-27T07:20:38.367751+0000","last_peered":"2024-05-27T07:20:38.367751+0000","last_clean":"2024-05-27T07:20:38.367751+0000","last_became_active":"2024-05-27T07:17:46.703496+0000","last_became_peered":"2024-05-27T07:17:46.703496+0000","last_unstale":"2024-05-27T07:20:38.367751+0000","last_undegraded":"2024-05-27T07:20:38.367751+0000","last_fullsized":"2024-05-27T07:20:38.367751+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:26.310524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62321,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":250,"num_read_kb":186,"num_write":37,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718304+0000","last_change":"2024-05-27T07:17:06.763141+0000","last_active":"2024-05-27T07:20:15.718304+0000","last_peered":"2024-05-27T07:20:15.718304+0000","last_clean":"2024-05-27T07:20:15.718304+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:20:15.718304+0000","last_undegraded":"2024-05-27T07:20:15.718304+0000","last_fullsized":"2024-05-27T07:20:15.718304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:47.072552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071470+0000","last_change":"2024-05-27T07:17:21.895342+0000","last_active":"2024-05-27T07:18:47.071470+0000","last_peered":"2024-05-27T07:18:47.071470+0000","last_clean":"2024-05-27T07:18:47.071470+0000","last_became_active":"2024-05-27T07:17:21.894630+0000","last_became_peered":"2024-05-27T07:17:21.894630+0000","last_unstale":"2024-05-27T07:18:47.071470+0000","last_undegraded":"2024-05-27T07:18:47.071470+0000","last_fullsized":"2024-05-27T07:18:47.071470+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:50.501683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062929+0000","last_change":"2024-05-27T07:17:02.341848+0000","last_active":"2024-05-27T07:18:47.062929+0000","last_peered":"2024-05-27T07:18:47.062929+0000","last_clean":"2024-05-27T07:18:47.062929+0000","last_became_active":"2024-05-27T07:17:02.339477+0000","last_became_peered":"2024-05-27T07:17:02.339477+0000","last_unstale":"2024-05-27T07:18:47.062929+0000","last_undegraded":"2024-05-27T07:18:47.062929+0000","last_fullsized":"2024-05-27T07:18:47.062929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:23.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718010+0000","last_change":"2024-05-27T07:17:06.761340+0000","last_active":"2024-05-27T07:20:15.718010+0000","last_peered":"2024-05-27T07:20:15.718010+0000","last_clean":"2024-05-27T07:20:15.718010+0000","last_became_active":"2024-05-27T07:17:06.759103+0000","last_became_peered":"2024-05-27T07:17:06.759103+0000","last_unstale":"2024-05-27T07:20:15.718010+0000","last_undegraded":"2024-05-27T07:20:15.718010+0000","last_fullsized":"2024-05-27T07:20:15.718010+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:42.263106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071473+0000","last_change":"2024-05-27T07:17:21.883252+0000","last_active":"2024-05-27T07:18:47.071473+0000","last_peered":"2024-05-27T07:18:47.071473+0000","last_clean":"2024-05-27T07:18:47.071473+0000","last_became_active":"2024-05-27T07:17:21.882468+0000","last_became_peered":"2024-05-27T07:17:21.882468+0000","last_unstale":"2024-05-27T07:18:47.071473+0000","last_undegraded":"2024-05-27T07:18:47.071473+0000","last_fullsized":"2024-05-27T07:18:47.071473+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:59.250357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069762+0000","last_change":"2024-05-27T07:17:02.343343+0000","last_active":"2024-05-27T07:18:47.069762+0000","last_peered":"2024-05-27T07:18:47.069762+0000","last_clean":"2024-05-27T07:18:47.069762+0000","last_became_active":"2024-05-27T07:17:02.342741+0000","last_became_peered":"2024-05-27T07:17:02.342741+0000","last_unstale":"2024-05-27T07:18:47.069762+0000","last_undegraded":"2024-05-27T07:18:47.069762+0000","last_fullsized":"2024-05-27T07:18:47.069762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:46.006865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.725298+0000","last_change":"2024-05-27T07:17:06.771473+0000","last_active":"2024-05-27T07:20:15.725298+0000","last_peered":"2024-05-27T07:20:15.725298+0000","last_clean":"2024-05-27T07:20:15.725298+0000","last_became_active":"2024-05-27T07:17:06.771082+0000","last_became_peered":"2024-05-27T07:17:06.771082+0000","last_unstale":"2024-05-27T07:20:15.725298+0000","last_undegraded":"2024-05-27T07:20:15.725298+0000","last_fullsized":"2024-05-27T07:20:15.725298+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:01.371182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060660+0000","last_change":"2024-05-27T07:17:21.897691+0000","last_active":"2024-05-27T07:18:47.060660+0000","last_peered":"2024-05-27T07:18:47.060660+0000","last_clean":"2024-05-27T07:18:47.060660+0000","last_became_active":"2024-05-27T07:17:21.895221+0000","last_became_peered":"2024-05-27T07:17:21.895221+0000","last_unstale":"2024-05-27T07:18:47.060660+0000","last_undegraded":"2024-05-27T07:18:47.060660+0000","last_fullsized":"2024-05-27T07:18:47.060660+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:14.626947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":174,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:37.370576+0000","last_change":"2024-05-27T07:17:02.343375+0000","last_active":"2024-05-27T07:20:37.370576+0000","last_peered":"2024-05-27T07:20:37.370576+0000","last_clean":"2024-05-27T07:20:37.370576+0000","last_became_active":"2024-05-27T07:17:02.341315+0000","last_became_peered":"2024-05-27T07:17:02.341315+0000","last_unstale":"2024-05-27T07:20:37.370576+0000","last_undegraded":"2024-05-27T07:20:37.370576+0000","last_fullsized":"2024-05-27T07:20:37.370576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:41.927265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.719109+0000","last_change":"2024-05-27T07:17:06.765862+0000","last_active":"2024-05-27T07:20:15.719109+0000","last_peered":"2024-05-27T07:20:15.719109+0000","last_clean":"2024-05-27T07:20:15.719109+0000","last_became_active":"2024-05-27T07:17:06.765439+0000","last_became_peered":"2024-05-27T07:17:06.765439+0000","last_unstale":"2024-05-27T07:20:15.719109+0000","last_undegraded":"2024-05-27T07:20:15.719109+0000","last_fullsized":"2024-05-27T07:20:15.719109+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:47.586833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130000+0000","last_change":"2024-05-27T07:17:21.891810+0000","last_active":"2024-05-27T07:18:47.130000+0000","last_peered":"2024-05-27T07:18:47.130000+0000","last_clean":"2024-05-27T07:18:47.130000+0000","last_became_active":"2024-05-27T07:17:21.890473+0000","last_became_peered":"2024-05-27T07:17:21.890473+0000","last_unstale":"2024-05-27T07:18:47.130000+0000","last_undegraded":"2024-05-27T07:18:47.130000+0000","last_fullsized":"2024-05-27T07:18:47.130000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:12.731189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065951+0000","last_change":"2024-05-27T07:17:46.715870+0000","last_active":"2024-05-27T07:18:47.065951+0000","last_peered":"2024-05-27T07:18:47.065951+0000","last_clean":"2024-05-27T07:18:47.065951+0000","last_became_active":"2024-05-27T07:17:46.701248+0000","last_became_peered":"2024-05-27T07:17:46.701248+0000","last_unstale":"2024-05-27T07:18:47.065951+0000","last_undegraded":"2024-05-27T07:18:47.065951+0000","last_fullsized":"2024-05-27T07:18:47.065951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:18.693752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715188+0000","last_change":"2024-05-27T07:17:06.759937+0000","last_active":"2024-05-27T07:20:15.715188+0000","last_peered":"2024-05-27T07:20:15.715188+0000","last_clean":"2024-05-27T07:20:15.715188+0000","last_became_active":"2024-05-27T07:17:06.759526+0000","last_became_peered":"2024-05-27T07:17:06.759526+0000","last_unstale":"2024-05-27T07:20:15.715188+0000","last_undegraded":"2024-05-27T07:20:15.715188+0000","last_fullsized":"2024-05-27T07:20:15.715188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:22.934317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068886+0000","last_change":"2024-05-27T07:17:21.898331+0000","last_active":"2024-05-27T07:18:47.068886+0000","last_peered":"2024-05-27T07:18:47.068886+0000","last_clean":"2024-05-27T07:18:47.068886+0000","last_became_active":"2024-05-27T07:17:21.897965+0000","last_became_peered":"2024-05-27T07:17:21.897965+0000","last_unstale":"2024-05-27T07:18:47.068886+0000","last_undegraded":"2024-05-27T07:18:47.068886+0000","last_fullsized":"2024-05-27T07:18:47.068886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:20.584175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128389+0000","last_change":"2024-05-27T07:17:02.346029+0000","last_active":"2024-05-27T07:18:47.128389+0000","last_peered":"2024-05-27T07:18:47.128389+0000","last_clean":"2024-05-27T07:18:47.128389+0000","last_became_active":"2024-05-27T07:17:02.341292+0000","last_became_peered":"2024-05-27T07:17:02.341292+0000","last_unstale":"2024-05-27T07:18:47.128389+0000","last_undegraded":"2024-05-27T07:18:47.128389+0000","last_fullsized":"2024-05-27T07:18:47.128389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:31.705741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.712879+0000","last_change":"2024-05-27T07:17:46.691785+0000","last_active":"2024-05-27T07:20:15.712879+0000","last_peered":"2024-05-27T07:20:15.712879+0000","last_clean":"2024-05-27T07:20:15.712879+0000","last_became_active":"2024-05-27T07:17:46.690445+0000","last_became_peered":"2024-05-27T07:17:46.690445+0000","last_unstale":"2024-05-27T07:20:15.712879+0000","last_undegraded":"2024-05-27T07:20:15.712879+0000","last_fullsized":"2024-05-27T07:20:15.712879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:59:27.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126890+0000","last_change":"2024-05-27T07:17:21.892434+0000","last_active":"2024-05-27T07:18:47.126890+0000","last_peered":"2024-05-27T07:18:47.126890+0000","last_clean":"2024-05-27T07:18:47.126890+0000","last_became_active":"2024-05-27T07:17:21.891736+0000","last_became_peered":"2024-05-27T07:17:21.891736+0000","last_unstale":"2024-05-27T07:18:47.126890+0000","last_undegraded":"2024-05-27T07:18:47.126890+0000","last_fullsized":"2024-05-27T07:18:47.126890+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:48.458819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059903+0000","last_change":"2024-05-27T07:17:02.436323+0000","last_active":"2024-05-27T07:18:47.059903+0000","last_peered":"2024-05-27T07:18:47.059903+0000","last_clean":"2024-05-27T07:18:47.059903+0000","last_became_active":"2024-05-27T07:17:02.435000+0000","last_became_peered":"2024-05-27T07:17:02.435000+0000","last_unstale":"2024-05-27T07:18:47.059903+0000","last_undegraded":"2024-05-27T07:18:47.059903+0000","last_fullsized":"2024-05-27T07:18:47.059903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:08.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716763+0000","last_change":"2024-05-27T07:17:06.760418+0000","last_active":"2024-05-27T07:20:15.716763+0000","last_peered":"2024-05-27T07:20:15.716763+0000","last_clean":"2024-05-27T07:20:15.716763+0000","last_became_active":"2024-05-27T07:17:06.759875+0000","last_became_peered":"2024-05-27T07:17:06.759875+0000","last_unstale":"2024-05-27T07:20:15.716763+0000","last_undegraded":"2024-05-27T07:20:15.716763+0000","last_fullsized":"2024-05-27T07:20:15.716763+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:12:54.324323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130559+0000","last_change":"2024-05-27T07:17:21.888048+0000","last_active":"2024-05-27T07:18:47.130559+0000","last_peered":"2024-05-27T07:18:47.130559+0000","last_clean":"2024-05-27T07:18:47.130559+0000","last_became_active":"2024-05-27T07:17:21.887579+0000","last_became_peered":"2024-05-27T07:17:21.887579+0000","last_unstale":"2024-05-27T07:18:47.130559+0000","last_undegraded":"2024-05-27T07:18:47.130559+0000","last_fullsized":"2024-05-27T07:18:47.130559+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:34:44.569959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066518+0000","last_change":"2024-05-27T07:17:02.337655+0000","last_active":"2024-05-27T07:18:47.066518+0000","last_peered":"2024-05-27T07:18:47.066518+0000","last_clean":"2024-05-27T07:18:47.066518+0000","last_became_active":"2024-05-27T07:17:02.337252+0000","last_became_peered":"2024-05-27T07:17:02.337252+0000","last_unstale":"2024-05-27T07:18:47.066518+0000","last_undegraded":"2024-05-27T07:18:47.066518+0000","last_fullsized":"2024-05-27T07:18:47.066518+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:11:25.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.703510+0000","last_change":"2024-05-27T07:17:06.762934+0000","last_active":"2024-05-27T07:20:15.703510+0000","last_peered":"2024-05-27T07:20:15.703510+0000","last_clean":"2024-05-27T07:20:15.703510+0000","last_became_active":"2024-05-27T07:17:06.762507+0000","last_became_peered":"2024-05-27T07:17:06.762507+0000","last_unstale":"2024-05-27T07:20:15.703510+0000","last_undegraded":"2024-05-27T07:20:15.703510+0000","last_fullsized":"2024-05-27T07:20:15.703510+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:23:35.164892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129953+0000","last_change":"2024-05-27T07:17:21.886991+0000","last_active":"2024-05-27T07:18:47.129953+0000","last_peered":"2024-05-27T07:18:47.129953+0000","last_clean":"2024-05-27T07:18:47.129953+0000","last_became_active":"2024-05-27T07:17:21.885945+0000","last_became_peered":"2024-05-27T07:17:21.885945+0000","last_unstale":"2024-05-27T07:18:47.129953+0000","last_undegraded":"2024-05-27T07:18:47.129953+0000","last_fullsized":"2024-05-27T07:18:47.129953+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:23:01.853601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717378+0000","last_change":"2024-05-27T07:17:46.864130+0000","last_active":"2024-05-27T07:20:15.717378+0000","last_peered":"2024-05-27T07:20:15.717378+0000","last_clean":"2024-05-27T07:20:15.717378+0000","last_became_active":"2024-05-27T07:17:46.862997+0000","last_became_peered":"2024-05-27T07:17:46.862997+0000","last_unstale":"2024-05-27T07:20:15.717378+0000","last_undegraded":"2024-05-27T07:20:15.717378+0000","last_fullsized":"2024-05-27T07:20:15.717378+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:10.594627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:36.365707+0000","last_change":"2024-05-27T07:17:02.435463+0000","last_active":"2024-05-27T07:20:36.365707+0000","last_peered":"2024-05-27T07:20:36.365707+0000","last_clean":"2024-05-27T07:20:36.365707+0000","last_became_active":"2024-05-27T07:17:02.432131+0000","last_became_peered":"2024-05-27T07:17:02.432131+0000","last_unstale":"2024-05-27T07:20:36.365707+0000","last_undegraded":"2024-05-27T07:20:36.365707+0000","last_fullsized":"2024-05-27T07:20:36.365707+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:41.874577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:51.173771+0000","last_change":"2024-05-27T07:18:47.134043+0000","last_active":"2024-05-27T07:18:51.173771+0000","last_peered":"2024-05-27T07:18:51.173771+0000","last_clean":"2024-05-27T07:18:51.173771+0000","last_became_active":"2024-05-27T07:18:47.132050+0000","last_became_peered":"2024-05-27T07:18:47.132050+0000","last_unstale":"2024-05-27T07:18:51.173771+0000","last_undegraded":"2024-05-27T07:18:51.173771+0000","last_fullsized":"2024-05-27T07:18:51.173771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:12.170883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718922+0000","last_change":"2024-05-27T07:17:06.751168+0000","last_active":"2024-05-27T07:20:15.718922+0000","last_peered":"2024-05-27T07:20:15.718922+0000","last_clean":"2024-05-27T07:20:15.718922+0000","last_became_active":"2024-05-27T07:17:06.750816+0000","last_became_peered":"2024-05-27T07:17:06.750816+0000","last_unstale":"2024-05-27T07:20:15.718922+0000","last_undegraded":"2024-05-27T07:20:15.718922+0000","last_fullsized":"2024-05-27T07:20:15.718922+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:33:45.607426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083905+0000","last_change":"2024-05-27T07:17:02.337490+0000","last_active":"2024-05-27T07:18:47.083905+0000","last_peered":"2024-05-27T07:18:47.083905+0000","last_clean":"2024-05-27T07:18:47.083905+0000","last_became_active":"2024-05-27T07:17:02.335317+0000","last_became_peered":"2024-05-27T07:17:02.335317+0000","last_unstale":"2024-05-27T07:18:47.083905+0000","last_undegraded":"2024-05-27T07:18:47.083905+0000","last_fullsized":"2024-05-27T07:18:47.083905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:33:46.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":120,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065144+0000","last_change":"2024-05-27T07:15:10.825312+0000","last_active":"2024-05-27T07:18:47.065144+0000","last_peered":"2024-05-27T07:18:47.065144+0000","last_clean":"2024-05-27T07:18:47.065144+0000","last_became_active":"2024-05-27T07:15:10.809245+0000","last_became_peered":"2024-05-27T07:15:10.809245+0000","last_unstale":"2024-05-27T07:18:47.065144+0000","last_undegraded":"2024-05-27T07:18:47.065144+0000","last_fullsized":"2024-05-27T07:18:47.065144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_clean_scrub_stamp":"2024-05-27T07:10:08.008565+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:32.769372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.137659+0000","last_change":"2024-05-27T07:18:47.133948+0000","last_active":"2024-05-27T07:18:47.137659+0000","last_peered":"2024-05-27T07:18:47.137659+0000","last_clean":"2024-05-27T07:18:47.137659+0000","last_became_active":"2024-05-27T07:18:47.132566+0000","last_became_peered":"2024-05-27T07:18:47.132566+0000","last_unstale":"2024-05-27T07:18:47.137659+0000","last_undegraded":"2024-05-27T07:18:47.137659+0000","last_fullsized":"2024-05-27T07:18:47.137659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:04.463121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715743+0000","last_change":"2024-05-27T07:17:06.766187+0000","last_active":"2024-05-27T07:20:15.715743+0000","last_peered":"2024-05-27T07:20:15.715743+0000","last_clean":"2024-05-27T07:20:15.715743+0000","last_became_active":"2024-05-27T07:17:06.763365+0000","last_became_peered":"2024-05-27T07:17:06.763365+0000","last_unstale":"2024-05-27T07:20:15.715743+0000","last_undegraded":"2024-05-27T07:20:15.715743+0000","last_fullsized":"2024-05-27T07:20:15.715743+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:13.668426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065681+0000","last_change":"2024-05-27T07:17:02.344452+0000","last_active":"2024-05-27T07:18:47.065681+0000","last_peered":"2024-05-27T07:18:47.065681+0000","last_clean":"2024-05-27T07:18:47.065681+0000","last_became_active":"2024-05-27T07:17:02.343885+0000","last_became_peered":"2024-05-27T07:17:02.343885+0000","last_unstale":"2024-05-27T07:18:47.065681+0000","last_undegraded":"2024-05-27T07:18:47.065681+0000","last_fullsized":"2024-05-27T07:18:47.065681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:26.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072673+0000","last_change":"2024-05-27T07:17:21.883024+0000","last_active":"2024-05-27T07:18:47.072673+0000","last_peered":"2024-05-27T07:18:47.072673+0000","last_clean":"2024-05-27T07:18:47.072673+0000","last_became_active":"2024-05-27T07:17:21.882429+0000","last_became_peered":"2024-05-27T07:17:21.882429+0000","last_unstale":"2024-05-27T07:18:47.072673+0000","last_undegraded":"2024-05-27T07:18:47.072673+0000","last_fullsized":"2024-05-27T07:18:47.072673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:36:40.443537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128649+0000","last_change":"2024-05-27T07:17:02.432768+0000","last_active":"2024-05-27T07:18:47.128649+0000","last_peered":"2024-05-27T07:18:47.128649+0000","last_clean":"2024-05-27T07:18:47.128649+0000","last_became_active":"2024-05-27T07:17:02.431551+0000","last_became_peered":"2024-05-27T07:17:02.431551+0000","last_unstale":"2024-05-27T07:18:47.128649+0000","last_undegraded":"2024-05-27T07:18:47.128649+0000","last_fullsized":"2024-05-27T07:18:47.128649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:39.759512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717103+0000","last_change":"2024-05-27T07:17:06.770992+0000","last_active":"2024-05-27T07:20:15.717103+0000","last_peered":"2024-05-27T07:20:15.717103+0000","last_clean":"2024-05-27T07:20:15.717103+0000","last_became_active":"2024-05-27T07:17:06.770560+0000","last_became_peered":"2024-05-27T07:17:06.770560+0000","last_unstale":"2024-05-27T07:20:15.717103+0000","last_undegraded":"2024-05-27T07:20:15.717103+0000","last_fullsized":"2024-05-27T07:20:15.717103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:07:34.443551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.124998+0000","last_change":"2024-05-27T07:17:21.895362+0000","last_active":"2024-05-27T07:18:47.124998+0000","last_peered":"2024-05-27T07:18:47.124998+0000","last_clean":"2024-05-27T07:18:47.124998+0000","last_became_active":"2024-05-27T07:17:21.894563+0000","last_became_peered":"2024-05-27T07:17:21.894563+0000","last_unstale":"2024-05-27T07:18:47.124998+0000","last_undegraded":"2024-05-27T07:18:47.124998+0000","last_fullsized":"2024-05-27T07:18:47.124998+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:44.394301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'13","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:15.306628+0000","last_change":"2024-05-27T07:17:02.338203+0000","last_active":"2024-05-27T07:19:15.306628+0000","last_peered":"2024-05-27T07:19:15.306628+0000","last_clean":"2024-05-27T07:19:15.306628+0000","last_became_active":"2024-05-27T07:17:02.336192+0000","last_became_peered":"2024-05-27T07:17:02.336192+0000","last_unstale":"2024-05-27T07:19:15.306628+0000","last_undegraded":"2024-05-27T07:19:15.306628+0000","last_fullsized":"2024-05-27T07:19:15.306628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:48.187517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718283+0000","last_change":"2024-05-27T07:17:06.759729+0000","last_active":"2024-05-27T07:20:15.718283+0000","last_peered":"2024-05-27T07:20:15.718283+0000","last_clean":"2024-05-27T07:20:15.718283+0000","last_became_active":"2024-05-27T07:17:06.756766+0000","last_became_peered":"2024-05-27T07:17:06.756766+0000","last_unstale":"2024-05-27T07:20:15.718283+0000","last_undegraded":"2024-05-27T07:20:15.718283+0000","last_fullsized":"2024-05-27T07:20:15.718283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:01.090586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063120+0000","last_change":"2024-05-27T07:17:21.878487+0000","last_active":"2024-05-27T07:18:47.063120+0000","last_peered":"2024-05-27T07:18:47.063120+0000","last_clean":"2024-05-27T07:18:47.063120+0000","last_became_active":"2024-05-27T07:17:21.877623+0000","last_became_peered":"2024-05-27T07:17:21.877623+0000","last_unstale":"2024-05-27T07:18:47.063120+0000","last_undegraded":"2024-05-27T07:18:47.063120+0000","last_fullsized":"2024-05-27T07:18:47.063120+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:28:52.618738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.085169+0000","last_change":"2024-05-27T07:17:02.486495+0000","last_active":"2024-05-27T07:18:47.085169+0000","last_peered":"2024-05-27T07:18:47.085169+0000","last_clean":"2024-05-27T07:18:47.085169+0000","last_became_active":"2024-05-27T07:17:02.485493+0000","last_became_peered":"2024-05-27T07:17:02.485493+0000","last_unstale":"2024-05-27T07:18:47.085169+0000","last_undegraded":"2024-05-27T07:18:47.085169+0000","last_fullsized":"2024-05-27T07:18:47.085169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:31.977210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714041+0000","last_change":"2024-05-27T07:17:06.760937+0000","last_active":"2024-05-27T07:20:15.714041+0000","last_peered":"2024-05-27T07:20:15.714041+0000","last_clean":"2024-05-27T07:20:15.714041+0000","last_became_active":"2024-05-27T07:17:06.760254+0000","last_became_peered":"2024-05-27T07:17:06.760254+0000","last_unstale":"2024-05-27T07:20:15.714041+0000","last_undegraded":"2024-05-27T07:20:15.714041+0000","last_fullsized":"2024-05-27T07:20:15.714041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:50.147319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084721+0000","last_change":"2024-05-27T07:17:21.901511+0000","last_active":"2024-05-27T07:18:47.084721+0000","last_peered":"2024-05-27T07:18:47.084721+0000","last_clean":"2024-05-27T07:18:47.084721+0000","last_became_active":"2024-05-27T07:17:21.899847+0000","last_became_peered":"2024-05-27T07:17:21.899847+0000","last_unstale":"2024-05-27T07:18:47.084721+0000","last_undegraded":"2024-05-27T07:18:47.084721+0000","last_fullsized":"2024-05-27T07:18:47.084721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:28.133509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080859+0000","last_change":"2024-05-27T07:17:02.432520+0000","last_active":"2024-05-27T07:18:47.080859+0000","last_peered":"2024-05-27T07:18:47.080859+0000","last_clean":"2024-05-27T07:18:47.080859+0000","last_became_active":"2024-05-27T07:17:02.431563+0000","last_became_peered":"2024-05-27T07:17:02.431563+0000","last_unstale":"2024-05-27T07:18:47.080859+0000","last_undegraded":"2024-05-27T07:18:47.080859+0000","last_fullsized":"2024-05-27T07:18:47.080859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:17:28.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.660913+0000","last_change":"2024-05-27T07:17:06.754408+0000","last_active":"2024-05-27T07:20:15.660913+0000","last_peered":"2024-05-27T07:20:15.660913+0000","last_clean":"2024-05-27T07:20:15.660913+0000","last_became_active":"2024-05-27T07:17:06.753787+0000","last_became_peered":"2024-05-27T07:17:06.753787+0000","last_unstale":"2024-05-27T07:20:15.660913+0000","last_undegraded":"2024-05-27T07:20:15.660913+0000","last_fullsized":"2024-05-27T07:20:15.660913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:21:50.461296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060453+0000","last_change":"2024-05-27T07:17:21.895277+0000","last_active":"2024-05-27T07:18:47.060453+0000","last_peered":"2024-05-27T07:18:47.060453+0000","last_clean":"2024-05-27T07:18:47.060453+0000","last_became_active":"2024-05-27T07:17:21.894518+0000","last_became_peered":"2024-05-27T07:17:21.894518+0000","last_unstale":"2024-05-27T07:18:47.060453+0000","last_undegraded":"2024-05-27T07:18:47.060453+0000","last_fullsized":"2024-05-27T07:18:47.060453+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:02.008209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125540+0000","last_change":"2024-05-27T07:17:02.343604+0000","last_active":"2024-05-27T07:18:47.125540+0000","last_peered":"2024-05-27T07:18:47.125540+0000","last_clean":"2024-05-27T07:18:47.125540+0000","last_became_active":"2024-05-27T07:17:02.341890+0000","last_became_peered":"2024-05-27T07:17:02.341890+0000","last_unstale":"2024-05-27T07:18:47.125540+0000","last_undegraded":"2024-05-27T07:18:47.125540+0000","last_fullsized":"2024-05-27T07:18:47.125540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:10:21.670193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.708722+0000","last_change":"2024-05-27T07:17:06.754767+0000","last_active":"2024-05-27T07:20:15.708722+0000","last_peered":"2024-05-27T07:20:15.708722+0000","last_clean":"2024-05-27T07:20:15.708722+0000","last_became_active":"2024-05-27T07:17:06.754291+0000","last_became_peered":"2024-05-27T07:17:06.754291+0000","last_unstale":"2024-05-27T07:20:15.708722+0000","last_undegraded":"2024-05-27T07:20:15.708722+0000","last_fullsized":"2024-05-27T07:20:15.708722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:46:48.145874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:59.283159+0000","last_change":"2024-05-27T07:17:21.892342+0000","last_active":"2024-05-27T07:18:59.283159+0000","last_peered":"2024-05-27T07:18:59.283159+0000","last_clean":"2024-05-27T07:18:59.283159+0000","last_became_active":"2024-05-27T07:17:21.891393+0000","last_became_peered":"2024-05-27T07:17:21.891393+0000","last_unstale":"2024-05-27T07:18:59.283159+0000","last_undegraded":"2024-05-27T07:18:59.283159+0000","last_fullsized":"2024-05-27T07:18:59.283159+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:30.510616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062370+0000","last_change":"2024-05-27T07:17:02.340238+0000","last_active":"2024-05-27T07:18:47.062370+0000","last_peered":"2024-05-27T07:18:47.062370+0000","last_clean":"2024-05-27T07:18:47.062370+0000","last_became_active":"2024-05-27T07:17:02.339561+0000","last_became_peered":"2024-05-27T07:17:02.339561+0000","last_unstale":"2024-05-27T07:18:47.062370+0000","last_undegraded":"2024-05-27T07:18:47.062370+0000","last_fullsized":"2024-05-27T07:18:47.062370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:32.451397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715306+0000","last_change":"2024-05-27T07:17:06.769397+0000","last_active":"2024-05-27T07:20:15.715306+0000","last_peered":"2024-05-27T07:20:15.715306+0000","last_clean":"2024-05-27T07:20:15.715306+0000","last_became_active":"2024-05-27T07:17:06.768936+0000","last_became_peered":"2024-05-27T07:17:06.768936+0000","last_unstale":"2024-05-27T07:20:15.715306+0000","last_undegraded":"2024-05-27T07:20:15.715306+0000","last_fullsized":"2024-05-27T07:20:15.715306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:52.767902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084477+0000","last_change":"2024-05-27T07:17:21.894225+0000","last_active":"2024-05-27T07:18:47.084477+0000","last_peered":"2024-05-27T07:18:47.084477+0000","last_clean":"2024-05-27T07:18:47.084477+0000","last_became_active":"2024-05-27T07:17:21.893499+0000","last_became_peered":"2024-05-27T07:17:21.893499+0000","last_unstale":"2024-05-27T07:18:47.084477+0000","last_undegraded":"2024-05-27T07:18:47.084477+0000","last_fullsized":"2024-05-27T07:18:47.084477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:54.985290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126297+0000","last_change":"2024-05-27T07:17:02.347736+0000","last_active":"2024-05-27T07:18:47.126297+0000","last_peered":"2024-05-27T07:18:47.126297+0000","last_clean":"2024-05-27T07:18:47.126297+0000","last_became_active":"2024-05-27T07:17:02.347237+0000","last_became_peered":"2024-05-27T07:17:02.347237+0000","last_unstale":"2024-05-27T07:18:47.126297+0000","last_undegraded":"2024-05-27T07:18:47.126297+0000","last_fullsized":"2024-05-27T07:18:47.126297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:36:15.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732204+0000","last_change":"2024-05-27T07:17:06.759348+0000","last_active":"2024-05-27T07:20:15.732204+0000","last_peered":"2024-05-27T07:20:15.732204+0000","last_clean":"2024-05-27T07:20:15.732204+0000","last_became_active":"2024-05-27T07:17:06.758967+0000","last_became_peered":"2024-05-27T07:17:06.758967+0000","last_unstale":"2024-05-27T07:20:15.732204+0000","last_undegraded":"2024-05-27T07:20:15.732204+0000","last_fullsized":"2024-05-27T07:20:15.732204+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:45.430719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080597+0000","last_change":"2024-05-27T07:17:21.882919+0000","last_active":"2024-05-27T07:18:47.080597+0000","last_peered":"2024-05-27T07:18:47.080597+0000","last_clean":"2024-05-27T07:18:47.080597+0000","last_became_active":"2024-05-27T07:17:21.882330+0000","last_became_peered":"2024-05-27T07:17:21.882330+0000","last_unstale":"2024-05-27T07:18:47.080597+0000","last_undegraded":"2024-05-27T07:18:47.080597+0000","last_fullsized":"2024-05-27T07:18:47.080597+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:51.409079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064865+0000","last_change":"2024-05-27T07:17:02.486099+0000","last_active":"2024-05-27T07:18:47.064865+0000","last_peered":"2024-05-27T07:18:47.064865+0000","last_clean":"2024-05-27T07:18:47.064865+0000","last_became_active":"2024-05-27T07:17:02.485116+0000","last_became_peered":"2024-05-27T07:17:02.485116+0000","last_unstale":"2024-05-27T07:18:47.064865+0000","last_undegraded":"2024-05-27T07:18:47.064865+0000","last_fullsized":"2024-05-27T07:18:47.064865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:36.338287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.731750+0000","last_change":"2024-05-27T07:17:06.773586+0000","last_active":"2024-05-27T07:20:15.731750+0000","last_peered":"2024-05-27T07:20:15.731750+0000","last_clean":"2024-05-27T07:20:15.731750+0000","last_became_active":"2024-05-27T07:17:06.773139+0000","last_became_peered":"2024-05-27T07:17:06.773139+0000","last_unstale":"2024-05-27T07:20:15.731750+0000","last_undegraded":"2024-05-27T07:20:15.731750+0000","last_fullsized":"2024-05-27T07:20:15.731750+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:19:36.373091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127485+0000","last_change":"2024-05-27T07:17:21.892351+0000","last_active":"2024-05-27T07:18:47.127485+0000","last_peered":"2024-05-27T07:18:47.127485+0000","last_clean":"2024-05-27T07:18:47.127485+0000","last_became_active":"2024-05-27T07:17:21.891862+0000","last_became_peered":"2024-05-27T07:17:21.891862+0000","last_unstale":"2024-05-27T07:18:47.127485+0000","last_undegraded":"2024-05-27T07:18:47.127485+0000","last_fullsized":"2024-05-27T07:18:47.127485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:50:18.193149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071128+0000","last_change":"2024-05-27T07:17:02.336917+0000","last_active":"2024-05-27T07:18:47.071128+0000","last_peered":"2024-05-27T07:18:47.071128+0000","last_clean":"2024-05-27T07:18:47.071128+0000","last_became_active":"2024-05-27T07:17:02.335836+0000","last_became_peered":"2024-05-27T07:17:02.335836+0000","last_unstale":"2024-05-27T07:18:47.071128+0000","last_undegraded":"2024-05-27T07:18:47.071128+0000","last_fullsized":"2024-05-27T07:18:47.071128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:54.109480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.719413+0000","last_change":"2024-05-27T07:17:46.705826+0000","last_active":"2024-05-27T07:20:15.719413+0000","last_peered":"2024-05-27T07:20:15.719413+0000","last_clean":"2024-05-27T07:20:15.719413+0000","last_became_active":"2024-05-27T07:17:46.704749+0000","last_became_peered":"2024-05-27T07:17:46.704749+0000","last_unstale":"2024-05-27T07:20:15.719413+0000","last_undegraded":"2024-05-27T07:20:15.719413+0000","last_fullsized":"2024-05-27T07:20:15.719413+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:26.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068148+0000","last_change":"2024-05-27T07:17:21.885535+0000","last_active":"2024-05-27T07:18:47.068148+0000","last_peered":"2024-05-27T07:18:47.068148+0000","last_clean":"2024-05-27T07:18:47.068148+0000","last_became_active":"2024-05-27T07:17:21.884118+0000","last_became_peered":"2024-05-27T07:17:21.884118+0000","last_unstale":"2024-05-27T07:18:47.068148+0000","last_undegraded":"2024-05-27T07:18:47.068148+0000","last_fullsized":"2024-05-27T07:18:47.068148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:26.600199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130129+0000","last_change":"2024-05-27T07:17:02.340037+0000","last_active":"2024-05-27T07:18:47.130129+0000","last_peered":"2024-05-27T07:18:47.130129+0000","last_clean":"2024-05-27T07:18:47.130129+0000","last_became_active":"2024-05-27T07:17:02.339378+0000","last_became_peered":"2024-05-27T07:17:02.339378+0000","last_unstale":"2024-05-27T07:18:47.130129+0000","last_undegraded":"2024-05-27T07:18:47.130129+0000","last_fullsized":"2024-05-27T07:18:47.130129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:33.774917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.722350+0000","last_change":"2024-05-27T07:17:06.763277+0000","last_active":"2024-05-27T07:20:15.722350+0000","last_peered":"2024-05-27T07:20:15.722350+0000","last_clean":"2024-05-27T07:20:15.722350+0000","last_became_active":"2024-05-27T07:17:06.762837+0000","last_became_peered":"2024-05-27T07:17:06.762837+0000","last_unstale":"2024-05-27T07:20:15.722350+0000","last_undegraded":"2024-05-27T07:20:15.722350+0000","last_fullsized":"2024-05-27T07:20:15.722350+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:52.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716297+0000","last_change":"2024-05-27T07:17:06.752556+0000","last_active":"2024-05-27T07:20:15.716297+0000","last_peered":"2024-05-27T07:20:15.716297+0000","last_clean":"2024-05-27T07:20:15.716297+0000","last_became_active":"2024-05-27T07:17:06.751085+0000","last_became_peered":"2024-05-27T07:17:06.751085+0000","last_unstale":"2024-05-27T07:20:15.716297+0000","last_undegraded":"2024-05-27T07:20:15.716297+0000","last_fullsized":"2024-05-27T07:20:15.716297+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:13:53.883845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716301+0000","last_change":"2024-05-27T07:17:06.753674+0000","last_active":"2024-05-27T07:20:15.716301+0000","last_peered":"2024-05-27T07:20:15.716301+0000","last_clean":"2024-05-27T07:20:15.716301+0000","last_became_active":"2024-05-27T07:17:06.752373+0000","last_became_peered":"2024-05-27T07:17:06.752373+0000","last_unstale":"2024-05-27T07:20:15.716301+0000","last_undegraded":"2024-05-27T07:20:15.716301+0000","last_fullsized":"2024-05-27T07:20:15.716301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:40:17.474462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.705046+0000","last_change":"2024-05-27T07:17:06.766796+0000","last_active":"2024-05-27T07:20:15.705046+0000","last_peered":"2024-05-27T07:20:15.705046+0000","last_clean":"2024-05-27T07:20:15.705046+0000","last_became_active":"2024-05-27T07:17:06.766435+0000","last_became_peered":"2024-05-27T07:17:06.766435+0000","last_unstale":"2024-05-27T07:20:15.705046+0000","last_undegraded":"2024-05-27T07:20:15.705046+0000","last_fullsized":"2024-05-27T07:20:15.705046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:38.747810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717873+0000","last_change":"2024-05-27T07:17:06.762804+0000","last_active":"2024-05-27T07:20:15.717873+0000","last_peered":"2024-05-27T07:20:15.717873+0000","last_clean":"2024-05-27T07:20:15.717873+0000","last_became_active":"2024-05-27T07:17:06.762490+0000","last_became_peered":"2024-05-27T07:17:06.762490+0000","last_unstale":"2024-05-27T07:20:15.717873+0000","last_undegraded":"2024-05-27T07:20:15.717873+0000","last_fullsized":"2024-05-27T07:20:15.717873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:59:33.584848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716897+0000","last_change":"2024-05-27T07:17:06.765640+0000","last_active":"2024-05-27T07:20:15.716897+0000","last_peered":"2024-05-27T07:20:15.716897+0000","last_clean":"2024-05-27T07:20:15.716897+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:20:15.716897+0000","last_undegraded":"2024-05-27T07:20:15.716897+0000","last_fullsized":"2024-05-27T07:20:15.716897+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:19.695007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706987+0000","last_change":"2024-05-27T07:17:06.759982+0000","last_active":"2024-05-27T07:20:15.706987+0000","last_peered":"2024-05-27T07:20:15.706987+0000","last_clean":"2024-05-27T07:20:15.706987+0000","last_became_active":"2024-05-27T07:17:06.759367+0000","last_became_peered":"2024-05-27T07:17:06.759367+0000","last_unstale":"2024-05-27T07:20:15.706987+0000","last_undegraded":"2024-05-27T07:20:15.706987+0000","last_fullsized":"2024-05-27T07:20:15.706987+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:35:04.686989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732188+0000","last_change":"2024-05-27T07:17:06.768264+0000","last_active":"2024-05-27T07:20:15.732188+0000","last_peered":"2024-05-27T07:20:15.732188+0000","last_clean":"2024-05-27T07:20:15.732188+0000","last_became_active":"2024-05-27T07:17:06.767952+0000","last_became_peered":"2024-05-27T07:17:06.767952+0000","last_unstale":"2024-05-27T07:20:15.732188+0000","last_undegraded":"2024-05-27T07:20:15.732188+0000","last_fullsized":"2024-05-27T07:20:15.732188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:34.867113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706251+0000","last_change":"2024-05-27T07:17:46.704705+0000","last_active":"2024-05-27T07:20:15.706251+0000","last_peered":"2024-05-27T07:20:15.706251+0000","last_clean":"2024-05-27T07:20:15.706251+0000","last_became_active":"2024-05-27T07:17:46.703503+0000","last_became_peered":"2024-05-27T07:17:46.703503+0000","last_unstale":"2024-05-27T07:20:15.706251+0000","last_undegraded":"2024-05-27T07:20:15.706251+0000","last_fullsized":"2024-05-27T07:20:15.706251+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:39.975009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.713613+0000","last_change":"2024-05-27T07:17:06.752080+0000","last_active":"2024-05-27T07:20:15.713613+0000","last_peered":"2024-05-27T07:20:15.713613+0000","last_clean":"2024-05-27T07:20:15.713613+0000","last_became_active":"2024-05-27T07:17:06.750486+0000","last_became_peered":"2024-05-27T07:17:06.750486+0000","last_unstale":"2024-05-27T07:20:15.713613+0000","last_undegraded":"2024-05-27T07:20:15.713613+0000","last_fullsized":"2024-05-27T07:20:15.713613+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:12.461727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715971+0000","last_change":"2024-05-27T07:17:06.759200+0000","last_active":"2024-05-27T07:20:15.715971+0000","last_peered":"2024-05-27T07:20:15.715971+0000","last_clean":"2024-05-27T07:20:15.715971+0000","last_became_active":"2024-05-27T07:17:06.758819+0000","last_became_peered":"2024-05-27T07:17:06.758819+0000","last_unstale":"2024-05-27T07:20:15.715971+0000","last_undegraded":"2024-05-27T07:20:15.715971+0000","last_fullsized":"2024-05-27T07:20:15.715971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:00.274229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732910+0000","last_change":"2024-05-27T07:17:06.760412+0000","last_active":"2024-05-27T07:20:15.732910+0000","last_peered":"2024-05-27T07:20:15.732910+0000","last_clean":"2024-05-27T07:20:15.732910+0000","last_became_active":"2024-05-27T07:17:06.759949+0000","last_became_peered":"2024-05-27T07:17:06.759949+0000","last_unstale":"2024-05-27T07:20:15.732910+0000","last_undegraded":"2024-05-27T07:20:15.732910+0000","last_fullsized":"2024-05-27T07:20:15.732910+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.235488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.725325+0000","last_change":"2024-05-27T07:17:06.775371+0000","last_active":"2024-05-27T07:20:15.725325+0000","last_peered":"2024-05-27T07:20:15.725325+0000","last_clean":"2024-05-27T07:20:15.725325+0000","last_became_active":"2024-05-27T07:17:06.774904+0000","last_became_peered":"2024-05-27T07:17:06.774904+0000","last_unstale":"2024-05-27T07:20:15.725325+0000","last_undegraded":"2024-05-27T07:20:15.725325+0000","last_fullsized":"2024-05-27T07:20:15.725325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:22:13.484733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714253+0000","last_change":"2024-05-27T07:17:06.757066+0000","last_active":"2024-05-27T07:20:15.714253+0000","last_peered":"2024-05-27T07:20:15.714253+0000","last_clean":"2024-05-27T07:20:15.714253+0000","last_became_active":"2024-05-27T07:17:06.756316+0000","last_became_peered":"2024-05-27T07:17:06.756316+0000","last_unstale":"2024-05-27T07:20:15.714253+0000","last_undegraded":"2024-05-27T07:20:15.714253+0000","last_fullsized":"2024-05-27T07:20:15.714253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:15:12.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66995,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":750,"num_read_kb":447,"num_write":179,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710782,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":515890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907012,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32852,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712300,"statfs":{"total":95995035648,"available":95961395200,"internally_reserved":0,"allocated":1716224,"data_stored":1165317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070541,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":519528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266773,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":519336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463003,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1630208,"data_stored":1111352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626530,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":510608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":572839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":511996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":511246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182522,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986290,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":579265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822763,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1638400,"data_stored":1102894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:20:41.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:20:42.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:41 smithi012 ceph-mon[31473]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:20:42.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:41 smithi162 ceph-mon[35676]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:20:42.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:41 smithi062 ceph-mon[35304]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:20:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:43.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:42 smithi012 ceph-mon[31473]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:43.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:42 smithi162 ceph-mon[35676]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:43.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:42 smithi062 ceph-mon[35304]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 -- 172.21.15.12:0/1703636418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340775b0 msgr2=0x7fcd34114e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 --2- 172.21.15.12:0/1703636418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340775b0 0x7fcd34114e00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009990 tx=0x7fcd20009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 -- 172.21.15.12:0/1703636418 shutdown_connections 2024-05-27T07:20:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 --2- 172.21.15.12:0/1703636418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd34076c60 0x7fcd34077070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 --2- 172.21.15.12:0/1703636418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd34115340 0x7fcd34117760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 --2- 172.21.15.12:0/1703636418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd340775b0 0x7fcd34114e00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 -- 172.21.15.12:0/1703636418 >> 172.21.15.12:0/1703636418 conn(0x7fcd340ffe90 msgr2=0x7fcd341022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 -- 172.21.15.12:0/1703636418 shutdown_connections 2024-05-27T07:20:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.689+0000 7fcd38a00700 1 -- 172.21.15.12:0/1703636418 wait complete. 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.690+0000 7fcd38a00700 1 Processor -- start 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.690+0000 7fcd38a00700 1 -- start start 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd38a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd34076c60 0x7fcd341a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd38a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd340775b0 0x7fcd341a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd38a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34115340 0x7fcd341a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd38a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3411a940 con 0x7fcd34115340 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd38a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3411a7c0 con 0x7fcd340775b0 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd38a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3411aac0 con 0x7fcd34076c60 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd32800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34115340 0x7fcd341a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd32800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34115340 0x7fcd341a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39148/0 (socket says 172.21.15.12:39148) 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd32800700 1 -- 172.21.15.12:0/31936660 learned_addr learned my addr 172.21.15.12:0/31936660 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd31e00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd34076c60 0x7fcd341a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.691+0000 7fcd31400700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd340775b0 0x7fcd341a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.692+0000 7fcd32800700 1 -- 172.21.15.12:0/31936660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd34076c60 msgr2=0x7fcd341a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.692+0000 7fcd32800700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd34076c60 0x7fcd341a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.692+0000 7fcd32800700 1 -- 172.21.15.12:0/31936660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd340775b0 msgr2=0x7fcd341a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.692+0000 7fcd32800700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd340775b0 0x7fcd341a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.692+0000 7fcd32800700 1 -- 172.21.15.12:0/31936660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd341a6240 con 0x7fcd34115340 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.692+0000 7fcd32800700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34115340 0x7fcd341a5ba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c007ee0 tx=0x7fcd2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.692+0000 7fcd2aa00700 1 -- 172.21.15.12:0/31936660 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c004bb0 con 0x7fcd34115340 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.693+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd341a6530 con 0x7fcd34115340 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.693+0000 7fcd2aa00700 1 -- 172.21.15.12:0/31936660 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcd2c004d50 con 0x7fcd34115340 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.693+0000 7fcd2aa00700 1 -- 172.21.15.12:0/31936660 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c0058b0 con 0x7fcd34115340 2024-05-27T07:20:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.693+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd341ade50 con 0x7fcd34115340 2024-05-27T07:20:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.695+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd340783a0 con 0x7fcd34115340 2024-05-27T07:20:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.697+0000 7fcd2aa00700 1 -- 172.21.15.12:0/31936660 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fcd2c01a020 con 0x7fcd34115340 2024-05-27T07:20:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.698+0000 7fcd2aa00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcd180b2900 0x7fcd180b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:44.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.698+0000 7fcd2aa00700 1 -- 172.21.15.12:0/31936660 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fcd2c0e1bd0 con 0x7fcd34115340 2024-05-27T07:20:44.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.699+0000 7fcd31e00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcd180b2900 0x7fcd180b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:44.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.699+0000 7fcd31e00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcd180b2900 0x7fcd180b4e10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009c60 tx=0x7fcd24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:44.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.703+0000 7fcd2aa00700 1 -- 172.21.15.12:0/31936660 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcd2c010070 con 0x7fcd34115340 2024-05-27T07:20:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.933+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcd341a26d0 con 0x7fcd180b2900 2024-05-27T07:20:44.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.945+0000 7fcd2aa00700 1 -- 172.21.15.12:0/31936660 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340480 (secure 0 0 0) 0x7fcd341a26d0 con 0x7fcd180b2900 2024-05-27T07:20:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:20:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcd180b2900 msgr2=0x7fcd180b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcd180b2900 0x7fcd180b4e10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009c60 tx=0x7fcd24009260 comp rx=0 tx=0).stop 2024-05-27T07:20:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34115340 msgr2=0x7fcd341a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34115340 0x7fcd341a5ba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c007ee0 tx=0x7fcd2c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 shutdown_connections 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fcd180b2900 0x7fcd180b4e10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34115340 0x7fcd341a5ba0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd34076c60 0x7fcd341a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 --2- 172.21.15.12:0/31936660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd340775b0 0x7fcd341a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.952+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 >> 172.21.15.12:0/31936660 conn(0x7fcd340ffe90 msgr2=0x7fcd341006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.953+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 shutdown_connections 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:44.953+0000 7fcd38a00700 1 -- 172.21.15.12:0/31936660 wait complete. 2024-05-27T07:20:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-05-27T07:20:45.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:44 smithi062 ceph-mon[35304]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:45.331 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:44 smithi012 ceph-mon[31473]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:45.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:44 smithi162 ceph-mon[35676]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:45.925 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-05-27T07:20:43.829816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657597,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":846,"num_read_kb":521,"num_write":304,"num_write_kb":2277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375728,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566096,"statfs":{"total":1151940427776,"available":1151555682304,"internally_reserved":0,"allocated":14434304,"data_stored":8131073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007318"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.726652+0000","last_change":"2024-05-27T07:17:06.763882+0000","last_active":"2024-05-27T07:20:15.726652+0000","last_peered":"2024-05-27T07:20:15.726652+0000","last_clean":"2024-05-27T07:20:15.726652+0000","last_became_active":"2024-05-27T07:17:06.759625+0000","last_became_peered":"2024-05-27T07:17:06.759625+0000","last_unstale":"2024-05-27T07:20:15.726652+0000","last_undegraded":"2024-05-27T07:20:15.726652+0000","last_fullsized":"2024-05-27T07:20:15.726652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:24.105867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.699117+0000","last_change":"2024-05-27T07:17:06.755737+0000","last_active":"2024-05-27T07:20:15.699117+0000","last_peered":"2024-05-27T07:20:15.699117+0000","last_clean":"2024-05-27T07:20:15.699117+0000","last_became_active":"2024-05-27T07:17:06.754588+0000","last_became_peered":"2024-05-27T07:17:06.754588+0000","last_unstale":"2024-05-27T07:20:15.699117+0000","last_undegraded":"2024-05-27T07:20:15.699117+0000","last_fullsized":"2024-05-27T07:20:15.699117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:02.417140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.730543+0000","last_change":"2024-05-27T07:17:06.772057+0000","last_active":"2024-05-27T07:20:15.730543+0000","last_peered":"2024-05-27T07:20:15.730543+0000","last_clean":"2024-05-27T07:20:15.730543+0000","last_became_active":"2024-05-27T07:17:06.771636+0000","last_became_peered":"2024-05-27T07:17:06.771636+0000","last_unstale":"2024-05-27T07:20:15.730543+0000","last_undegraded":"2024-05-27T07:20:15.730543+0000","last_fullsized":"2024-05-27T07:20:15.730543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:45.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714732+0000","last_change":"2024-05-27T07:17:06.763599+0000","last_active":"2024-05-27T07:20:15.714732+0000","last_peered":"2024-05-27T07:20:15.714732+0000","last_clean":"2024-05-27T07:20:15.714732+0000","last_became_active":"2024-05-27T07:17:06.763125+0000","last_became_peered":"2024-05-27T07:17:06.763125+0000","last_unstale":"2024-05-27T07:20:15.714732+0000","last_undegraded":"2024-05-27T07:20:15.714732+0000","last_fullsized":"2024-05-27T07:20:15.714732+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:43.359673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717624+0000","last_change":"2024-05-27T07:17:06.755896+0000","last_active":"2024-05-27T07:20:15.717624+0000","last_peered":"2024-05-27T07:20:15.717624+0000","last_clean":"2024-05-27T07:20:15.717624+0000","last_became_active":"2024-05-27T07:17:06.755495+0000","last_became_peered":"2024-05-27T07:17:06.755495+0000","last_unstale":"2024-05-27T07:20:15.717624+0000","last_undegraded":"2024-05-27T07:20:15.717624+0000","last_fullsized":"2024-05-27T07:20:15.717624+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:41:37.632012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706194+0000","last_change":"2024-05-27T07:17:46.699881+0000","last_active":"2024-05-27T07:20:15.706194+0000","last_peered":"2024-05-27T07:20:15.706194+0000","last_clean":"2024-05-27T07:20:15.706194+0000","last_became_active":"2024-05-27T07:17:46.699080+0000","last_became_peered":"2024-05-27T07:17:46.699080+0000","last_unstale":"2024-05-27T07:20:15.706194+0000","last_undegraded":"2024-05-27T07:20:15.706194+0000","last_fullsized":"2024-05-27T07:20:15.706194+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:43.580762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714004+0000","last_change":"2024-05-27T07:17:06.770468+0000","last_active":"2024-05-27T07:20:15.714004+0000","last_peered":"2024-05-27T07:20:15.714004+0000","last_clean":"2024-05-27T07:20:15.714004+0000","last_became_active":"2024-05-27T07:17:06.768345+0000","last_became_peered":"2024-05-27T07:17:06.768345+0000","last_unstale":"2024-05-27T07:20:15.714004+0000","last_undegraded":"2024-05-27T07:20:15.714004+0000","last_fullsized":"2024-05-27T07:20:15.714004+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:12:27.646654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717873+0000","last_change":"2024-05-27T07:17:06.758769+0000","last_active":"2024-05-27T07:20:15.717873+0000","last_peered":"2024-05-27T07:20:15.717873+0000","last_clean":"2024-05-27T07:20:15.717873+0000","last_became_active":"2024-05-27T07:17:06.758393+0000","last_became_peered":"2024-05-27T07:17:06.758393+0000","last_unstale":"2024-05-27T07:20:15.717873+0000","last_undegraded":"2024-05-27T07:20:15.717873+0000","last_fullsized":"2024-05-27T07:20:15.717873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:56.450569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.725941+0000","last_change":"2024-05-27T07:17:06.770681+0000","last_active":"2024-05-27T07:20:15.725941+0000","last_peered":"2024-05-27T07:20:15.725941+0000","last_clean":"2024-05-27T07:20:15.725941+0000","last_became_active":"2024-05-27T07:17:06.766520+0000","last_became_peered":"2024-05-27T07:17:06.766520+0000","last_unstale":"2024-05-27T07:20:15.725941+0000","last_undegraded":"2024-05-27T07:20:15.725941+0000","last_fullsized":"2024-05-27T07:20:15.725941+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:56.931743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.713503+0000","last_change":"2024-05-27T07:17:06.761462+0000","last_active":"2024-05-27T07:20:15.713503+0000","last_peered":"2024-05-27T07:20:15.713503+0000","last_clean":"2024-05-27T07:20:15.713503+0000","last_became_active":"2024-05-27T07:17:06.759344+0000","last_became_peered":"2024-05-27T07:17:06.759344+0000","last_unstale":"2024-05-27T07:20:15.713503+0000","last_undegraded":"2024-05-27T07:20:15.713503+0000","last_fullsized":"2024-05-27T07:20:15.713503+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:12:22.688477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.731716+0000","last_change":"2024-05-27T07:17:06.761947+0000","last_active":"2024-05-27T07:20:15.731716+0000","last_peered":"2024-05-27T07:20:15.731716+0000","last_clean":"2024-05-27T07:20:15.731716+0000","last_became_active":"2024-05-27T07:17:06.761464+0000","last_became_peered":"2024-05-27T07:17:06.761464+0000","last_unstale":"2024-05-27T07:20:15.731716+0000","last_undegraded":"2024-05-27T07:20:15.731716+0000","last_fullsized":"2024-05-27T07:20:15.731716+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:08:19.939311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715044+0000","last_change":"2024-05-27T07:17:06.772039+0000","last_active":"2024-05-27T07:20:15.715044+0000","last_peered":"2024-05-27T07:20:15.715044+0000","last_clean":"2024-05-27T07:20:15.715044+0000","last_became_active":"2024-05-27T07:17:06.771303+0000","last_became_peered":"2024-05-27T07:17:06.771303+0000","last_unstale":"2024-05-27T07:20:15.715044+0000","last_undegraded":"2024-05-27T07:20:15.715044+0000","last_fullsized":"2024-05-27T07:20:15.715044+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:30.254848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.719705+0000","last_change":"2024-05-27T07:17:06.774228+0000","last_active":"2024-05-27T07:20:15.719705+0000","last_peered":"2024-05-27T07:20:15.719705+0000","last_clean":"2024-05-27T07:20:15.719705+0000","last_became_active":"2024-05-27T07:17:06.771436+0000","last_became_peered":"2024-05-27T07:17:06.771436+0000","last_unstale":"2024-05-27T07:20:15.719705+0000","last_undegraded":"2024-05-27T07:20:15.719705+0000","last_fullsized":"2024-05-27T07:20:15.719705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:25.749143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706852+0000","last_change":"2024-05-27T07:17:06.755558+0000","last_active":"2024-05-27T07:20:15.706852+0000","last_peered":"2024-05-27T07:20:15.706852+0000","last_clean":"2024-05-27T07:20:15.706852+0000","last_became_active":"2024-05-27T07:17:06.754858+0000","last_became_peered":"2024-05-27T07:17:06.754858+0000","last_unstale":"2024-05-27T07:20:15.706852+0000","last_undegraded":"2024-05-27T07:20:15.706852+0000","last_fullsized":"2024-05-27T07:20:15.706852+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:09.853982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.709868+0000","last_change":"2024-05-27T07:17:06.770731+0000","last_active":"2024-05-27T07:20:15.709868+0000","last_peered":"2024-05-27T07:20:15.709868+0000","last_clean":"2024-05-27T07:20:15.709868+0000","last_became_active":"2024-05-27T07:17:06.766923+0000","last_became_peered":"2024-05-27T07:17:06.766923+0000","last_unstale":"2024-05-27T07:20:15.709868+0000","last_undegraded":"2024-05-27T07:20:15.709868+0000","last_fullsized":"2024-05-27T07:20:15.709868+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.110121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.731693+0000","last_change":"2024-05-27T07:17:06.758928+0000","last_active":"2024-05-27T07:20:15.731693+0000","last_peered":"2024-05-27T07:20:15.731693+0000","last_clean":"2024-05-27T07:20:15.731693+0000","last_became_active":"2024-05-27T07:17:06.758328+0000","last_became_peered":"2024-05-27T07:17:06.758328+0000","last_unstale":"2024-05-27T07:20:15.731693+0000","last_undegraded":"2024-05-27T07:20:15.731693+0000","last_fullsized":"2024-05-27T07:20:15.731693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:36.284395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.726854+0000","last_change":"2024-05-27T07:17:06.760006+0000","last_active":"2024-05-27T07:20:15.726854+0000","last_peered":"2024-05-27T07:20:15.726854+0000","last_clean":"2024-05-27T07:20:15.726854+0000","last_became_active":"2024-05-27T07:17:06.758177+0000","last_became_peered":"2024-05-27T07:17:06.758177+0000","last_unstale":"2024-05-27T07:20:15.726854+0000","last_undegraded":"2024-05-27T07:20:15.726854+0000","last_fullsized":"2024-05-27T07:20:15.726854+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.516289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.729927+0000","last_change":"2024-05-27T07:17:06.768484+0000","last_active":"2024-05-27T07:20:15.729927+0000","last_peered":"2024-05-27T07:20:15.729927+0000","last_clean":"2024-05-27T07:20:15.729927+0000","last_became_active":"2024-05-27T07:17:06.768105+0000","last_became_peered":"2024-05-27T07:17:06.768105+0000","last_unstale":"2024-05-27T07:20:15.729927+0000","last_undegraded":"2024-05-27T07:20:15.729927+0000","last_fullsized":"2024-05-27T07:20:15.729927+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:54:48.751678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717458+0000","last_change":"2024-05-27T07:17:06.762782+0000","last_active":"2024-05-27T07:20:15.717458+0000","last_peered":"2024-05-27T07:20:15.717458+0000","last_clean":"2024-05-27T07:20:15.717458+0000","last_became_active":"2024-05-27T07:17:06.760129+0000","last_became_peered":"2024-05-27T07:17:06.760129+0000","last_unstale":"2024-05-27T07:20:15.717458+0000","last_undegraded":"2024-05-27T07:20:15.717458+0000","last_fullsized":"2024-05-27T07:20:15.717458+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:47:19.360662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063670+0000","last_change":"2024-05-27T07:17:21.899772+0000","last_active":"2024-05-27T07:18:47.063670+0000","last_peered":"2024-05-27T07:18:47.063670+0000","last_clean":"2024-05-27T07:18:47.063670+0000","last_became_active":"2024-05-27T07:17:21.899038+0000","last_became_peered":"2024-05-27T07:17:21.899038+0000","last_unstale":"2024-05-27T07:18:47.063670+0000","last_undegraded":"2024-05-27T07:18:47.063670+0000","last_fullsized":"2024-05-27T07:18:47.063670+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:38.784433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059878+0000","last_change":"2024-05-27T07:17:46.708798+0000","last_active":"2024-05-27T07:18:47.059878+0000","last_peered":"2024-05-27T07:18:47.059878+0000","last_clean":"2024-05-27T07:18:47.059878+0000","last_became_active":"2024-05-27T07:17:46.696433+0000","last_became_peered":"2024-05-27T07:17:46.696433+0000","last_unstale":"2024-05-27T07:18:47.059878+0000","last_undegraded":"2024-05-27T07:18:47.059878+0000","last_fullsized":"2024-05-27T07:18:47.059878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:40.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717457+0000","last_change":"2024-05-27T07:17:06.760865+0000","last_active":"2024-05-27T07:20:15.717457+0000","last_peered":"2024-05-27T07:20:15.717457+0000","last_clean":"2024-05-27T07:20:15.717457+0000","last_became_active":"2024-05-27T07:17:06.759491+0000","last_became_peered":"2024-05-27T07:17:06.759491+0000","last_unstale":"2024-05-27T07:20:15.717457+0000","last_undegraded":"2024-05-27T07:20:15.717457+0000","last_fullsized":"2024-05-27T07:20:15.717457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:31.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082530+0000","last_change":"2024-05-27T07:17:21.898583+0000","last_active":"2024-05-27T07:18:47.082530+0000","last_peered":"2024-05-27T07:18:47.082530+0000","last_clean":"2024-05-27T07:18:47.082530+0000","last_became_active":"2024-05-27T07:17:21.897937+0000","last_became_peered":"2024-05-27T07:17:21.897937+0000","last_unstale":"2024-05-27T07:18:47.082530+0000","last_undegraded":"2024-05-27T07:18:47.082530+0000","last_fullsized":"2024-05-27T07:18:47.082530+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:45:00.412914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126886+0000","last_change":"2024-05-27T07:17:02.432465+0000","last_active":"2024-05-27T07:18:47.126886+0000","last_peered":"2024-05-27T07:18:47.126886+0000","last_clean":"2024-05-27T07:18:47.126886+0000","last_became_active":"2024-05-27T07:17:02.431638+0000","last_became_peered":"2024-05-27T07:17:02.431638+0000","last_unstale":"2024-05-27T07:18:47.126886+0000","last_undegraded":"2024-05-27T07:18:47.126886+0000","last_fullsized":"2024-05-27T07:18:47.126886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:03:13.167952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.730513+0000","last_change":"2024-05-27T07:17:06.763133+0000","last_active":"2024-05-27T07:20:15.730513+0000","last_peered":"2024-05-27T07:20:15.730513+0000","last_clean":"2024-05-27T07:20:15.730513+0000","last_became_active":"2024-05-27T07:17:06.762770+0000","last_became_peered":"2024-05-27T07:17:06.762770+0000","last_unstale":"2024-05-27T07:20:15.730513+0000","last_undegraded":"2024-05-27T07:20:15.730513+0000","last_fullsized":"2024-05-27T07:20:15.730513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:33.223722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083574+0000","last_change":"2024-05-27T07:17:21.900369+0000","last_active":"2024-05-27T07:18:47.083574+0000","last_peered":"2024-05-27T07:18:47.083574+0000","last_clean":"2024-05-27T07:18:47.083574+0000","last_became_active":"2024-05-27T07:17:21.899758+0000","last_became_peered":"2024-05-27T07:17:21.899758+0000","last_unstale":"2024-05-27T07:18:47.083574+0000","last_undegraded":"2024-05-27T07:18:47.083574+0000","last_fullsized":"2024-05-27T07:18:47.083574+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:42.116302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128098+0000","last_change":"2024-05-27T07:17:02.347047+0000","last_active":"2024-05-27T07:18:47.128098+0000","last_peered":"2024-05-27T07:18:47.128098+0000","last_clean":"2024-05-27T07:18:47.128098+0000","last_became_active":"2024-05-27T07:17:02.346562+0000","last_became_peered":"2024-05-27T07:17:02.346562+0000","last_unstale":"2024-05-27T07:18:47.128098+0000","last_undegraded":"2024-05-27T07:18:47.128098+0000","last_fullsized":"2024-05-27T07:18:47.128098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:25:50.133089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732803+0000","last_change":"2024-05-27T07:17:06.767930+0000","last_active":"2024-05-27T07:20:15.732803+0000","last_peered":"2024-05-27T07:20:15.732803+0000","last_clean":"2024-05-27T07:20:15.732803+0000","last_became_active":"2024-05-27T07:17:06.767439+0000","last_became_peered":"2024-05-27T07:17:06.767439+0000","last_unstale":"2024-05-27T07:20:15.732803+0000","last_undegraded":"2024-05-27T07:20:15.732803+0000","last_fullsized":"2024-05-27T07:20:15.732803+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:18:47.464602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.082887+0000","last_change":"2024-05-27T07:17:21.889199+0000","last_active":"2024-05-27T07:18:47.082887+0000","last_peered":"2024-05-27T07:18:47.082887+0000","last_clean":"2024-05-27T07:18:47.082887+0000","last_became_active":"2024-05-27T07:17:21.888667+0000","last_became_peered":"2024-05-27T07:17:21.888667+0000","last_unstale":"2024-05-27T07:18:47.082887+0000","last_undegraded":"2024-05-27T07:18:47.082887+0000","last_fullsized":"2024-05-27T07:18:47.082887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.029820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127096+0000","last_change":"2024-05-27T07:17:02.339872+0000","last_active":"2024-05-27T07:18:47.127096+0000","last_peered":"2024-05-27T07:18:47.127096+0000","last_clean":"2024-05-27T07:18:47.127096+0000","last_became_active":"2024-05-27T07:17:02.338975+0000","last_became_peered":"2024-05-27T07:17:02.338975+0000","last_unstale":"2024-05-27T07:18:47.127096+0000","last_undegraded":"2024-05-27T07:18:47.127096+0000","last_fullsized":"2024-05-27T07:18:47.127096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:14.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.723806+0000","last_change":"2024-05-27T07:17:06.758234+0000","last_active":"2024-05-27T07:20:15.723806+0000","last_peered":"2024-05-27T07:20:15.723806+0000","last_clean":"2024-05-27T07:20:15.723806+0000","last_became_active":"2024-05-27T07:17:06.757827+0000","last_became_peered":"2024-05-27T07:17:06.757827+0000","last_unstale":"2024-05-27T07:20:15.723806+0000","last_undegraded":"2024-05-27T07:20:15.723806+0000","last_fullsized":"2024-05-27T07:20:15.723806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:21.688435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064804+0000","last_change":"2024-05-27T07:17:21.882540+0000","last_active":"2024-05-27T07:18:47.064804+0000","last_peered":"2024-05-27T07:18:47.064804+0000","last_clean":"2024-05-27T07:18:47.064804+0000","last_became_active":"2024-05-27T07:17:21.882087+0000","last_became_peered":"2024-05-27T07:17:21.882087+0000","last_unstale":"2024-05-27T07:18:47.064804+0000","last_undegraded":"2024-05-27T07:18:47.064804+0000","last_fullsized":"2024-05-27T07:18:47.064804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:53:04.363521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'7","reported_seq":186,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:41.382053+0000","last_change":"2024-05-27T07:17:46.704997+0000","last_active":"2024-05-27T07:20:41.382053+0000","last_peered":"2024-05-27T07:20:41.382053+0000","last_clean":"2024-05-27T07:20:41.382053+0000","last_became_active":"2024-05-27T07:17:46.704355+0000","last_became_peered":"2024-05-27T07:17:46.704355+0000","last_unstale":"2024-05-27T07:20:41.382053+0000","last_undegraded":"2024-05-27T07:20:41.382053+0000","last_fullsized":"2024-05-27T07:20:41.382053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:43:02.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.705254+0000","last_change":"2024-05-27T07:17:06.765809+0000","last_active":"2024-05-27T07:20:15.705254+0000","last_peered":"2024-05-27T07:20:15.705254+0000","last_clean":"2024-05-27T07:20:15.705254+0000","last_became_active":"2024-05-27T07:17:06.763845+0000","last_became_peered":"2024-05-27T07:17:06.763845+0000","last_unstale":"2024-05-27T07:20:15.705254+0000","last_undegraded":"2024-05-27T07:20:15.705254+0000","last_fullsized":"2024-05-27T07:20:15.705254+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:25.399181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064484+0000","last_change":"2024-05-27T07:17:21.886316+0000","last_active":"2024-05-27T07:18:47.064484+0000","last_peered":"2024-05-27T07:18:47.064484+0000","last_clean":"2024-05-27T07:18:47.064484+0000","last_became_active":"2024-05-27T07:17:21.885129+0000","last_became_peered":"2024-05-27T07:17:21.885129+0000","last_unstale":"2024-05-27T07:18:47.064484+0000","last_undegraded":"2024-05-27T07:18:47.064484+0000","last_fullsized":"2024-05-27T07:18:47.064484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:10.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064424+0000","last_change":"2024-05-27T07:17:02.338765+0000","last_active":"2024-05-27T07:18:47.064424+0000","last_peered":"2024-05-27T07:18:47.064424+0000","last_clean":"2024-05-27T07:18:47.064424+0000","last_became_active":"2024-05-27T07:17:02.337103+0000","last_became_peered":"2024-05-27T07:17:02.337103+0000","last_unstale":"2024-05-27T07:18:47.064424+0000","last_undegraded":"2024-05-27T07:18:47.064424+0000","last_fullsized":"2024-05-27T07:18:47.064424+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:15:07.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.713789+0000","last_change":"2024-05-27T07:17:06.766839+0000","last_active":"2024-05-27T07:20:15.713789+0000","last_peered":"2024-05-27T07:20:15.713789+0000","last_clean":"2024-05-27T07:20:15.713789+0000","last_became_active":"2024-05-27T07:17:06.766373+0000","last_became_peered":"2024-05-27T07:17:06.766373+0000","last_unstale":"2024-05-27T07:20:15.713789+0000","last_undegraded":"2024-05-27T07:20:15.713789+0000","last_fullsized":"2024-05-27T07:20:15.713789+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:09:55.064045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.138290+0000","last_change":"2024-05-27T07:18:47.133692+0000","last_active":"2024-05-27T07:18:47.138290+0000","last_peered":"2024-05-27T07:18:47.138290+0000","last_clean":"2024-05-27T07:18:47.138290+0000","last_became_active":"2024-05-27T07:18:47.132122+0000","last_became_peered":"2024-05-27T07:18:47.132122+0000","last_unstale":"2024-05-27T07:18:47.138290+0000","last_undegraded":"2024-05-27T07:18:47.138290+0000","last_fullsized":"2024-05-27T07:18:47.138290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:57.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130263+0000","last_change":"2024-05-27T07:17:02.329809+0000","last_active":"2024-05-27T07:18:47.130263+0000","last_peered":"2024-05-27T07:18:47.130263+0000","last_clean":"2024-05-27T07:18:47.130263+0000","last_became_active":"2024-05-27T07:17:02.328758+0000","last_became_peered":"2024-05-27T07:17:02.328758+0000","last_unstale":"2024-05-27T07:18:47.130263+0000","last_undegraded":"2024-05-27T07:18:47.130263+0000","last_fullsized":"2024-05-27T07:18:47.130263+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:52.479444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716195+0000","last_change":"2024-05-27T07:17:06.749836+0000","last_active":"2024-05-27T07:20:15.716195+0000","last_peered":"2024-05-27T07:20:15.716195+0000","last_clean":"2024-05-27T07:20:15.716195+0000","last_became_active":"2024-05-27T07:17:06.749512+0000","last_became_peered":"2024-05-27T07:17:06.749512+0000","last_unstale":"2024-05-27T07:20:15.716195+0000","last_undegraded":"2024-05-27T07:20:15.716195+0000","last_fullsized":"2024-05-27T07:20:15.716195+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:41:51.000415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127857+0000","last_change":"2024-05-27T07:17:21.888376+0000","last_active":"2024-05-27T07:18:47.127857+0000","last_peered":"2024-05-27T07:18:47.127857+0000","last_clean":"2024-05-27T07:18:47.127857+0000","last_became_active":"2024-05-27T07:17:21.886183+0000","last_became_peered":"2024-05-27T07:17:21.886183+0000","last_unstale":"2024-05-27T07:18:47.127857+0000","last_undegraded":"2024-05-27T07:18:47.127857+0000","last_fullsized":"2024-05-27T07:18:47.127857+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:34.541338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070176+0000","last_change":"2024-05-27T07:17:02.334113+0000","last_active":"2024-05-27T07:18:47.070176+0000","last_peered":"2024-05-27T07:18:47.070176+0000","last_clean":"2024-05-27T07:18:47.070176+0000","last_became_active":"2024-05-27T07:17:02.333201+0000","last_became_peered":"2024-05-27T07:17:02.333201+0000","last_unstale":"2024-05-27T07:18:47.070176+0000","last_undegraded":"2024-05-27T07:18:47.070176+0000","last_fullsized":"2024-05-27T07:18:47.070176+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:13.484654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716170+0000","last_change":"2024-05-27T07:17:06.751676+0000","last_active":"2024-05-27T07:20:15.716170+0000","last_peered":"2024-05-27T07:20:15.716170+0000","last_clean":"2024-05-27T07:20:15.716170+0000","last_became_active":"2024-05-27T07:17:06.751221+0000","last_became_peered":"2024-05-27T07:17:06.751221+0000","last_unstale":"2024-05-27T07:20:15.716170+0000","last_undegraded":"2024-05-27T07:20:15.716170+0000","last_fullsized":"2024-05-27T07:20:15.716170+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:40.012561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125183+0000","last_change":"2024-05-27T07:17:21.897640+0000","last_active":"2024-05-27T07:18:47.125183+0000","last_peered":"2024-05-27T07:18:47.125183+0000","last_clean":"2024-05-27T07:18:47.125183+0000","last_became_active":"2024-05-27T07:17:21.895355+0000","last_became_peered":"2024-05-27T07:17:21.895355+0000","last_unstale":"2024-05-27T07:18:47.125183+0000","last_undegraded":"2024-05-27T07:18:47.125183+0000","last_fullsized":"2024-05-27T07:18:47.125183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:51:12.030069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:37.367376+0000","last_change":"2024-05-27T07:17:02.339464+0000","last_active":"2024-05-27T07:20:37.367376+0000","last_peered":"2024-05-27T07:20:37.367376+0000","last_clean":"2024-05-27T07:20:37.367376+0000","last_became_active":"2024-05-27T07:17:02.338577+0000","last_became_peered":"2024-05-27T07:17:02.338577+0000","last_unstale":"2024-05-27T07:20:37.367376+0000","last_undegraded":"2024-05-27T07:20:37.367376+0000","last_fullsized":"2024-05-27T07:20:37.367376+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:29:59.210157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.729317+0000","last_change":"2024-05-27T07:17:06.766883+0000","last_active":"2024-05-27T07:20:15.729317+0000","last_peered":"2024-05-27T07:20:15.729317+0000","last_clean":"2024-05-27T07:20:15.729317+0000","last_became_active":"2024-05-27T07:17:06.766166+0000","last_became_peered":"2024-05-27T07:17:06.766166+0000","last_unstale":"2024-05-27T07:20:15.729317+0000","last_undegraded":"2024-05-27T07:20:15.729317+0000","last_fullsized":"2024-05-27T07:20:15.729317+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:28.936499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.058915+0000","last_change":"2024-05-27T07:17:21.893930+0000","last_active":"2024-05-27T07:18:47.058915+0000","last_peered":"2024-05-27T07:18:47.058915+0000","last_clean":"2024-05-27T07:18:47.058915+0000","last_became_active":"2024-05-27T07:17:21.893029+0000","last_became_peered":"2024-05-27T07:17:21.893029+0000","last_unstale":"2024-05-27T07:18:47.058915+0000","last_undegraded":"2024-05-27T07:18:47.058915+0000","last_fullsized":"2024-05-27T07:18:47.058915+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:32.453078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068465+0000","last_change":"2024-05-27T07:17:02.346531+0000","last_active":"2024-05-27T07:18:47.068465+0000","last_peered":"2024-05-27T07:18:47.068465+0000","last_clean":"2024-05-27T07:18:47.068465+0000","last_became_active":"2024-05-27T07:17:02.345844+0000","last_became_peered":"2024-05-27T07:17:02.345844+0000","last_unstale":"2024-05-27T07:18:47.068465+0000","last_undegraded":"2024-05-27T07:18:47.068465+0000","last_fullsized":"2024-05-27T07:18:47.068465+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:28.544616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.712857+0000","last_change":"2024-05-27T07:17:06.769845+0000","last_active":"2024-05-27T07:20:15.712857+0000","last_peered":"2024-05-27T07:20:15.712857+0000","last_clean":"2024-05-27T07:20:15.712857+0000","last_became_active":"2024-05-27T07:17:06.768467+0000","last_became_peered":"2024-05-27T07:17:06.768467+0000","last_unstale":"2024-05-27T07:20:15.712857+0000","last_undegraded":"2024-05-27T07:20:15.712857+0000","last_fullsized":"2024-05-27T07:20:15.712857+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:30.311824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084110+0000","last_change":"2024-05-27T07:17:21.899885+0000","last_active":"2024-05-27T07:18:47.084110+0000","last_peered":"2024-05-27T07:18:47.084110+0000","last_clean":"2024-05-27T07:18:47.084110+0000","last_became_active":"2024-05-27T07:17:21.899340+0000","last_became_peered":"2024-05-27T07:17:21.899340+0000","last_unstale":"2024-05-27T07:18:47.084110+0000","last_undegraded":"2024-05-27T07:18:47.084110+0000","last_fullsized":"2024-05-27T07:18:47.084110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:12.665409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.070793+0000","last_change":"2024-05-27T07:17:02.435770+0000","last_active":"2024-05-27T07:18:47.070793+0000","last_peered":"2024-05-27T07:18:47.070793+0000","last_clean":"2024-05-27T07:18:47.070793+0000","last_became_active":"2024-05-27T07:17:02.434535+0000","last_became_peered":"2024-05-27T07:17:02.434535+0000","last_unstale":"2024-05-27T07:18:47.070793+0000","last_undegraded":"2024-05-27T07:18:47.070793+0000","last_fullsized":"2024-05-27T07:18:47.070793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:53.745583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.733492+0000","last_change":"2024-05-27T07:17:46.704100+0000","last_active":"2024-05-27T07:20:15.733492+0000","last_peered":"2024-05-27T07:20:15.733492+0000","last_clean":"2024-05-27T07:20:15.733492+0000","last_became_active":"2024-05-27T07:17:46.703475+0000","last_became_peered":"2024-05-27T07:17:46.703475+0000","last_unstale":"2024-05-27T07:20:15.733492+0000","last_undegraded":"2024-05-27T07:20:15.733492+0000","last_fullsized":"2024-05-27T07:20:15.733492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:20:29.173923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.133397+0000","last_change":"2024-05-27T07:18:47.131521+0000","last_active":"2024-05-27T07:18:47.133397+0000","last_peered":"2024-05-27T07:18:47.133397+0000","last_clean":"2024-05-27T07:18:47.133397+0000","last_became_active":"2024-05-27T07:18:47.129995+0000","last_became_peered":"2024-05-27T07:18:47.129995+0000","last_unstale":"2024-05-27T07:18:47.133397+0000","last_undegraded":"2024-05-27T07:18:47.133397+0000","last_fullsized":"2024-05-27T07:18:47.133397+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:45:27.268768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'36","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:38.367751+0000","last_change":"2024-05-27T07:17:46.719293+0000","last_active":"2024-05-27T07:20:38.367751+0000","last_peered":"2024-05-27T07:20:38.367751+0000","last_clean":"2024-05-27T07:20:38.367751+0000","last_became_active":"2024-05-27T07:17:46.703496+0000","last_became_peered":"2024-05-27T07:17:46.703496+0000","last_unstale":"2024-05-27T07:20:38.367751+0000","last_undegraded":"2024-05-27T07:20:38.367751+0000","last_fullsized":"2024-05-27T07:20:38.367751+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:26.310524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62321,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":250,"num_read_kb":186,"num_write":37,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718304+0000","last_change":"2024-05-27T07:17:06.763141+0000","last_active":"2024-05-27T07:20:15.718304+0000","last_peered":"2024-05-27T07:20:15.718304+0000","last_clean":"2024-05-27T07:20:15.718304+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:20:15.718304+0000","last_undegraded":"2024-05-27T07:20:15.718304+0000","last_fullsized":"2024-05-27T07:20:15.718304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:47.072552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071470+0000","last_change":"2024-05-27T07:17:21.895342+0000","last_active":"2024-05-27T07:18:47.071470+0000","last_peered":"2024-05-27T07:18:47.071470+0000","last_clean":"2024-05-27T07:18:47.071470+0000","last_became_active":"2024-05-27T07:17:21.894630+0000","last_became_peered":"2024-05-27T07:17:21.894630+0000","last_unstale":"2024-05-27T07:18:47.071470+0000","last_undegraded":"2024-05-27T07:18:47.071470+0000","last_fullsized":"2024-05-27T07:18:47.071470+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:50.501683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062929+0000","last_change":"2024-05-27T07:17:02.341848+0000","last_active":"2024-05-27T07:18:47.062929+0000","last_peered":"2024-05-27T07:18:47.062929+0000","last_clean":"2024-05-27T07:18:47.062929+0000","last_became_active":"2024-05-27T07:17:02.339477+0000","last_became_peered":"2024-05-27T07:17:02.339477+0000","last_unstale":"2024-05-27T07:18:47.062929+0000","last_undegraded":"2024-05-27T07:18:47.062929+0000","last_fullsized":"2024-05-27T07:18:47.062929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:23.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718010+0000","last_change":"2024-05-27T07:17:06.761340+0000","last_active":"2024-05-27T07:20:15.718010+0000","last_peered":"2024-05-27T07:20:15.718010+0000","last_clean":"2024-05-27T07:20:15.718010+0000","last_became_active":"2024-05-27T07:17:06.759103+0000","last_became_peered":"2024-05-27T07:17:06.759103+0000","last_unstale":"2024-05-27T07:20:15.718010+0000","last_undegraded":"2024-05-27T07:20:15.718010+0000","last_fullsized":"2024-05-27T07:20:15.718010+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:42.263106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071473+0000","last_change":"2024-05-27T07:17:21.883252+0000","last_active":"2024-05-27T07:18:47.071473+0000","last_peered":"2024-05-27T07:18:47.071473+0000","last_clean":"2024-05-27T07:18:47.071473+0000","last_became_active":"2024-05-27T07:17:21.882468+0000","last_became_peered":"2024-05-27T07:17:21.882468+0000","last_unstale":"2024-05-27T07:18:47.071473+0000","last_undegraded":"2024-05-27T07:18:47.071473+0000","last_fullsized":"2024-05-27T07:18:47.071473+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:59.250357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.069762+0000","last_change":"2024-05-27T07:17:02.343343+0000","last_active":"2024-05-27T07:18:47.069762+0000","last_peered":"2024-05-27T07:18:47.069762+0000","last_clean":"2024-05-27T07:18:47.069762+0000","last_became_active":"2024-05-27T07:17:02.342741+0000","last_became_peered":"2024-05-27T07:17:02.342741+0000","last_unstale":"2024-05-27T07:18:47.069762+0000","last_undegraded":"2024-05-27T07:18:47.069762+0000","last_fullsized":"2024-05-27T07:18:47.069762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:46.006865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.725298+0000","last_change":"2024-05-27T07:17:06.771473+0000","last_active":"2024-05-27T07:20:15.725298+0000","last_peered":"2024-05-27T07:20:15.725298+0000","last_clean":"2024-05-27T07:20:15.725298+0000","last_became_active":"2024-05-27T07:17:06.771082+0000","last_became_peered":"2024-05-27T07:17:06.771082+0000","last_unstale":"2024-05-27T07:20:15.725298+0000","last_undegraded":"2024-05-27T07:20:15.725298+0000","last_fullsized":"2024-05-27T07:20:15.725298+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:01.371182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060660+0000","last_change":"2024-05-27T07:17:21.897691+0000","last_active":"2024-05-27T07:18:47.060660+0000","last_peered":"2024-05-27T07:18:47.060660+0000","last_clean":"2024-05-27T07:18:47.060660+0000","last_became_active":"2024-05-27T07:17:21.895221+0000","last_became_peered":"2024-05-27T07:17:21.895221+0000","last_unstale":"2024-05-27T07:18:47.060660+0000","last_undegraded":"2024-05-27T07:18:47.060660+0000","last_fullsized":"2024-05-27T07:18:47.060660+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:14.626947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":179,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:42.387180+0000","last_change":"2024-05-27T07:17:02.343375+0000","last_active":"2024-05-27T07:20:42.387180+0000","last_peered":"2024-05-27T07:20:42.387180+0000","last_clean":"2024-05-27T07:20:42.387180+0000","last_became_active":"2024-05-27T07:17:02.341315+0000","last_became_peered":"2024-05-27T07:17:02.341315+0000","last_unstale":"2024-05-27T07:20:42.387180+0000","last_undegraded":"2024-05-27T07:20:42.387180+0000","last_fullsized":"2024-05-27T07:20:42.387180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:41.927265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.719109+0000","last_change":"2024-05-27T07:17:06.765862+0000","last_active":"2024-05-27T07:20:15.719109+0000","last_peered":"2024-05-27T07:20:15.719109+0000","last_clean":"2024-05-27T07:20:15.719109+0000","last_became_active":"2024-05-27T07:17:06.765439+0000","last_became_peered":"2024-05-27T07:17:06.765439+0000","last_unstale":"2024-05-27T07:20:15.719109+0000","last_undegraded":"2024-05-27T07:20:15.719109+0000","last_fullsized":"2024-05-27T07:20:15.719109+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:47.586833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130000+0000","last_change":"2024-05-27T07:17:21.891810+0000","last_active":"2024-05-27T07:18:47.130000+0000","last_peered":"2024-05-27T07:18:47.130000+0000","last_clean":"2024-05-27T07:18:47.130000+0000","last_became_active":"2024-05-27T07:17:21.890473+0000","last_became_peered":"2024-05-27T07:17:21.890473+0000","last_unstale":"2024-05-27T07:18:47.130000+0000","last_undegraded":"2024-05-27T07:18:47.130000+0000","last_fullsized":"2024-05-27T07:18:47.130000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:12.731189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065951+0000","last_change":"2024-05-27T07:17:46.715870+0000","last_active":"2024-05-27T07:18:47.065951+0000","last_peered":"2024-05-27T07:18:47.065951+0000","last_clean":"2024-05-27T07:18:47.065951+0000","last_became_active":"2024-05-27T07:17:46.701248+0000","last_became_peered":"2024-05-27T07:17:46.701248+0000","last_unstale":"2024-05-27T07:18:47.065951+0000","last_undegraded":"2024-05-27T07:18:47.065951+0000","last_fullsized":"2024-05-27T07:18:47.065951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:18.693752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715188+0000","last_change":"2024-05-27T07:17:06.759937+0000","last_active":"2024-05-27T07:20:15.715188+0000","last_peered":"2024-05-27T07:20:15.715188+0000","last_clean":"2024-05-27T07:20:15.715188+0000","last_became_active":"2024-05-27T07:17:06.759526+0000","last_became_peered":"2024-05-27T07:17:06.759526+0000","last_unstale":"2024-05-27T07:20:15.715188+0000","last_undegraded":"2024-05-27T07:20:15.715188+0000","last_fullsized":"2024-05-27T07:20:15.715188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:22.934317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068886+0000","last_change":"2024-05-27T07:17:21.898331+0000","last_active":"2024-05-27T07:18:47.068886+0000","last_peered":"2024-05-27T07:18:47.068886+0000","last_clean":"2024-05-27T07:18:47.068886+0000","last_became_active":"2024-05-27T07:17:21.897965+0000","last_became_peered":"2024-05-27T07:17:21.897965+0000","last_unstale":"2024-05-27T07:18:47.068886+0000","last_undegraded":"2024-05-27T07:18:47.068886+0000","last_fullsized":"2024-05-27T07:18:47.068886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:20.584175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128389+0000","last_change":"2024-05-27T07:17:02.346029+0000","last_active":"2024-05-27T07:18:47.128389+0000","last_peered":"2024-05-27T07:18:47.128389+0000","last_clean":"2024-05-27T07:18:47.128389+0000","last_became_active":"2024-05-27T07:17:02.341292+0000","last_became_peered":"2024-05-27T07:17:02.341292+0000","last_unstale":"2024-05-27T07:18:47.128389+0000","last_undegraded":"2024-05-27T07:18:47.128389+0000","last_fullsized":"2024-05-27T07:18:47.128389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:31.705741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.712879+0000","last_change":"2024-05-27T07:17:46.691785+0000","last_active":"2024-05-27T07:20:15.712879+0000","last_peered":"2024-05-27T07:20:15.712879+0000","last_clean":"2024-05-27T07:20:15.712879+0000","last_became_active":"2024-05-27T07:17:46.690445+0000","last_became_peered":"2024-05-27T07:17:46.690445+0000","last_unstale":"2024-05-27T07:20:15.712879+0000","last_undegraded":"2024-05-27T07:20:15.712879+0000","last_fullsized":"2024-05-27T07:20:15.712879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:59:27.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126890+0000","last_change":"2024-05-27T07:17:21.892434+0000","last_active":"2024-05-27T07:18:47.126890+0000","last_peered":"2024-05-27T07:18:47.126890+0000","last_clean":"2024-05-27T07:18:47.126890+0000","last_became_active":"2024-05-27T07:17:21.891736+0000","last_became_peered":"2024-05-27T07:17:21.891736+0000","last_unstale":"2024-05-27T07:18:47.126890+0000","last_undegraded":"2024-05-27T07:18:47.126890+0000","last_fullsized":"2024-05-27T07:18:47.126890+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:48.458819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.059903+0000","last_change":"2024-05-27T07:17:02.436323+0000","last_active":"2024-05-27T07:18:47.059903+0000","last_peered":"2024-05-27T07:18:47.059903+0000","last_clean":"2024-05-27T07:18:47.059903+0000","last_became_active":"2024-05-27T07:17:02.435000+0000","last_became_peered":"2024-05-27T07:17:02.435000+0000","last_unstale":"2024-05-27T07:18:47.059903+0000","last_undegraded":"2024-05-27T07:18:47.059903+0000","last_fullsized":"2024-05-27T07:18:47.059903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:08.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716763+0000","last_change":"2024-05-27T07:17:06.760418+0000","last_active":"2024-05-27T07:20:15.716763+0000","last_peered":"2024-05-27T07:20:15.716763+0000","last_clean":"2024-05-27T07:20:15.716763+0000","last_became_active":"2024-05-27T07:17:06.759875+0000","last_became_peered":"2024-05-27T07:17:06.759875+0000","last_unstale":"2024-05-27T07:20:15.716763+0000","last_undegraded":"2024-05-27T07:20:15.716763+0000","last_fullsized":"2024-05-27T07:20:15.716763+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:12:54.324323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130559+0000","last_change":"2024-05-27T07:17:21.888048+0000","last_active":"2024-05-27T07:18:47.130559+0000","last_peered":"2024-05-27T07:18:47.130559+0000","last_clean":"2024-05-27T07:18:47.130559+0000","last_became_active":"2024-05-27T07:17:21.887579+0000","last_became_peered":"2024-05-27T07:17:21.887579+0000","last_unstale":"2024-05-27T07:18:47.130559+0000","last_undegraded":"2024-05-27T07:18:47.130559+0000","last_fullsized":"2024-05-27T07:18:47.130559+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:34:44.569959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.066518+0000","last_change":"2024-05-27T07:17:02.337655+0000","last_active":"2024-05-27T07:18:47.066518+0000","last_peered":"2024-05-27T07:18:47.066518+0000","last_clean":"2024-05-27T07:18:47.066518+0000","last_became_active":"2024-05-27T07:17:02.337252+0000","last_became_peered":"2024-05-27T07:17:02.337252+0000","last_unstale":"2024-05-27T07:18:47.066518+0000","last_undegraded":"2024-05-27T07:18:47.066518+0000","last_fullsized":"2024-05-27T07:18:47.066518+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:11:25.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.703510+0000","last_change":"2024-05-27T07:17:06.762934+0000","last_active":"2024-05-27T07:20:15.703510+0000","last_peered":"2024-05-27T07:20:15.703510+0000","last_clean":"2024-05-27T07:20:15.703510+0000","last_became_active":"2024-05-27T07:17:06.762507+0000","last_became_peered":"2024-05-27T07:17:06.762507+0000","last_unstale":"2024-05-27T07:20:15.703510+0000","last_undegraded":"2024-05-27T07:20:15.703510+0000","last_fullsized":"2024-05-27T07:20:15.703510+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:23:35.164892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.129953+0000","last_change":"2024-05-27T07:17:21.886991+0000","last_active":"2024-05-27T07:18:47.129953+0000","last_peered":"2024-05-27T07:18:47.129953+0000","last_clean":"2024-05-27T07:18:47.129953+0000","last_became_active":"2024-05-27T07:17:21.885945+0000","last_became_peered":"2024-05-27T07:17:21.885945+0000","last_unstale":"2024-05-27T07:18:47.129953+0000","last_undegraded":"2024-05-27T07:18:47.129953+0000","last_fullsized":"2024-05-27T07:18:47.129953+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:23:01.853601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717378+0000","last_change":"2024-05-27T07:17:46.864130+0000","last_active":"2024-05-27T07:20:15.717378+0000","last_peered":"2024-05-27T07:20:15.717378+0000","last_clean":"2024-05-27T07:20:15.717378+0000","last_became_active":"2024-05-27T07:17:46.862997+0000","last_became_peered":"2024-05-27T07:17:46.862997+0000","last_unstale":"2024-05-27T07:20:15.717378+0000","last_undegraded":"2024-05-27T07:20:15.717378+0000","last_fullsized":"2024-05-27T07:20:15.717378+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:10.594627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":174,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:41.382799+0000","last_change":"2024-05-27T07:17:02.435463+0000","last_active":"2024-05-27T07:20:41.382799+0000","last_peered":"2024-05-27T07:20:41.382799+0000","last_clean":"2024-05-27T07:20:41.382799+0000","last_became_active":"2024-05-27T07:17:02.432131+0000","last_became_peered":"2024-05-27T07:17:02.432131+0000","last_unstale":"2024-05-27T07:20:41.382799+0000","last_undegraded":"2024-05-27T07:20:41.382799+0000","last_fullsized":"2024-05-27T07:20:41.382799+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:41.874577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:51.173771+0000","last_change":"2024-05-27T07:18:47.134043+0000","last_active":"2024-05-27T07:18:51.173771+0000","last_peered":"2024-05-27T07:18:51.173771+0000","last_clean":"2024-05-27T07:18:51.173771+0000","last_became_active":"2024-05-27T07:18:47.132050+0000","last_became_peered":"2024-05-27T07:18:47.132050+0000","last_unstale":"2024-05-27T07:18:51.173771+0000","last_undegraded":"2024-05-27T07:18:51.173771+0000","last_fullsized":"2024-05-27T07:18:51.173771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:12.170883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718922+0000","last_change":"2024-05-27T07:17:06.751168+0000","last_active":"2024-05-27T07:20:15.718922+0000","last_peered":"2024-05-27T07:20:15.718922+0000","last_clean":"2024-05-27T07:20:15.718922+0000","last_became_active":"2024-05-27T07:17:06.750816+0000","last_became_peered":"2024-05-27T07:17:06.750816+0000","last_unstale":"2024-05-27T07:20:15.718922+0000","last_undegraded":"2024-05-27T07:20:15.718922+0000","last_fullsized":"2024-05-27T07:20:15.718922+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:33:45.607426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.083905+0000","last_change":"2024-05-27T07:17:02.337490+0000","last_active":"2024-05-27T07:18:47.083905+0000","last_peered":"2024-05-27T07:18:47.083905+0000","last_clean":"2024-05-27T07:18:47.083905+0000","last_became_active":"2024-05-27T07:17:02.335317+0000","last_became_peered":"2024-05-27T07:17:02.335317+0000","last_unstale":"2024-05-27T07:18:47.083905+0000","last_undegraded":"2024-05-27T07:18:47.083905+0000","last_fullsized":"2024-05-27T07:18:47.083905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:33:46.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":120,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065144+0000","last_change":"2024-05-27T07:15:10.825312+0000","last_active":"2024-05-27T07:18:47.065144+0000","last_peered":"2024-05-27T07:18:47.065144+0000","last_clean":"2024-05-27T07:18:47.065144+0000","last_became_active":"2024-05-27T07:15:10.809245+0000","last_became_peered":"2024-05-27T07:15:10.809245+0000","last_unstale":"2024-05-27T07:18:47.065144+0000","last_undegraded":"2024-05-27T07:18:47.065144+0000","last_fullsized":"2024-05-27T07:18:47.065144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_clean_scrub_stamp":"2024-05-27T07:10:08.008565+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:32.769372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.137659+0000","last_change":"2024-05-27T07:18:47.133948+0000","last_active":"2024-05-27T07:18:47.137659+0000","last_peered":"2024-05-27T07:18:47.137659+0000","last_clean":"2024-05-27T07:18:47.137659+0000","last_became_active":"2024-05-27T07:18:47.132566+0000","last_became_peered":"2024-05-27T07:18:47.132566+0000","last_unstale":"2024-05-27T07:18:47.137659+0000","last_undegraded":"2024-05-27T07:18:47.137659+0000","last_fullsized":"2024-05-27T07:18:47.137659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:04.463121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715743+0000","last_change":"2024-05-27T07:17:06.766187+0000","last_active":"2024-05-27T07:20:15.715743+0000","last_peered":"2024-05-27T07:20:15.715743+0000","last_clean":"2024-05-27T07:20:15.715743+0000","last_became_active":"2024-05-27T07:17:06.763365+0000","last_became_peered":"2024-05-27T07:17:06.763365+0000","last_unstale":"2024-05-27T07:20:15.715743+0000","last_undegraded":"2024-05-27T07:20:15.715743+0000","last_fullsized":"2024-05-27T07:20:15.715743+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:13.668426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.065681+0000","last_change":"2024-05-27T07:17:02.344452+0000","last_active":"2024-05-27T07:18:47.065681+0000","last_peered":"2024-05-27T07:18:47.065681+0000","last_clean":"2024-05-27T07:18:47.065681+0000","last_became_active":"2024-05-27T07:17:02.343885+0000","last_became_peered":"2024-05-27T07:17:02.343885+0000","last_unstale":"2024-05-27T07:18:47.065681+0000","last_undegraded":"2024-05-27T07:18:47.065681+0000","last_fullsized":"2024-05-27T07:18:47.065681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:26.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.072673+0000","last_change":"2024-05-27T07:17:21.883024+0000","last_active":"2024-05-27T07:18:47.072673+0000","last_peered":"2024-05-27T07:18:47.072673+0000","last_clean":"2024-05-27T07:18:47.072673+0000","last_became_active":"2024-05-27T07:17:21.882429+0000","last_became_peered":"2024-05-27T07:17:21.882429+0000","last_unstale":"2024-05-27T07:18:47.072673+0000","last_undegraded":"2024-05-27T07:18:47.072673+0000","last_fullsized":"2024-05-27T07:18:47.072673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:36:40.443537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.128649+0000","last_change":"2024-05-27T07:17:02.432768+0000","last_active":"2024-05-27T07:18:47.128649+0000","last_peered":"2024-05-27T07:18:47.128649+0000","last_clean":"2024-05-27T07:18:47.128649+0000","last_became_active":"2024-05-27T07:17:02.431551+0000","last_became_peered":"2024-05-27T07:17:02.431551+0000","last_unstale":"2024-05-27T07:18:47.128649+0000","last_undegraded":"2024-05-27T07:18:47.128649+0000","last_fullsized":"2024-05-27T07:18:47.128649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:39.759512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717103+0000","last_change":"2024-05-27T07:17:06.770992+0000","last_active":"2024-05-27T07:20:15.717103+0000","last_peered":"2024-05-27T07:20:15.717103+0000","last_clean":"2024-05-27T07:20:15.717103+0000","last_became_active":"2024-05-27T07:17:06.770560+0000","last_became_peered":"2024-05-27T07:17:06.770560+0000","last_unstale":"2024-05-27T07:20:15.717103+0000","last_undegraded":"2024-05-27T07:20:15.717103+0000","last_fullsized":"2024-05-27T07:20:15.717103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:07:34.443551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.124998+0000","last_change":"2024-05-27T07:17:21.895362+0000","last_active":"2024-05-27T07:18:47.124998+0000","last_peered":"2024-05-27T07:18:47.124998+0000","last_clean":"2024-05-27T07:18:47.124998+0000","last_became_active":"2024-05-27T07:17:21.894563+0000","last_became_peered":"2024-05-27T07:17:21.894563+0000","last_unstale":"2024-05-27T07:18:47.124998+0000","last_undegraded":"2024-05-27T07:18:47.124998+0000","last_fullsized":"2024-05-27T07:18:47.124998+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:44.394301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'13","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:19:15.306628+0000","last_change":"2024-05-27T07:17:02.338203+0000","last_active":"2024-05-27T07:19:15.306628+0000","last_peered":"2024-05-27T07:19:15.306628+0000","last_clean":"2024-05-27T07:19:15.306628+0000","last_became_active":"2024-05-27T07:17:02.336192+0000","last_became_peered":"2024-05-27T07:17:02.336192+0000","last_unstale":"2024-05-27T07:19:15.306628+0000","last_undegraded":"2024-05-27T07:19:15.306628+0000","last_fullsized":"2024-05-27T07:19:15.306628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:48.187517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.718283+0000","last_change":"2024-05-27T07:17:06.759729+0000","last_active":"2024-05-27T07:20:15.718283+0000","last_peered":"2024-05-27T07:20:15.718283+0000","last_clean":"2024-05-27T07:20:15.718283+0000","last_became_active":"2024-05-27T07:17:06.756766+0000","last_became_peered":"2024-05-27T07:17:06.756766+0000","last_unstale":"2024-05-27T07:20:15.718283+0000","last_undegraded":"2024-05-27T07:20:15.718283+0000","last_fullsized":"2024-05-27T07:20:15.718283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:01.090586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.063120+0000","last_change":"2024-05-27T07:17:21.878487+0000","last_active":"2024-05-27T07:18:47.063120+0000","last_peered":"2024-05-27T07:18:47.063120+0000","last_clean":"2024-05-27T07:18:47.063120+0000","last_became_active":"2024-05-27T07:17:21.877623+0000","last_became_peered":"2024-05-27T07:17:21.877623+0000","last_unstale":"2024-05-27T07:18:47.063120+0000","last_undegraded":"2024-05-27T07:18:47.063120+0000","last_fullsized":"2024-05-27T07:18:47.063120+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:28:52.618738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.085169+0000","last_change":"2024-05-27T07:17:02.486495+0000","last_active":"2024-05-27T07:18:47.085169+0000","last_peered":"2024-05-27T07:18:47.085169+0000","last_clean":"2024-05-27T07:18:47.085169+0000","last_became_active":"2024-05-27T07:17:02.485493+0000","last_became_peered":"2024-05-27T07:17:02.485493+0000","last_unstale":"2024-05-27T07:18:47.085169+0000","last_undegraded":"2024-05-27T07:18:47.085169+0000","last_fullsized":"2024-05-27T07:18:47.085169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:31.977210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714041+0000","last_change":"2024-05-27T07:17:06.760937+0000","last_active":"2024-05-27T07:20:15.714041+0000","last_peered":"2024-05-27T07:20:15.714041+0000","last_clean":"2024-05-27T07:20:15.714041+0000","last_became_active":"2024-05-27T07:17:06.760254+0000","last_became_peered":"2024-05-27T07:17:06.760254+0000","last_unstale":"2024-05-27T07:20:15.714041+0000","last_undegraded":"2024-05-27T07:20:15.714041+0000","last_fullsized":"2024-05-27T07:20:15.714041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:50.147319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084721+0000","last_change":"2024-05-27T07:17:21.901511+0000","last_active":"2024-05-27T07:18:47.084721+0000","last_peered":"2024-05-27T07:18:47.084721+0000","last_clean":"2024-05-27T07:18:47.084721+0000","last_became_active":"2024-05-27T07:17:21.899847+0000","last_became_peered":"2024-05-27T07:17:21.899847+0000","last_unstale":"2024-05-27T07:18:47.084721+0000","last_undegraded":"2024-05-27T07:18:47.084721+0000","last_fullsized":"2024-05-27T07:18:47.084721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:28.133509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080859+0000","last_change":"2024-05-27T07:17:02.432520+0000","last_active":"2024-05-27T07:18:47.080859+0000","last_peered":"2024-05-27T07:18:47.080859+0000","last_clean":"2024-05-27T07:18:47.080859+0000","last_became_active":"2024-05-27T07:17:02.431563+0000","last_became_peered":"2024-05-27T07:17:02.431563+0000","last_unstale":"2024-05-27T07:18:47.080859+0000","last_undegraded":"2024-05-27T07:18:47.080859+0000","last_fullsized":"2024-05-27T07:18:47.080859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:17:28.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.660913+0000","last_change":"2024-05-27T07:17:06.754408+0000","last_active":"2024-05-27T07:20:15.660913+0000","last_peered":"2024-05-27T07:20:15.660913+0000","last_clean":"2024-05-27T07:20:15.660913+0000","last_became_active":"2024-05-27T07:17:06.753787+0000","last_became_peered":"2024-05-27T07:17:06.753787+0000","last_unstale":"2024-05-27T07:20:15.660913+0000","last_undegraded":"2024-05-27T07:20:15.660913+0000","last_fullsized":"2024-05-27T07:20:15.660913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:21:50.461296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.060453+0000","last_change":"2024-05-27T07:17:21.895277+0000","last_active":"2024-05-27T07:18:47.060453+0000","last_peered":"2024-05-27T07:18:47.060453+0000","last_clean":"2024-05-27T07:18:47.060453+0000","last_became_active":"2024-05-27T07:17:21.894518+0000","last_became_peered":"2024-05-27T07:17:21.894518+0000","last_unstale":"2024-05-27T07:18:47.060453+0000","last_undegraded":"2024-05-27T07:18:47.060453+0000","last_fullsized":"2024-05-27T07:18:47.060453+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:02.008209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.125540+0000","last_change":"2024-05-27T07:17:02.343604+0000","last_active":"2024-05-27T07:18:47.125540+0000","last_peered":"2024-05-27T07:18:47.125540+0000","last_clean":"2024-05-27T07:18:47.125540+0000","last_became_active":"2024-05-27T07:17:02.341890+0000","last_became_peered":"2024-05-27T07:17:02.341890+0000","last_unstale":"2024-05-27T07:18:47.125540+0000","last_undegraded":"2024-05-27T07:18:47.125540+0000","last_fullsized":"2024-05-27T07:18:47.125540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:10:21.670193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.708722+0000","last_change":"2024-05-27T07:17:06.754767+0000","last_active":"2024-05-27T07:20:15.708722+0000","last_peered":"2024-05-27T07:20:15.708722+0000","last_clean":"2024-05-27T07:20:15.708722+0000","last_became_active":"2024-05-27T07:17:06.754291+0000","last_became_peered":"2024-05-27T07:17:06.754291+0000","last_unstale":"2024-05-27T07:20:15.708722+0000","last_undegraded":"2024-05-27T07:20:15.708722+0000","last_fullsized":"2024-05-27T07:20:15.708722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:46:48.145874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:59.283159+0000","last_change":"2024-05-27T07:17:21.892342+0000","last_active":"2024-05-27T07:18:59.283159+0000","last_peered":"2024-05-27T07:18:59.283159+0000","last_clean":"2024-05-27T07:18:59.283159+0000","last_became_active":"2024-05-27T07:17:21.891393+0000","last_became_peered":"2024-05-27T07:17:21.891393+0000","last_unstale":"2024-05-27T07:18:59.283159+0000","last_undegraded":"2024-05-27T07:18:59.283159+0000","last_fullsized":"2024-05-27T07:18:59.283159+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:30.510616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.062370+0000","last_change":"2024-05-27T07:17:02.340238+0000","last_active":"2024-05-27T07:18:47.062370+0000","last_peered":"2024-05-27T07:18:47.062370+0000","last_clean":"2024-05-27T07:18:47.062370+0000","last_became_active":"2024-05-27T07:17:02.339561+0000","last_became_peered":"2024-05-27T07:17:02.339561+0000","last_unstale":"2024-05-27T07:18:47.062370+0000","last_undegraded":"2024-05-27T07:18:47.062370+0000","last_fullsized":"2024-05-27T07:18:47.062370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:32.451397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715306+0000","last_change":"2024-05-27T07:17:06.769397+0000","last_active":"2024-05-27T07:20:15.715306+0000","last_peered":"2024-05-27T07:20:15.715306+0000","last_clean":"2024-05-27T07:20:15.715306+0000","last_became_active":"2024-05-27T07:17:06.768936+0000","last_became_peered":"2024-05-27T07:17:06.768936+0000","last_unstale":"2024-05-27T07:20:15.715306+0000","last_undegraded":"2024-05-27T07:20:15.715306+0000","last_fullsized":"2024-05-27T07:20:15.715306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:52.767902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.084477+0000","last_change":"2024-05-27T07:17:21.894225+0000","last_active":"2024-05-27T07:18:47.084477+0000","last_peered":"2024-05-27T07:18:47.084477+0000","last_clean":"2024-05-27T07:18:47.084477+0000","last_became_active":"2024-05-27T07:17:21.893499+0000","last_became_peered":"2024-05-27T07:17:21.893499+0000","last_unstale":"2024-05-27T07:18:47.084477+0000","last_undegraded":"2024-05-27T07:18:47.084477+0000","last_fullsized":"2024-05-27T07:18:47.084477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:54.985290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.126297+0000","last_change":"2024-05-27T07:17:02.347736+0000","last_active":"2024-05-27T07:18:47.126297+0000","last_peered":"2024-05-27T07:18:47.126297+0000","last_clean":"2024-05-27T07:18:47.126297+0000","last_became_active":"2024-05-27T07:17:02.347237+0000","last_became_peered":"2024-05-27T07:17:02.347237+0000","last_unstale":"2024-05-27T07:18:47.126297+0000","last_undegraded":"2024-05-27T07:18:47.126297+0000","last_fullsized":"2024-05-27T07:18:47.126297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:36:15.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732204+0000","last_change":"2024-05-27T07:17:06.759348+0000","last_active":"2024-05-27T07:20:15.732204+0000","last_peered":"2024-05-27T07:20:15.732204+0000","last_clean":"2024-05-27T07:20:15.732204+0000","last_became_active":"2024-05-27T07:17:06.758967+0000","last_became_peered":"2024-05-27T07:17:06.758967+0000","last_unstale":"2024-05-27T07:20:15.732204+0000","last_undegraded":"2024-05-27T07:20:15.732204+0000","last_fullsized":"2024-05-27T07:20:15.732204+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:45.430719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.080597+0000","last_change":"2024-05-27T07:17:21.882919+0000","last_active":"2024-05-27T07:18:47.080597+0000","last_peered":"2024-05-27T07:18:47.080597+0000","last_clean":"2024-05-27T07:18:47.080597+0000","last_became_active":"2024-05-27T07:17:21.882330+0000","last_became_peered":"2024-05-27T07:17:21.882330+0000","last_unstale":"2024-05-27T07:18:47.080597+0000","last_undegraded":"2024-05-27T07:18:47.080597+0000","last_fullsized":"2024-05-27T07:18:47.080597+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:51.409079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.064865+0000","last_change":"2024-05-27T07:17:02.486099+0000","last_active":"2024-05-27T07:18:47.064865+0000","last_peered":"2024-05-27T07:18:47.064865+0000","last_clean":"2024-05-27T07:18:47.064865+0000","last_became_active":"2024-05-27T07:17:02.485116+0000","last_became_peered":"2024-05-27T07:17:02.485116+0000","last_unstale":"2024-05-27T07:18:47.064865+0000","last_undegraded":"2024-05-27T07:18:47.064865+0000","last_fullsized":"2024-05-27T07:18:47.064865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:36.338287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.731750+0000","last_change":"2024-05-27T07:17:06.773586+0000","last_active":"2024-05-27T07:20:15.731750+0000","last_peered":"2024-05-27T07:20:15.731750+0000","last_clean":"2024-05-27T07:20:15.731750+0000","last_became_active":"2024-05-27T07:17:06.773139+0000","last_became_peered":"2024-05-27T07:17:06.773139+0000","last_unstale":"2024-05-27T07:20:15.731750+0000","last_undegraded":"2024-05-27T07:20:15.731750+0000","last_fullsized":"2024-05-27T07:20:15.731750+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:19:36.373091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.127485+0000","last_change":"2024-05-27T07:17:21.892351+0000","last_active":"2024-05-27T07:18:47.127485+0000","last_peered":"2024-05-27T07:18:47.127485+0000","last_clean":"2024-05-27T07:18:47.127485+0000","last_became_active":"2024-05-27T07:17:21.891862+0000","last_became_peered":"2024-05-27T07:17:21.891862+0000","last_unstale":"2024-05-27T07:18:47.127485+0000","last_undegraded":"2024-05-27T07:18:47.127485+0000","last_fullsized":"2024-05-27T07:18:47.127485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:50:18.193149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.071128+0000","last_change":"2024-05-27T07:17:02.336917+0000","last_active":"2024-05-27T07:18:47.071128+0000","last_peered":"2024-05-27T07:18:47.071128+0000","last_clean":"2024-05-27T07:18:47.071128+0000","last_became_active":"2024-05-27T07:17:02.335836+0000","last_became_peered":"2024-05-27T07:17:02.335836+0000","last_unstale":"2024-05-27T07:18:47.071128+0000","last_undegraded":"2024-05-27T07:18:47.071128+0000","last_fullsized":"2024-05-27T07:18:47.071128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:54.109480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.719413+0000","last_change":"2024-05-27T07:17:46.705826+0000","last_active":"2024-05-27T07:20:15.719413+0000","last_peered":"2024-05-27T07:20:15.719413+0000","last_clean":"2024-05-27T07:20:15.719413+0000","last_became_active":"2024-05-27T07:17:46.704749+0000","last_became_peered":"2024-05-27T07:17:46.704749+0000","last_unstale":"2024-05-27T07:20:15.719413+0000","last_undegraded":"2024-05-27T07:20:15.719413+0000","last_fullsized":"2024-05-27T07:20:15.719413+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:26.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.068148+0000","last_change":"2024-05-27T07:17:21.885535+0000","last_active":"2024-05-27T07:18:47.068148+0000","last_peered":"2024-05-27T07:18:47.068148+0000","last_clean":"2024-05-27T07:18:47.068148+0000","last_became_active":"2024-05-27T07:17:21.884118+0000","last_became_peered":"2024-05-27T07:17:21.884118+0000","last_unstale":"2024-05-27T07:18:47.068148+0000","last_undegraded":"2024-05-27T07:18:47.068148+0000","last_fullsized":"2024-05-27T07:18:47.068148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:26.600199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:18:47.130129+0000","last_change":"2024-05-27T07:17:02.340037+0000","last_active":"2024-05-27T07:18:47.130129+0000","last_peered":"2024-05-27T07:18:47.130129+0000","last_clean":"2024-05-27T07:18:47.130129+0000","last_became_active":"2024-05-27T07:17:02.339378+0000","last_became_peered":"2024-05-27T07:17:02.339378+0000","last_unstale":"2024-05-27T07:18:47.130129+0000","last_undegraded":"2024-05-27T07:18:47.130129+0000","last_fullsized":"2024-05-27T07:18:47.130129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:33.774917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.722350+0000","last_change":"2024-05-27T07:17:06.763277+0000","last_active":"2024-05-27T07:20:15.722350+0000","last_peered":"2024-05-27T07:20:15.722350+0000","last_clean":"2024-05-27T07:20:15.722350+0000","last_became_active":"2024-05-27T07:17:06.762837+0000","last_became_peered":"2024-05-27T07:17:06.762837+0000","last_unstale":"2024-05-27T07:20:15.722350+0000","last_undegraded":"2024-05-27T07:20:15.722350+0000","last_fullsized":"2024-05-27T07:20:15.722350+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:52.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716297+0000","last_change":"2024-05-27T07:17:06.752556+0000","last_active":"2024-05-27T07:20:15.716297+0000","last_peered":"2024-05-27T07:20:15.716297+0000","last_clean":"2024-05-27T07:20:15.716297+0000","last_became_active":"2024-05-27T07:17:06.751085+0000","last_became_peered":"2024-05-27T07:17:06.751085+0000","last_unstale":"2024-05-27T07:20:15.716297+0000","last_undegraded":"2024-05-27T07:20:15.716297+0000","last_fullsized":"2024-05-27T07:20:15.716297+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:13:53.883845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716301+0000","last_change":"2024-05-27T07:17:06.753674+0000","last_active":"2024-05-27T07:20:15.716301+0000","last_peered":"2024-05-27T07:20:15.716301+0000","last_clean":"2024-05-27T07:20:15.716301+0000","last_became_active":"2024-05-27T07:17:06.752373+0000","last_became_peered":"2024-05-27T07:17:06.752373+0000","last_unstale":"2024-05-27T07:20:15.716301+0000","last_undegraded":"2024-05-27T07:20:15.716301+0000","last_fullsized":"2024-05-27T07:20:15.716301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:40:17.474462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.705046+0000","last_change":"2024-05-27T07:17:06.766796+0000","last_active":"2024-05-27T07:20:15.705046+0000","last_peered":"2024-05-27T07:20:15.705046+0000","last_clean":"2024-05-27T07:20:15.705046+0000","last_became_active":"2024-05-27T07:17:06.766435+0000","last_became_peered":"2024-05-27T07:17:06.766435+0000","last_unstale":"2024-05-27T07:20:15.705046+0000","last_undegraded":"2024-05-27T07:20:15.705046+0000","last_fullsized":"2024-05-27T07:20:15.705046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:38.747810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.717873+0000","last_change":"2024-05-27T07:17:06.762804+0000","last_active":"2024-05-27T07:20:15.717873+0000","last_peered":"2024-05-27T07:20:15.717873+0000","last_clean":"2024-05-27T07:20:15.717873+0000","last_became_active":"2024-05-27T07:17:06.762490+0000","last_became_peered":"2024-05-27T07:17:06.762490+0000","last_unstale":"2024-05-27T07:20:15.717873+0000","last_undegraded":"2024-05-27T07:20:15.717873+0000","last_fullsized":"2024-05-27T07:20:15.717873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:59:33.584848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.716897+0000","last_change":"2024-05-27T07:17:06.765640+0000","last_active":"2024-05-27T07:20:15.716897+0000","last_peered":"2024-05-27T07:20:15.716897+0000","last_clean":"2024-05-27T07:20:15.716897+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:20:15.716897+0000","last_undegraded":"2024-05-27T07:20:15.716897+0000","last_fullsized":"2024-05-27T07:20:15.716897+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:19.695007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706987+0000","last_change":"2024-05-27T07:17:06.759982+0000","last_active":"2024-05-27T07:20:15.706987+0000","last_peered":"2024-05-27T07:20:15.706987+0000","last_clean":"2024-05-27T07:20:15.706987+0000","last_became_active":"2024-05-27T07:17:06.759367+0000","last_became_peered":"2024-05-27T07:17:06.759367+0000","last_unstale":"2024-05-27T07:20:15.706987+0000","last_undegraded":"2024-05-27T07:20:15.706987+0000","last_fullsized":"2024-05-27T07:20:15.706987+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:35:04.686989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732188+0000","last_change":"2024-05-27T07:17:06.768264+0000","last_active":"2024-05-27T07:20:15.732188+0000","last_peered":"2024-05-27T07:20:15.732188+0000","last_clean":"2024-05-27T07:20:15.732188+0000","last_became_active":"2024-05-27T07:17:06.767952+0000","last_became_peered":"2024-05-27T07:17:06.767952+0000","last_unstale":"2024-05-27T07:20:15.732188+0000","last_undegraded":"2024-05-27T07:20:15.732188+0000","last_fullsized":"2024-05-27T07:20:15.732188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:34.867113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.706251+0000","last_change":"2024-05-27T07:17:46.704705+0000","last_active":"2024-05-27T07:20:15.706251+0000","last_peered":"2024-05-27T07:20:15.706251+0000","last_clean":"2024-05-27T07:20:15.706251+0000","last_became_active":"2024-05-27T07:17:46.703503+0000","last_became_peered":"2024-05-27T07:17:46.703503+0000","last_unstale":"2024-05-27T07:20:15.706251+0000","last_undegraded":"2024-05-27T07:20:15.706251+0000","last_fullsized":"2024-05-27T07:20:15.706251+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:39.975009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.713613+0000","last_change":"2024-05-27T07:17:06.752080+0000","last_active":"2024-05-27T07:20:15.713613+0000","last_peered":"2024-05-27T07:20:15.713613+0000","last_clean":"2024-05-27T07:20:15.713613+0000","last_became_active":"2024-05-27T07:17:06.750486+0000","last_became_peered":"2024-05-27T07:17:06.750486+0000","last_unstale":"2024-05-27T07:20:15.713613+0000","last_undegraded":"2024-05-27T07:20:15.713613+0000","last_fullsized":"2024-05-27T07:20:15.713613+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:12.461727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.715971+0000","last_change":"2024-05-27T07:17:06.759200+0000","last_active":"2024-05-27T07:20:15.715971+0000","last_peered":"2024-05-27T07:20:15.715971+0000","last_clean":"2024-05-27T07:20:15.715971+0000","last_became_active":"2024-05-27T07:17:06.758819+0000","last_became_peered":"2024-05-27T07:17:06.758819+0000","last_unstale":"2024-05-27T07:20:15.715971+0000","last_undegraded":"2024-05-27T07:20:15.715971+0000","last_fullsized":"2024-05-27T07:20:15.715971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:00.274229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.732910+0000","last_change":"2024-05-27T07:17:06.760412+0000","last_active":"2024-05-27T07:20:15.732910+0000","last_peered":"2024-05-27T07:20:15.732910+0000","last_clean":"2024-05-27T07:20:15.732910+0000","last_became_active":"2024-05-27T07:17:06.759949+0000","last_became_peered":"2024-05-27T07:17:06.759949+0000","last_unstale":"2024-05-27T07:20:15.732910+0000","last_undegraded":"2024-05-27T07:20:15.732910+0000","last_fullsized":"2024-05-27T07:20:15.732910+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.235488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.725325+0000","last_change":"2024-05-27T07:17:06.775371+0000","last_active":"2024-05-27T07:20:15.725325+0000","last_peered":"2024-05-27T07:20:15.725325+0000","last_clean":"2024-05-27T07:20:15.725325+0000","last_became_active":"2024-05-27T07:17:06.774904+0000","last_became_peered":"2024-05-27T07:17:06.774904+0000","last_unstale":"2024-05-27T07:20:15.725325+0000","last_undegraded":"2024-05-27T07:20:15.725325+0000","last_fullsized":"2024-05-27T07:20:15.725325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:22:13.484733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-27T07:20:15.714253+0000","last_change":"2024-05-27T07:17:06.757066+0000","last_active":"2024-05-27T07:20:15.714253+0000","last_peered":"2024-05-27T07:20:15.714253+0000","last_clean":"2024-05-27T07:20:15.714253+0000","last_became_active":"2024-05-27T07:17:06.756316+0000","last_became_peered":"2024-05-27T07:17:06.756316+0000","last_unstale":"2024-05-27T07:20:15.714253+0000","last_undegraded":"2024-05-27T07:20:15.714253+0000","last_fullsized":"2024-05-27T07:20:15.714253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:15:12.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66995,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":765,"num_read_kb":452,"num_write":179,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710783,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":515890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907012,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32852,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712300,"statfs":{"total":95995035648,"available":95961395200,"internally_reserved":0,"allocated":1716224,"data_stored":1165317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070542,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":519528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266774,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":519336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463004,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1630208,"data_stored":1111352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626531,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":510608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":572839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":511996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":511246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182523,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986291,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":579265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822764,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1638400,"data_stored":1102894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:20:45.933 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:20:45.934 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-27T07:20:45.934 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-27T07:20:45.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-05-27T07:20:46.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:45 smithi162 ceph-mon[35676]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:20:46.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:45 smithi062 ceph-mon[35304]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:20:46.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:45 smithi012 ceph-mon[31473]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:20:46.437 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:47.055 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:47 smithi012 ceph-mon[31473]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:47.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:47 smithi162 ceph-mon[35676]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:47.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:47 smithi062 ceph-mon[35304]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:48.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 -- 172.21.15.12:0/2479012661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8678106ec0 msgr2=0x7f8678113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:48.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 --2- 172.21.15.12:0/2479012661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8678106ec0 0x7f8678113350 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f867000b0a0 tx=0x7f867000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:48.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 -- 172.21.15.12:0/2479012661 shutdown_connections 2024-05-27T07:20:48.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 --2- 172.21.15.12:0/2479012661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867810c5b0 0x7f867810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:48.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 --2- 172.21.15.12:0/2479012661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8678106ec0 0x7f8678113350 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:48.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 --2- 172.21.15.12:0/2479012661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8678106570 0x7f8678106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:48.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 -- 172.21.15.12:0/2479012661 >> 172.21.15.12:0/2479012661 conn(0x7f8678102070 msgr2=0x7f86781044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:48.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 -- 172.21.15.12:0/2479012661 shutdown_connections 2024-05-27T07:20:48.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.974+0000 7f867fe00700 1 -- 172.21.15.12:0/2479012661 wait complete. 2024-05-27T07:20:48.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.975+0000 7f867fe00700 1 Processor -- start 2024-05-27T07:20:48.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.976+0000 7f867fe00700 1 -- start start 2024-05-27T07:20:48.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.976+0000 7f867fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8678106570 0x7f8678078dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:48.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.976+0000 7f867fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8678106ec0 0x7f8678079310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:48.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.976+0000 7f867fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f867810c5b0 0x7f8678079850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:48.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.976+0000 7f867fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86781165e0 con 0x7f867810c5b0 2024-05-27T07:20:48.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.976+0000 7f867fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8678116460 con 0x7f8678106570 2024-05-27T07:20:48.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.976+0000 7f867fe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8678116760 con 0x7f8678106ec0 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f867810c5b0 0x7f8678079850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f867810c5b0 0x7f8678079850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39168/0 (socket says 172.21.15.12:39168) 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 -- 172.21.15.12:0/2997090693 learned_addr learned my addr 172.21.15.12:0/2997090693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867d400700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8678106570 0x7f8678078dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867ca00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8678106ec0 0x7f8678079310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 -- 172.21.15.12:0/2997090693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8678106ec0 msgr2=0x7f8678079310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8678106ec0 0x7f8678079310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 -- 172.21.15.12:0/2997090693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8678106570 msgr2=0x7f8678078dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8678106570 0x7f8678078dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.977+0000 7f867de00700 1 -- 172.21.15.12:0/2997090693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86781ad470 con 0x7f867810c5b0 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.978+0000 7f867de00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f867810c5b0 0x7f8678079850 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8670004dc0 tx=0x7f8670004ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:48.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.978+0000 7f8676000700 1 -- 172.21.15.12:0/2997090693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670021020 con 0x7f867810c5b0 2024-05-27T07:20:48.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.978+0000 7f8676000700 1 -- 172.21.15.12:0/2997090693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8670007cf0 con 0x7f867810c5b0 2024-05-27T07:20:48.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.978+0000 7f8676000700 1 -- 172.21.15.12:0/2997090693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670005370 con 0x7f867810c5b0 2024-05-27T07:20:48.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.978+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86781ad640 con 0x7f867810c5b0 2024-05-27T07:20:48.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.978+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86781ada40 con 0x7f867810c5b0 2024-05-27T07:20:48.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.980+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f867810e970 con 0x7f867810c5b0 2024-05-27T07:20:48.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.982+0000 7f8676000700 1 -- 172.21.15.12:0/2997090693 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8670024020 con 0x7f867810c5b0 2024-05-27T07:20:48.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.983+0000 7f8676000700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f86600b2900 0x7f86600b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:48.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.984+0000 7f8676000700 1 -- 172.21.15.12:0/2997090693 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f867000f020 con 0x7f867810c5b0 2024-05-27T07:20:48.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.984+0000 7f867d400700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f86600b2900 0x7f86600b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:48.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.985+0000 7f867d400700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f86600b2900 0x7f86600b4e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f866c005ea0 tx=0x7f866c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:48.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:48.987+0000 7f8676000700 1 -- 172.21.15.12:0/2997090693 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f86700ab6f0 con 0x7f867810c5b0 2024-05-27T07:20:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:49 smithi012 ceph-mon[31473]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:49.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.281+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f867810e350 con 0x7f867810c5b0 2024-05-27T07:20:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.281+0000 7f8676000700 1 -- 172.21.15.12:0/2997090693 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f86700aaff0 con 0x7f867810c5b0 2024-05-27T07:20:49.271 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:20:49.271 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.284+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f86600b2900 msgr2=0x7f86600b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.284+0000 7f867fe00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f86600b2900 0x7f86600b4e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f866c005ea0 tx=0x7f866c005e30 comp rx=0 tx=0).stop 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.284+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f867810c5b0 msgr2=0x7f8678079850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f867810c5b0 0x7f8678079850 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8670004dc0 tx=0x7f8670004ea0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 shutdown_connections 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f86600b2900 0x7f86600b4e10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8678106570 0x7f8678078dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f867810c5b0 0x7f8678079850 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 --2- 172.21.15.12:0/2997090693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8678106ec0 0x7f8678079310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 >> 172.21.15.12:0/2997090693 conn(0x7f8678102070 msgr2=0x7f86781046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 shutdown_connections 2024-05-27T07:20:49.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:49.285+0000 7f867fe00700 1 -- 172.21.15.12:0/2997090693 wait complete. 2024-05-27T07:20:49.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:49 smithi162 ceph-mon[35676]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:49.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:49 smithi062 ceph-mon[35304]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:50.197 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-27T07:20:50.198 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:20:50.283 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:50 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2997090693' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:20:50.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:50 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2997090693' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:20:50.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:50 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2997090693' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:20:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:51.071 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:51 smithi012 ceph-mon[31473]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:51.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:51 smithi162 ceph-mon[35676]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:51.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:51 smithi062 ceph-mon[35304]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:20:53.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.148+0000 7fdb59400700 1 -- 172.21.15.12:0/3853510457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 msgr2=0x7fdb54110d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:53.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.148+0000 7fdb59400700 1 --2- 172.21.15.12:0/3853510457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb54110d90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009990 tx=0x7fdb44009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.148+0000 7fdb59400700 1 -- 172.21.15.12:0/3853510457 shutdown_connections 2024-05-27T07:20:53.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.148+0000 7fdb59400700 1 --2- 172.21.15.12:0/3853510457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb5410b2a0 0x7fdb54117820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.148+0000 7fdb59400700 1 --2- 172.21.15.12:0/3853510457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb54110d90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.148+0000 7fdb59400700 1 --2- 172.21.15.12:0/3853510457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5410a950 0x7fdb5410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.148+0000 7fdb59400700 1 -- 172.21.15.12:0/3853510457 >> 172.21.15.12:0/3853510457 conn(0x7fdb54076d20 msgr2=0x7fdb54077130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:53.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.149+0000 7fdb59400700 1 -- 172.21.15.12:0/3853510457 shutdown_connections 2024-05-27T07:20:53.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.149+0000 7fdb59400700 1 -- 172.21.15.12:0/3853510457 wait complete. 2024-05-27T07:20:53.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.150+0000 7fdb59400700 1 Processor -- start 2024-05-27T07:20:53.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.150+0000 7fdb59400700 1 -- start start 2024-05-27T07:20:53.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.150+0000 7fdb59400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb5410a950 0x7fdb541a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:53.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.150+0000 7fdb59400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5410b2a0 0x7fdb541a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:53.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.150+0000 7fdb59400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb541a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:53.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.150+0000 7fdb59400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5411a950 con 0x7fdb54110980 2024-05-27T07:20:53.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.150+0000 7fdb59400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5411a7d0 con 0x7fdb5410a950 2024-05-27T07:20:53.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb59400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5411aad0 con 0x7fdb5410b2a0 2024-05-27T07:20:53.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb541a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:53.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb541a5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55396/0 (socket says 172.21.15.12:55396) 2024-05-27T07:20:53.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 -- 172.21.15.12:0/2430427922 learned_addr learned my addr 172.21.15.12:0/2430427922 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:53.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb51e00700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5410b2a0 0x7fdb541a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:53.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb52800700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb5410a950 0x7fdb541a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:53.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 -- 172.21.15.12:0/2430427922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5410b2a0 msgr2=0x7fdb541a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:53.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5410b2a0 0x7fdb541a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 -- 172.21.15.12:0/2430427922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb5410a950 msgr2=0x7fdb541a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:53.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb5410a950 0x7fdb541a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.151+0000 7fdb53200700 1 -- 172.21.15.12:0/2430427922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb541a6180 con 0x7fdb54110980 2024-05-27T07:20:53.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.152+0000 7fdb53200700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb541a5b90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c007d70 tx=0x7fdb4c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:53.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.152+0000 7fdb4b400700 1 -- 172.21.15.12:0/2430427922 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c016020 con 0x7fdb54110980 2024-05-27T07:20:53.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.152+0000 7fdb4b400700 1 -- 172.21.15.12:0/2430427922 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb4c00ad40 con 0x7fdb54110980 2024-05-27T07:20:53.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.152+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb541a6470 con 0x7fdb54110980 2024-05-27T07:20:53.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.152+0000 7fdb4b400700 1 -- 172.21.15.12:0/2430427922 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0115e0 con 0x7fdb54110980 2024-05-27T07:20:53.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.152+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb541a6960 con 0x7fdb54110980 2024-05-27T07:20:53.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.154+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb540512b0 con 0x7fdb54110980 2024-05-27T07:20:53.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.156+0000 7fdb4b400700 1 -- 172.21.15.12:0/2430427922 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fdb4c019020 con 0x7fdb54110980 2024-05-27T07:20:53.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.158+0000 7fdb4b400700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdb380b2880 0x7fdb380b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:53.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.158+0000 7fdb4b400700 1 -- 172.21.15.12:0/2430427922 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdb4c0e2610 con 0x7fdb54110980 2024-05-27T07:20:53.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.158+0000 7fdb52800700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdb380b2880 0x7fdb380b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:53.180 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:53 smithi012 ceph-mon[31473]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:53.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.159+0000 7fdb52800700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdb380b2880 0x7fdb380b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006010 tx=0x7fdb4400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:53.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.164+0000 7fdb4b400700 1 -- 172.21.15.12:0/2430427922 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdb4c020020 con 0x7fdb54110980 2024-05-27T07:20:53.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.456+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb5402e5e0 con 0x7fdb54110980 2024-05-27T07:20:53.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.458+0000 7fdb4b400700 1 -- 172.21.15.12:0/2430427922 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11867 (secure 0 0 0) 0x7fdb4c0a9f00 con 0x7fdb54110980 2024-05-27T07:20:53.448 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:20:53.448 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:18:42.889436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:20:53.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdb380b2880 msgr2=0x7fdb380b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:53.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdb380b2880 0x7fdb380b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006010 tx=0x7fdb4400b540 comp rx=0 tx=0).stop 2024-05-27T07:20:53.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 msgr2=0x7fdb541a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:53.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb541a5b90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c007d70 tx=0x7fdb4c00a340 comp rx=0 tx=0).stop 2024-05-27T07:20:53.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 shutdown_connections 2024-05-27T07:20:53.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdb380b2880 0x7fdb380b4d90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb5410a950 0x7fdb541a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb54110980 0x7fdb541a5b90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 --2- 172.21.15.12:0/2430427922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5410b2a0 0x7fdb541a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:53.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.461+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 >> 172.21.15.12:0/2430427922 conn(0x7fdb54076d20 msgr2=0x7fdb54109250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:53.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.462+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 shutdown_connections 2024-05-27T07:20:53.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:53.462+0000 7fdb59400700 1 -- 172.21.15.12:0/2430427922 wait complete. 2024-05-27T07:20:53.453 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 26 2024-05-27T07:20:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:53 smithi162 ceph-mon[35676]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:53.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:53 smithi062 ceph-mon[35304]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:20:54.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:54 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2430427922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:54.372 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:20:54.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:54 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2430427922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:54.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:54 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2430427922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:55.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:55 smithi012 ceph-mon[31473]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:55.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:55 smithi162 ceph-mon[35676]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:55.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:55 smithi062 ceph-mon[35304]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:20:57.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 -- 172.21.15.12:0/3657835076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 msgr2=0x7f0e701038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 --2- 172.21.15.12:0/3657835076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e701038c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0e640099c0 tx=0x7f0e64009cd0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 -- 172.21.15.12:0/3657835076 shutdown_connections 2024-05-27T07:20:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 --2- 172.21.15.12:0/3657835076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e70108730 0x7f0e70110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 --2- 172.21.15.12:0/3657835076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e701038c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 --2- 172.21.15.12:0/3657835076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e70107de0 0x7f0e701081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 -- 172.21.15.12:0/3657835076 >> 172.21.15.12:0/3657835076 conn(0x7f0e700fedb0 msgr2=0x7f0e701011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:57.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 -- 172.21.15.12:0/3657835076 shutdown_connections 2024-05-27T07:20:57.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:57 smithi012 ceph-mon[31473]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.347+0000 7f0e78800700 1 -- 172.21.15.12:0/3657835076 wait complete. 2024-05-27T07:20:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 Processor -- start 2024-05-27T07:20:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 -- start start 2024-05-27T07:20:57.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e7019d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e70107de0 0x7f0e7019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e70108730 0x7f0e701a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0e70114380 con 0x7f0e701034b0 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e70114200 con 0x7f0e70108730 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.349+0000 7f0e78800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e70114500 con 0x7f0e70107de0 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e7019d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e7019d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55414/0 (socket says 172.21.15.12:55414) 2024-05-27T07:20:57.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 -- 172.21.15.12:0/1821144725 learned_addr learned my addr 172.21.15.12:0/1821144725 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:20:57.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e76800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e70108730 0x7f0e701a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75400700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e70107de0 0x7f0e7019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 -- 172.21.15.12:0/1821144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e70107de0 msgr2=0x7f0e7019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e70107de0 0x7f0e7019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 -- 172.21.15.12:0/1821144725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e70108730 msgr2=0x7f0e701a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e70108730 0x7f0e701a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e75e00700 1 -- 172.21.15.12:0/1821144725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e701a1f20 con 0x7f0e701034b0 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.350+0000 7f0e76800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e70108730 0x7f0e701a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.351+0000 7f0e75e00700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e7019d050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0e64000c00 tx=0x7f0e6400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.351+0000 7f0e6aa00700 1 -- 172.21.15.12:0/1821144725 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64026070 con 0x7f0e701034b0 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.351+0000 7f0e6aa00700 1 -- 172.21.15.12:0/1821144725 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e640054c0 con 0x7f0e701034b0 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.351+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e701a21b0 con 0x7f0e701034b0 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.351+0000 7f0e6aa00700 1 -- 172.21.15.12:0/1821144725 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64004c20 con 0x7f0e701034b0 2024-05-27T07:20:57.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.351+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e701a9b30 con 0x7f0e701034b0 2024-05-27T07:20:57.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.352+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e700512b0 con 0x7f0e701034b0 2024-05-27T07:20:57.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.355+0000 7f0e6aa00700 1 -- 172.21.15.12:0/1821144725 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0e64029070 con 0x7f0e701034b0 2024-05-27T07:20:57.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.357+0000 7f0e6aa00700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e580b2900 0x7f0e580b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.357+0000 7f0e6aa00700 1 -- 172.21.15.12:0/1821144725 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f0e640e3540 con 0x7f0e701034b0 2024-05-27T07:20:57.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.357+0000 7f0e75400700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e580b2900 0x7f0e580b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.358+0000 7f0e75400700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e580b2900 0x7f0e580b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e60005fd0 tx=0x7f0e6000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:57.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.359+0000 7f0e6aa00700 1 -- 172.21.15.12:0/1821144725 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0e64030080 con 0x7f0e701034b0 2024-05-27T07:20:57.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:57 smithi162 ceph-mon[35676]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:57.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:57 smithi062 ceph-mon[35304]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:20:57.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.649+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e701a9d00 con 0x7f0e701034b0 2024-05-27T07:20:57.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.650+0000 7f0e6aa00700 1 -- 172.21.15.12:0/1821144725 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11867 (secure 0 0 0) 0x7f0e64013070 con 0x7f0e701034b0 2024-05-27T07:20:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:20:57.641 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:18:42.889436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:20:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e580b2900 msgr2=0x7f0e580b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e580b2900 0x7f0e580b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e60005fd0 tx=0x7f0e6000a560 comp rx=0 tx=0).stop 2024-05-27T07:20:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 msgr2=0x7f0e7019d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e7019d050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0e64000c00 tx=0x7f0e6400f740 comp rx=0 tx=0).stop 2024-05-27T07:20:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 shutdown_connections 2024-05-27T07:20:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0e580b2900 0x7f0e580b4e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e70108730 0x7f0e701a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e701034b0 0x7f0e7019d050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 --2- 172.21.15.12:0/1821144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e70107de0 0x7f0e7019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 >> 172.21.15.12:0/1821144725 conn(0x7f0e700fedb0 msgr2=0x7f0e701064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 shutdown_connections 2024-05-27T07:20:57.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:20:57.654+0000 7f0e78800700 1 -- 172.21.15.12:0/1821144725 wait complete. 2024-05-27T07:20:57.646 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 26 2024-05-27T07:20:58.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:58 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1821144725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:58.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:58 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1821144725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:58.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:58 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1821144725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:58.503 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:20:59.010 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:59.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:20:59 smithi012 ceph-mon[31473]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:59.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:20:59 smithi162 ceph-mon[35676]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:20:59.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:20:59 smithi062 ceph-mon[35304]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:01.107 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:01 smithi012 ceph-mon[31473]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:01.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:01 smithi162 ceph-mon[35676]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:01.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:01 smithi062 ceph-mon[35304]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.501+0000 7fce59600700 1 -- 172.21.15.12:0/1724458367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54107c30 msgr2=0x7fce54108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.501+0000 7fce59600700 1 --2- 172.21.15.12:0/1724458367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54107c30 0x7fce54108040 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fce400099c0 tx=0x7fce40009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.501+0000 7fce59600700 1 -- 172.21.15.12:0/1724458367 shutdown_connections 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.501+0000 7fce59600700 1 --2- 172.21.15.12:0/1724458367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce54108580 0x7fce5410fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.501+0000 7fce59600700 1 --2- 172.21.15.12:0/1724458367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce54103300 0x7fce54103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.501+0000 7fce59600700 1 --2- 172.21.15.12:0/1724458367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54107c30 0x7fce54108040 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.502+0000 7fce59600700 1 -- 172.21.15.12:0/1724458367 >> 172.21.15.12:0/1724458367 conn(0x7fce54078ae0 msgr2=0x7fce54078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.502+0000 7fce59600700 1 -- 172.21.15.12:0/1724458367 shutdown_connections 2024-05-27T07:21:01.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.502+0000 7fce59600700 1 -- 172.21.15.12:0/1724458367 wait complete. 2024-05-27T07:21:01.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.503+0000 7fce59600700 1 Processor -- start 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.503+0000 7fce59600700 1 -- start start 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.503+0000 7fce59600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce54103300 0x7fce541a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce59600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce54107c30 0x7fce541a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce59600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54108580 0x7fce5419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce59600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fce54113090 con 0x7fce54108580 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce59600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fce54112f10 con 0x7fce54107c30 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce59600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce54113210 con 0x7fce54103300 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce53400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54108580 0x7fce5419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce53400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54108580 0x7fce5419aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55426/0 (socket says 172.21.15.12:55426) 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce53400700 1 -- 172.21.15.12:0/1593733284 learned_addr learned my addr 172.21.15.12:0/1593733284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce52a00700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce54103300 0x7fce541a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.504+0000 7fce52000700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce54107c30 0x7fce541a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.505+0000 7fce53400700 1 -- 172.21.15.12:0/1593733284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce54103300 msgr2=0x7fce541a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.505+0000 7fce53400700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce54103300 0x7fce541a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.505+0000 7fce53400700 1 -- 172.21.15.12:0/1593733284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce54107c30 msgr2=0x7fce541a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.505+0000 7fce53400700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce54107c30 0x7fce541a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.505+0000 7fce53400700 1 -- 172.21.15.12:0/1593733284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5419b130 con 0x7fce54108580 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.505+0000 7fce53400700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54108580 0x7fce5419aa80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00ef30 tx=0x7fce4c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.506+0000 7fce4b400700 1 -- 172.21.15.12:0/1593733284 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c0092e0 con 0x7fce54108580 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.506+0000 7fce4b400700 1 -- 172.21.15.12:0/1593733284 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce4c004bb0 con 0x7fce54108580 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.506+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5419b3c0 con 0x7fce54108580 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.506+0000 7fce4b400700 1 -- 172.21.15.12:0/1593733284 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c004020 con 0x7fce54108580 2024-05-27T07:21:01.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.506+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5419b880 con 0x7fce54108580 2024-05-27T07:21:01.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.507+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce541047b0 con 0x7fce54108580 2024-05-27T07:21:01.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.511+0000 7fce4b400700 1 -- 172.21.15.12:0/1593733284 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fce4c01b030 con 0x7fce54108580 2024-05-27T07:21:01.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.513+0000 7fce4b400700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fce380b2880 0x7fce380b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.513+0000 7fce4b400700 1 -- 172.21.15.12:0/1593733284 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fce4c0e2ab0 con 0x7fce54108580 2024-05-27T07:21:01.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.513+0000 7fce52a00700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fce380b2880 0x7fce380b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.514+0000 7fce4b400700 1 -- 172.21.15.12:0/1593733284 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fce4c022020 con 0x7fce54108580 2024-05-27T07:21:01.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.514+0000 7fce52a00700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fce380b2880 0x7fce380b4d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fce5419bbc0 tx=0x7fce44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:01.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.755+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce5419bfb0 con 0x7fce54108580 2024-05-27T07:21:01.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.757+0000 7fce4b400700 1 -- 172.21.15.12:0/1593733284 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24806 (secure 0 0 0) 0x7fce4c010040 con 0x7fce54108580 2024-05-27T07:21:01.748 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:01.749 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:18:47.037205+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:21:01.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fce380b2880 msgr2=0x7fce380b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fce380b2880 0x7fce380b4d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fce5419bbc0 tx=0x7fce44009260 comp rx=0 tx=0).stop 2024-05-27T07:21:01.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54108580 msgr2=0x7fce5419aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54108580 0x7fce5419aa80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00ef30 tx=0x7fce4c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 shutdown_connections 2024-05-27T07:21:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fce380b2880 0x7fce380b4d90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fce54108580 0x7fce5419aa80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce54103300 0x7fce541a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 --2- 172.21.15.12:0/1593733284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce54107c30 0x7fce541a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.760+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 >> 172.21.15.12:0/1593733284 conn(0x7fce54078ae0 msgr2=0x7fce54107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.761+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 shutdown_connections 2024-05-27T07:21:01.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:01.761+0000 7fce59600700 1 -- 172.21.15.12:0/1593733284 wait complete. 2024-05-27T07:21:02.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:02 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1593733284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:21:02.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:02 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1593733284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:21:02.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:02 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1593733284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:21:02.686 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:03.363 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:03 smithi012 ceph-mon[31473]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:03.375 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:03 smithi162 ceph-mon[35676]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:03.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:03 smithi062 ceph-mon[35304]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:04.375 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:04 smithi012 ceph-mon[31473]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:04.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:04 smithi162 ceph-mon[35676]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:04.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:04 smithi062 ceph-mon[35304]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 -- 172.21.15.12:0/2127237495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 msgr2=0x7fa53c077010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 --2- 172.21.15.12:0/2127237495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c077010 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009990 tx=0x7fa52c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 -- 172.21.15.12:0/2127237495 shutdown_connections 2024-05-27T07:21:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 --2- 172.21.15.12:0/2127237495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa53c110bd0 0x7fa53c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 --2- 172.21.15.12:0/2127237495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c077010 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 --2- 172.21.15.12:0/2127237495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa53c077550 0x7fa53c110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 -- 172.21.15.12:0/2127237495 >> 172.21.15.12:0/2127237495 conn(0x7fa53c0ffe90 msgr2=0x7fa53c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 -- 172.21.15.12:0/2127237495 shutdown_connections 2024-05-27T07:21:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.516+0000 7fa540e00700 1 -- 172.21.15.12:0/2127237495 wait complete. 2024-05-27T07:21:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.517+0000 7fa540e00700 1 Processor -- start 2024-05-27T07:21:05.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa540e00700 1 -- start start 2024-05-27T07:21:05.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa540e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa540e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa53c077550 0x7fa53c19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa540e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa53c110bd0 0x7fa53c1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa540e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c116580 con 0x7fa53c076c00 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa540e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c116400 con 0x7fa53c110bd0 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa540e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c116700 con 0x7fa53c077550 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa539e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa539e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c19cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37494/0 (socket says 172.21.15.12:37494) 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.518+0000 7fa539e00700 1 -- 172.21.15.12:0/1085752981 learned_addr learned my addr 172.21.15.12:0/1085752981 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.519+0000 7fa53a800700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa53c110bd0 0x7fa53c1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.519+0000 7fa539400700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa53c077550 0x7fa53c19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.519+0000 7fa539e00700 1 -- 172.21.15.12:0/1085752981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa53c077550 msgr2=0x7fa53c19d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.519+0000 7fa539e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa53c077550 0x7fa53c19d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.519+0000 7fa539e00700 1 -- 172.21.15.12:0/1085752981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa53c110bd0 msgr2=0x7fa53c1a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.519+0000 7fa539e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa53c110bd0 0x7fa53c1a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.519+0000 7fa539e00700 1 -- 172.21.15.12:0/1085752981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c1a1e60 con 0x7fa53c076c00 2024-05-27T07:21:05.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.520+0000 7fa539e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c19cf90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0057c0 tx=0x7fa52c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.520+0000 7fa532a00700 1 -- 172.21.15.12:0/1085752981 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c027070 con 0x7fa53c076c00 2024-05-27T07:21:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.520+0000 7fa532a00700 1 -- 172.21.15.12:0/1085752981 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa52c00fda0 con 0x7fa53c076c00 2024-05-27T07:21:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.520+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53c1a20f0 con 0x7fa53c076c00 2024-05-27T07:21:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.520+0000 7fa532a00700 1 -- 172.21.15.12:0/1085752981 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c022c70 con 0x7fa53c076c00 2024-05-27T07:21:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.520+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53c1a9a70 con 0x7fa53c076c00 2024-05-27T07:21:05.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.521+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa53c078340 con 0x7fa53c076c00 2024-05-27T07:21:05.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.524+0000 7fa532a00700 1 -- 172.21.15.12:0/1085752981 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa52c02a070 con 0x7fa53c076c00 2024-05-27T07:21:05.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.526+0000 7fa532a00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5200b2900 0x7fa5200b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.526+0000 7fa532a00700 1 -- 172.21.15.12:0/1085752981 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fa52c0e3820 con 0x7fa53c076c00 2024-05-27T07:21:05.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.526+0000 7fa539400700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5200b2900 0x7fa5200b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.527+0000 7fa539400700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5200b2900 0x7fa5200b4e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa528005950 tx=0x7fa5280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:05.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.528+0000 7fa532a00700 1 -- 172.21.15.12:0/1085752981 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa52c05fdf0 con 0x7fa53c076c00 2024-05-27T07:21:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.815+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa53c02e5e0 con 0x7fa53c076c00 2024-05-27T07:21:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.816+0000 7fa532a00700 1 -- 172.21.15.12:0/1085752981 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11867 (secure 0 0 0) 0x7fa52c013070 con 0x7fa53c076c00 2024-05-27T07:21:05.807 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:05.808 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:18:42.889436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:21:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5200b2900 msgr2=0x7fa5200b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5200b2900 0x7fa5200b4e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa528005950 tx=0x7fa5280058e0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 msgr2=0x7fa53c19cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c19cf90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0057c0 tx=0x7fa52c009f90 comp rx=0 tx=0).stop 2024-05-27T07:21:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 shutdown_connections 2024-05-27T07:21:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5200b2900 0x7fa5200b4e10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa53c110bd0 0x7fa53c1a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa53c076c00 0x7fa53c19cf90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 --2- 172.21.15.12:0/1085752981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa53c077550 0x7fa53c19d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.819+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 >> 172.21.15.12:0/1085752981 conn(0x7fa53c0ffe90 msgr2=0x7fa53c100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.820+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 shutdown_connections 2024-05-27T07:21:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:05.820+0000 7fa540e00700 1 -- 172.21.15.12:0/1085752981 wait complete. 2024-05-27T07:21:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 26 2024-05-27T07:21:06.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:05 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1085752981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:06.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:05 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1085752981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:06.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:05 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1085752981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:06.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:17:09.825542+0000', 'modified': '2024-05-27T07:18:42.889436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 34475, 'mds_1': 34439, 'mds_2': 34415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6837/1296782695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 1296782695}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 1296782695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6839/884400330', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 884400330}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 884400330}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34391': {'gid': 34391, 'name': 'k', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.62:6833/1529598052', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1529598052}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 1529598052}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34415': {'gid': 34415, 'name': 'e', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.62:6835/989260451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 989260451}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 989260451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34439': {'gid': 34439, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.62:6837/1993227402', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6836', 'nonce': 1993227402}, {'type': 'v1', 'addr': '172.21.15.62:6837', 'nonce': 1993227402}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34475': {'gid': 34475, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.62:6839/994488749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6838', 'nonce': 994488749}, {'type': 'v1', 'addr': '172.21.15.62:6839', 'nonce': 994488749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34475, 'qdb_cluster': [34439, 34475, 34415]} 2024-05-27T07:21:06.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-27T07:21:06.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.e status 2024-05-27T07:21:07.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:06 smithi012 ceph-mon[31473]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:07.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:06 smithi162 ceph-mon[35676]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:07.225 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:07.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:06 smithi062 ceph-mon[35304]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:09.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:08 smithi012 ceph-mon[31473]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:09.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:08 smithi162 ceph-mon[35676]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:09.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:08 smithi062 ceph-mon[35304]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.697+0000 7fbc53200700 1 -- 172.21.15.12:0/2915530418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 msgr2=0x7fbc4c110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.697+0000 7fbc53200700 1 --2- 172.21.15.12:0/2915530418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c110fd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800b0d0 tx=0x7fbc4800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.698+0000 7fbc53200700 1 -- 172.21.15.12:0/2915530418 shutdown_connections 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.698+0000 7fbc53200700 1 --2- 172.21.15.12:0/2915530418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c110fd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.698+0000 7fbc53200700 1 --2- 172.21.15.12:0/2915530418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc4c1043f0 0x7fbc4c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.698+0000 7fbc53200700 1 --2- 172.21.15.12:0/2915530418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108d20 0x7fbc4c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.698+0000 7fbc53200700 1 -- 172.21.15.12:0/2915530418 >> 172.21.15.12:0/2915530418 conn(0x7fbc4c079d00 msgr2=0x7fbc4c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.698+0000 7fbc53200700 1 -- 172.21.15.12:0/2915530418 shutdown_connections 2024-05-27T07:21:09.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.698+0000 7fbc53200700 1 -- 172.21.15.12:0/2915530418 wait complete. 2024-05-27T07:21:09.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.700+0000 7fbc53200700 1 Processor -- start 2024-05-27T07:21:09.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.700+0000 7fbc53200700 1 -- start start 2024-05-27T07:21:09.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.700+0000 7fbc53200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc4c1043f0 0x7fbc4c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.700+0000 7fbc53200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108d20 0x7fbc4c19d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.700+0000 7fbc53200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c1a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.700+0000 7fbc53200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c114150 con 0x7fbc4c109670 2024-05-27T07:21:09.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc53200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c113fd0 con 0x7fbc4c108d20 2024-05-27T07:21:09.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc53200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1142d0 con 0x7fbc4c1043f0 2024-05-27T07:21:09.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c1a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c1a1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37524/0 (socket says 172.21.15.12:37524) 2024-05-27T07:21:09.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 -- 172.21.15.12:0/3329998449 learned_addr learned my addr 172.21.15.12:0/3329998449 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:09.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc51e00700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc4c1043f0 0x7fbc4c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc51400700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108d20 0x7fbc4c19d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 -- 172.21.15.12:0/3329998449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc4c1043f0 msgr2=0x7fbc4c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc4c1043f0 0x7fbc4c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 -- 172.21.15.12:0/3329998449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108d20 msgr2=0x7fbc4c19d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108d20 0x7fbc4c19d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.701+0000 7fbc52800700 1 -- 172.21.15.12:0/3329998449 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c1a1b40 con 0x7fbc4c109670 2024-05-27T07:21:09.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.702+0000 7fbc52800700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c1a1550 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009850 tx=0x7fbc48007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:09.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.702+0000 7fbc46a00700 1 -- 172.21.15.12:0/3329998449 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4801f030 con 0x7fbc4c109670 2024-05-27T07:21:09.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.702+0000 7fbc46a00700 1 -- 172.21.15.12:0/3329998449 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc4801bad0 con 0x7fbc4c109670 2024-05-27T07:21:09.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.702+0000 7fbc46a00700 1 -- 172.21.15.12:0/3329998449 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48012bb0 con 0x7fbc4c109670 2024-05-27T07:21:09.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.703+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c1a1dd0 con 0x7fbc4c109670 2024-05-27T07:21:09.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.703+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c1a2290 con 0x7fbc4c109670 2024-05-27T07:21:09.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.707+0000 7fbc46a00700 1 -- 172.21.15.12:0/3329998449 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fbc48022030 con 0x7fbc4c109670 2024-05-27T07:21:09.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.708+0000 7fbc6329eb80 1 Processor -- start 2024-05-27T07:21:09.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.708+0000 7fbc6329eb80 1 -- start start 2024-05-27T07:21:09.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.708+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab40dcdf0 0x558ab425ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.708+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b3c0 0x558ab4265a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558ab4189df0 con 0x558ab425b7d0 2024-05-27T07:21:09.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558ab4179990 con 0x558ab425b3c0 2024-05-27T07:21:09.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558ab417d9d0 con 0x558ab40dcdf0 2024-05-27T07:21:09.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc46a00700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbc340b2880 0x7fbc340b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc46a00700 1 -- 172.21.15.12:0/3329998449 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fbc4800f030 con 0x7fbc4c109670 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc52800700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbc340b2880 0x7fbc340b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc51e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b3c0 0x558ab4265a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37540/0 (socket says 172.21.15.12:37540) 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc51400700 1 -- 172.21.15.12:0/2962206797 learned_addr learned my addr 172.21.15.12:0/2962206797 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.709+0000 7fbc51400700 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab40dcdf0 0x558ab425ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.710+0000 7fbc51400700 1 -- 172.21.15.12:0/2962206797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab40dcdf0 msgr2=0x558ab425ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.710+0000 7fbc51400700 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab40dcdf0 0x558ab425ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.710+0000 7fbc51400700 1 -- 172.21.15.12:0/2962206797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b3c0 msgr2=0x558ab4265a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.710+0000 7fbc51400700 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b3c0 0x558ab4265a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.710+0000 7fbc51400700 1 -- 172.21.15.12:0/2962206797 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ab41423d0 con 0x558ab425b7d0 2024-05-27T07:21:09.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.710+0000 7fbc52800700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbc340b2880 0x7fbc340b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbc48089490 tx=0x7fbc4800b3e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:09.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.710+0000 7fbc51400700 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267fd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x558ab4257640 tx=0x7fbc3c00b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27596bcea20480da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.711+0000 7fbc3b400700 1 -- 172.21.15.12:0/2962206797 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c010640 con 0x558ab425b7d0 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.711+0000 7fbc3b400700 1 -- 172.21.15.12:0/2962206797 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc3c010c70 con 0x558ab425b7d0 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.711+0000 7fbc3b400700 1 -- 172.21.15.12:0/2962206797 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c011910 con 0x558ab425b7d0 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 -- 172.21.15.12:0/2962206797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 msgr2=0x558ab4267fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267fd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x558ab4257640 tx=0x7fbc3c00b440 comp rx=0 tx=0).stop 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 -- 172.21.15.12:0/2962206797 shutdown_connections 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267fd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab40dcdf0 0x558ab425ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/2962206797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b3c0 0x558ab4265a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 -- 172.21.15.12:0/2962206797 >> 172.21.15.12:0/2962206797 conn(0x558ab413cf70 msgr2=0x558ab4256ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:09.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 -- 172.21.15.12:0/2962206797 shutdown_connections 2024-05-27T07:21:09.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.712+0000 7fbc6329eb80 1 -- 172.21.15.12:0/2962206797 wait complete. 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.713+0000 7fbc6329eb80 1 -- start start 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.713+0000 7fbc6329eb80 1 Processor -- start 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.713+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4265500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.713+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4265a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.713+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4265f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.713+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558ab407ff50 con 0x558ab425b7d0 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558ab411a710 con 0x558ab40dcdf0 2024-05-27T07:21:09.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558ab42688e0 con 0x558ab425b3c0 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc6329eb80 10 client.0 _ll_register_callbacks cb 0x558ab4123c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4265f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4265f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37546/0 (socket says 172.21.15.12:37546) 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc51400700 1 -- 172.21.15.12:0/1978739865 learned_addr learned my addr 172.21.15.12:0/1978739865 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc51e00700 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4265a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.714+0000 7fbc51400700 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4265500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc51400700 1 -- 172.21.15.12:0/1978739865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 msgr2=0x558ab4265a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc51400700 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4265a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc51400700 1 -- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 msgr2=0x558ab4265500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc51400700 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4265500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc51400700 1 -- 172.21.15.12:0/1978739865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ab40c9880 con 0x558ab425b7d0 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc51400700 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4265f80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00bfd0 tx=0x7fbc3c00b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd740dc7a6a499c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:09.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc2fe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc2fe00700 1 -- 172.21.15.12:0/1978739865 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c011cb0 con 0x558ab425b7d0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc2fe00700 1 -- 172.21.15.12:0/1978739865 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fbc3c010640 con 0x558ab425b7d0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.715+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc3c010c70 con 0x558ab425b7d0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc2fe00700 1 -- 172.21.15.12:0/1978739865 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc3c011e50 con 0x558ab425b7d0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc6329eb80 10 client.15069 fetch_fsmap learned FSMap version 26 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ab414a010 con 0x558ab425b7d0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc2fe00700 1 -- 172.21.15.12:0/1978739865 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc3c010c70 con 0x558ab425b7d0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc2fe00700 1 -- 172.21.15.12:0/1978739865 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fbc3c005100 con 0x558ab425b7d0 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc6329eb80 10 client.15069 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc6329eb80 10 client.15069 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc6329eb80 20 client.15069 populate_metadata read hostname 'smithi012' 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab4275600 0x558ab4276a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.716+0000 7fbc6329eb80 4 client.15069 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:09.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.717+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ab418bd40 con 0x558ab4275600 2024-05-27T07:21:09.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.717+0000 7fbc51e00700 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab4275600 0x558ab4276a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.718+0000 7fbc51e00700 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab4275600 0x558ab4276a70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1120387270680f7 server_cookie=fb4d6aefc21d8d49 in_seq=0 out_seq=0 2024-05-27T07:21:09.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.718+0000 7fbc2fe00700 10 client.15069 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:21:09.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.721+0000 7fbc2fe00700 1 -- 172.21.15.12:0/1978739865 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558ab418bd40 con 0x558ab4275600 2024-05-27T07:21:09.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.721+0000 7fbc2fe00700 10 client.15069 handle_command_reply: tid=0 2024-05-27T07:21:09.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.721+0000 7fbc6329eb80 1 client.15069 shutdown 2024-05-27T07:21:09.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.721+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 msgr2=0x558ab4265f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.721+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4265f80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00bfd0 tx=0x7fbc3c00b3a0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 shutdown_connections 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab4275600 0x558ab4276a70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4265500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4265f80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc2fe00700 0 client.15069 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/1978739865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4265a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 >> 172.21.15.12:0/1978739865 conn(0x558ab413cf70 msgr2=0x558ab4255fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 shutdown_connections 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 1 -- 172.21.15.12:0/1978739865 wait complete. 2024-05-27T07:21:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.722+0000 7fbc6329eb80 20 client.15069 trim_cache size 0 max 16384 2024-05-27T07:21:09.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 Processor -- start 2024-05-27T07:21:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 -- start start 2024-05-27T07:21:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4260ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4271be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558ab4274db0 con 0x558ab425b7d0 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558ab4275480 con 0x558ab40dcdf0 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.746+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558ab4275b50 con 0x558ab425b3c0 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37562/0 (socket says 172.21.15.12:37562) 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 -- 172.21.15.12:0/789939414 learned_addr learned my addr 172.21.15.12:0/789939414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51e00700 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4271be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4260ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 -- 172.21.15.12:0/789939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 msgr2=0x558ab4271be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4271be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 -- 172.21.15.12:0/789939414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 msgr2=0x558ab4260ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4260ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.747+0000 7fbc51400700 1 -- 172.21.15.12:0/789939414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ab4258300 con 0x558ab425b7d0 2024-05-27T07:21:09.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.748+0000 7fbc51400700 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267860 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00b480 tx=0x7fbc3c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3711f2179a12bd14 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.748+0000 7fbc3aa00700 1 -- 172.21.15.12:0/789939414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c03a070 con 0x558ab425b7d0 2024-05-27T07:21:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.748+0000 7fbc3aa00700 1 -- 172.21.15.12:0/789939414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc3c011e50 con 0x558ab425b7d0 2024-05-27T07:21:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.748+0000 7fbc3aa00700 1 -- 172.21.15.12:0/789939414 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c004a50 con 0x558ab425b7d0 2024-05-27T07:21:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 -- 172.21.15.12:0/789939414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 msgr2=0x558ab4267860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267860 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00b480 tx=0x7fbc3c00b410 comp rx=0 tx=0).stop 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 -- 172.21.15.12:0/789939414 shutdown_connections 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab40dcdf0 0x558ab4260ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab425b7d0 0x558ab4267860 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/789939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab4271be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 -- 172.21.15.12:0/789939414 >> 172.21.15.12:0/789939414 conn(0x558ab413cf70 msgr2=0x558ab4274640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 -- 172.21.15.12:0/789939414 shutdown_connections 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.749+0000 7fbc6329eb80 1 -- 172.21.15.12:0/789939414 wait complete. 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.750+0000 7fbc6329eb80 1 -- start start 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.750+0000 7fbc6329eb80 1 Processor -- start 2024-05-27T07:21:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.750+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab40dcdf0 0x558ab4280480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.750+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab42797c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc6329eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b7d0 0x558ab4279d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558ab42768f0 con 0x558ab40dcdf0 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558ab4276220 con 0x558ab425b7d0 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc6329eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558ab4276fc0 con 0x558ab425b3c0 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc6329eb80 10 client.0 _ll_register_callbacks cb 0x558ab4123c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab40dcdf0 0x558ab4280480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab40dcdf0 0x558ab4280480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37564/0 (socket says 172.21.15.12:37564) 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 -- 172.21.15.12:0/3976176005 learned_addr learned my addr 172.21.15.12:0/3976176005 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51e00700 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab42797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 -- 172.21.15.12:0/3976176005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 msgr2=0x558ab42797c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab42797c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 -- 172.21.15.12:0/3976176005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b7d0 msgr2=0x558ab4279d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b7d0 0x558ab4279d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51400700 1 -- 172.21.15.12:0/3976176005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ab42662f0 con 0x558ab40dcdf0 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.751+0000 7fbc51e00700 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab42797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc51400700 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab40dcdf0 0x558ab4280480 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c038040 tx=0x7fbc3c00bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34cfe01023bfae18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc2fe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc2fe00700 1 -- 172.21.15.12:0/3976176005 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c03a070 con 0x558ab40dcdf0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc2fe00700 1 -- 172.21.15.12:0/3976176005 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fbc3c010910 con 0x558ab40dcdf0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc3c011e50 con 0x558ab40dcdf0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc2fe00700 1 -- 172.21.15.12:0/3976176005 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc3c046050 con 0x558ab40dcdf0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc6329eb80 10 client.15075 fetch_fsmap learned FSMap version 26 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.752+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ab427eb60 con 0x558ab40dcdf0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc2fe00700 1 -- 172.21.15.12:0/3976176005 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbc3c011e50 con 0x558ab40dcdf0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc2fe00700 1 -- 172.21.15.12:0/3976176005 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fbc3c03e070 con 0x558ab40dcdf0 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc6329eb80 10 client.15075 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc6329eb80 10 client.15075 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc6329eb80 20 client.15075 populate_metadata read hostname 'smithi012' 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab42e27a0 0x558ab42e3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc6329eb80 4 client.15075 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:21:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.753+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "status"}) -- 0x558ab4185520 con 0x558ab42e27a0 2024-05-27T07:21:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.754+0000 7fbc51e00700 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab42e27a0 0x558ab42e3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.755+0000 7fbc51e00700 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab42e27a0 0x558ab42e3c10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=313a440c0baa8543 server_cookie=bddc85b6ba30ac6 in_seq=0 out_seq=0 2024-05-27T07:21:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.755+0000 7fbc2fe00700 10 client.15075 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.755+0000 7fbc2fe00700 1 -- 172.21.15.12:0/3976176005 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558ab4185520 con 0x558ab42e27a0 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.755+0000 7fbc2fe00700 10 client.15075 handle_command_reply: tid=0 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.755+0000 7fbc6329eb80 1 client.15075 shutdown 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab40dcdf0 msgr2=0x558ab4280480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab40dcdf0 0x558ab4280480 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c038040 tx=0x7fbc3c00bae0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 shutdown_connections 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558ab42e27a0 0x558ab42e3c10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558ab425b7d0 0x558ab4279d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558ab40dcdf0 0x558ab4280480 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 --2- 172.21.15.12:0/3976176005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558ab425b3c0 0x558ab42797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc2fe00700 0 client.15075 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 >> 172.21.15.12:0/3976176005 conn(0x558ab413cf70 msgr2=0x558ab4274640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 shutdown_connections 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 1 -- 172.21.15.12:0/3976176005 wait complete. 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.756+0000 7fbc6329eb80 20 client.15075 trim_cache size 0 max 16384 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 34415, 2024-05-27T07:21:09.749 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 153.81690021200001, 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 26, 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 89, 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 289.47153451999998 2024-05-27T07:21:09.750 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbc340b2880 msgr2=0x7fbc340b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbc340b2880 0x7fbc340b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbc48089490 tx=0x7fbc4800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 msgr2=0x7fbc4c1a1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c1a1550 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009850 tx=0x7fbc48007e40 comp rx=0 tx=0).stop 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 shutdown_connections 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fbc340b2880 0x7fbc340b4d90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc4c109670 0x7fbc4c1a1550 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc4c1043f0 0x7fbc4c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 --2- 172.21.15.12:0/3329998449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108d20 0x7fbc4c19d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 >> 172.21.15.12:0/3329998449 conn(0x7fbc4c079d00 msgr2=0x7fbc4c101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:09.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 shutdown_connections 2024-05-27T07:21:09.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:09.757+0000 7fbc53200700 1 -- 172.21.15.12:0/3329998449 wait complete. 2024-05-27T07:21:10.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-05-27T07:21:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:11.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:10 smithi162 ceph-mon[35676]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:11.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:10 smithi062 ceph-mon[35304]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:11.236 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:10 smithi012 ceph-mon[31473]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:13.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:12 smithi162 ceph-mon[35676]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:13.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:12 smithi062 ceph-mon[35304]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:13.272 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:12 smithi012 ceph-mon[31473]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 -- 172.21.15.12:0/4099365133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075af0 msgr2=0x7fdc6c075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 --2- 172.21.15.12:0/4099365133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075af0 0x7fdc6c075f00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009990 tx=0x7fdc60009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 -- 172.21.15.12:0/4099365133 shutdown_connections 2024-05-27T07:21:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 --2- 172.21.15.12:0/4099365133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c110bf0 0x7fdc6c113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 --2- 172.21.15.12:0/4099365133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075af0 0x7fdc6c075f00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 --2- 172.21.15.12:0/4099365133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076440 0x7fdc6c1106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 -- 172.21.15.12:0/4099365133 >> 172.21.15.12:0/4099365133 conn(0x7fdc6c100030 msgr2=0x7fdc6c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 -- 172.21.15.12:0/4099365133 shutdown_connections 2024-05-27T07:21:13.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.537+0000 7fdc73400700 1 -- 172.21.15.12:0/4099365133 wait complete. 2024-05-27T07:21:13.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.538+0000 7fdc73400700 1 Processor -- start 2024-05-27T07:21:13.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.539+0000 7fdc73400700 1 -- start start 2024-05-27T07:21:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.539+0000 7fdc73400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c075af0 0x7fdc6c19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.539+0000 7fdc73400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076440 0x7fdc6c19d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.539+0000 7fdc73400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c110bf0 0x7fdc6c1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.539+0000 7fdc73400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1165d0 con 0x7fdc6c110bf0 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.540+0000 7fdc73400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c116450 con 0x7fdc6c076440 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.540+0000 7fdc73400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c116750 con 0x7fdc6c075af0 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.540+0000 7fdc71400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c110bf0 0x7fdc6c1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.540+0000 7fdc71400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c110bf0 0x7fdc6c1a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46292/0 (socket says 172.21.15.12:46292) 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.540+0000 7fdc71400700 1 -- 172.21.15.12:0/2203028498 learned_addr learned my addr 172.21.15.12:0/2203028498 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.540+0000 7fdc70a00700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c075af0 0x7fdc6c19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.540+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076440 0x7fdc6c19d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc71400700 1 -- 172.21.15.12:0/2203028498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c075af0 msgr2=0x7fdc6c19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc71400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c075af0 0x7fdc6c19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc71400700 1 -- 172.21.15.12:0/2203028498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076440 msgr2=0x7fdc6c19d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc71400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076440 0x7fdc6c19d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc71400700 1 -- 172.21.15.12:0/2203028498 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c1a1e50 con 0x7fdc6c110bf0 2024-05-27T07:21:13.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076440 0x7fdc6c19d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:13.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc71400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c110bf0 0x7fdc6c1a17b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdc640094f0 tx=0x7fdc6400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc69600700 1 -- 172.21.15.12:0/2203028498 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400ddd0 con 0x7fdc6c110bf0 2024-05-27T07:21:13.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc69600700 1 -- 172.21.15.12:0/2203028498 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc64009e20 con 0x7fdc6c110bf0 2024-05-27T07:21:13.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6c1a2140 con 0x7fdc6c110bf0 2024-05-27T07:21:13.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.541+0000 7fdc69600700 1 -- 172.21.15.12:0/2203028498 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64015400 con 0x7fdc6c110bf0 2024-05-27T07:21:13.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.542+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6c1a9a60 con 0x7fdc6c110bf0 2024-05-27T07:21:13.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.547+0000 7fdc69600700 1 -- 172.21.15.12:0/2203028498 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fdc64005020 con 0x7fdc6c110bf0 2024-05-27T07:21:13.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.548+0000 7fdc81e7fb80 1 Processor -- start 2024-05-27T07:21:13.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.548+0000 7fdc81e7fb80 1 -- start start 2024-05-27T07:21:13.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ec6e9a0 0x56246ed012e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246eb82df0 0x56246ed01820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ed01e20 0x56246ed0e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56246ec2fdf0 con 0x56246ec6e9a0 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56246ec1f990 con 0x56246eb82df0 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56246ec239d0 con 0x56246ed01e20 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ec6e9a0 0x56246ed012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ec6e9a0 0x56246ed012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46298/0 (socket says 172.21.15.12:46298) 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc6be00700 1 -- 172.21.15.12:0/2239185464 learned_addr learned my addr 172.21.15.12:0/2239185464 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:13.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.549+0000 7fdc70a00700 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246eb82df0 0x56246ed01820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ed01e20 0x56246ed0e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc69600700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdc540b2880 0x7fdc540b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc6be00700 1 -- 172.21.15.12:0/2239185464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ed01e20 msgr2=0x56246ed0e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ed01e20 0x56246ed0e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc6be00700 1 -- 172.21.15.12:0/2239185464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246eb82df0 msgr2=0x56246ed01820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246eb82df0 0x56246ed01820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc6be00700 1 -- 172.21.15.12:0/2239185464 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56246ebe83d0 con 0x56246ec6e9a0 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc71400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdc540b2880 0x7fdc540b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc69600700 1 -- 172.21.15.12:0/2203028498 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdc640e2070 con 0x7fdc6c110bf0 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.550+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ec6e9a0 0x56246ed012e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdc580135e0 tx=0x7fdc5800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b560ffb915fc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.551+0000 7fdc5e000700 1 -- 172.21.15.12:0/2239185464 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58019030 con 0x56246ec6e9a0 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.551+0000 7fdc5e000700 1 -- 172.21.15.12:0/2239185464 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc58013e40 con 0x56246ec6e9a0 2024-05-27T07:21:13.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.551+0000 7fdc5e000700 1 -- 172.21.15.12:0/2239185464 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58003d50 con 0x56246ec6e9a0 2024-05-27T07:21:13.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2239185464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ec6e9a0 msgr2=0x56246ed012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ec6e9a0 0x56246ed012e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdc580135e0 tx=0x7fdc5800b730 comp rx=0 tx=0).stop 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc71400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdc540b2880 0x7fdc540b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdc640a52e0 tx=0x7fdc6400df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2239185464 shutdown_connections 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ed01e20 0x56246ed0e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ec6e9a0 0x56246ed012e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2239185464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246eb82df0 0x56246ed01820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2239185464 >> 172.21.15.12:0/2239185464 conn(0x56246ec1c340 msgr2=0x56246ecfd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2239185464 shutdown_connections 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.552+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2239185464 wait complete. 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.553+0000 7fdc81e7fb80 1 -- start start 2024-05-27T07:21:13.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.553+0000 7fdc81e7fb80 1 Processor -- start 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed0b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed0bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 0x56246ed0c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56246eb25f50 con 0x56246eb82df0 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56246ebc0710 con 0x56246ed01e20 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56246ed0eb80 con 0x56246ec6e9a0 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc81e7fb80 10 client.0 _ll_register_callbacks cb 0x56246ebc9c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed0b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed0b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46312/0 (socket says 172.21.15.12:46312) 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc6be00700 1 -- 172.21.15.12:0/3058235229 learned_addr learned my addr 172.21.15.12:0/3058235229 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.554+0000 7fdc70a00700 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed0bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc6be00700 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 0x56246ed0c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc6be00700 1 -- 172.21.15.12:0/3058235229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 msgr2=0x56246ed0bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc6be00700 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed0bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc6be00700 1 -- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 msgr2=0x56246ed0c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc6be00700 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 0x56246ed0c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc6be00700 1 -- 172.21.15.12:0/3058235229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56246eb6f880 con 0x56246eb82df0 2024-05-27T07:21:13.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc6be00700 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed0b630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdc580038d0 tx=0x7fdc58005c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a7941d6d3d9aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc52a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:21:13.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.555+0000 7fdc52a00700 1 -- 172.21.15.12:0/3058235229 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58019040 con 0x56246eb82df0 2024-05-27T07:21:13.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.556+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56246ec31d40 con 0x56246eb82df0 2024-05-27T07:21:13.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.556+0000 7fdc52a00700 1 -- 172.21.15.12:0/3058235229 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdc58003b30 con 0x56246eb82df0 2024-05-27T07:21:13.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.556+0000 7fdc52a00700 1 -- 172.21.15.12:0/3058235229 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdc58032050 con 0x56246eb82df0 2024-05-27T07:21:13.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.556+0000 7fdc81e7fb80 10 client.15087 fetch_fsmap learned FSMap version 26 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.556+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56246ebf0010 con 0x56246eb82df0 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.556+0000 7fdc52a00700 1 -- 172.21.15.12:0/3058235229 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56246ec31d40 con 0x56246eb82df0 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc52a00700 1 -- 172.21.15.12:0/3058235229 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fdc58004750 con 0x56246eb82df0 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc81e7fb80 10 client.15087 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc81e7fb80 10 client.15087 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc81e7fb80 20 client.15087 populate_metadata read hostname 'smithi012' 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed1b870 0x56246ed1cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc81e7fb80 4 client.15087 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56246ec2b520 con 0x56246ed1b870 2024-05-27T07:21:13.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.557+0000 7fdc70a00700 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed1b870 0x56246ed1cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.559+0000 7fdc70a00700 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed1b870 0x56246ed1cce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b77d31a29a9949e0 server_cookie=7463b805abffc1b8 in_seq=0 out_seq=0 2024-05-27T07:21:13.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.559+0000 7fdc52a00700 10 client.15087 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:21:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc52a00700 1 -- 172.21.15.12:0/3058235229 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56246ec2b520 con 0x56246ed1b870 2024-05-27T07:21:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc52a00700 10 client.15087 handle_command_reply: tid=0 2024-05-27T07:21:13.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 client.15087 shutdown 2024-05-27T07:21:13.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 msgr2=0x56246ed0b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed0b630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdc580038d0 tx=0x7fdc58005c40 comp rx=0 tx=0).stop 2024-05-27T07:21:13.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 shutdown_connections 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed1b870 0x56246ed1cce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 0x56246ed0c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed0b630 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc52a00700 0 client.15087 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/3058235229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed0bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.561+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 >> 172.21.15.12:0/3058235229 conn(0x56246ec1c340 msgr2=0x56246ed01320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.562+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 shutdown_connections 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.562+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/3058235229 wait complete. 2024-05-27T07:21:13.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.562+0000 7fdc81e7fb80 20 client.15087 trim_cache size 0 max 16384 2024-05-27T07:21:13.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.585+0000 7fdc81e7fb80 1 Processor -- start 2024-05-27T07:21:13.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.585+0000 7fdc81e7fb80 1 -- start start 2024-05-27T07:21:13.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246eb82df0 0x56246ed06e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ec6e9a0 0x56246ed0a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ed01e20 0x56246ed01770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56246ed1b020 con 0x56246ed01e20 2024-05-27T07:21:13.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56246ed1b6f0 con 0x56246ec6e9a0 2024-05-27T07:21:13.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56246ed1bdc0 con 0x56246eb82df0 2024-05-27T07:21:13.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ed01e20 0x56246ed01770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.586+0000 7fdc70a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ec6e9a0 0x56246ed0a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc70a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ec6e9a0 0x56246ed0a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:59972/0 (socket says 172.21.15.12:59972) 2024-05-27T07:21:13.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc70a00700 1 -- 172.21.15.12:0/2245270575 learned_addr learned my addr 172.21.15.12:0/2245270575 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:13.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ed01e20 0x56246ed01770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46320/0 (socket says 172.21.15.12:46320) 2024-05-27T07:21:13.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246eb82df0 0x56246ed06e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 -- 172.21.15.12:0/2245270575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246eb82df0 msgr2=0x56246ed06e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246eb82df0 0x56246ed06e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 -- 172.21.15.12:0/2245270575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ec6e9a0 msgr2=0x56246ed0a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ec6e9a0 0x56246ed0a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 -- 172.21.15.12:0/2245270575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56246ecfe800 con 0x56246ed01e20 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.587+0000 7fdc6be00700 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ed01e20 0x56246ed01770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdc58004b50 tx=0x7fdc5800b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422943ee3db1b75 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.588+0000 7fdc5d600700 1 -- 172.21.15.12:0/2245270575 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58004d40 con 0x56246ed01e20 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.588+0000 7fdc5d600700 1 -- 172.21.15.12:0/2245270575 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc58004450 con 0x56246ed01e20 2024-05-27T07:21:13.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.588+0000 7fdc5d600700 1 -- 172.21.15.12:0/2245270575 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58013280 con 0x56246ed01e20 2024-05-27T07:21:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2245270575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ed01e20 msgr2=0x56246ed01770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ed01e20 0x56246ed01770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdc58004b50 tx=0x7fdc5800b3c0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2245270575 shutdown_connections 2024-05-27T07:21:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246ed01e20 0x56246ed01770 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246eb82df0 0x56246ed06e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/2245270575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ec6e9a0 0x56246ed0a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2245270575 >> 172.21.15.12:0/2245270575 conn(0x56246ec1c340 msgr2=0x56246ed01060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2245270575 shutdown_connections 2024-05-27T07:21:13.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/2245270575 wait complete. 2024-05-27T07:21:13.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.589+0000 7fdc81e7fb80 1 -- start start 2024-05-27T07:21:13.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 1 Processor -- start 2024-05-27T07:21:13.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed262a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed26840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 0x56246ed1f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56246ed1cb60 con 0x56246eb82df0 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56246ed1c490 con 0x56246ed01e20 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56246ed1d230 con 0x56246ec6e9a0 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc81e7fb80 10 client.0 _ll_register_callbacks cb 0x56246ebc9c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.590+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed262a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed262a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46330/0 (socket says 172.21.15.12:46330) 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 -- 172.21.15.12:0/4088428085 learned_addr learned my addr 172.21.15.12:0/4088428085 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc70a00700 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed26840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 -- 172.21.15.12:0/4088428085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 msgr2=0x56246ed26840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed26840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 -- 172.21.15.12:0/4088428085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 msgr2=0x56246ed1f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 0x56246ed1f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 -- 172.21.15.12:0/4088428085 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56246ed1fe80 con 0x56246eb82df0 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc70a00700 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed26840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc6be00700 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed262a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc58016040 tx=0x7fdc58004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba3404f671ca6da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.591+0000 7fdc52a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.592+0000 7fdc52a00700 1 -- 172.21.15.12:0/4088428085 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58019070 con 0x56246eb82df0 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.592+0000 7fdc52a00700 1 -- 172.21.15.12:0/4088428085 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdc58013230 con 0x56246eb82df0 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.592+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc58004450 con 0x56246eb82df0 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.592+0000 7fdc52a00700 1 -- 172.21.15.12:0/4088428085 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdc58003900 con 0x56246eb82df0 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.592+0000 7fdc81e7fb80 10 client.15093 fetch_fsmap learned FSMap version 26 2024-05-27T07:21:13.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.592+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56246ed247e0 con 0x56246eb82df0 2024-05-27T07:21:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.592+0000 7fdc52a00700 1 -- 172.21.15.12:0/4088428085 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc58004450 con 0x56246eb82df0 2024-05-27T07:21:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc52a00700 1 -- 172.21.15.12:0/4088428085 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fdc5801e460 con 0x56246eb82df0 2024-05-27T07:21:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc81e7fb80 10 client.15093 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:21:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc81e7fb80 10 client.15093 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:21:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc81e7fb80 20 client.15093 populate_metadata read hostname 'smithi012' 2024-05-27T07:21:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed88bf0 0x56246ed8a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc81e7fb80 4 client.15093 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:21:13.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "status"}) -- 0x56246eaa2200 con 0x56246ed88bf0 2024-05-27T07:21:13.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.593+0000 7fdc70a00700 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed88bf0 0x56246ed8a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.594+0000 7fdc70a00700 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed88bf0 0x56246ed8a060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e2ac96d5d720538 server_cookie=4dd62c8898b40f65 in_seq=0 out_seq=0 2024-05-27T07:21:13.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.594+0000 7fdc52a00700 10 client.15093 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc52a00700 1 -- 172.21.15.12:0/4088428085 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56246eaa2200 con 0x56246ed88bf0 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc52a00700 10 client.15093 handle_command_reply: tid=0 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 client.15093 shutdown 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 msgr2=0x56246ed262a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed262a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc58016040 tx=0x7fdc58004df0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 shutdown_connections 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56246ed88bf0 0x56246ed8a060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56246ed01e20 0x56246ed1f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56246eb82df0 0x56246ed262a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.595+0000 7fdc81e7fb80 1 --2- 172.21.15.12:0/4088428085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56246ec6e9a0 0x56246ed26840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.596+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 >> 172.21.15.12:0/4088428085 conn(0x56246ec1c340 msgr2=0x56246ed16e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.596+0000 7fdc52a00700 0 client.15093 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:21:13.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.596+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 shutdown_connections 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.596+0000 7fdc81e7fb80 1 -- 172.21.15.12:0/4088428085 wait complete. 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.596+0000 7fdc81e7fb80 20 client.15093 trim_cache size 0 max 16384 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 34439, 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 158.68285809100001, 2024-05-27T07:21:13.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 26, 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 89, 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 280.99185677200001 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdc540b2880 msgr2=0x7fdc540b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdc540b2880 0x7fdc540b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdc640a52e0 tx=0x7fdc6400df90 comp rx=0 tx=0).stop 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c110bf0 msgr2=0x7fdc6c1a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c110bf0 0x7fdc6c1a17b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdc640094f0 tx=0x7fdc6400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 shutdown_connections 2024-05-27T07:21:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdc540b2880 0x7fdc540b4d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c110bf0 0x7fdc6c1a17b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c075af0 0x7fdc6c19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 --2- 172.21.15.12:0/2203028498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076440 0x7fdc6c19d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 >> 172.21.15.12:0/2203028498 conn(0x7fdc6c100030 msgr2=0x7fdc6c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 shutdown_connections 2024-05-27T07:21:13.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:13.597+0000 7fdc73400700 1 -- 172.21.15.12:0/2203028498 wait complete. 2024-05-27T07:21:14.592 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:21:15.061 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:15.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:14 smithi012 ceph-mon[31473]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:15.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:14 smithi162 ceph-mon[35676]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:15.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:14 smithi062 ceph-mon[35304]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:17.146 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:16 smithi012 ceph-mon[31473]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:17.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:16 smithi162 ceph-mon[35676]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:17.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:16 smithi062 ceph-mon[35304]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.516+0000 7f1a5da00700 1 -- 172.21.15.12:0/1080374540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58110c60 msgr2=0x7f1a58113080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.516+0000 7f1a5da00700 1 --2- 172.21.15.12:0/1080374540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58110c60 0x7f1a58113080 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000b0a0 tx=0x7f1a5000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.517+0000 7f1a5da00700 1 -- 172.21.15.12:0/1080374540 shutdown_connections 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.517+0000 7f1a5da00700 1 --2- 172.21.15.12:0/1080374540 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58110c60 0x7f1a58113080 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.517+0000 7f1a5da00700 1 --2- 172.21.15.12:0/1080374540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a581030b0 0x7f1a581034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.517+0000 7f1a5da00700 1 --2- 172.21.15.12:0/1080374540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a58103a00 0x7f1a58110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.517+0000 7f1a5da00700 1 -- 172.21.15.12:0/1080374540 >> 172.21.15.12:0/1080374540 conn(0x7f1a580fedb0 msgr2=0x7f1a581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.517+0000 7f1a5da00700 1 -- 172.21.15.12:0/1080374540 shutdown_connections 2024-05-27T07:21:17.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.517+0000 7f1a5da00700 1 -- 172.21.15.12:0/1080374540 wait complete. 2024-05-27T07:21:17.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.519+0000 7f1a5da00700 1 Processor -- start 2024-05-27T07:21:17.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.519+0000 7f1a5da00700 1 -- start start 2024-05-27T07:21:17.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.520+0000 7f1a5da00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a581030b0 0x7f1a5819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.520+0000 7f1a5da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58103a00 0x7f1a5819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.520+0000 7f1a5da00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a58110c60 0x7f1a581a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.520+0000 7f1a5da00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58116570 con 0x7f1a58103a00 2024-05-27T07:21:17.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.520+0000 7f1a5da00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581163f0 con 0x7f1a58110c60 2024-05-27T07:21:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.520+0000 7f1a5da00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581166f0 con 0x7f1a581030b0 2024-05-27T07:21:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.520+0000 7f1a56000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58103a00 0x7f1a5819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58103a00 0x7f1a5819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46356/0 (socket says 172.21.15.12:46356) 2024-05-27T07:21:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 -- 172.21.15.12:0/115815907 learned_addr learned my addr 172.21.15.12:0/115815907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56a00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a581030b0 0x7f1a5819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a57400700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a58110c60 0x7f1a581a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 -- 172.21.15.12:0/115815907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a581030b0 msgr2=0x7f1a5819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a581030b0 0x7f1a5819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 -- 172.21.15.12:0/115815907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a58110c60 msgr2=0x7f1a581a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a58110c60 0x7f1a581a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 -- 172.21.15.12:0/115815907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a581a1d90 con 0x7f1a58103a00 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.521+0000 7f1a56000700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58103a00 0x7f1a5819d460 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400d980 tx=0x7f1a4400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.522+0000 7f1a4f400700 1 -- 172.21.15.12:0/115815907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44017070 con 0x7f1a58103a00 2024-05-27T07:21:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.522+0000 7f1a4f400700 1 -- 172.21.15.12:0/115815907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1a44010460 con 0x7f1a58103a00 2024-05-27T07:21:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.522+0000 7f1a4f400700 1 -- 172.21.15.12:0/115815907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4400de60 con 0x7f1a58103a00 2024-05-27T07:21:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.522+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a581a2080 con 0x7f1a58103a00 2024-05-27T07:21:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.522+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a581a99a0 con 0x7f1a58103a00 2024-05-27T07:21:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 Processor -- start 2024-05-27T07:21:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 -- start start 2024-05-27T07:21:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a4f400700 1 -- 172.21.15.12:0/115815907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f1a44010610 con 0x7f1a58103a00 2024-05-27T07:21:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fb6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fb6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fcb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557591ee4df0 con 0x557591f239a0 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557591ed4990 con 0x557591e37df0 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.527+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557591ed89d0 con 0x557591fb6d60 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fb6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fb6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46372/0 (socket says 172.21.15.12:46372) 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 -- 172.21.15.12:0/3325478389 learned_addr learned my addr 172.21.15.12:0/3325478389 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a56a00700 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fb6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a56000700 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fcb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 -- 172.21.15.12:0/3325478389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 msgr2=0x557591fcb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fcb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 -- 172.21.15.12:0/3325478389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 msgr2=0x557591fb6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fb6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.528+0000 7f1a57400700 1 -- 172.21.15.12:0/3325478389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557591e9d3d0 con 0x557591f239a0 2024-05-27T07:21:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.529+0000 7f1a57400700 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fb6200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009d10 tx=0x7f1a50009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d026593fcae74f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.529+0000 7f1a4f400700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1a3c0b2880 0x7f1a3c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.529+0000 7f1a43e00700 1 -- 172.21.15.12:0/3325478389 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50009450 con 0x557591f239a0 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.529+0000 7f1a4f400700 1 -- 172.21.15.12:0/115815907 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f1a440e2150 con 0x7f1a58103a00 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.529+0000 7f1a43e00700 1 -- 172.21.15.12:0/3325478389 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1a500075c0 con 0x557591f239a0 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.529+0000 7f1a43e00700 1 -- 172.21.15.12:0/3325478389 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5001c6a0 con 0x557591f239a0 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.529+0000 7f1a56a00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1a3c0b2880 0x7f1a3c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/3325478389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 msgr2=0x557591fb6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fb6200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009d10 tx=0x7f1a50009d40 comp rx=0 tx=0).stop 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a56a00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1a3c0b2880 0x7f1a3c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1a48005950 tx=0x7f1a480058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/3325478389 shutdown_connections 2024-05-27T07:21:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fcb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fb6200 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/3325478389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fb6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/3325478389 >> 172.21.15.12:0/3325478389 conn(0x557591ed1340 msgr2=0x557591fb2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/3325478389 shutdown_connections 2024-05-27T07:21:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.530+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/3325478389 wait complete. 2024-05-27T07:21:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.531+0000 7f1a6c4c3b80 1 -- start start 2024-05-27T07:21:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.531+0000 7f1a6c4c3b80 1 Processor -- start 2024-05-27T07:21:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fc6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591f239a0 0x557591fc6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fc73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557591ddaf50 con 0x557591e37df0 2024-05-27T07:21:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557591e75710 con 0x557591f239a0 2024-05-27T07:21:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557591fcc330 con 0x557591fb6d60 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a6c4c3b80 10 client.0 _ll_register_callbacks cb 0x557591e7ec20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a56a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fc6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a56a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fc6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46380/0 (socket says 172.21.15.12:46380) 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a56a00700 1 -- 172.21.15.12:0/1182283718 learned_addr learned my addr 172.21.15.12:0/1182283718 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a56000700 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591f239a0 0x557591fc6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.532+0000 7f1a57400700 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fc73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a56a00700 1 -- 172.21.15.12:0/1182283718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 msgr2=0x557591fc73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a56a00700 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fc73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a56a00700 1 -- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591f239a0 msgr2=0x557591fc6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a56a00700 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591f239a0 0x557591fc6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a56a00700 1 -- 172.21.15.12:0/1182283718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557591e24880 con 0x557591e37df0 2024-05-27T07:21:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a56a00700 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fc6920 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1a480042b0 tx=0x7f1a48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f1407903bcdecf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a40c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a40c00700 1 -- 172.21.15.12:0/1182283718 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48003c90 con 0x557591e37df0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a500075c0 con 0x557591e37df0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.533+0000 7f1a40c00700 1 -- 172.21.15.12:0/1182283718 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f1a48012450 con 0x557591e37df0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a40c00700 1 -- 172.21.15.12:0/1182283718 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1a4802e050 con 0x557591e37df0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 10 client.15105 fetch_fsmap learned FSMap version 26 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557591ea5010 con 0x557591e37df0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a40c00700 1 -- 172.21.15.12:0/1182283718 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1a500075c0 con 0x557591e37df0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a40c00700 1 -- 172.21.15.12:0/1182283718 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f1a48020430 con 0x557591e37df0 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 10 client.15105 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 10 client.15105 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:21:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 20 client.15105 populate_metadata read hostname 'smithi012' 2024-05-27T07:21:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557591fd93a0 0x557591fda810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 4 client.15105 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.534+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557591ee6d40 con 0x557591fd93a0 2024-05-27T07:21:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.535+0000 7f1a56000700 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557591fd93a0 0x557591fda810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.536+0000 7f1a40c00700 10 client.15105 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:21:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.536+0000 7f1a56000700 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557591fd93a0 0x557591fda810 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1f456ded917ab21 server_cookie=c35c7bdbcc2078d4 in_seq=0 out_seq=0 2024-05-27T07:21:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.538+0000 7f1a40c00700 1 -- 172.21.15.12:0/1182283718 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557591ee6d40 con 0x557591fd93a0 2024-05-27T07:21:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.538+0000 7f1a40c00700 10 client.15105 handle_command_reply: tid=0 2024-05-27T07:21:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.538+0000 7f1a6c4c3b80 1 client.15105 shutdown 2024-05-27T07:21:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.538+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 msgr2=0x557591fc6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.538+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fc6920 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1a480042b0 tx=0x7f1a48004750 comp rx=0 tx=0).stop 2024-05-27T07:21:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 shutdown_connections 2024-05-27T07:21:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fc73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fc6920 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557591fd93a0 0x557591fda810 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1182283718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591f239a0 0x557591fc6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 >> 172.21.15.12:0/1182283718 conn(0x557591ed1340 msgr2=0x557591fb1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a40c00700 0 client.15105 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 shutdown_connections 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1182283718 wait complete. 2024-05-27T07:21:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.539+0000 7f1a6c4c3b80 20 client.15105 trim_cache size 0 max 16384 2024-05-27T07:21:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 Processor -- start 2024-05-27T07:21:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 -- start start 2024-05-27T07:21:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fb35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591f239a0 0x557591fc5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591fb6d60 0x557591fc5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557591fd8b50 con 0x557591e37df0 2024-05-27T07:21:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557591fd9220 con 0x557591fb6d60 2024-05-27T07:21:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.562+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557591fd98f0 con 0x557591f239a0 2024-05-27T07:21:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fb35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fb35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46382/0 (socket says 172.21.15.12:46382) 2024-05-27T07:21:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 -- 172.21.15.12:0/711358195 learned_addr learned my addr 172.21.15.12:0/711358195 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a57400700 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591fb6d60 0x557591fc5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 -- 172.21.15.12:0/711358195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591f239a0 msgr2=0x557591fc5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591f239a0 0x557591fc5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 -- 172.21.15.12:0/711358195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591fb6d60 msgr2=0x557591fc5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591fb6d60 0x557591fc5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.563+0000 7f1a56a00700 1 -- 172.21.15.12:0/711358195 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557591fc7710 con 0x557591e37df0 2024-05-27T07:21:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.564+0000 7f1a56a00700 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fb35b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1a48020850 tx=0x7f1a4800b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33777174ffc8a61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.564+0000 7f1a43400700 1 -- 172.21.15.12:0/711358195 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48020430 con 0x557591e37df0 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.564+0000 7f1a43400700 1 -- 172.21.15.12:0/711358195 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1a48016070 con 0x557591e37df0 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.564+0000 7f1a43400700 1 -- 172.21.15.12:0/711358195 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48005350 con 0x557591e37df0 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/711358195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 msgr2=0x557591fb35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fb35b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1a48020850 tx=0x7f1a4800b3a0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/711358195 shutdown_connections 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591fb6d60 0x557591fc5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591e37df0 0x557591fb35b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/711358195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591f239a0 0x557591fc5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/711358195 >> 172.21.15.12:0/711358195 conn(0x557591ed1340 msgr2=0x557591fb1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/711358195 shutdown_connections 2024-05-27T07:21:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.565+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/711358195 wait complete. 2024-05-27T07:21:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 -- start start 2024-05-27T07:21:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 Processor -- start 2024-05-27T07:21:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fde210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fde750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fe3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557591fda690 con 0x557591f239a0 2024-05-27T07:21:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557591fd9fc0 con 0x557591e37df0 2024-05-27T07:21:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557591fdad60 con 0x557591fb6d60 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.566+0000 7f1a6c4c3b80 10 client.0 _ll_register_callbacks cb 0x557591e7ec20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fde750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fde750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46398/0 (socket says 172.21.15.12:46398) 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 -- 172.21.15.12:0/1326804799 learned_addr learned my addr 172.21.15.12:0/1326804799 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56a00700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fde210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a57400700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fe3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 -- 172.21.15.12:0/1326804799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 msgr2=0x557591fe3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fe3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 -- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 msgr2=0x557591fde210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fde210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 -- 172.21.15.12:0/1326804799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557591fdf000 con 0x557591f239a0 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56a00700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fde210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a56000700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fde750 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1a440df820 tx=0x7f1a440a8ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5e13ac0345802 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.567+0000 7f1a40c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a40c00700 1 -- 172.21.15.12:0/1326804799 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44020040 con 0x557591f239a0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a48016070 con 0x557591f239a0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a40c00700 1 -- 172.21.15.12:0/1326804799 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f1a440a97b0 con 0x557591f239a0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a40c00700 1 -- 172.21.15.12:0/1326804799 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1a440a9a40 con 0x557591f239a0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a6c4c3b80 10 client.15111 fetch_fsmap learned FSMap version 26 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557591fe4840 con 0x557591f239a0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a40c00700 1 -- 172.21.15.12:0/1326804799 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1a48016070 con 0x557591f239a0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a40c00700 1 -- 172.21.15.12:0/1326804799 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f1a440dfce0 con 0x557591f239a0 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.568+0000 7f1a6c4c3b80 10 client.15111 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.569+0000 7f1a6c4c3b80 10 client.15111 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.569+0000 7f1a6c4c3b80 20 client.15111 populate_metadata read hostname 'smithi012' 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.569+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557592046d40 0x5575920481b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.569+0000 7f1a6c4c3b80 4 client.15111 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:21:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.569+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "status"}) -- 0x557591ee0520 con 0x557592046d40 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.569+0000 7f1a56a00700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557592046d40 0x5575920481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.570+0000 7f1a56a00700 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557592046d40 0x5575920481b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=369080fa37c6332e server_cookie=4db90a3c27baffaa in_seq=0 out_seq=0 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.570+0000 7f1a40c00700 10 client.15111 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a40c00700 1 -- 172.21.15.12:0/1326804799 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557591ee0520 con 0x557592046d40 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a40c00700 10 client.15111 handle_command_reply: tid=0 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 client.15111 shutdown 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 msgr2=0x557591fde750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fde750 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1a440df820 tx=0x7f1a440a8ab0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 shutdown_connections 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557592046d40 0x5575920481b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557591e37df0 0x557591fde210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557591fb6d60 0x557591fe3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 --2- 172.21.15.12:0/1326804799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557591f239a0 0x557591fde750 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 >> 172.21.15.12:0/1326804799 conn(0x557591ed1340 msgr2=0x557591fb1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a40c00700 0 client.15111 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 shutdown_connections 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.571+0000 7f1a6c4c3b80 1 -- 172.21.15.12:0/1326804799 wait complete. 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a6c4c3b80 20 client.15111 trim_cache size 0 max 16384 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:21:17.564 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 34475, 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 246.78444832700001, 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 26, 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 89, 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 267.02830451400001 2024-05-27T07:21:17.565 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:21:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1a3c0b2880 msgr2=0x7f1a3c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1a3c0b2880 0x7f1a3c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1a48005950 tx=0x7f1a480058e0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58103a00 msgr2=0x7f1a5819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58103a00 0x7f1a5819d460 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400d980 tx=0x7f1a4400dd40 comp rx=0 tx=0).stop 2024-05-27T07:21:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 shutdown_connections 2024-05-27T07:21:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f1a3c0b2880 0x7f1a3c0b4d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a58110c60 0x7f1a581a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a581030b0 0x7f1a5819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 --2- 172.21.15.12:0/115815907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a58103a00 0x7f1a5819d460 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.572+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 >> 172.21.15.12:0/115815907 conn(0x7f1a580fedb0 msgr2=0x7f1a58112730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.573+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 shutdown_connections 2024-05-27T07:21:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:17.573+0000 7f1a5da00700 1 -- 172.21.15.12:0/115815907 wait complete. 2024-05-27T07:21:18.428 INFO:tasks.cephadm:Setup complete, yielding 2024-05-27T07:21:18.428 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:21:18.438 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:21:18.439 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-05-27T07:21:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:19.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:18 smithi162 ceph-mon[35676]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:19.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:18 smithi062 ceph-mon[35304]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:19.277 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:18 smithi012 ceph-mon[31473]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:21.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:20 smithi012 ceph-mon[31473]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:21.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:20 smithi162 ceph-mon[35676]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:21.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:20 smithi062 ceph-mon[35304]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:21.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.351+0000 7f0c59c00700 1 -- 172.21.15.12:0/3042954305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104330 msgr2=0x7f0c54104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.351+0000 7f0c59c00700 1 --2- 172.21.15.12:0/3042954305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104330 0x7f0c54104740 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009990 tx=0x7f0c44009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.351+0000 7f0c59c00700 1 -- 172.21.15.12:0/3042954305 shutdown_connections 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.351+0000 7f0c59c00700 1 --2- 172.21.15.12:0/3042954305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c54105e00 0x7f0c5410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.351+0000 7f0c59c00700 1 --2- 172.21.15.12:0/3042954305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54104330 0x7f0c54104740 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.351+0000 7f0c59c00700 1 --2- 172.21.15.12:0/3042954305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c54105460 0x7f0c54105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.351+0000 7f0c59c00700 1 -- 172.21.15.12:0/3042954305 >> 172.21.15.12:0/3042954305 conn(0x7f0c540ff9b0 msgr2=0x7f0c54101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.352+0000 7f0c59c00700 1 -- 172.21.15.12:0/3042954305 shutdown_connections 2024-05-27T07:21:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.352+0000 7f0c59c00700 1 -- 172.21.15.12:0/3042954305 wait complete. 2024-05-27T07:21:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.353+0000 7f0c59c00700 1 Processor -- start 2024-05-27T07:21:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c59c00700 1 -- start start 2024-05-27T07:21:21.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c59c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54105460 0x7f0c54077450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c59c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c54105e00 0x7f0c54075b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c59c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c54076040 0x7f0c54076450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c59c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5410fc70 con 0x7f0c54105460 2024-05-27T07:21:21.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c59c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5410faf0 con 0x7f0c54076040 2024-05-27T07:21:21.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c59c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5410fdf0 con 0x7f0c54105e00 2024-05-27T07:21:21.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c52a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54105460 0x7f0c54077450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.354+0000 7f0c52a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54105460 0x7f0c54077450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46428/0 (socket says 172.21.15.12:46428) 2024-05-27T07:21:21.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c53400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c54076040 0x7f0c54076450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c52a00700 1 -- 172.21.15.12:0/2250596530 learned_addr learned my addr 172.21.15.12:0/2250596530 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:21.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c52000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c54105e00 0x7f0c54075b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c52a00700 1 -- 172.21.15.12:0/2250596530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c54105e00 msgr2=0x7f0c54075b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c52a00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c54105e00 0x7f0c54075b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c52a00700 1 -- 172.21.15.12:0/2250596530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c54076040 msgr2=0x7f0c54076450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c52a00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c54076040 0x7f0c54076450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c52a00700 1 -- 172.21.15.12:0/2250596530 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c541a4c20 con 0x7f0c54105460 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.355+0000 7f0c53400700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c54076040 0x7f0c54076450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.356+0000 7f0c52a00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54105460 0x7f0c54077450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0c440098e0 tx=0x7f0c4400b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:21.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.356+0000 7f0c4b400700 1 -- 172.21.15.12:0/2250596530 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c44026070 con 0x7f0c54105460 2024-05-27T07:21:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.356+0000 7f0c4b400700 1 -- 172.21.15.12:0/2250596530 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c44003d20 con 0x7f0c54105460 2024-05-27T07:21:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.356+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c541a4eb0 con 0x7f0c54105460 2024-05-27T07:21:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.356+0000 7f0c4b400700 1 -- 172.21.15.12:0/2250596530 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4400fe10 con 0x7f0c54105460 2024-05-27T07:21:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.356+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c541a5250 con 0x7f0c54105460 2024-05-27T07:21:21.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.358+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c540512b0 con 0x7f0c54105460 2024-05-27T07:21:21.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.361+0000 7f0c4b400700 1 -- 172.21.15.12:0/2250596530 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0c44009d10 con 0x7f0c54105460 2024-05-27T07:21:21.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.363+0000 7f0c4b400700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0c380b2900 0x7f0c380b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.364+0000 7f0c4b400700 1 -- 172.21.15.12:0/2250596530 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f0c440e3930 con 0x7f0c54105460 2024-05-27T07:21:21.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.364+0000 7f0c52000700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0c380b2900 0x7f0c380b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.365+0000 7f0c52000700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0c380b2900 0x7f0c380b4e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009c60 tx=0x7f0c40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:21.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.367+0000 7f0c4b400700 1 -- 172.21.15.12:0/2250596530 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0c44013070 con 0x7f0c54105460 2024-05-27T07:21:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.636+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f0c54063bf0 con 0x7f0c380b2900 2024-05-27T07:21:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.638+0000 7f0c4b400700 1 -- 172.21.15.12:0/2250596530 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f0c54063bf0 con 0x7f0c380b2900 2024-05-27T07:21:21.629 INFO:teuthology.orchestra.run.smithi012.stdout:Backend: cephadm 2024-05-27T07:21:21.630 INFO:teuthology.orchestra.run.smithi012.stdout:Available: Yes 2024-05-27T07:21:21.630 INFO:teuthology.orchestra.run.smithi012.stdout:Paused: No 2024-05-27T07:21:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0c380b2900 msgr2=0x7f0c380b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0c380b2900 0x7f0c380b4e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009c60 tx=0x7f0c40009260 comp rx=0 tx=0).stop 2024-05-27T07:21:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54105460 msgr2=0x7f0c54077450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54105460 0x7f0c54077450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0c440098e0 tx=0x7f0c4400b9c0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 shutdown_connections 2024-05-27T07:21:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f0c380b2900 0x7f0c380b4e10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c54076040 0x7f0c54076450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c54105460 0x7f0c54077450 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 --2- 172.21.15.12:0/2250596530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c54105e00 0x7f0c54075b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.642+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 >> 172.21.15.12:0/2250596530 conn(0x7f0c540ff9b0 msgr2=0x7f0c541087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.643+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 shutdown_connections 2024-05-27T07:21:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:21.643+0000 7f0c59c00700 1 -- 172.21.15.12:0/2250596530 wait complete. 2024-05-27T07:21:22.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:21 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:22.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:21 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:22.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:21 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:22.635 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-27T07:21:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:23.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:22 smithi162 ceph-mon[35676]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:22 smithi162 ceph-mon[35676]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:23.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:22 smithi162 ceph-mon[35676]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:21:23.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:22 smithi062 ceph-mon[35304]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:22 smithi062 ceph-mon[35304]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:23.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:22 smithi062 ceph-mon[35304]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:21:23.250 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:22 smithi012 ceph-mon[31473]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:23.250 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:22 smithi012 ceph-mon[31473]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:23.250 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:23.250 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:23.250 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:22 smithi012 ceph-mon[31473]: from='mgr.14152 172.21.15.12:0/158433943' entity='mgr.x' 2024-05-27T07:21:25.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:24 smithi012 ceph-mon[31473]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:25.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:24 smithi162 ceph-mon[35676]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:25.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:24 smithi062 ceph-mon[35304]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 -- 172.21.15.12:0/3222608206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac1077a0 msgr2=0x7fc4ac109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 --2- 172.21.15.12:0/3222608206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac1077a0 0x7fc4ac109bc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc494009990 tx=0x7fc494009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 -- 172.21.15.12:0/3222608206 shutdown_connections 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 --2- 172.21.15.12:0/3222608206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4ac10a100 0x7fc4ac10c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 --2- 172.21.15.12:0/3222608206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ac104e40 0x7fc4ac107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 --2- 172.21.15.12:0/3222608206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac1077a0 0x7fc4ac109bc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 -- 172.21.15.12:0/3222608206 >> 172.21.15.12:0/3222608206 conn(0x7fc4ac0fe880 msgr2=0x7fc4ac100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 -- 172.21.15.12:0/3222608206 shutdown_connections 2024-05-27T07:21:25.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.438+0000 7fc4b1200700 1 -- 172.21.15.12:0/3222608206 wait complete. 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.439+0000 7fc4b1200700 1 Processor -- start 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.439+0000 7fc4b1200700 1 -- start start 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4b1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac104e40 0x7fc4ac19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4b1200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ac1077a0 0x7fc4ac19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4b1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4ac10a100 0x7fc4ac1a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4b1200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac10fc50 con 0x7fc4ac104e40 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4b1200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac10fad0 con 0x7fc4ac10a100 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4b1200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac10fdd0 con 0x7fc4ac1077a0 2024-05-27T07:21:25.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac104e40 0x7fc4ac19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:25.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4a9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac104e40 0x7fc4ac19ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55188/0 (socket says 172.21.15.12:55188) 2024-05-27T07:21:25.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4a9e00700 1 -- 172.21.15.12:0/669944305 learned_addr learned my addr 172.21.15.12:0/669944305 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:25.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4a1400700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ac1077a0 0x7fc4ac19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:25.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.440+0000 7fc4aa800700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4ac10a100 0x7fc4ac1a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a9e00700 1 -- 172.21.15.12:0/669944305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ac1077a0 msgr2=0x7fc4ac19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a9e00700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ac1077a0 0x7fc4ac19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a9e00700 1 -- 172.21.15.12:0/669944305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4ac10a100 msgr2=0x7fc4ac1a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a9e00700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4ac10a100 0x7fc4ac1a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a9e00700 1 -- 172.21.15.12:0/669944305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac1a4560 con 0x7fc4ac104e40 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a1400700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ac1077a0 0x7fc4ac19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4aa800700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4ac10a100 0x7fc4ac1a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:25.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a9e00700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac104e40 0x7fc4ac19ca90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c8f0 tx=0x7fc49c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:25.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a3400700 1 -- 172.21.15.12:0/669944305 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c007b00 con 0x7fc4ac104e40 2024-05-27T07:21:25.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a3400700 1 -- 172.21.15.12:0/669944305 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc49c007ca0 con 0x7fc4ac104e40 2024-05-27T07:21:25.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ac1a4850 con 0x7fc4ac104e40 2024-05-27T07:21:25.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.441+0000 7fc4a3400700 1 -- 172.21.15.12:0/669944305 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c018660 con 0x7fc4ac104e40 2024-05-27T07:21:25.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.442+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ac1a4d90 con 0x7fc4ac104e40 2024-05-27T07:21:25.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.443+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ac0512b0 con 0x7fc4ac104e40 2024-05-27T07:21:25.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.446+0000 7fc4a3400700 1 -- 172.21.15.12:0/669944305 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc49c00f450 con 0x7fc4ac104e40 2024-05-27T07:21:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.447+0000 7fc4a3400700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc4900b2880 0x7fc4900b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.447+0000 7fc4a3400700 1 -- 172.21.15.12:0/669944305 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fc49c0e2690 con 0x7fc4ac104e40 2024-05-27T07:21:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.447+0000 7fc4a1400700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc4900b2880 0x7fc4900b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.448+0000 7fc4a1400700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc4900b2880 0x7fc4900b4d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc49400b5c0 tx=0x7fc49401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:25.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.450+0000 7fc4a3400700 1 -- 172.21.15.12:0/669944305 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc49c014030 con 0x7fc4ac104e40 2024-05-27T07:21:25.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.699+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fc4ac1a4ff0 con 0x7fc4900b2880 2024-05-27T07:21:25.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.742+0000 7fc4a3400700 1 -- 172.21.15.12:0/669944305 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fc4ac1a4ff0 con 0x7fc4900b2880 2024-05-27T07:21:25.733 INFO:teuthology.orchestra.run.smithi012.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-27T07:21:25.733 INFO:teuthology.orchestra.run.smithi012.stdout:mds.a smithi012 running (4m) 2m ago 4m 80.3M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 de5f65ec6cc6 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.b smithi062 running (4m) 2m ago 4m 92.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 7e9040046a30 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.c smithi162 running (4m) 2m ago 4m 82.5M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 6506cb73b580 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.d smithi012 running (5m) 2m ago 5m 80.5M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 bdd79a7e3870 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.e smithi062 running (5m) 2m ago 5m 84.1M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 c6bd01fa6bb2 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.f smithi162 running (5m) 2m ago 5m 80.4M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 ebb2488adfa9 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.g smithi012 running (4m) 2m ago 4m 80.3M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 49c7c89e3875 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.h smithi062 running (4m) 2m ago 4m 83.9M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 6a0fbf2d97ca 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.i smithi162 running (4m) 2m ago 4m 82.3M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 ce8edb43337e 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.j smithi012 running (5m) 2m ago 5m 82.7M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 f20e208d4b35 2024-05-27T07:21:25.734 INFO:teuthology.orchestra.run.smithi012.stdout:mds.k smithi062 running (5m) 2m ago 5m 83.8M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 ff5ae73d7394 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:mds.l smithi162 running (5m) 2m ago 5m 80.6M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 e113b963d71c 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.x smithi012 *:9283,8765 running (15m) 2m ago 15m 598M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 0587ea20bc32 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.y smithi062 *:8443,8765 running (13m) 2m ago 12m 534M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 9873a08d4594 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.z smithi162 *:8443,8765 running (13m) 2m ago 13m 534M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 f3a2800d7af6 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:mon.a smithi012 running (15m) 2m ago 16m 118M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 f4a74345a656 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:mon.b smithi062 running (13m) 2m ago 13m 109M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 ccbee1f2bf47 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:mon.c smithi162 running (13m) 2m ago 13m 108M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 87b3e6a82711 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 smithi012 running (12m) 2m ago 12m 112M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 8fe08bb84110 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 smithi062 running (12m) 2m ago 12m 119M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 a3e90de9255b 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 smithi162 running (11m) 2m ago 11m 120M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 63c6536b358d 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 smithi012 running (10m) 2m ago 10m 111M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 861e74be575c 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 smithi062 running (10m) 2m ago 10m 118M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 1e8cc33a729a 2024-05-27T07:21:25.735 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 smithi162 running (9m) 2m ago 9m 111M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 c61963426e76 2024-05-27T07:21:25.736 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 smithi012 running (8m) 2m ago 8m 109M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 cbca1932ba4f 2024-05-27T07:21:25.736 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 smithi062 running (8m) 2m ago 8m 121M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 a45fa7621ee4 2024-05-27T07:21:25.736 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 smithi162 running (7m) 2m ago 7m 110M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 bdc96db15df1 2024-05-27T07:21:25.736 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 smithi012 running (7m) 2m ago 7m 117M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 453f2bb9eec6 2024-05-27T07:21:25.736 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 smithi062 running (6m) 2m ago 6m 119M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 3cad76387a2a 2024-05-27T07:21:25.736 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 smithi162 running (5m) 2m ago 5m 116M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 c3ad7d65d312 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.745+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc4900b2880 msgr2=0x7fc4900b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.745+0000 7fc4b1200700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc4900b2880 0x7fc4900b4d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc49400b5c0 tx=0x7fc49401a040 comp rx=0 tx=0).stop 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.745+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac104e40 msgr2=0x7fc4ac19ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.745+0000 7fc4b1200700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac104e40 0x7fc4ac19ca90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c8f0 tx=0x7fc49c00ccb0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 shutdown_connections 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fc4900b2880 0x7fc4900b4d90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4ac10a100 0x7fc4ac1a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4ac104e40 0x7fc4ac19ca90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 --2- 172.21.15.12:0/669944305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ac1077a0 0x7fc4ac19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 >> 172.21.15.12:0/669944305 conn(0x7fc4ac0fe880 msgr2=0x7fc4ac102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 shutdown_connections 2024-05-27T07:21:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:25.746+0000 7fc4b1200700 1 -- 172.21.15.12:0/669944305 wait complete. 2024-05-27T07:21:26.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-27T07:21:27.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:26 smithi062 ceph-mon[35304]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:26 smithi062 ceph-mon[35304]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:27.251 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:27.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:26 smithi012 ceph-mon[31473]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:27.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:26 smithi012 ceph-mon[31473]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:27.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:26 smithi162 ceph-mon[35676]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:27.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:26 smithi162 ceph-mon[35676]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:28.456 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:28 smithi012 ceph-mon[31473]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:28.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:28 smithi162 ceph-mon[35676]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:28.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:28 smithi062 ceph-mon[35304]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:29.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 -- 172.21.15.12:0/1983884553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 msgr2=0x7f22f010c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:29.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 --2- 172.21.15.12:0/1983884553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f010c510 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009990 tx=0x7f22e8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 -- 172.21.15.12:0/1983884553 shutdown_connections 2024-05-27T07:21:29.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 --2- 172.21.15.12:0/1983884553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22f0104e30 0x7f22f0107250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 --2- 172.21.15.12:0/1983884553 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f010c510 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 --2- 172.21.15.12:0/1983884553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0107790 0x7f22f0109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 -- 172.21.15.12:0/1983884553 >> 172.21.15.12:0/1983884553 conn(0x7f22f00fe870 msgr2=0x7f22f0100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 -- 172.21.15.12:0/1983884553 shutdown_connections 2024-05-27T07:21:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.691+0000 7f22f6800700 1 -- 172.21.15.12:0/1983884553 wait complete. 2024-05-27T07:21:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.692+0000 7f22f6800700 1 Processor -- start 2024-05-27T07:21:29.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.693+0000 7f22f6800700 1 -- start start 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.693+0000 7f22f6800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0104e30 0x7f22f019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.693+0000 7f22f6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22f0107790 0x7f22f019ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.693+0000 7f22f6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f01a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.693+0000 7f22f6800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f22f010fc80 con 0x7f22f010a0f0 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.693+0000 7f22f6800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f22f010fb00 con 0x7f22f0107790 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.693+0000 7f22f6800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22f010fe00 con 0x7f22f0104e30 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f01a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f01a1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55214/0 (socket says 172.21.15.12:55214) 2024-05-27T07:21:29.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 -- 172.21.15.12:0/668428449 learned_addr learned my addr 172.21.15.12:0/668428449 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22e6a00700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22f0107790 0x7f22f019ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22ef400700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0104e30 0x7f22f019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 -- 172.21.15.12:0/668428449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0104e30 msgr2=0x7f22f019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0104e30 0x7f22f019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 -- 172.21.15.12:0/668428449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22f0107790 msgr2=0x7f22f019ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22f0107790 0x7f22f019ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22efe00700 1 -- 172.21.15.12:0/668428449 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f01a2350 con 0x7f22f010a0f0 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22e6a00700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22f0107790 0x7f22f019ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.694+0000 7f22ef400700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0104e30 0x7f22f019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.695+0000 7f22efe00700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f01a1cd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f22e80047b0 tx=0x7f22e8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.695+0000 7f22ecc00700 1 -- 172.21.15.12:0/668428449 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8026070 con 0x7f22f010a0f0 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.695+0000 7f22ecc00700 1 -- 172.21.15.12:0/668428449 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22e8004c70 con 0x7f22f010a0f0 2024-05-27T07:21:29.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.695+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f01a25e0 con 0x7f22f010a0f0 2024-05-27T07:21:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.695+0000 7f22ecc00700 1 -- 172.21.15.12:0/668428449 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8003b50 con 0x7f22f010a0f0 2024-05-27T07:21:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.695+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f01a2a30 con 0x7f22f010a0f0 2024-05-27T07:21:29.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.696+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f00512b0 con 0x7f22f010a0f0 2024-05-27T07:21:29.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.699+0000 7f22ecc00700 1 -- 172.21.15.12:0/668428449 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f22e8029070 con 0x7f22f010a0f0 2024-05-27T07:21:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.701+0000 7f22ecc00700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f22d40b2880 0x7f22d40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.701+0000 7f22ecc00700 1 -- 172.21.15.12:0/668428449 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f22e80e3500 con 0x7f22f010a0f0 2024-05-27T07:21:29.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.701+0000 7f22ef400700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f22d40b2880 0x7f22d40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:29.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.702+0000 7f22ef400700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f22d40b2880 0x7f22d40b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f22f006c020 tx=0x7f22e0009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:29.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.703+0000 7f22ecc00700 1 -- 172.21.15.12:0/668428449 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f22e8013070 con 0x7f22f010a0f0 2024-05-27T07:21:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.959+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f22f0063cb0 con 0x7f22d40b2880 2024-05-27T07:21:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.967+0000 7f22ecc00700 1 -- 172.21.15.12:0/668428449 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f22f0063cb0 con 0x7f22d40b2880 2024-05-27T07:21:29.959 INFO:teuthology.orchestra.run.smithi012.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-27T07:21:29.959 INFO:teuthology.orchestra.run.smithi012.stdout:mds.all 12/12 2m ago 5m smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-05-27T07:21:29.959 INFO:teuthology.orchestra.run.smithi012.stdout:mgr 3/3 2m ago 13m smithi012=x;smithi062=y;smithi162=z;count:3 2024-05-27T07:21:29.959 INFO:teuthology.orchestra.run.smithi012.stdout:mon 3/3 2m ago 13m smithi012:172.21.15.12=a;smithi062:172.21.15.62=b;smithi162:172.21.15.162=c;count:3 2024-05-27T07:21:29.959 INFO:teuthology.orchestra.run.smithi012.stdout:osd 12 2m ago - 2024-05-27T07:21:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f22d40b2880 msgr2=0x7f22d40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f22d40b2880 0x7f22d40b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f22f006c020 tx=0x7f22e0009040 comp rx=0 tx=0).stop 2024-05-27T07:21:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 msgr2=0x7f22f01a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f01a1cd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f22e80047b0 tx=0x7f22e8005e80 comp rx=0 tx=0).stop 2024-05-27T07:21:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 shutdown_connections 2024-05-27T07:21:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f22d40b2880 0x7f22d40b4d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22f010a0f0 0x7f22f01a1cd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0104e30 0x7f22f019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 --2- 172.21.15.12:0/668428449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22f0107790 0x7f22f019ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 >> 172.21.15.12:0/668428449 conn(0x7f22f00fe870 msgr2=0x7f22f01059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.970+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 shutdown_connections 2024-05-27T07:21:29.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:29.971+0000 7f22f6800700 1 -- 172.21.15.12:0/668428449 wait complete. 2024-05-27T07:21:30.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-27T07:21:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:30 smithi012 ceph-mon[31473]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:30 smithi012 ceph-mon[31473]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:30 smithi162 ceph-mon[35676]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:30 smithi162 ceph-mon[35676]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:31.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:30 smithi062 ceph-mon[35304]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:31.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:30 smithi062 ceph-mon[35304]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:31.292 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:33.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:32 smithi012 ceph-mon[31473]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:33.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:32 smithi162 ceph-mon[35676]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:33.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:32 smithi062 ceph-mon[35304]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:33.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 -- 172.21.15.12:0/2555433089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 msgr2=0x7f65e410c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:33.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2555433089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e410c320 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00b0a0 tx=0x7f65dc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 -- 172.21.15.12:0/2555433089 shutdown_connections 2024-05-27T07:21:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2555433089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e410c320 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2555433089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e4104c40 0x7f65e4107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2555433089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e41075a0 0x7f65e41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 -- 172.21.15.12:0/2555433089 >> 172.21.15.12:0/2555433089 conn(0x7f65e40fe680 msgr2=0x7f65e4100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 -- 172.21.15.12:0/2555433089 shutdown_connections 2024-05-27T07:21:33.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.862+0000 7f65e9c00700 1 -- 172.21.15.12:0/2555433089 wait complete. 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.863+0000 7f65e9c00700 1 Processor -- start 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9c00700 1 -- start start 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e4104c40 0x7f65e419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e41075a0 0x7f65e419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e41a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f65e410fa50 con 0x7f65e4109f00 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65e410f8d0 con 0x7f65e41075a0 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f65e410fbd0 con 0x7f65e4104c40 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e41a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:33.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e41a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44978/0 (socket says 172.21.15.12:44978) 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.864+0000 7f65e9200700 1 -- 172.21.15.12:0/2636281407 learned_addr learned my addr 172.21.15.12:0/2636281407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e3400700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e41075a0 0x7f65e419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e3e00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e4104c40 0x7f65e419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e9200700 1 -- 172.21.15.12:0/2636281407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e4104c40 msgr2=0x7f65e419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e9200700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e4104c40 0x7f65e419a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e9200700 1 -- 172.21.15.12:0/2636281407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e41075a0 msgr2=0x7f65e419ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e9200700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e41075a0 0x7f65e419ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e9200700 1 -- 172.21.15.12:0/2636281407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e41a2120 con 0x7f65e4109f00 2024-05-27T07:21:33.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.865+0000 7f65e9200700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e41a1aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009010 tx=0x7f65dc009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:33.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.866+0000 7f65e0c00700 1 -- 172.21.15.12:0/2636281407 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc021020 con 0x7f65e4109f00 2024-05-27T07:21:33.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.866+0000 7f65e0c00700 1 -- 172.21.15.12:0/2636281407 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65dc007c00 con 0x7f65e4109f00 2024-05-27T07:21:33.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.866+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e41a23b0 con 0x7f65e4109f00 2024-05-27T07:21:33.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.866+0000 7f65e0c00700 1 -- 172.21.15.12:0/2636281407 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc01b7b0 con 0x7f65e4109f00 2024-05-27T07:21:33.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.866+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e41a2860 con 0x7f65e4109f00 2024-05-27T07:21:33.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.867+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e40512b0 con 0x7f65e4109f00 2024-05-27T07:21:33.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.870+0000 7f65e0c00700 1 -- 172.21.15.12:0/2636281407 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f65dc024020 con 0x7f65e4109f00 2024-05-27T07:21:33.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.871+0000 7f65e0c00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65c80b2900 0x7f65c80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:33.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.871+0000 7f65e0c00700 1 -- 172.21.15.12:0/2636281407 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f65dc00f020 con 0x7f65e4109f00 2024-05-27T07:21:33.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.872+0000 7f65e3e00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65c80b2900 0x7f65c80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:33.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.872+0000 7f65e3e00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65c80b2900 0x7f65c80b4e10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f65d4005fd0 tx=0x7f65d400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:33.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:33.874+0000 7f65e0c00700 1 -- 172.21.15.12:0/2636281407 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f65dc0ab3e0 con 0x7f65e4109f00 2024-05-27T07:21:34.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.115+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f65e4063bf0 con 0x7f65c80b2900 2024-05-27T07:21:34.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.118+0000 7f65e0c00700 1 -- 172.21.15.12:0/2636281407 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f65e4063bf0 con 0x7f65c80b2900 2024-05-27T07:21:34.109 INFO:teuthology.orchestra.run.smithi012.stdout:HOST ADDR LABELS STATUS 2024-05-27T07:21:34.110 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 172.21.15.12 2024-05-27T07:21:34.110 INFO:teuthology.orchestra.run.smithi012.stdout:smithi062 172.21.15.62 2024-05-27T07:21:34.110 INFO:teuthology.orchestra.run.smithi012.stdout:smithi162 172.21.15.162 2024-05-27T07:21:34.110 INFO:teuthology.orchestra.run.smithi012.stdout:3 hosts in cluster 2024-05-27T07:21:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65c80b2900 msgr2=0x7f65c80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65c80b2900 0x7f65c80b4e10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f65d4005fd0 tx=0x7f65d400a560 comp rx=0 tx=0).stop 2024-05-27T07:21:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 msgr2=0x7f65e41a1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:34.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e41a1aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009010 tx=0x7f65dc009bf0 comp rx=0 tx=0).stop 2024-05-27T07:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 shutdown_connections 2024-05-27T07:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f65c80b2900 0x7f65c80b4e10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65e4109f00 0x7f65e41a1aa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e4104c40 0x7f65e419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 --2- 172.21.15.12:0/2636281407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e41075a0 0x7f65e419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.122+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 >> 172.21.15.12:0/2636281407 conn(0x7f65e40fe680 msgr2=0x7f65e4106be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:34.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.123+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 shutdown_connections 2024-05-27T07:21:34.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:34.123+0000 7f65e9c00700 1 -- 172.21.15.12:0/2636281407 wait complete. 2024-05-27T07:21:34.986 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-27T07:21:35.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:34 smithi162 ceph-mon[35676]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:35.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:34 smithi062 ceph-mon[35304]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:35.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:34 smithi012 ceph-mon[31473]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:21:35.445 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:36.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:35 smithi162 ceph-mon[35676]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:36.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:35 smithi062 ceph-mon[35304]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:36.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:35 smithi012 ceph-mon[31473]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:37.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:36 smithi012 ceph-mon[31473]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:37.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:36 smithi162 ceph-mon[35676]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:37.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:36 smithi062 ceph-mon[35304]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.824+0000 7f2fcd800700 1 -- 172.21.15.12:0/428237079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8104180 msgr2=0x7f2fc8104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 --2- 172.21.15.12:0/428237079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8104180 0x7f2fc8104590 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009990 tx=0x7f2fb8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 -- 172.21.15.12:0/428237079 shutdown_connections 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 --2- 172.21.15.12:0/428237079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fc8105c50 0x7f2fc810c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 --2- 172.21.15.12:0/428237079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8104180 0x7f2fc8104590 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 --2- 172.21.15.12:0/428237079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc81052b0 0x7f2fc81056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 -- 172.21.15.12:0/428237079 >> 172.21.15.12:0/428237079 conn(0x7f2fc80ff7c0 msgr2=0x7f2fc8101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 -- 172.21.15.12:0/428237079 shutdown_connections 2024-05-27T07:21:37.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.825+0000 7f2fcd800700 1 -- 172.21.15.12:0/428237079 wait complete. 2024-05-27T07:21:37.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.826+0000 7f2fcd800700 1 Processor -- start 2024-05-27T07:21:37.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.826+0000 7f2fcd800700 1 -- start start 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcd800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fc8104180 0x7f2fc819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcd800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc81052b0 0x7f2fc819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcd800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8105c50 0x7f2fc81a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcd800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc810fa80 con 0x7f2fc8105c50 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcd800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc810f900 con 0x7f2fc8104180 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcd800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc810fc00 con 0x7f2fc81052b0 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8105c50 0x7f2fc81a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8105c50 0x7f2fc81a1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44994/0 (socket says 172.21.15.12:44994) 2024-05-27T07:21:37.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcce00700 1 -- 172.21.15.12:0/1592105155 learned_addr learned my addr 172.21.15.12:0/1592105155 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fc7e00700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fc8104180 0x7f2fc819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fc7400700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc81052b0 0x7f2fc819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcce00700 1 -- 172.21.15.12:0/1592105155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc81052b0 msgr2=0x7f2fc819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcce00700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc81052b0 0x7f2fc819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcce00700 1 -- 172.21.15.12:0/1592105155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fc8104180 msgr2=0x7f2fc819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.827+0000 7f2fcce00700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fc8104180 0x7f2fc819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.828+0000 7f2fcce00700 1 -- 172.21.15.12:0/1592105155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc81a2200 con 0x7f2fc8105c50 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.828+0000 7f2fcce00700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8105c50 0x7f2fc81a1b80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0007ee0 tx=0x7f2fc000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:37.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.828+0000 7f2fc4c00700 1 -- 172.21.15.12:0/1592105155 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc000dbd0 con 0x7f2fc8105c50 2024-05-27T07:21:37.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.828+0000 7f2fc4c00700 1 -- 172.21.15.12:0/1592105155 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2fc000dd70 con 0x7f2fc8105c50 2024-05-27T07:21:37.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.828+0000 7f2fc4c00700 1 -- 172.21.15.12:0/1592105155 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00058d0 con 0x7f2fc8105c50 2024-05-27T07:21:37.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.828+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc81a24f0 con 0x7f2fc8105c50 2024-05-27T07:21:37.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.828+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc81a29e0 con 0x7f2fc8105c50 2024-05-27T07:21:37.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.830+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc80512b0 con 0x7f2fc8105c50 2024-05-27T07:21:37.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.832+0000 7f2fc4c00700 1 -- 172.21.15.12:0/1592105155 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2fc001a020 con 0x7f2fc8105c50 2024-05-27T07:21:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.834+0000 7f2fc4c00700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2fac0b2900 0x7f2fac0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.834+0000 7f2fc4c00700 1 -- 172.21.15.12:0/1592105155 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f2fc00e2420 con 0x7f2fc8105c50 2024-05-27T07:21:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.834+0000 7f2fc7e00700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2fac0b2900 0x7f2fac0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.835+0000 7f2fc7e00700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2fac0b2900 0x7f2fac0b4e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8003820 tx=0x7f2fb801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:37.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:37.837+0000 7f2fc4c00700 1 -- 172.21.15.12:0/1592105155 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2fc0010070 con 0x7f2fc8105c50 2024-05-27T07:21:38.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.093+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 --> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f2fc8063bf0 con 0x7f2fac0b2900 2024-05-27T07:21:38.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.095+0000 7f2fc4c00700 1 -- 172.21.15.12:0/1592105155 <== mgr.14152 v2:172.21.15.12:6800/2856894494 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f2fc8063bf0 con 0x7f2fac0b2900 2024-05-27T07:21:38.086 INFO:teuthology.orchestra.run.smithi012.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-27T07:21:38.086 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331005J400BGN 372G No 7m ago Has a FileSystem, LVM detected 2024-05-27T07:21:38.086 INFO:teuthology.orchestra.run.smithi012.stdout:smithi062 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440094400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-05-27T07:21:38.086 INFO:teuthology.orchestra.run.smithi012.stdout:smithi162 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JU400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-05-27T07:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.098+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2fac0b2900 msgr2=0x7f2fac0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.098+0000 7f2fcd800700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2fac0b2900 0x7f2fac0b4e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8003820 tx=0x7f2fb801a040 comp rx=0 tx=0).stop 2024-05-27T07:21:38.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.098+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8105c50 msgr2=0x7f2fc81a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.098+0000 7f2fcd800700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8105c50 0x7f2fc81a1b80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0007ee0 tx=0x7f2fc000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 shutdown_connections 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f2fac0b2900 0x7f2fac0b4e10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fc8104180 0x7f2fc819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fc8105c50 0x7f2fc81a1b80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 --2- 172.21.15.12:0/1592105155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc81052b0 0x7f2fc819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 >> 172.21.15.12:0/1592105155 conn(0x7f2fc80ff7c0 msgr2=0x7f2fc81084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 shutdown_connections 2024-05-27T07:21:38.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:38.099+0000 7f2fcd800700 1 -- 172.21.15.12:0/1592105155 wait complete. 2024-05-27T07:21:39.010 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:21:39.020 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:21:39.021 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-27T07:21:39.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:38 smithi162 ceph-mon[35676]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:39.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:38 smithi162 ceph-mon[35676]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:39.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:38 smithi062 ceph-mon[35304]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:39.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:38 smithi062 ceph-mon[35304]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:39.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:38 smithi012 ceph-mon[31473]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:39.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:38 smithi012 ceph-mon[31473]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:21:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:41.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:40 smithi162 ceph-mon[35676]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:41.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:40 smithi062 ceph-mon[35304]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:41.315 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:40 smithi012 ceph-mon[31473]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:21:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.952+0000 7fec56c00700 1 -- 172.21.15.12:0/1231379752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50104360 msgr2=0x7fec50104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.952+0000 7fec56c00700 1 --2- 172.21.15.12:0/1231379752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50104360 0x7fec50104770 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fec40009990 tx=0x7fec40009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.952+0000 7fec56c00700 1 -- 172.21.15.12:0/1231379752 shutdown_connections 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.952+0000 7fec56c00700 1 --2- 172.21.15.12:0/1231379752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50105e30 0x7fec5010c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.952+0000 7fec56c00700 1 --2- 172.21.15.12:0/1231379752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50104360 0x7fec50104770 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.952+0000 7fec56c00700 1 --2- 172.21.15.12:0/1231379752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec50105490 0x7fec501058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.953+0000 7fec56c00700 1 -- 172.21.15.12:0/1231379752 >> 172.21.15.12:0/1231379752 conn(0x7fec500ff9c0 msgr2=0x7fec50101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.953+0000 7fec56c00700 1 -- 172.21.15.12:0/1231379752 shutdown_connections 2024-05-27T07:21:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.953+0000 7fec56c00700 1 -- 172.21.15.12:0/1231379752 wait complete. 2024-05-27T07:21:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.954+0000 7fec56c00700 1 Processor -- start 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.954+0000 7fec56c00700 1 -- start start 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec56c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec50104360 0x7fec50078510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec56c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50105490 0x7fec50076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec56c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50105e30 0x7fec50077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec56c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010fca0 con 0x7fec50105490 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec56c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010fb20 con 0x7fec50105e30 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec56c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010fe20 con 0x7fec50104360 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec47e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50105490 0x7fec50076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec47e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50105490 0x7fec50076c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44260/0 (socket says 172.21.15.12:44260) 2024-05-27T07:21:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec47e00700 1 -- 172.21.15.12:0/2072671463 learned_addr learned my addr 172.21.15.12:0/2072671463 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:41.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.955+0000 7fec4fe00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec50104360 0x7fec50078510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec54c00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50105e30 0x7fec50077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec47e00700 1 -- 172.21.15.12:0/2072671463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec50104360 msgr2=0x7fec50078510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec47e00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec50104360 0x7fec50078510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec47e00700 1 -- 172.21.15.12:0/2072671463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50105e30 msgr2=0x7fec50077150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec47e00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50105e30 0x7fec50077150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec47e00700 1 -- 172.21.15.12:0/2072671463 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec50077860 con 0x7fec50105490 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec47e00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50105490 0x7fec50076c10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fec3800d980 tx=0x7fec3800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.956+0000 7fec4d600700 1 -- 172.21.15.12:0/2072671463 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38016070 con 0x7fec50105490 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.957+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec501a4970 con 0x7fec50105490 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.957+0000 7fec4d600700 1 -- 172.21.15.12:0/2072671463 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec38004020 con 0x7fec50105490 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.957+0000 7fec4d600700 1 -- 172.21.15.12:0/2072671463 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38004b90 con 0x7fec50105490 2024-05-27T07:21:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.957+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec501a4e50 con 0x7fec50105490 2024-05-27T07:21:41.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.958+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec5002e5e0 con 0x7fec50105490 2024-05-27T07:21:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.960+0000 7fec4d600700 1 -- 172.21.15.12:0/2072671463 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fec3800bb60 con 0x7fec50105490 2024-05-27T07:21:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.962+0000 7fec4d600700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fec340b2880 0x7fec340b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.962+0000 7fec4d600700 1 -- 172.21.15.12:0/2072671463 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fec380e2a70 con 0x7fec50105490 2024-05-27T07:21:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.962+0000 7fec4fe00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fec340b2880 0x7fec340b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.963+0000 7fec4fe00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fec340b2880 0x7fec340b4d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fec40006010 tx=0x7fec4001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:41.964+0000 7fec4d600700 1 -- 172.21.15.12:0/2072671463 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fec3801f020 con 0x7fec50105490 2024-05-27T07:21:42.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.252+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fec500643c0 con 0x7fec50105490 2024-05-27T07:21:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.253+0000 7fec4d600700 1 -- 172.21.15.12:0/2072671463 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3048 (secure 0 0 0) 0x7fec380aa360 con 0x7fec50105490 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout:e26 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout:legacy client fscid: 1 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout:Filesystem 'cephfs' (1) 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout:fs_name cephfs 2024-05-27T07:21:42.245 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 26 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-05-27T07:17:09.825542+0000 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-05-27T07:18:42.889436+0000 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:tableserver 0 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:root 0 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:session_timeout 300 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:session_autoclose 300 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:max_file_size 1099511627776 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:max_xattr_size 65536 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:required_client_features {} 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure 0 2024-05-27T07:21:42.246 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure_osd_epoch 0 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:max_mds 3 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:in 0,1,2 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:up {0=34475,1=34439,2=34415} 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:failed 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:damaged 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:stopped 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:data_pools [3,4] 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:metadata_pool 2 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:inline_data disabled 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:balancer 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:bal_rank_mask -1 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:standby_count_wanted 0 2024-05-27T07:21:42.247 INFO:teuthology.orchestra.run.smithi012.stdout:qdb_cluster leader: 34475 members: 34439,34475,34415 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b{0:34475} state up:active seq 7 addr [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.k{0:34391} state up:standby-replay seq 1 addr [v2:172.21.15.62:6832/1529598052,v1:172.21.15.62:6833/1529598052] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.h{1:34439} state up:active seq 32 addr [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.i{1:24493} state up:standby-replay seq 1 addr [v2:172.21.15.162:6838/884400330,v1:172.21.15.162:6839/884400330] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.e{2:34415} state up:active seq 35 addr [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c{2:24481} state up:standby-replay seq 1 addr [v2:172.21.15.162:6836/1296782695,v1:172.21.15.162:6837/1296782695] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:Standby daemons: 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.12:6834/2219705283,v1:172.21.15.12:6835/2219705283] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.12:6836/4064394372,v1:172.21.15.12:6837/4064394372] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.g{-1:14607} state up:standby seq 1 addr [v2:172.21.15.12:6838/2663671830,v1:172.21.15.12:6839/2663671830] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a{-1:14613} state up:standby seq 1 addr [v2:172.21.15.12:6840/3587822997,v1:172.21.15.12:6841/3587822997] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.l{-1:24436} state up:standby seq 1 addr [v2:172.21.15.162:6832/1597502189,v1:172.21.15.162:6833/1597502189] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.248 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.f{-1:24469} state up:standby seq 1 addr [v2:172.21.15.162:6834/2577053624,v1:172.21.15.162:6835/2577053624] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:21:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.256+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fec340b2880 msgr2=0x7fec340b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.256+0000 7fec56c00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fec340b2880 0x7fec340b4d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fec40006010 tx=0x7fec4001a040 comp rx=0 tx=0).stop 2024-05-27T07:21:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.256+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50105490 msgr2=0x7fec50076c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.256+0000 7fec56c00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50105490 0x7fec50076c10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fec3800d980 tx=0x7fec3800dd40 comp rx=0 tx=0).stop 2024-05-27T07:21:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 shutdown_connections 2024-05-27T07:21:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fec340b2880 0x7fec340b4d90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50105e30 0x7fec50077150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec50104360 0x7fec50078510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 --2- 172.21.15.12:0/2072671463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fec50105490 0x7fec50076c10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 >> 172.21.15.12:0/2072671463 conn(0x7fec500ff9c0 msgr2=0x7fec501086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 shutdown_connections 2024-05-27T07:21:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:42.257+0000 7fec56c00700 1 -- 172.21.15.12:0/2072671463 wait complete. 2024-05-27T07:21:42.250 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 26 2024-05-27T07:21:43.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-27T07:21:43.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:42 smithi162 ceph-mon[35676]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:43.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:42 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2072671463' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:21:43.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:42 smithi062 ceph-mon[35304]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:43.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:42 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2072671463' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:21:43.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:42 smithi012 ceph-mon[31473]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:21:43.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:42 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2072671463' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:21:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:45.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:44 smithi162 ceph-mon[35676]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-27T07:21:45.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:44 smithi062 ceph-mon[35304]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-27T07:21:45.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:44 smithi012 ceph-mon[31473]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-27T07:21:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 -- 172.21.15.12:0/1480746882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 msgr2=0x7fa5ac075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1480746882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac075ba0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0099c0 tx=0x7fa59c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 -- 172.21.15.12:0/1480746882 shutdown_connections 2024-05-27T07:21:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1480746882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5ac110980 0x7fa5ac112da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1480746882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac075ba0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1480746882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5ac0760e0 0x7fa5ac076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 -- 172.21.15.12:0/1480746882 >> 172.21.15.12:0/1480746882 conn(0x7fa5ac0ff970 msgr2=0x7fa5ac101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 -- 172.21.15.12:0/1480746882 shutdown_connections 2024-05-27T07:21:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.070+0000 7fa5b2000700 1 -- 172.21.15.12:0/1480746882 wait complete. 2024-05-27T07:21:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.071+0000 7fa5b2000700 1 Processor -- start 2024-05-27T07:21:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5b2000700 1 -- start start 2024-05-27T07:21:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5b2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5ac0760e0 0x7fa5ac1a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5b2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac1a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5b2000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5ac110980 0x7fa5ac1a8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5b2000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac1162a0 con 0x7fa5ac0776d0 2024-05-27T07:21:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5b2000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac116120 con 0x7fa5ac0760e0 2024-05-27T07:21:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5b2000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac116420 con 0x7fa5ac110980 2024-05-27T07:21:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.072+0000 7fa5a2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac1a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5a2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac1a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44274/0 (socket says 172.21.15.12:44274) 2024-05-27T07:21:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5a2a00700 1 -- 172.21.15.12:0/1135634485 learned_addr learned my addr 172.21.15.12:0/1135634485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5ab400700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5ac0760e0 0x7fa5ac1a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5abe00700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5ac110980 0x7fa5ac1a8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5a2a00700 1 -- 172.21.15.12:0/1135634485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5ac110980 msgr2=0x7fa5ac1a8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5ac110980 0x7fa5ac1a8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5a2a00700 1 -- 172.21.15.12:0/1135634485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5ac0760e0 msgr2=0x7fa5ac1a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5ac0760e0 0x7fa5ac1a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5a2a00700 1 -- 172.21.15.12:0/1135634485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ac1a8970 con 0x7fa5ac0776d0 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.073+0000 7fa5abe00700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5ac110980 0x7fa5ac1a8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.074+0000 7fa5a2a00700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac1a13e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa59400b530 tx=0x7fa59400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.074+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1135634485 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594015020 con 0x7fa5ac0776d0 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.074+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1135634485 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa594011cc0 con 0x7fa5ac0776d0 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.074+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1135634485 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594010dc0 con 0x7fa5ac0776d0 2024-05-27T07:21:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.074+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5ac1a8c60 con 0x7fa5ac0776d0 2024-05-27T07:21:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.074+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5ac1a9150 con 0x7fa5ac0776d0 2024-05-27T07:21:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.075+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5ac0512b0 con 0x7fa5ac0776d0 2024-05-27T07:21:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.078+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1135634485 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa594018020 con 0x7fa5ac0776d0 2024-05-27T07:21:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.079+0000 7fa5a8c00700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5900b2900 0x7fa5900b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.080+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1135634485 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fa5940e23d0 con 0x7fa5ac0776d0 2024-05-27T07:21:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.080+0000 7fa5ab400700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5900b2900 0x7fa5900b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.081+0000 7fa5ab400700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5900b2900 0x7fa5900b4e10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009990 tx=0x7fa59c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.082+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1135634485 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa59401f020 con 0x7fa5ac0776d0 2024-05-27T07:21:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.322+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fa5ac02e660 con 0x7fa5ac0776d0 2024-05-27T07:21:46.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.324+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1135634485 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) ==== 56+0+5693 (secure 0 0 0) 0x7fa5940a9cc0 con 0x7fa5ac0776d0 2024-05-27T07:21:46.316 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 89 2024-05-27T07:21:46.316 INFO:teuthology.orchestra.run.smithi012.stdout:fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:21:46.316 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-05-27T07:05:24.014715+0000 2024-05-27T07:21:46.316 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-05-27T07:18:47.037205+0000 2024-05-27T07:21:46.316 INFO:teuthology.orchestra.run.smithi012.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-27T07:21:46.316 INFO:teuthology.orchestra.run.smithi012.stdout:crush_version 27 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:full_ratio 0.95 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:backfillfull_ratio 0.9 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:nearfull_ratio 0.85 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:require_min_compat_client luminous 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:min_compat_client luminous 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:require_osd_release squid 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:stretch_mode_enabled false 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:max_osd 12 2024-05-27T07:21:46.317 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6802/667573268,v1:172.21.15.12:6803/667573268] [v2:172.21.15.12:6804/667573268,v1:172.21.15.12:6805/667573268] exists,up e5c0c75c-b9be-4c85-8ad2-8c4555dfce22 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6800/1268848674,v1:172.21.15.62:6801/1268848674] [v2:172.21.15.62:6802/1268848674,v1:172.21.15.62:6803/1268848674] exists,up afd3e256-9590-46d2-b846-c3b3539d8de5 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/2540739935,v1:172.21.15.162:6801/2540739935] [v2:172.21.15.162:6802/2540739935,v1:172.21.15.162:6803/2540739935] exists,up 3814d7d7-2bd7-4e53-958f-3fa857d074dd 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 up in weight 1 up_from 25 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6810/3716699952,v1:172.21.15.12:6811/3716699952] [v2:172.21.15.12:6812/3716699952,v1:172.21.15.12:6813/3716699952] exists,up e6c89349-9353-4f0c-9a8d-07e7305be298 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 up in weight 1 up_from 31 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6808/2898003339,v1:172.21.15.62:6809/2898003339] [v2:172.21.15.62:6810/2898003339,v1:172.21.15.62:6811/2898003339] exists,up 50344032-dab9-44e9-a549-f57d219d7959 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 up in weight 1 up_from 36 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/1419812101,v1:172.21.15.162:6809/1419812101] [v2:172.21.15.162:6810/1419812101,v1:172.21.15.162:6811/1419812101] exists,up c7e9391c-de97-406f-aa88-e12252972a6b 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 up in weight 1 up_from 42 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6818/3556220791,v1:172.21.15.12:6819/3556220791] [v2:172.21.15.12:6820/3556220791,v1:172.21.15.12:6821/3556220791] exists,up b9511154-fa9d-432a-b0be-c67dacf0d0f9 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 up in weight 1 up_from 47 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6816/3439452424,v1:172.21.15.62:6817/3439452424] [v2:172.21.15.62:6818/3439452424,v1:172.21.15.62:6819/3439452424] exists,up ff91153a-4b0f-4275-96a2-6bde3d6f192c 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 up in weight 1 up_from 53 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/1474732705,v1:172.21.15.162:6817/1474732705] [v2:172.21.15.162:6818/1474732705,v1:172.21.15.162:6819/1474732705] exists,up 0775d1a0-a6b0-4a31-9f58-7b310ad4eeba 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6826/667220154,v1:172.21.15.12:6827/667220154] [v2:172.21.15.12:6828/667220154,v1:172.21.15.12:6829/667220154] exists,up c7623211-c464-44f4-b7bd-9bd337ae5c80 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 up in weight 1 up_from 64 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6824/3686530646,v1:172.21.15.62:6825/3686530646] [v2:172.21.15.62:6826/3686530646,v1:172.21.15.62:6827/3686530646] exists,up 9449f46e-d354-4305-9f57-ff89f196c55a 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 up in weight 1 up_from 70 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/1037138310,v1:172.21.15.162:6825/1037138310] [v2:172.21.15.162:6826/1037138310,v1:172.21.15.162:6827/1037138310] exists,up ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6 2024-05-27T07:21:46.318 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.10 [8,11] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.14 [2,4] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.1b [2,4] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.1f [7,11] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.0 [4,6] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.c [3,9] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.e [5,9] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.14 [4,7] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.27 [3,9] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.3a [4,8] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.5 [5,8] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.6 [9,11] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.12 [9,10] 2024-05-27T07:21:46.319 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 4.1f [9,8] 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/4121651169 expires 2024-05-28T07:06:45.507849+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/2115905110 expires 2024-05-28T07:06:45.507849+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/2115905110 expires 2024-05-28T07:06:45.507849+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1199837030 expires 2024-05-28T07:06:09.346476+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/949043977 expires 2024-05-28T07:06:09.346476+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1631546627 expires 2024-05-28T07:06:45.507849+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/579288494 expires 2024-05-28T07:06:09.346476+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/1064527569 expires 2024-05-28T07:06:45.507849+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/579288494 expires 2024-05-28T07:06:09.346476+0000 2024-05-27T07:21:46.320 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/3780513737 expires 2024-05-28T07:06:09.346476+0000 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5900b2900 msgr2=0x7fa5900b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5900b2900 0x7fa5900b4e10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009990 tx=0x7fa59c005fb0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 msgr2=0x7fa5ac1a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac1a13e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa59400b530 tx=0x7fa59400b840 comp rx=0 tx=0).stop 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 shutdown_connections 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa5900b2900 0x7fa5900b4e10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5ac0760e0 0x7fa5ac1a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5ac110980 0x7fa5ac1a8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 --2- 172.21.15.12:0/1135634485 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5ac0776d0 0x7fa5ac1a13e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 >> 172.21.15.12:0/1135634485 conn(0x7fa5ac0ff970 msgr2=0x7fa5ac103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 shutdown_connections 2024-05-27T07:21:46.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:46.328+0000 7fa5b2000700 1 -- 172.21.15.12:0/1135634485 wait complete. 2024-05-27T07:21:47.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:46 smithi062 ceph-mon[35304]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T07:21:47.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:46 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1135634485' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:21:47.253 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:46 smithi012 ceph-mon[31473]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T07:21:47.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:46 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1135634485' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:21:47.257 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-27T07:21:47.271 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:47.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:46 smithi162 ceph-mon[35676]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T07:21:47.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:46 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1135634485' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:21:47.711 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:48 smithi012 ceph-mon[31473]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T07:21:49.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:48 smithi062 ceph-mon[35304]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T07:21:49.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:48 smithi162 ceph-mon[35676]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-27T07:21:50.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 -- 172.21.15.12:0/3575570314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618109670 msgr2=0x7fb618110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 --2- 172.21.15.12:0/3575570314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618109670 0x7fb618110fd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb6100099c0 tx=0x7fb610009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 -- 172.21.15.12:0/3575570314 shutdown_connections 2024-05-27T07:21:50.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 --2- 172.21.15.12:0/3575570314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6181043f0 0x7fb618104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 --2- 172.21.15.12:0/3575570314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618109670 0x7fb618110fd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 --2- 172.21.15.12:0/3575570314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb618108d20 0x7fb618109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 -- 172.21.15.12:0/3575570314 >> 172.21.15.12:0/3575570314 conn(0x7fb618079d00 msgr2=0x7fb61807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:50.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 -- 172.21.15.12:0/3575570314 shutdown_connections 2024-05-27T07:21:50.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.155+0000 7fb617e00700 1 -- 172.21.15.12:0/3575570314 wait complete. 2024-05-27T07:21:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.156+0000 7fb617e00700 1 Processor -- start 2024-05-27T07:21:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.156+0000 7fb617e00700 1 -- start start 2024-05-27T07:21:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb617e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6181043f0 0x7fb618199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb617e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618108d20 0x7fb618199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb617e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb618109670 0x7fb6181a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb617e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb6181141a0 con 0x7fb618108d20 2024-05-27T07:21:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb617e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb618114020 con 0x7fb618109670 2024-05-27T07:21:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb617e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb618114320 con 0x7fb6181043f0 2024-05-27T07:21:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb60e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618108d20 0x7fb618199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb60e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618108d20 0x7fb618199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44294/0 (socket says 172.21.15.12:44294) 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb60e000700 1 -- 172.21.15.12:0/3507639206 learned_addr learned my addr 172.21.15.12:0/3507639206 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.157+0000 7fb616a00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6181043f0 0x7fb618199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb617400700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb618109670 0x7fb6181a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb60e000700 1 -- 172.21.15.12:0/3507639206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6181043f0 msgr2=0x7fb618199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb60e000700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6181043f0 0x7fb618199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb60e000700 1 -- 172.21.15.12:0/3507639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb618109670 msgr2=0x7fb6181a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb60e000700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb618109670 0x7fb6181a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb60e000700 1 -- 172.21.15.12:0/3507639206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6181a0c10 con 0x7fb618108d20 2024-05-27T07:21:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb617400700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb618109670 0x7fb6181a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.158+0000 7fb60e000700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618108d20 0x7fb618199bd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb60000c8f0 tx=0x7fb60000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.159+0000 7fb61ca00700 1 -- 172.21.15.12:0/3507639206 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb600007940 con 0x7fb618108d20 2024-05-27T07:21:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.159+0000 7fb61ca00700 1 -- 172.21.15.12:0/3507639206 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb60000f450 con 0x7fb618108d20 2024-05-27T07:21:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.159+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6181a0f00 con 0x7fb618108d20 2024-05-27T07:21:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.159+0000 7fb61ca00700 1 -- 172.21.15.12:0/3507639206 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb600018660 con 0x7fb618108d20 2024-05-27T07:21:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.159+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6181a1440 con 0x7fb618108d20 2024-05-27T07:21:50.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.160+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6181058a0 con 0x7fb618108d20 2024-05-27T07:21:50.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.163+0000 7fb61ca00700 1 -- 172.21.15.12:0/3507639206 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb60000f900 con 0x7fb618108d20 2024-05-27T07:21:50.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.164+0000 7fb61ca00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5fc0b2880 0x7fb5fc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.165+0000 7fb61ca00700 1 -- 172.21.15.12:0/3507639206 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fb6000e2550 con 0x7fb618108d20 2024-05-27T07:21:50.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.165+0000 7fb616a00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5fc0b2880 0x7fb5fc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.166+0000 7fb616a00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5fc0b2880 0x7fb5fc0b4d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb6080060b0 tx=0x7fb60800c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:50.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.167+0000 7fb61ca00700 1 -- 172.21.15.12:0/3507639206 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb600014030 con 0x7fb618108d20 2024-05-27T07:21:50.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.462+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6180512b0 con 0x7fb618108d20 2024-05-27T07:21:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.464+0000 7fb61ca00700 1 -- 172.21.15.12:0/3507639206 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11867 (secure 0 0 0) 0x7fb6000a9ed0 con 0x7fb618108d20 2024-05-27T07:21:50.455 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:50.455 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:18:42.889436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:21:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5fc0b2880 msgr2=0x7fb5fc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5fc0b2880 0x7fb5fc0b4d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb6080060b0 tx=0x7fb60800c040 comp rx=0 tx=0).stop 2024-05-27T07:21:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618108d20 msgr2=0x7fb618199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618108d20 0x7fb618199bd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb60000c8f0 tx=0x7fb60000cc00 comp rx=0 tx=0).stop 2024-05-27T07:21:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 shutdown_connections 2024-05-27T07:21:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fb5fc0b2880 0x7fb5fc0b4d90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb618109670 0x7fb6181a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6181043f0 0x7fb618199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 --2- 172.21.15.12:0/3507639206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb618108d20 0x7fb618199bd0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.467+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 >> 172.21.15.12:0/3507639206 conn(0x7fb618079d00 msgr2=0x7fb618101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.468+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 shutdown_connections 2024-05-27T07:21:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:50.468+0000 7fb617e00700 1 -- 172.21.15.12:0/3507639206 wait complete. 2024-05-27T07:21:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 26 2024-05-27T07:21:51.324 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:50 smithi012 ceph-mon[31473]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T07:21:51.324 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:50 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3507639206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:51.383 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:51.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:50 smithi162 ceph-mon[35676]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T07:21:51.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:51 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3507639206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:51.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:50 smithi062 ceph-mon[35304]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-27T07:21:51.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:50 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3507639206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:53.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:53 smithi012 ceph-mon[31473]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-27T07:21:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:53 smithi162 ceph-mon[35676]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-27T07:21:53.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:53 smithi062 ceph-mon[35304]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-27T07:21:54.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 -- 172.21.15.12:0/2422730002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1178109860 msgr2=0x7f11781111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:54.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 --2- 172.21.15.12:0/2422730002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1178109860 0x7f11781111c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f117000b0d0 tx=0x7f117000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 -- 172.21.15.12:0/2422730002 shutdown_connections 2024-05-27T07:21:54.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 --2- 172.21.15.12:0/2422730002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1178109860 0x7f11781111c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 --2- 172.21.15.12:0/2422730002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11781045e0 0x7f11781049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 --2- 172.21.15.12:0/2422730002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1178108f10 0x7f1178109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 -- 172.21.15.12:0/2422730002 >> 172.21.15.12:0/2422730002 conn(0x7f1178100030 msgr2=0x7f1178102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 -- 172.21.15.12:0/2422730002 shutdown_connections 2024-05-27T07:21:54.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.391+0000 7f117ca00700 1 -- 172.21.15.12:0/2422730002 wait complete. 2024-05-27T07:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.392+0000 7f117ca00700 1 Processor -- start 2024-05-27T07:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.392+0000 7f117ca00700 1 -- start start 2024-05-27T07:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f117ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11781045e0 0x7f117810b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:54.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f117ca00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1178108f10 0x7f117810b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f117ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1178109860 0x7f117810bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f117ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1178114390 con 0x7f11781045e0 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f117ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1178114210 con 0x7f1178109860 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f117ca00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1178114510 con 0x7f1178108f10 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f1175e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11781045e0 0x7f117810b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f1175e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11781045e0 0x7f117810b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50268/0 (socket says 172.21.15.12:50268) 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f1175e00700 1 -- 172.21.15.12:0/1423992847 learned_addr learned my addr 172.21.15.12:0/1423992847 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f1176800700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1178109860 0x7f117810bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.393+0000 7f116d400700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1178108f10 0x7f117810b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:54.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.394+0000 7f1175e00700 1 -- 172.21.15.12:0/1423992847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1178108f10 msgr2=0x7f117810b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.394+0000 7f1175e00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1178108f10 0x7f117810b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.394+0000 7f1175e00700 1 -- 172.21.15.12:0/1423992847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1178109860 msgr2=0x7f117810bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.394+0000 7f1175e00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1178109860 0x7f117810bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.394+0000 7f1175e00700 1 -- 172.21.15.12:0/1423992847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11781ad540 con 0x7f11781045e0 2024-05-27T07:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.394+0000 7f1175e00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11781045e0 0x7f117810b240 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f116800d950 tx=0x7f116800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.394+0000 7f116f400700 1 -- 172.21.15.12:0/1423992847 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168016070 con 0x7f11781045e0 2024-05-27T07:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.395+0000 7f116f400700 1 -- 172.21.15.12:0/1423992847 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1168010460 con 0x7f11781045e0 2024-05-27T07:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.395+0000 7f116f400700 1 -- 172.21.15.12:0/1423992847 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116800f620 con 0x7f11781045e0 2024-05-27T07:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.395+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11781ad7d0 con 0x7f11781045e0 2024-05-27T07:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.395+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11781adcc0 con 0x7f11781045e0 2024-05-27T07:21:54.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.396+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1178105a90 con 0x7f11781045e0 2024-05-27T07:21:54.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.399+0000 7f116f400700 1 -- 172.21.15.12:0/1423992847 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f1168019070 con 0x7f11781045e0 2024-05-27T07:21:54.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.400+0000 7f116f400700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f115c0b2900 0x7f115c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:54.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.400+0000 7f116f400700 1 -- 172.21.15.12:0/1423992847 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f11680e2200 con 0x7f11781045e0 2024-05-27T07:21:54.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.401+0000 7f116d400700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f115c0b2900 0x7f115c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:54.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.401+0000 7f116d400700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f115c0b2900 0x7f115c0b4e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1160005950 tx=0x7f11600058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:54.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.403+0000 7f116f400700 1 -- 172.21.15.12:0/1423992847 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f11680a9af0 con 0x7f11781045e0 2024-05-27T07:21:54.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.695+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11781adf50 con 0x7f11781045e0 2024-05-27T07:21:54.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.697+0000 7f116f400700 1 -- 172.21.15.12:0/1423992847 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11867 (secure 0 0 0) 0x7f11680a93f0 con 0x7f11781045e0 2024-05-27T07:21:54.688 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:54.689 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:18:42.889436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f115c0b2900 msgr2=0x7f115c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f115c0b2900 0x7f115c0b4e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1160005950 tx=0x7f11600058e0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11781045e0 msgr2=0x7f117810b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11781045e0 0x7f117810b240 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f116800d950 tx=0x7f116800dd10 comp rx=0 tx=0).stop 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 shutdown_connections 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f115c0b2900 0x7f115c0b4e10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1178109860 0x7f117810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11781045e0 0x7f117810b240 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 --2- 172.21.15.12:0/1423992847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1178108f10 0x7f117810b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:54.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 >> 172.21.15.12:0/1423992847 conn(0x7f1178100030 msgr2=0x7f1178110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:54.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 shutdown_connections 2024-05-27T07:21:54.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:54.702+0000 7f117ca00700 1 -- 172.21.15.12:0/1423992847 wait complete. 2024-05-27T07:21:54.695 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 26 2024-05-27T07:21:55.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:55 smithi012 ceph-mon[31473]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-27T07:21:55.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:55 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1423992847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:55.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:55 smithi162 ceph-mon[35676]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-27T07:21:55.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:55 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1423992847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:55.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:55 smithi062 ceph-mon[35304]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-27T07:21:55.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:55 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1423992847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:55.644 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:21:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:57.306 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:57 smithi012 ceph-mon[31473]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:57.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:57 smithi162 ceph-mon[35676]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:57.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:57 smithi062 ceph-mon[35304]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.632+0000 7fe1e8e00700 1 -- 172.21.15.12:0/4117195693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 msgr2=0x7fe1e410fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.632+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/4117195693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e410fac0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009990 tx=0x7fe1d4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.632+0000 7fe1e8e00700 1 -- 172.21.15.12:0/4117195693 shutdown_connections 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.632+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/4117195693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e410b100 0x7fe1e4117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.632+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/4117195693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e410fac0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.632+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/4117195693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1e410a7b0 0x7fe1e410abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.632+0000 7fe1e8e00700 1 -- 172.21.15.12:0/4117195693 >> 172.21.15.12:0/4117195693 conn(0x7fe1e4076af0 msgr2=0x7fe1e4076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.633+0000 7fe1e8e00700 1 -- 172.21.15.12:0/4117195693 shutdown_connections 2024-05-27T07:21:58.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.633+0000 7fe1e8e00700 1 -- 172.21.15.12:0/4117195693 wait complete. 2024-05-27T07:21:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.634+0000 7fe1e8e00700 1 Processor -- start 2024-05-27T07:21:58.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.634+0000 7fe1e8e00700 1 -- start start 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e8e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1e410a7b0 0x7fe1e41a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e410b100 0x7fe1e41a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e41a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e8e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e411a740 con 0x7fe1e410f6b0 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e8e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e411a5c0 con 0x7fe1e410b100 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e8e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e411a8c0 con 0x7fe1e410a7b0 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e41a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e41a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50284/0 (socket says 172.21.15.12:50284) 2024-05-27T07:21:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e3e00700 1 -- 172.21.15.12:0/3694994263 learned_addr learned my addr 172.21.15.12:0/3694994263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:21:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1e3400700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1e410a7b0 0x7fe1e41a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.635+0000 7fe1daa00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e410b100 0x7fe1e41a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1e3e00700 1 -- 172.21.15.12:0/3694994263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1e410a7b0 msgr2=0x7fe1e41a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1e3e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1e410a7b0 0x7fe1e41a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1e3e00700 1 -- 172.21.15.12:0/3694994263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e410b100 msgr2=0x7fe1e41a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1e3e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e410b100 0x7fe1e41a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1e3e00700 1 -- 172.21.15.12:0/3694994263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e41a5f10 con 0x7fe1e410f6b0 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1daa00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e410b100 0x7fe1e41a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1e3e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e41a5920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00d980 tx=0x7fe1dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.636+0000 7fe1e0c00700 1 -- 172.21.15.12:0/3694994263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc017070 con 0x7fe1e410f6b0 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.637+0000 7fe1e0c00700 1 -- 172.21.15.12:0/3694994263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1dc005020 con 0x7fe1e410f6b0 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.637+0000 7fe1e0c00700 1 -- 172.21.15.12:0/3694994263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc009990 con 0x7fe1e410f6b0 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.637+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e41a6200 con 0x7fe1e410f6b0 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.637+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e41a66f0 con 0x7fe1e410f6b0 2024-05-27T07:21:58.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.638+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e40512b0 con 0x7fe1e410f6b0 2024-05-27T07:21:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.641+0000 7fe1e0c00700 1 -- 172.21.15.12:0/3694994263 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe1dc004a30 con 0x7fe1e410f6b0 2024-05-27T07:21:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.642+0000 7fe1e0c00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1c80b2880 0x7fe1c80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.642+0000 7fe1e0c00700 1 -- 172.21.15.12:0/3694994263 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fe1dc0e27c0 con 0x7fe1e410f6b0 2024-05-27T07:21:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.643+0000 7fe1e3400700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1c80b2880 0x7fe1c80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.643+0000 7fe1e3400700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1c80b2880 0x7fe1c80b4d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400b5c0 tx=0x7fe1d4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:58.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.645+0000 7fe1e0c00700 1 -- 172.21.15.12:0/3694994263 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe1dc020020 con 0x7fe1e410f6b0 2024-05-27T07:21:58.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.883+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1e402e5e0 con 0x7fe1e410f6b0 2024-05-27T07:21:58.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.886+0000 7fe1e0c00700 1 -- 172.21.15.12:0/3694994263 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24806 (secure 0 0 0) 0x7fe1dc0aa0b0 con 0x7fe1e410f6b0 2024-05-27T07:21:58.878 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:21:58.878 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:18:47.037205+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:21:58.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1c80b2880 msgr2=0x7fe1c80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1c80b2880 0x7fe1c80b4d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400b5c0 tx=0x7fe1d4005960 comp rx=0 tx=0).stop 2024-05-27T07:21:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 msgr2=0x7fe1e41a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e41a5920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00d980 tx=0x7fe1dc00dc90 comp rx=0 tx=0).stop 2024-05-27T07:21:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 shutdown_connections 2024-05-27T07:21:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fe1c80b2880 0x7fe1c80b4d90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1e410f6b0 0x7fe1e41a5920 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1e410a7b0 0x7fe1e41a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 --2- 172.21.15.12:0/3694994263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e410b100 0x7fe1e41a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 >> 172.21.15.12:0/3694994263 conn(0x7fe1e4076af0 msgr2=0x7fe1e41088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:58.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 shutdown_connections 2024-05-27T07:21:58.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:21:58.890+0000 7fe1e8e00700 1 -- 172.21.15.12:0/3694994263 wait complete. 2024-05-27T07:21:59.266 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:59 smithi012 ceph-mon[31473]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:59.266 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:21:59 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3694994263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:21:59.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:59 smithi162 ceph-mon[35676]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:59.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:21:59 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3694994263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:21:59.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:59 smithi062 ceph-mon[35304]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:21:59.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:21:59 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3694994263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:21:59.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:17:09.825542+0000', 'modified': '2024-05-27T07:18:42.889436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 34475, 'mds_1': 34439, 'mds_2': 34415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6837/1296782695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 1296782695}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 1296782695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6839/884400330', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 884400330}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 884400330}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34391': {'gid': 34391, 'name': 'k', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.62:6833/1529598052', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1529598052}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 1529598052}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34415': {'gid': 34415, 'name': 'e', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 35, 'addr': '172.21.15.62:6835/989260451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 989260451}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 989260451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34439': {'gid': 34439, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.62:6837/1993227402', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6836', 'nonce': 1993227402}, {'type': 'v1', 'addr': '172.21.15.62:6837', 'nonce': 1993227402}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34475': {'gid': 34475, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.62:6839/994488749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6838', 'nonce': 994488749}, {'type': 'v1', 'addr': '172.21.15.62:6839', 'nonce': 994488749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34475, 'qdb_cluster': [34439, 34475, 34415]} 2024-05-27T07:21:59.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-27T07:21:59.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.e status 2024-05-27T07:22:00.256 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:01.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:01 smithi012 ceph-mon[31473]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:01 smithi162 ceph-mon[35676]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:22:01.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:01 smithi062 ceph-mon[35304]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:22:02.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 -- 172.21.15.12:0/301754242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 msgr2=0x7f710c113080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 --2- 172.21.15.12:0/301754242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c113080 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f710400b0a0 tx=0x7f710400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 -- 172.21.15.12:0/301754242 shutdown_connections 2024-05-27T07:22:02.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 --2- 172.21.15.12:0/301754242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f710c1030b0 0x7f710c1034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 --2- 172.21.15.12:0/301754242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c113080 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 --2- 172.21.15.12:0/301754242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f710c103a00 0x7f710c110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 -- 172.21.15.12:0/301754242 >> 172.21.15.12:0/301754242 conn(0x7f710c0fedb0 msgr2=0x7f710c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:02.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 -- 172.21.15.12:0/301754242 shutdown_connections 2024-05-27T07:22:02.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.654+0000 7f7112400700 1 -- 172.21.15.12:0/301754242 wait complete. 2024-05-27T07:22:02.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.655+0000 7f7112400700 1 Processor -- start 2024-05-27T07:22:02.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f7112400700 1 -- start start 2024-05-27T07:22:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f7112400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f710c1030b0 0x7f710c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f7112400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f710c103a00 0x7f710c19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f7112400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f7112400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f710c116520 con 0x7f710c110c60 2024-05-27T07:22:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f7112400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f710c1163a0 con 0x7f710c1030b0 2024-05-27T07:22:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f7112400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f710c1166a0 con 0x7f710c103a00 2024-05-27T07:22:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f710be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f710be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c1a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53660/0 (socket says 172.21.15.12:53660) 2024-05-27T07:22:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.656+0000 7f710be00700 1 -- 172.21.15.12:0/3637997953 learned_addr learned my addr 172.21.15.12:0/3637997953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710b400700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f710c1030b0 0x7f710c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710aa00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f710c103a00 0x7f710c19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710be00700 1 -- 172.21.15.12:0/3637997953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f710c103a00 msgr2=0x7f710c19d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710be00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f710c103a00 0x7f710c19d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710be00700 1 -- 172.21.15.12:0/3637997953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f710c1030b0 msgr2=0x7f710c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710be00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f710c1030b0 0x7f710c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710be00700 1 -- 172.21.15.12:0/3637997953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710c1a1d10 con 0x7f710c110c60 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710aa00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f710c103a00 0x7f710c19d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.657+0000 7f710be00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c1a1720 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f71040062a0 tx=0x7f7104004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.658+0000 7f7103e00700 1 -- 172.21.15.12:0/3637997953 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104021070 con 0x7f710c110c60 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.658+0000 7f7103e00700 1 -- 172.21.15.12:0/3637997953 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7104007d00 con 0x7f710c110c60 2024-05-27T07:22:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.658+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710c1a1fa0 con 0x7f710c110c60 2024-05-27T07:22:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.658+0000 7f7103e00700 1 -- 172.21.15.12:0/3637997953 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104005280 con 0x7f710c110c60 2024-05-27T07:22:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.658+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710c1a2480 con 0x7f710c110c60 2024-05-27T07:22:02.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.662+0000 7f7103e00700 1 -- 172.21.15.12:0/3637997953 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7104024070 con 0x7f710c110c60 2024-05-27T07:22:02.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.663+0000 7f7121000b80 1 Processor -- start 2024-05-27T07:22:02.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.663+0000 7f7121000b80 1 -- start start 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7103e00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f70f00b2880 0x7f70f00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7121000b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a56b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7121000b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a565820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7121000b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a566170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7121000b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c95a493df0 con 0x55c95a3e6df0 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7121000b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c95a483990 con 0x55c95a56bce0 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7121000b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c95a4879d0 con 0x55c95a565d60 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f7103e00700 1 -- 172.21.15.12:0/3637997953 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f710400f020 con 0x7f710c110c60 2024-05-27T07:22:02.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f710aa00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f70f00b2880 0x7f70f00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.664+0000 7f710b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a56b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a56b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53670/0 (socket says 172.21.15.12:53670) 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710b400700 1 -- 172.21.15.12:0/1254276112 learned_addr learned my addr 172.21.15.12:0/1254276112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710aa00700 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a565820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710be00700 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a566170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710b400700 1 -- 172.21.15.12:0/1254276112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 msgr2=0x55c95a566170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710b400700 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a566170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710b400700 1 -- 172.21.15.12:0/1254276112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 msgr2=0x55c95a565820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710b400700 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a565820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710b400700 1 -- 172.21.15.12:0/1254276112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c95a44c3d0 con 0x55c95a3e6df0 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.665+0000 7f710be00700 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a566170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:02.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.666+0000 7f710aa00700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f70f00b2880 0x7f70f00b4d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f70f80059d0 tx=0x7f70f800a9f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.666+0000 7f710b400700 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a56b7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f70fc008c60 tx=0x7f70fc008f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ff53a911a110b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.666+0000 7f7100c00700 1 -- 172.21.15.12:0/1254276112 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc014070 con 0x55c95a3e6df0 2024-05-27T07:22:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.666+0000 7f7100c00700 1 -- 172.21.15.12:0/1254276112 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70fc00ed70 con 0x55c95a3e6df0 2024-05-27T07:22:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.666+0000 7f7100c00700 1 -- 172.21.15.12:0/1254276112 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc0056c0 con 0x55c95a3e6df0 2024-05-27T07:22:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 -- 172.21.15.12:0/1254276112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 msgr2=0x55c95a56b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a56b7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f70fc008c60 tx=0x7f70fc008f70 comp rx=0 tx=0).stop 2024-05-27T07:22:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 -- 172.21.15.12:0/1254276112 shutdown_connections 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a566170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a56b7a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 --2- 172.21.15.12:0/1254276112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a565820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 -- 172.21.15.12:0/1254276112 >> 172.21.15.12:0/1254276112 conn(0x55c95a446f70 msgr2=0x55c95a560880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 -- 172.21.15.12:0/1254276112 shutdown_connections 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.667+0000 7f7121000b80 1 -- 172.21.15.12:0/1254276112 wait complete. 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.668+0000 7f7121000b80 1 -- start start 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.668+0000 7f7121000b80 1 Processor -- start 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f7121000b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a3e6df0 0x55c95a56f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f7121000b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a565d60 0x55c95a56f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f7121000b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a56fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f7121000b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c95a389f50 con 0x55c95a565d60 2024-05-27T07:22:02.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f7121000b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c95a424710 con 0x55c95a3e6df0 2024-05-27T07:22:02.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f7121000b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c95a5727b0 con 0x55c95a56bce0 2024-05-27T07:22:02.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f7121000b80 10 client.0 _ll_register_callbacks cb 0x55c95a42dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:02.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f710aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a565d60 0x55c95a56f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f710aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a565d60 0x55c95a56f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53684/0 (socket says 172.21.15.12:53684) 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f710aa00700 1 -- 172.21.15.12:0/993163125 learned_addr learned my addr 172.21.15.12:0/993163125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f710b400700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a3e6df0 0x55c95a56f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.669+0000 7f710be00700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a56fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f710aa00700 1 -- 172.21.15.12:0/993163125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 msgr2=0x55c95a56fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f710aa00700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a56fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f710aa00700 1 -- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a3e6df0 msgr2=0x55c95a56f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f710aa00700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a3e6df0 0x55c95a56f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f710aa00700 1 -- 172.21.15.12:0/993163125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c95a3d3880 con 0x55c95a565d60 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f710be00700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a56fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f710aa00700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a565d60 0x55c95a56f960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f70f800f9f0 tx=0x7f70f8005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4411910daa4f22d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:02.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f70f5600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f70f5600700 1 -- 172.21.15.12:0/993163125 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8010960 con 0x55c95a565d60 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f70f5600700 1 -- 172.21.15.12:0/993163125 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f70f8010b00 con 0x55c95a565d60 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.670+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70fc00ed70 con 0x55c95a565d60 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f70f5600700 1 -- 172.21.15.12:0/993163125 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70f800fe70 con 0x55c95a565d60 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 10 client.15183 fetch_fsmap learned FSMap version 26 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c95a454010 con 0x55c95a565d60 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f70f5600700 1 -- 172.21.15.12:0/993163125 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70fc00ed70 con 0x55c95a565d60 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f70f5600700 1 -- 172.21.15.12:0/993163125 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f70f8028c30 con 0x55c95a565d60 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 10 client.15183 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 10 client.15183 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 20 client.15183 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:02.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a57f5f0 0x55c95a580a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 4 client.15183 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:02.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.671+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c95a495d40 con 0x55c95a57f5f0 2024-05-27T07:22:02.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.672+0000 7f710b400700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a57f5f0 0x55c95a580a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.673+0000 7f710b400700 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a57f5f0 0x55c95a580a60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6092976cedddf62a server_cookie=d4580354eb28dfdf in_seq=0 out_seq=0 2024-05-27T07:22:02.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.673+0000 7f70f5600700 10 client.15183 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:22:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f70f5600700 1 -- 172.21.15.12:0/993163125 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c95a495d40 con 0x55c95a57f5f0 2024-05-27T07:22:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f70f5600700 10 client.15183 handle_command_reply: tid=0 2024-05-27T07:22:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 client.15183 shutdown 2024-05-27T07:22:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a565d60 msgr2=0x55c95a56f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a565d60 0x55c95a56f960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f70f800f9f0 tx=0x7f70f8005bb0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 shutdown_connections 2024-05-27T07:22:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a57f5f0 0x55c95a580a60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a3e6df0 0x55c95a56f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a56fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 --2- 172.21.15.12:0/993163125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a565d60 0x55c95a56f960 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f70f5600700 0 client.15183 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:22:02.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 >> 172.21.15.12:0/993163125 conn(0x55c95a446f70 msgr2=0x55c95a560530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:02.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 shutdown_connections 2024-05-27T07:22:02.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.676+0000 7f7121000b80 1 -- 172.21.15.12:0/993163125 wait complete. 2024-05-27T07:22:02.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.677+0000 7f7121000b80 20 client.15183 trim_cache size 0 max 16384 2024-05-27T07:22:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.700+0000 7f7121000b80 1 Processor -- start 2024-05-27T07:22:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.700+0000 7f7121000b80 1 -- start start 2024-05-27T07:22:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f7121000b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a565820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f7121000b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a57ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f7121000b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a56ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f7121000b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c95a57eda0 con 0x55c95a3e6df0 2024-05-27T07:22:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f7121000b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c95a57f470 con 0x55c95a56bce0 2024-05-27T07:22:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f7121000b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c95a57fb40 con 0x55c95a565d60 2024-05-27T07:22:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f710b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a565820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f710b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a565820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53692/0 (socket says 172.21.15.12:53692) 2024-05-27T07:22:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f710b400700 1 -- 172.21.15.12:0/1354209991 learned_addr learned my addr 172.21.15.12:0/1354209991 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f710be00700 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a56ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.701+0000 7f710aa00700 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a57ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f710b400700 1 -- 172.21.15.12:0/1354209991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 msgr2=0x55c95a57ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f710b400700 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a57ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f710b400700 1 -- 172.21.15.12:0/1354209991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 msgr2=0x55c95a56ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f710b400700 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a56ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f710b400700 1 -- 172.21.15.12:0/1354209991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c95a570210 con 0x55c95a3e6df0 2024-05-27T07:22:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f710aa00700 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a57ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f710b400700 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a565820 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f70fc008c60 tx=0x7f70fc008550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9f72a889c62ad3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f70f7e00700 1 -- 172.21.15.12:0/1354209991 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc014070 con 0x55c95a3e6df0 2024-05-27T07:22:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f70f7e00700 1 -- 172.21.15.12:0/1354209991 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70fc004710 con 0x55c95a3e6df0 2024-05-27T07:22:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.702+0000 7f70f7e00700 1 -- 172.21.15.12:0/1354209991 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc025670 con 0x55c95a3e6df0 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.703+0000 7f7121000b80 1 -- 172.21.15.12:0/1354209991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 msgr2=0x55c95a565820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.703+0000 7f7121000b80 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a565820 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f70fc008c60 tx=0x7f70fc008550 comp rx=0 tx=0).stop 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.703+0000 7f7121000b80 1 -- 172.21.15.12:0/1354209991 shutdown_connections 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.703+0000 7f7121000b80 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a56bce0 0x55c95a56ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.703+0000 7f7121000b80 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a565820 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.703+0000 7f7121000b80 1 --2- 172.21.15.12:0/1354209991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a565d60 0x55c95a57ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.703+0000 7f7121000b80 1 -- 172.21.15.12:0/1354209991 >> 172.21.15.12:0/1354209991 conn(0x55c95a446f70 msgr2=0x55c95a56a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.704+0000 7f7121000b80 1 -- 172.21.15.12:0/1354209991 shutdown_connections 2024-05-27T07:22:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.704+0000 7f7121000b80 1 -- 172.21.15.12:0/1354209991 wait complete. 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.704+0000 7f7121000b80 1 -- start start 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.704+0000 7f7121000b80 1 Processor -- start 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f7121000b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a570d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f7121000b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a565d60 0x55c95a5712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f7121000b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a5717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f7121000b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c95a5808e0 con 0x55c95a3e6df0 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f7121000b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c95a580210 con 0x55c95a565d60 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f7121000b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c95a580fb0 con 0x55c95a56bce0 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f7121000b80 10 client.0 _ll_register_callbacks cb 0x55c95a42dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f710b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a570d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f710b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a570d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53696/0 (socket says 172.21.15.12:53696) 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.705+0000 7f710b400700 1 -- 172.21.15.12:0/2224054578 learned_addr learned my addr 172.21.15.12:0/2224054578 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710aa00700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a565d60 0x55c95a5712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710be00700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a5717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710b400700 1 -- 172.21.15.12:0/2224054578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 msgr2=0x55c95a5717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710b400700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a5717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710b400700 1 -- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a565d60 msgr2=0x55c95a5712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710b400700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a565d60 0x55c95a5712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710b400700 1 -- 172.21.15.12:0/2224054578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c95a5884f0 con 0x55c95a3e6df0 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710be00700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a5717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710aa00700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a565d60 0x55c95a5712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f710b400700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a570d70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009010 tx=0x7f70fc0083c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb205b4846139ba8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.706+0000 7f70f5600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.707+0000 7f70f5600700 1 -- 172.21.15.12:0/2224054578 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc033070 con 0x55c95a3e6df0 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.707+0000 7f70f5600700 1 -- 172.21.15.12:0/2224054578 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f70fc004ac0 con 0x55c95a3e6df0 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.707+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70fc004710 con 0x55c95a3e6df0 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.707+0000 7f70f5600700 1 -- 172.21.15.12:0/2224054578 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70fc041050 con 0x55c95a3e6df0 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.707+0000 7f7121000b80 10 client.15189 fetch_fsmap learned FSMap version 26 2024-05-27T07:22:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.707+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c95a5eaf90 con 0x55c95a3e6df0 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f70f5600700 1 -- 172.21.15.12:0/2224054578 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70fc004710 con 0x55c95a3e6df0 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f70f5600700 1 -- 172.21.15.12:0/2224054578 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f70fc038c10 con 0x55c95a3e6df0 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f7121000b80 10 client.15189 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f7121000b80 10 client.15189 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f7121000b80 20 client.15189 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f7121000b80 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a5ecd60 0x55c95a5ee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f7121000b80 4 client.15189 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.708+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "status"}) -- 0x55c95a48f520 con 0x55c95a5ecd60 2024-05-27T07:22:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.709+0000 7f710aa00700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a5ecd60 0x55c95a5ee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.710+0000 7f710aa00700 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a5ecd60 0x55c95a5ee1d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75ddb9f488419f52 server_cookie=a6a4065c9aade45c in_seq=0 out_seq=0 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.710+0000 7f70f5600700 10 client.15189 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.710+0000 7f70f5600700 1 -- 172.21.15.12:0/2224054578 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55c95a48f520 con 0x55c95a5ecd60 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.710+0000 7f70f5600700 10 client.15189 handle_command_reply: tid=0 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.710+0000 7f7121000b80 1 client.15189 shutdown 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.710+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 msgr2=0x55c95a570d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.710+0000 7f7121000b80 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a570d70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009010 tx=0x7f70fc0083c0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 shutdown_connections 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c95a56bce0 0x55c95a5717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c95a3e6df0 0x55c95a570d70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55c95a5ecd60 0x55c95a5ee1d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 --2- 172.21.15.12:0/2224054578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c95a565d60 0x55c95a5712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 >> 172.21.15.12:0/2224054578 conn(0x55c95a446f70 msgr2=0x55c95a5691c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f70f5600700 0 client.15189 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 shutdown_connections 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 1 -- 172.21.15.12:0/2224054578 wait complete. 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.711+0000 7f7121000b80 20 client.15189 trim_cache size 0 max 16384 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 34415, 2024-05-27T07:22:02.704 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 206.771731596, 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 26, 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 89, 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 342.426365616 2024-05-27T07:22:02.705 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f70f00b2880 msgr2=0x7f70f00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f70f00b2880 0x7f70f00b4d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f70f80059d0 tx=0x7f70f800a9f0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 msgr2=0x7f710c1a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c1a1720 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f71040062a0 tx=0x7f7104004ed0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 shutdown_connections 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f70f00b2880 0x7f70f00b4d90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f710c1030b0 0x7f710c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f710c110c60 0x7f710c1a1720 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 --2- 172.21.15.12:0/3637997953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f710c103a00 0x7f710c19d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 >> 172.21.15.12:0/3637997953 conn(0x7f710c0fedb0 msgr2=0x7f710c112890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 shutdown_connections 2024-05-27T07:22:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:02.712+0000 7f7112400700 1 -- 172.21.15.12:0/3637997953 wait complete. 2024-05-27T07:22:03.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:03 smithi012 ceph-mon[31473]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:22:03.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:03 smithi162 ceph-mon[35676]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:22:03.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:03 smithi062 ceph-mon[35304]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:22:03.711 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-05-27T07:22:04.182 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:05.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:05 smithi012 ceph-mon[31473]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:22:05.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:05 smithi162 ceph-mon[35676]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:22:05.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:05 smithi062 ceph-mon[35304]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-27T07:22:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.644+0000 7f4743e00700 1 -- 172.21.15.12:0/1210345947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744076670 msgr2=0x7f4744110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.644+0000 7f4743e00700 1 --2- 172.21.15.12:0/1210345947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744076670 0x7f4744110460 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f472c009990 tx=0x7f472c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.645+0000 7f4743e00700 1 -- 172.21.15.12:0/1210345947 shutdown_connections 2024-05-27T07:22:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.645+0000 7f4743e00700 1 --2- 172.21.15.12:0/1210345947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47441109a0 0x7f4744112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.645+0000 7f4743e00700 1 --2- 172.21.15.12:0/1210345947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4744075d20 0x7f4744076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.645+0000 7f4743e00700 1 --2- 172.21.15.12:0/1210345947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744076670 0x7f4744110460 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.645+0000 7f4743e00700 1 -- 172.21.15.12:0/1210345947 >> 172.21.15.12:0/1210345947 conn(0x7f4744079d00 msgr2=0x7f474407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:06.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.645+0000 7f4743e00700 1 -- 172.21.15.12:0/1210345947 shutdown_connections 2024-05-27T07:22:06.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.645+0000 7f4743e00700 1 -- 172.21.15.12:0/1210345947 wait complete. 2024-05-27T07:22:06.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.646+0000 7f4743e00700 1 Processor -- start 2024-05-27T07:22:06.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.646+0000 7f4743e00700 1 -- start start 2024-05-27T07:22:06.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4743e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744075d20 0x7f474419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4743e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4744076670 0x7f474419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4743e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47441109a0 0x7f47441a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4743e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f47441163a0 con 0x7f4744075d20 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4743e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4744116220 con 0x7f47441109a0 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4743e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4744116520 con 0x7f4744076670 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4742a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744075d20 0x7f474419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4742a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744075d20 0x7f474419ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53714/0 (socket says 172.21.15.12:53714) 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.647+0000 7f4742a00700 1 -- 172.21.15.12:0/80454330 learned_addr learned my addr 172.21.15.12:0/80454330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f473a000700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4744076670 0x7f474419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f4743400700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47441109a0 0x7f47441a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f4742a00700 1 -- 172.21.15.12:0/80454330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4744076670 msgr2=0x7f474419d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f4742a00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4744076670 0x7f474419d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f4742a00700 1 -- 172.21.15.12:0/80454330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47441109a0 msgr2=0x7f47441a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f4742a00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47441109a0 0x7f47441a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f4742a00700 1 -- 172.21.15.12:0/80454330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47441a1c50 con 0x7f4744075d20 2024-05-27T07:22:06.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.648+0000 7f4742a00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744075d20 0x7f474419ce30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4734007e20 tx=0x7f473400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:06.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.649+0000 7f473be00700 1 -- 172.21.15.12:0/80454330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4734016020 con 0x7f4744075d20 2024-05-27T07:22:06.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.649+0000 7f473be00700 1 -- 172.21.15.12:0/80454330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f473400ae00 con 0x7f4744075d20 2024-05-27T07:22:06.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.649+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47441a1f40 con 0x7f4744075d20 2024-05-27T07:22:06.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.649+0000 7f473be00700 1 -- 172.21.15.12:0/80454330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4734011670 con 0x7f4744075d20 2024-05-27T07:22:06.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.649+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47441a2480 con 0x7f4744075d20 2024-05-27T07:22:06.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.653+0000 7f473be00700 1 -- 172.21.15.12:0/80454330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f4734019020 con 0x7f4744075d20 2024-05-27T07:22:06.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 Processor -- start 2024-05-27T07:22:06.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 -- start start 2024-05-27T07:22:06.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2892f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a2892fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2893b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558a2885cdf0 con 0x558a287afdf0 2024-05-27T07:22:06.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558a2884c990 con 0x558a28939270 2024-05-27T07:22:06.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.654+0000 7f475811ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a288509d0 con 0x558a2889b9a0 2024-05-27T07:22:06.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f4742a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a2892fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f473be00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f47280b2880 0x7f47280b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f4742a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a2892fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53716/0 (socket says 172.21.15.12:53716) 2024-05-27T07:22:06.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f4742a00700 1 -- 172.21.15.12:0/3097435888 learned_addr learned my addr 172.21.15.12:0/3097435888 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:06.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f4743400700 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2892f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f473a000700 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2893b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f473be00700 1 -- 172.21.15.12:0/80454330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f47340e2450 con 0x7f4744075d20 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.655+0000 7f4743400700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f47280b2880 0x7f47280b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.656+0000 7f4742a00700 1 -- 172.21.15.12:0/3097435888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 msgr2=0x558a2892f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.656+0000 7f4742a00700 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2892f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.656+0000 7f4742a00700 1 -- 172.21.15.12:0/3097435888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 msgr2=0x558a2893b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.656+0000 7f4742a00700 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2893b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.656+0000 7f4742a00700 1 -- 172.21.15.12:0/3097435888 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a288153d0 con 0x558a287afdf0 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.656+0000 7f4743400700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f47280b2880 0x7f47280b4d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f473c005fd0 tx=0x7f473c005930 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:06.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.656+0000 7f4742a00700 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a2892fc40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4734007d00 tx=0x7f47340a5630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca82fd499199aaf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.657+0000 7f4733e00700 1 -- 172.21.15.12:0/3097435888 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47340a9640 con 0x558a287afdf0 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.657+0000 7f4733e00700 1 -- 172.21.15.12:0/3097435888 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47340ad9f0 con 0x558a287afdf0 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.657+0000 7f4733e00700 1 -- 172.21.15.12:0/3097435888 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47340b2270 con 0x558a287afdf0 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 -- 172.21.15.12:0/3097435888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 msgr2=0x558a2892fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a2892fc40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4734007d00 tx=0x7f47340a5630 comp rx=0 tx=0).stop 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 -- 172.21.15.12:0/3097435888 shutdown_connections 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2893b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2892f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 --2- 172.21.15.12:0/3097435888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a2892fc40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 -- 172.21.15.12:0/3097435888 >> 172.21.15.12:0/3097435888 conn(0x558a28849340 msgr2=0x558a2892a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:06.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 -- 172.21.15.12:0/3097435888 shutdown_connections 2024-05-27T07:22:06.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.658+0000 7f475811ab80 1 -- 172.21.15.12:0/3097435888 wait complete. 2024-05-27T07:22:06.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.659+0000 7f475811ab80 1 -- start start 2024-05-27T07:22:06.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.659+0000 7f475811ab80 1 Processor -- start 2024-05-27T07:22:06.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f475811ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28936200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f475811ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28936740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f475811ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a28936c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f475811ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558a28752f50 con 0x558a28939270 2024-05-27T07:22:06.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f475811ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558a287ed710 con 0x558a287afdf0 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f475811ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a2893bc30 con 0x558a2889b9a0 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f475811ab80 10 client.0 _ll_register_callbacks cb 0x558a287f6c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f4743400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a28936c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f4743400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a28936c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53722/0 (socket says 172.21.15.12:53722) 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f4743400700 1 -- 172.21.15.12:0/3858209970 learned_addr learned my addr 172.21.15.12:0/3858209970 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f473a000700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28936740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.660+0000 7f4742a00700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28936200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4743400700 1 -- 172.21.15.12:0/3858209970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 msgr2=0x558a28936740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4743400700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28936740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4743400700 1 -- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 msgr2=0x558a28936200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4743400700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28936200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4743400700 1 -- 172.21.15.12:0/3858209970 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a2879c880 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4742a00700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28936200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4743400700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a28936c80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f473c010df0 tx=0x7f473c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c947f4d4012da77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.661+0000 7f4730c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f4730c00700 1 -- 172.21.15.12:0/3858209970 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c01dd90 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f4730c00700 1 -- 172.21.15.12:0/3858209970 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f473c01f450 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47340ad9f0 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f4730c00700 1 -- 172.21.15.12:0/3858209970 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f473c00e8a0 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f475811ab80 10 client.15201 fetch_fsmap learned FSMap version 26 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a2881d010 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f4730c00700 1 -- 172.21.15.12:0/3858209970 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47340ad9f0 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.662+0000 7f4730c00700 1 -- 172.21.15.12:0/3858209970 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f473c01c040 con 0x558a28939270 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.663+0000 7f475811ab80 10 client.15201 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.663+0000 7f475811ab80 10 client.15201 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:22:06.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.663+0000 7f475811ab80 20 client.15201 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:06.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.663+0000 7f475811ab80 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a28948bb0 0x558a2894a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.663+0000 7f475811ab80 4 client.15201 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:06.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.663+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a2885ed40 con 0x558a28948bb0 2024-05-27T07:22:06.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.663+0000 7f4742a00700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a28948bb0 0x558a2894a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.664+0000 7f4742a00700 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a28948bb0 0x558a2894a020 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa84b2b951d4693a server_cookie=5a9423107cc4f7c7 in_seq=0 out_seq=0 2024-05-27T07:22:06.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.664+0000 7f4730c00700 10 client.15201 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f4730c00700 1 -- 172.21.15.12:0/3858209970 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558a2885ed40 con 0x558a28948bb0 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f4730c00700 10 client.15201 handle_command_reply: tid=0 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 client.15201 shutdown 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 msgr2=0x558a28936c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a28936c80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f473c010df0 tx=0x7f473c004750 comp rx=0 tx=0).stop 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 shutdown_connections 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a28948bb0 0x558a2894a020 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28936200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a28936c80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 --2- 172.21.15.12:0/3858209970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28936740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 >> 172.21.15.12:0/3858209970 conn(0x558a28849340 msgr2=0x558a2892f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f4730c00700 0 client.15201 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:22:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 shutdown_connections 2024-05-27T07:22:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.667+0000 7f475811ab80 1 -- 172.21.15.12:0/3858209970 wait complete. 2024-05-27T07:22:06.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.668+0000 7f475811ab80 20 client.15201 trim_cache size 0 max 16384 2024-05-27T07:22:06.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.691+0000 7f475811ab80 1 Processor -- start 2024-05-27T07:22:06.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.691+0000 7f475811ab80 1 -- start start 2024-05-27T07:22:06.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f475811ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28949280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f475811ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28944ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f475811ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a2892f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f475811ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558a28948360 con 0x558a28939270 2024-05-27T07:22:06.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f475811ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558a28948a30 con 0x558a287afdf0 2024-05-27T07:22:06.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f475811ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a28949100 con 0x558a2889b9a0 2024-05-27T07:22:06.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f4743400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a2892f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f4743400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a2892f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53734/0 (socket says 172.21.15.12:53734) 2024-05-27T07:22:06.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.692+0000 7f4743400700 1 -- 172.21.15.12:0/3125701527 learned_addr learned my addr 172.21.15.12:0/3125701527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:06.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f4742a00700 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28949280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f473a000700 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28944ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f4743400700 1 -- 172.21.15.12:0/3125701527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 msgr2=0x558a28944ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f4743400700 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28944ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f4743400700 1 -- 172.21.15.12:0/3125701527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 msgr2=0x558a28949280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f4743400700 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28949280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f4743400700 1 -- 172.21.15.12:0/3125701527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a28948850 con 0x558a28939270 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.693+0000 7f4743400700 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a2892f700 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f473c010dc0 tx=0x7f473c00efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c70acb6e11fcaa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.694+0000 7f4733400700 1 -- 172.21.15.12:0/3125701527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c01d130 con 0x558a28939270 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.694+0000 7f4733400700 1 -- 172.21.15.12:0/3125701527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f473c029b40 con 0x558a28939270 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.694+0000 7f4733400700 1 -- 172.21.15.12:0/3125701527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c004280 con 0x558a28939270 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.694+0000 7f475811ab80 1 -- 172.21.15.12:0/3125701527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 msgr2=0x558a2892f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.694+0000 7f475811ab80 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a2892f700 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f473c010dc0 tx=0x7f473c00efb0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 -- 172.21.15.12:0/3125701527 shutdown_connections 2024-05-27T07:22:06.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a287afdf0 0x558a28949280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a28939270 0x558a2892f700 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 --2- 172.21.15.12:0/3125701527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a28944ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 -- 172.21.15.12:0/3125701527 >> 172.21.15.12:0/3125701527 conn(0x558a28849340 msgr2=0x558a28947a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:06.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 -- 172.21.15.12:0/3125701527 shutdown_connections 2024-05-27T07:22:06.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 -- 172.21.15.12:0/3125701527 wait complete. 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 -- start start 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.695+0000 7f475811ab80 1 Processor -- start 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f475811ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a28944b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f475811ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2894da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f475811ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2894df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f475811ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558a28949ea0 con 0x558a287afdf0 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f475811ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558a289497d0 con 0x558a28939270 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f475811ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a2894a570 con 0x558a2889b9a0 2024-05-27T07:22:06.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f475811ab80 10 client.0 _ll_register_callbacks cb 0x558a287f6c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f4742a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a28944b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f4742a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a28944b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53736/0 (socket says 172.21.15.12:53736) 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.696+0000 7f4742a00700 1 -- 172.21.15.12:0/1800212195 learned_addr learned my addr 172.21.15.12:0/1800212195 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4743400700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2894df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f473a000700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2894da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4742a00700 1 -- 172.21.15.12:0/1800212195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 msgr2=0x558a2894da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4742a00700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2894da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4742a00700 1 -- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 msgr2=0x558a2894df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4742a00700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2894df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4742a00700 1 -- 172.21.15.12:0/1800212195 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a28953ef0 con 0x558a287afdf0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4743400700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2894df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4742a00700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a28944b70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f47340a68b0 tx=0x7f473400a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336bcbbf066c5a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4730c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.697+0000 7f4730c00700 1 -- 172.21.15.12:0/1800212195 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47340ad9f0 con 0x558a287afdf0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f4730c00700 1 -- 172.21.15.12:0/1800212195 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f47340a98a0 con 0x558a287afdf0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473c029b40 con 0x558a287afdf0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f4730c00700 1 -- 172.21.15.12:0/1800212195 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f47340a9aa0 con 0x558a287afdf0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f475811ab80 10 client.15207 fetch_fsmap learned FSMap version 26 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a28953430 con 0x558a287afdf0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f4730c00700 1 -- 172.21.15.12:0/1800212195 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f473c029b40 con 0x558a287afdf0 2024-05-27T07:22:06.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f4730c00700 1 -- 172.21.15.12:0/1800212195 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f4734023060 con 0x558a287afdf0 2024-05-27T07:22:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f475811ab80 10 client.15207 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:22:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f475811ab80 10 client.15207 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:22:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f475811ab80 20 client.15207 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.698+0000 7f475811ab80 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a289be340 0x558a289bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.699+0000 7f475811ab80 4 client.15207 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:22:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.699+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "status"}) -- 0x558a28858520 con 0x558a289be340 2024-05-27T07:22:06.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.699+0000 7f473a000700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a289be340 0x558a289bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.700+0000 7f473a000700 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a289be340 0x558a289bf7b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=657f53aefea26df1 server_cookie=95ccbd1311f1c5b1 in_seq=0 out_seq=0 2024-05-27T07:22:06.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.700+0000 7f4730c00700 10 client.15207 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.701+0000 7f4730c00700 1 -- 172.21.15.12:0/1800212195 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558a28858520 con 0x558a289be340 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.701+0000 7f4730c00700 10 client.15207 handle_command_reply: tid=0 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.701+0000 7f475811ab80 1 client.15207 shutdown 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.701+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 msgr2=0x558a28944b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.701+0000 7f475811ab80 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a28944b70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f47340a68b0 tx=0x7f473400a8b0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 shutdown_connections 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558a289be340 0x558a289bf7b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a28939270 0x558a2894df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a287afdf0 0x558a28944b70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 --2- 172.21.15.12:0/1800212195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a2889b9a0 0x558a2894da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f4730c00700 0 client.15207 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 >> 172.21.15.12:0/1800212195 conn(0x558a28849340 msgr2=0x558a2892f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:06.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 shutdown_connections 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 1 -- 172.21.15.12:0/1800212195 wait complete. 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.702+0000 7f475811ab80 20 client.15207 trim_cache size 0 max 16384 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 34439, 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 211.78873048899999, 2024-05-27T07:22:06.696 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 26, 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 89, 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 334.09772897699997 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f47280b2880 msgr2=0x7f47280b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f47280b2880 0x7f47280b4d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f473c005fd0 tx=0x7f473c005930 comp rx=0 tx=0).stop 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744075d20 msgr2=0x7f474419ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744075d20 0x7f474419ce30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4734007e20 tx=0x7f473400a340 comp rx=0 tx=0).stop 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 shutdown_connections 2024-05-27T07:22:06.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f47280b2880 0x7f47280b4d90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47441109a0 0x7f47441a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4744075d20 0x7f474419ce30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 --2- 172.21.15.12:0/80454330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4744076670 0x7f474419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:06.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 >> 172.21.15.12:0/80454330 conn(0x7f4744079d00 msgr2=0x7f4744101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:06.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 shutdown_connections 2024-05-27T07:22:06.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:06.703+0000 7f4743e00700 1 -- 172.21.15.12:0/80454330 wait complete. 2024-05-27T07:22:07.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:07 smithi012 ceph-mon[31473]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:22:07.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:07 smithi162 ceph-mon[35676]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:22:07.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:07 smithi062 ceph-mon[35304]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-27T07:22:07.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:22:08.153 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:09.336 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:09 smithi012 ceph-mon[31473]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:22:09.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:09 smithi162 ceph-mon[35676]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:22:09.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:09 smithi062 ceph-mon[35304]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:22:10.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 -- 172.21.15.12:0/1560636522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 msgr2=0x7f7e5c10e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 --2- 172.21.15.12:0/1560636522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c10e120 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009990 tx=0x7f7e48009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 -- 172.21.15.12:0/1560636522 shutdown_connections 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 --2- 172.21.15.12:0/1560636522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c105280 0x7f7e5c105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 --2- 172.21.15.12:0/1560636522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e5c10e6f0 0x7f7e5c110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 --2- 172.21.15.12:0/1560636522 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c10e120 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 -- 172.21.15.12:0/1560636522 >> 172.21.15.12:0/1560636522 conn(0x7f7e5c078af0 msgr2=0x7f7e5c078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 -- 172.21.15.12:0/1560636522 shutdown_connections 2024-05-27T07:22:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.466+0000 7f7e62800700 1 -- 172.21.15.12:0/1560636522 wait complete. 2024-05-27T07:22:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.468+0000 7f7e62800700 1 Processor -- start 2024-05-27T07:22:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.468+0000 7f7e62800700 1 -- start start 2024-05-27T07:22:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e62800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e5c105280 0x7f7e5c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e62800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c19d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e62800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c10e6f0 0x7f7e5c1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e62800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c114150 con 0x7f7e5c105bd0 2024-05-27T07:22:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e62800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c113fd0 con 0x7f7e5c10e6f0 2024-05-27T07:22:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e62800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c1142d0 con 0x7f7e5c105280 2024-05-27T07:22:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e60a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c19d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e60a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c19d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53756/0 (socket says 172.21.15.12:53756) 2024-05-27T07:22:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e60a00700 1 -- 172.21.15.12:0/3701161261 learned_addr learned my addr 172.21.15.12:0/3701161261 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.469+0000 7f7e61400700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e5c105280 0x7f7e5c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.470+0000 7f7e61e00700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c10e6f0 0x7f7e5c1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.470+0000 7f7e60a00700 1 -- 172.21.15.12:0/3701161261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e5c105280 msgr2=0x7f7e5c19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.470+0000 7f7e60a00700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e5c105280 0x7f7e5c19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.470+0000 7f7e60a00700 1 -- 172.21.15.12:0/3701161261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c10e6f0 msgr2=0x7f7e5c1a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.470+0000 7f7e60a00700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c10e6f0 0x7f7e5c1a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.470+0000 7f7e60a00700 1 -- 172.21.15.12:0/3701161261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5c1a1bc0 con 0x7f7e5c105bd0 2024-05-27T07:22:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.470+0000 7f7e60a00700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c19d340 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7e48000c00 tx=0x7f7e480036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.471+0000 7f7e5a000700 1 -- 172.21.15.12:0/3701161261 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48027070 con 0x7f7e5c105bd0 2024-05-27T07:22:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.471+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5c1a1e50 con 0x7f7e5c105bd0 2024-05-27T07:22:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.471+0000 7f7e5a000700 1 -- 172.21.15.12:0/3701161261 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e4800fc80 con 0x7f7e5c105bd0 2024-05-27T07:22:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.471+0000 7f7e5a000700 1 -- 172.21.15.12:0/3701161261 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48022d60 con 0x7f7e5c105bd0 2024-05-27T07:22:10.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.471+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5c1a2330 con 0x7f7e5c105bd0 2024-05-27T07:22:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.475+0000 7f7e5a000700 1 -- 172.21.15.12:0/3701161261 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7e4802a070 con 0x7f7e5c105bd0 2024-05-27T07:22:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.476+0000 7f7e72999b80 1 Processor -- start 2024-05-27T07:22:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.476+0000 7f7e72999b80 1 -- start start 2024-05-27T07:22:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc7387e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc7394fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e72999b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561bc72b6df0 con 0x561bc73887d0 2024-05-27T07:22:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e72999b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561bc72a6990 con 0x561bc7209df0 2024-05-27T07:22:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e72999b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bc72aa9d0 con 0x561bc73883c0 2024-05-27T07:22:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e5a000700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7e440b2880 0x7f7e440b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e61e00700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7e440b2880 0x7f7e440b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e60a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc7394fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e61400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e61400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:52476/0 (socket says 172.21.15.12:52476) 2024-05-27T07:22:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e61400700 1 -- 172.21.15.12:0/2270131338 learned_addr learned my addr 172.21.15.12:0/2270131338 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e5a000700 1 -- 172.21.15.12:0/3701161261 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f7e480e35c0 con 0x7f7e5c105bd0 2024-05-27T07:22:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.477+0000 7f7e61e00700 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc7387e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.478+0000 7f7e60a00700 1 -- 172.21.15.12:0/2270131338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 msgr2=0x561bc7392a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.478+0000 7f7e60a00700 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.478+0000 7f7e60a00700 1 -- 172.21.15.12:0/2270131338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 msgr2=0x561bc7387e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.478+0000 7f7e60a00700 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc7387e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.478+0000 7f7e60a00700 1 -- 172.21.15.12:0/2270131338 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc726f3d0 con 0x561bc73887d0 2024-05-27T07:22:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.478+0000 7f7e61e00700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7e440b2880 0x7f7e440b4d90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400adc0 tx=0x7f7e54007f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.478+0000 7f7e60a00700 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc7394fd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7e480a7b10 tx=0x7f7e48020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6463345ac87920e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.479+0000 7f7e4ea00700 1 -- 172.21.15.12:0/2270131338 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480aa1f0 con 0x561bc73887d0 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.479+0000 7f7e4ea00700 1 -- 172.21.15.12:0/2270131338 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e480aa3b0 con 0x561bc73887d0 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.479+0000 7f7e4ea00700 1 -- 172.21.15.12:0/2270131338 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480aa010 con 0x561bc73887d0 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 -- 172.21.15.12:0/2270131338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 msgr2=0x561bc7394fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc7394fd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7e480a7b10 tx=0x7f7e48020000 comp rx=0 tx=0).stop 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 -- 172.21.15.12:0/2270131338 shutdown_connections 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc7387e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc7394fd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 --2- 172.21.15.12:0/2270131338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 -- 172.21.15.12:0/2270131338 >> 172.21.15.12:0/2270131338 conn(0x561bc7269f70 msgr2=0x561bc7383ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 -- 172.21.15.12:0/2270131338 shutdown_connections 2024-05-27T07:22:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.480+0000 7f7e72999b80 1 -- 172.21.15.12:0/2270131338 wait complete. 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.481+0000 7f7e72999b80 1 -- start start 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.481+0000 7f7e72999b80 1 Processor -- start 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.481+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc7209df0 0x561bc7392580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.481+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc73887d0 0x561bc7393000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e72999b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561bc71acf50 con 0x561bc7209df0 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e72999b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561bc7247710 con 0x561bc73887d0 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e72999b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bc7395700 con 0x561bc73883c0 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e72999b80 10 client.0 _ll_register_callbacks cb 0x561bc7250c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc7209df0 0x561bc7392580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc7209df0 0x561bc7392580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53772/0 (socket says 172.21.15.12:53772) 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 -- 172.21.15.12:0/1734556712 learned_addr learned my addr 172.21.15.12:0/1734556712 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e60a00700 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61e00700 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc73887d0 0x561bc7393000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 -- 172.21.15.12:0/1734556712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 msgr2=0x561bc7392ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 -- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc73887d0 msgr2=0x561bc7393000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc73887d0 0x561bc7393000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.482+0000 7f7e61400700 1 -- 172.21.15.12:0/1734556712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc71f6880 con 0x561bc7209df0 2024-05-27T07:22:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e61400700 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc7209df0 0x561bc7392580 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7e50011850 tx=0x7f7e50011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8059cd270e2cc7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e43400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e43400700 1 -- 172.21.15.12:0/1734556712 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50004460 con 0x561bc7209df0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e43400700 1 -- 172.21.15.12:0/1734556712 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f7e5000f8b0 con 0x561bc7209df0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e480aa3b0 con 0x561bc7209df0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e43400700 1 -- 172.21.15.12:0/1734556712 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7e50023050 con 0x561bc7209df0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e72999b80 10 client.15219 fetch_fsmap learned FSMap version 26 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.483+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561bc7277010 con 0x561bc7209df0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e43400700 1 -- 172.21.15.12:0/1734556712 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e480aa3b0 con 0x561bc7209df0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e43400700 1 -- 172.21.15.12:0/1734556712 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f7e50018020 con 0x561bc7209df0 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e72999b80 10 client.15219 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e72999b80 10 client.15219 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e72999b80 20 client.15219 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e72999b80 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc73a2650 0x561bc73a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e72999b80 4 client.15219 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.484+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bc72b8d40 con 0x561bc73a2650 2024-05-27T07:22:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.485+0000 7f7e60a00700 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc73a2650 0x561bc73a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.486+0000 7f7e60a00700 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc73a2650 0x561bc73a3ac0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=999b537ae2a07ba9 server_cookie=41c7a5bbad8624c5 in_seq=0 out_seq=0 2024-05-27T07:22:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.486+0000 7f7e43400700 10 client.15219 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e43400700 1 -- 172.21.15.12:0/1734556712 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561bc72b8d40 con 0x561bc73a2650 2024-05-27T07:22:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e43400700 10 client.15219 handle_command_reply: tid=0 2024-05-27T07:22:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 client.15219 shutdown 2024-05-27T07:22:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc7209df0 msgr2=0x561bc7392580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc7209df0 0x561bc7392580 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7e50011850 tx=0x7f7e50011c10 comp rx=0 tx=0).stop 2024-05-27T07:22:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 shutdown_connections 2024-05-27T07:22:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc73a2650 0x561bc73a3ac0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc73887d0 0x561bc7393000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc7209df0 0x561bc7392580 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 --2- 172.21.15.12:0/1734556712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc7392ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 >> 172.21.15.12:0/1734556712 conn(0x561bc7269f70 msgr2=0x561bc7382fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.488+0000 7f7e43400700 0 client.15219 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.489+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 shutdown_connections 2024-05-27T07:22:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.489+0000 7f7e72999b80 1 -- 172.21.15.12:0/1734556712 wait complete. 2024-05-27T07:22:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.489+0000 7f7e72999b80 20 client.15219 trim_cache size 0 max 16384 2024-05-27T07:22:10.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.512+0000 7f7e72999b80 1 Processor -- start 2024-05-27T07:22:10.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.512+0000 7f7e72999b80 1 -- start start 2024-05-27T07:22:10.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc738c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73883c0 0x561bc738dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73887d0 0x561bc7383770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e72999b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a1e00 con 0x561bc73883c0 2024-05-27T07:22:10.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e72999b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a24d0 con 0x561bc7209df0 2024-05-27T07:22:10.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e72999b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a2ba0 con 0x561bc73887d0 2024-05-27T07:22:10.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e60a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73883c0 0x561bc738dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e60a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73883c0 0x561bc738dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53774/0 (socket says 172.21.15.12:53774) 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e60a00700 1 -- 172.21.15.12:0/4229472256 learned_addr learned my addr 172.21.15.12:0/4229472256 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e61e00700 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73887d0 0x561bc7383770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.513+0000 7f7e61400700 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc738c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e60a00700 1 -- 172.21.15.12:0/4229472256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73887d0 msgr2=0x561bc7383770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e60a00700 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73887d0 0x561bc7383770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e60a00700 1 -- 172.21.15.12:0/4229472256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 msgr2=0x561bc738c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e60a00700 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc738c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e60a00700 1 -- 172.21.15.12:0/4229472256 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc737e180 con 0x561bc73883c0 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e61e00700 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73887d0 0x561bc7383770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:10.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e61400700 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc738c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e60a00700 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73883c0 0x561bc738dd80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7e480e1060 tx=0x7f7e48022470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30c27e4f9f182a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e4e000700 1 -- 172.21.15.12:0/4229472256 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480aa8a0 con 0x561bc73883c0 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e4e000700 1 -- 172.21.15.12:0/4229472256 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e480aec50 con 0x561bc73883c0 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.514+0000 7f7e4e000700 1 -- 172.21.15.12:0/4229472256 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480e32d0 con 0x561bc73883c0 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 -- 172.21.15.12:0/4229472256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73883c0 msgr2=0x561bc738dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73883c0 0x561bc738dd80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7e480e1060 tx=0x7f7e48022470 comp rx=0 tx=0).stop 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 -- 172.21.15.12:0/4229472256 shutdown_connections 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc738c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73887d0 0x561bc7383770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 --2- 172.21.15.12:0/4229472256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73883c0 0x561bc738dd80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 -- 172.21.15.12:0/4229472256 >> 172.21.15.12:0/4229472256 conn(0x561bc7269f70 msgr2=0x561bc7390e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.515+0000 7f7e72999b80 1 -- 172.21.15.12:0/4229472256 shutdown_connections 2024-05-27T07:22:10.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.516+0000 7f7e72999b80 1 -- 172.21.15.12:0/4229472256 wait complete. 2024-05-27T07:22:10.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.516+0000 7f7e72999b80 1 -- start start 2024-05-27T07:22:10.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.516+0000 7f7e72999b80 1 Processor -- start 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc73a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc73a7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e72999b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc73a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e72999b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a3940 con 0x561bc73887d0 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e72999b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a3270 con 0x561bc7209df0 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e72999b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a4010 con 0x561bc73883c0 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e72999b80 10 client.0 _ll_register_callbacks cb 0x561bc7250c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.517+0000 7f7e61e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc73a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc73a75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53776/0 (socket says 172.21.15.12:53776) 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 -- 172.21.15.12:0/3527392635 learned_addr learned my addr 172.21.15.12:0/3527392635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61400700 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc73a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e60a00700 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc73a7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 -- 172.21.15.12:0/3527392635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 msgr2=0x561bc73a7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc73a7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 -- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 msgr2=0x561bc73a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:10.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc73a6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 -- 172.21.15.12:0/3527392635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc7385300 con 0x561bc73887d0 2024-05-27T07:22:10.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e61e00700 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc73a75d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e540079b0 tx=0x7f7e54006090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbda4c91c24c4a35 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.518+0000 7f7e43400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:10.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.519+0000 7f7e43400700 1 -- 172.21.15.12:0/3527392635 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e54029030 con 0x561bc73887d0 2024-05-27T07:22:10.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.519+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561bc72b2520 con 0x561bc73887d0 2024-05-27T07:22:10.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.519+0000 7f7e43400700 1 -- 172.21.15.12:0/3527392635 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f7e54020050 con 0x561bc73887d0 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.519+0000 7f7e43400700 1 -- 172.21.15.12:0/3527392635 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7e54033050 con 0x561bc73887d0 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.519+0000 7f7e72999b80 10 client.15225 fetch_fsmap learned FSMap version 26 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.519+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561bc740dde0 con 0x561bc73887d0 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e43400700 1 -- 172.21.15.12:0/3527392635 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x561bc72b2520 con 0x561bc73887d0 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e43400700 1 -- 172.21.15.12:0/3527392635 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f7e5401a460 con 0x561bc73887d0 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e72999b80 10 client.15225 fetch_fsmap finished waiting for FSMap version 26 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e72999b80 10 client.15225 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e72999b80 20 client.15225 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e72999b80 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc74101e0 0x561bc7411650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e72999b80 4 client.15225 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.520+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "status"}) -- 0x561bc7129200 con 0x561bc74101e0 2024-05-27T07:22:10.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.521+0000 7f7e61400700 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc74101e0 0x561bc7411650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.521+0000 7f7e61400700 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc74101e0 0x561bc7411650 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62d027d6948ff51b server_cookie=61c1702fbe5468bf in_seq=0 out_seq=0 2024-05-27T07:22:10.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.521+0000 7f7e43400700 10 client.15225 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e43400700 1 -- 172.21.15.12:0/3527392635 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561bc7129200 con 0x561bc74101e0 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e43400700 10 client.15225 handle_command_reply: tid=0 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 client.15225 shutdown 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 msgr2=0x561bc73a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc73a75d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e540079b0 tx=0x7f7e54006090 comp rx=0 tx=0).stop 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 shutdown_connections 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561bc74101e0 0x561bc7411650 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc7209df0 0x561bc73a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561bc73887d0 0x561bc73a75d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.522+0000 7f7e72999b80 1 --2- 172.21.15.12:0/3527392635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bc73883c0 0x561bc73a7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.523+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 >> 172.21.15.12:0/3527392635 conn(0x561bc7269f70 msgr2=0x561bc7383610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.523+0000 7f7e43400700 0 client.15225 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:10.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.523+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 shutdown_connections 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.523+0000 7f7e72999b80 1 -- 172.21.15.12:0/3527392635 wait complete. 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.523+0000 7f7e72999b80 20 client.15225 trim_cache size 0 max 16384 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "55c251ba-1bf7-11ef-bc9b-c7b262605968", 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 34475, 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-05-27T07:22:10.515 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 299.73569859200001, 2024-05-27T07:22:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 26, 2024-05-27T07:22:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 89, 2024-05-27T07:22:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 87, 2024-05-27T07:22:10.516 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 319.97955435799997 2024-05-27T07:22:10.516 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:22:10.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7e440b2880 msgr2=0x7f7e440b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7e440b2880 0x7f7e440b4d90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400adc0 tx=0x7f7e54007f50 comp rx=0 tx=0).stop 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 msgr2=0x7f7e5c19d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c19d340 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7e48000c00 tx=0x7f7e480036f0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 shutdown_connections 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f7e440b2880 0x7f7e440b4d90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c10e6f0 0x7f7e5c1a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e5c105280 0x7f7e5c19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 --2- 172.21.15.12:0/3701161261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e5c105bd0 0x7f7e5c19d340 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 >> 172.21.15.12:0/3701161261 conn(0x7f7e5c078af0 msgr2=0x7f7e5c0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 shutdown_connections 2024-05-27T07:22:10.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:10.524+0000 7f7e62800700 1 -- 172.21.15.12:0/3701161261 wait complete. 2024-05-27T07:22:11.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:11 smithi012 ceph-mon[31473]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:22:11.443 INFO:teuthology.run_tasks:Running task sequential... 2024-05-27T07:22:11.454 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-27T07:22:11.454 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-27T07:22:11.455 INFO:teuthology.task.print:Enabling mgr modules 2024-05-27T07:22:11.455 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-27T07:22:11.455 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:22:11.456 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:22:11.456 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-27T07:22:11.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:11 smithi162 ceph-mon[35676]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:22:11.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:11 smithi062 ceph-mon[35304]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-27T07:22:11.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.751+0000 7fa967200640 1 -- 172.21.15.12:0/2842259859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa968110060 msgr2=0x7fa968112480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.751+0000 7fa967200640 1 --2- 172.21.15.12:0/2842259859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa968110060 0x7fa968112480 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa960009ca0 tx=0x7fa96002a8c0 comp rx=0 tx=0).stop 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.752+0000 7fa967200640 1 -- 172.21.15.12:0/2842259859 shutdown_connections 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.752+0000 7fa967200640 1 --2- 172.21.15.12:0/2842259859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa968110060 0x7fa968112480 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.752+0000 7fa967200640 1 --2- 172.21.15.12:0/2842259859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa96810b360 0x7fa96810b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.752+0000 7fa967200640 1 --2- 172.21.15.12:0/2842259859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa96810a3f0 0x7fa96810ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.752+0000 7fa967200640 1 -- 172.21.15.12:0/2842259859 >> 172.21.15.12:0/2842259859 conn(0x7fa968078d20 msgr2=0x7fa96807b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.752+0000 7fa967200640 1 -- 172.21.15.12:0/2842259859 shutdown_connections 2024-05-27T07:22:11.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.752+0000 7fa967200640 1 -- 172.21.15.12:0/2842259859 wait complete. 2024-05-27T07:22:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.753+0000 7fa967200640 1 Processor -- start 2024-05-27T07:22:11.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.753+0000 7fa967200640 1 -- start start 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa967200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa96810a3f0 0x7fa9681a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa967200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa96810b360 0x7fa9681a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa967200640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa968110060 0x7fa9681a7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa967200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa968115aa0 con 0x7fa96810a3f0 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa967200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa968115920 con 0x7fa96810b360 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa967200640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa968115c20 con 0x7fa968110060 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa965e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa96810a3f0 0x7fa9681a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa965e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa96810a3f0 0x7fa9681a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53788/0 (socket says 172.21.15.12:53788) 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.754+0000 7fa965e00640 1 -- 172.21.15.12:0/2550405011 learned_addr learned my addr 172.21.15.12:0/2550405011 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:11.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa966800640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa968110060 0x7fa9681a7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:11.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965400640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa96810b360 0x7fa9681a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965e00640 1 -- 172.21.15.12:0/2550405011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa968110060 msgr2=0x7fa9681a7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965e00640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa968110060 0x7fa9681a7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965e00640 1 -- 172.21.15.12:0/2550405011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa96810b360 msgr2=0x7fa9681a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965e00640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa96810b360 0x7fa9681a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965e00640 1 -- 172.21.15.12:0/2550405011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9681a8180 con 0x7fa96810a3f0 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965400640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa96810b360 0x7fa9681a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.755+0000 7fa965e00640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa96810a3f0 0x7fa9681a0640 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa95800d770 tx=0x7fa95800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.756+0000 7fa95ea00640 1 -- 172.21.15.12:0/2550405011 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa958017070 con 0x7fa96810a3f0 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.756+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9681a8470 con 0x7fa96810a3f0 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.756+0000 7fa95ea00640 1 -- 172.21.15.12:0/2550405011 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa958009c30 con 0x7fa96810a3f0 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.756+0000 7fa95ea00640 1 -- 172.21.15.12:0/2550405011 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95800f620 con 0x7fa96810a3f0 2024-05-27T07:22:11.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.756+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9681a8a10 con 0x7fa96810a3f0 2024-05-27T07:22:11.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.757+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa96810ee70 con 0x7fa96810a3f0 2024-05-27T07:22:11.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.760+0000 7fa95ea00640 1 -- 172.21.15.12:0/2550405011 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa95801a070 con 0x7fa96810a3f0 2024-05-27T07:22:11.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.761+0000 7fa95ea00640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa94c0b2860 0x7fa94c0b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.762+0000 7fa95ea00640 1 -- 172.21.15.12:0/2550405011 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fa9580e2160 con 0x7fa96810a3f0 2024-05-27T07:22:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.762+0000 7fa965400640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa94c0b2860 0x7fa94c0b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.762+0000 7fa965400640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa94c0b2860 0x7fa94c0b4d70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa954005fd0 tx=0x7fa954005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:11.767+0000 7fa95ea00640 1 -- 172.21.15.12:0/2550405011 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa958009de0 con 0x7fa96810a3f0 2024-05-27T07:22:12.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.037+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fa9680638c0 con 0x7fa96810a3f0 2024-05-27T07:22:12.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.085+0000 7fa95ea00640 1 -- 172.21.15.12:0/2550405011 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fa9580a9a50 con 0x7fa96810a3f0 2024-05-27T07:22:12.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa94c0b2860 msgr2=0x7fa94c0b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:12.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa94c0b2860 0x7fa94c0b4d70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa954005fd0 tx=0x7fa954005850 comp rx=0 tx=0).stop 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa96810a3f0 msgr2=0x7fa9681a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa96810a3f0 0x7fa9681a0640 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa95800d770 tx=0x7fa95800dc30 comp rx=0 tx=0).stop 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 shutdown_connections 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fa94c0b2860 0x7fa94c0b4d70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa968110060 0x7fa9681a7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa96810b360 0x7fa9681a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 --2- 172.21.15.12:0/2550405011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa96810a3f0 0x7fa9681a0640 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 >> 172.21.15.12:0/2550405011 conn(0x7fa968078d20 msgr2=0x7fa96810c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.090+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 shutdown_connections 2024-05-27T07:22:12.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:12.091+0000 7fa967200640 1 -- 172.21.15.12:0/2550405011 wait complete. 2024-05-27T07:22:12.133 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-27T07:22:12.146 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-27T07:22:12.146 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-05-27T07:22:12.146 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-05-27T07:22:12.146 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:22:12.146 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:12.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:12.147 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-05-27T07:22:12.199 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-27T07:22:12.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:12.200 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:22:12.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:12.278 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-05-27T07:22:12.316 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:12 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2550405011' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:22:12.345 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:12 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2550405011' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:22:12.390 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-27T07:22:12.390 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:22:12.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:12 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2550405011' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:22:12.473 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: ignoring --setuser ceph since I am not root 2024-05-27T07:22:12.474 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: ignoring --setgroup ceph since I am not root 2024-05-27T07:22:12.474 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:12 smithi162 ceph-mgr[37559]: -- 172.21.15.162:0/339693397 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b7ac165a0 con 0x556b7f9a0480 2024-05-27T07:22:12.590 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: ignoring --setuser ceph since I am not root 2024-05-27T07:22:12.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: ignoring --setgroup ceph since I am not root 2024-05-27T07:22:12.591 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:12.526+0000 7f94a3b61200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:22:12.736 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: ignoring --setuser ceph since I am not root 2024-05-27T07:22:12.736 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: ignoring --setgroup ceph since I am not root 2024-05-27T07:22:12.736 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:12 smithi062 ceph-mgr[36682]: -- 172.21.15.62:0/3473888837 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cf36c7c5a0 con 0x55cf3ba06480 2024-05-27T07:22:12.736 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:12.543+0000 7f927e77d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:22:12.746 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:12.495+0000 7fdc9de53200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:22:12.746 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:12.755+0000 7fdc9de53200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:22:12.845 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:13.051 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:12.797+0000 7f94a3b61200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:22:13.051 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:12.966+0000 7f94a3b61200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:22:13.079 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:12.890+0000 7fdc9de53200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:22:13.080 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:12.807+0000 7f927e77d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:22:13.080 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:12.946+0000 7f927e77d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:22:13.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:13 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2550405011' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:22:13.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:13 smithi012 ceph-mon[31473]: mgrmap e19: x(active, since 15m), standbys: z, y 2024-05-27T07:22:13.351 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:13 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2550405011' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:22:13.351 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:13 smithi062 ceph-mon[35304]: mgrmap e19: x(active, since 15m), standbys: z, y 2024-05-27T07:22:13.397 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:13 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2550405011' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:22:13.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:13 smithi162 ceph-mon[35676]: mgrmap e19: x(active, since 15m), standbys: z, y 2024-05-27T07:22:13.398 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:13.288+0000 7fdc9de53200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:22:13.723 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:13.406+0000 7fdc9de53200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:22:13.735 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:13.359+0000 7f927e77d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:22:13.735 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:13.479+0000 7f927e77d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:22:13.840 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:13.412+0000 7f94a3b61200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:22:13.840 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:13.535+0000 7f94a3b61200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:22:15.223 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:14.958+0000 7fdc9de53200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:22:15.223 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:15.190+0000 7fdc9de53200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:22:15.340 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:15.182+0000 7f94a3b61200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.380+0000 7fe001000700 1 -- 172.21.15.12:0/647691501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 msgr2=0x7fdffc1049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.380+0000 7fe001000700 1 --2- 172.21.15.12:0/647691501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc1049d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0099c0 tx=0x7fdfec009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.381+0000 7fe001000700 1 -- 172.21.15.12:0/647691501 shutdown_connections 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.381+0000 7fe001000700 1 --2- 172.21.15.12:0/647691501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdffc109840 0x7fdffc1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.381+0000 7fe001000700 1 --2- 172.21.15.12:0/647691501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc1049d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.381+0000 7fe001000700 1 --2- 172.21.15.12:0/647691501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdffc108ef0 0x7fdffc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.381+0000 7fe001000700 1 -- 172.21.15.12:0/647691501 >> 172.21.15.12:0/647691501 conn(0x7fdffc0ffe90 msgr2=0x7fdffc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.381+0000 7fe001000700 1 -- 172.21.15.12:0/647691501 shutdown_connections 2024-05-27T07:22:15.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.381+0000 7fe001000700 1 -- 172.21.15.12:0/647691501 wait complete. 2024-05-27T07:22:15.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.382+0000 7fe001000700 1 Processor -- start 2024-05-27T07:22:15.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fe001000700 1 -- start start 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fe001000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fe001000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdffc108ef0 0x7fdffc077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fe001000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdffc109840 0x7fdffc077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fe001000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc114320 con 0x7fdffc1045c0 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fe001000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc1141a0 con 0x7fdffc108ef0 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fe001000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc1144a0 con 0x7fdffc109840 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fdff9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fdff9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc076bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50466/0 (socket says 172.21.15.12:50466) 2024-05-27T07:22:15.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.383+0000 7fdff9e00700 1 -- 172.21.15.12:0/1064382632 learned_addr learned my addr 172.21.15.12:0/1064382632 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.384+0000 7fdff9400700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdffc108ef0 0x7fdffc077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.384+0000 7fdffa800700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdffc109840 0x7fdffc077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.384+0000 7fdff9e00700 1 -- 172.21.15.12:0/1064382632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdffc109840 msgr2=0x7fdffc077640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.384+0000 7fdff9e00700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdffc109840 0x7fdffc077640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.384+0000 7fdff9e00700 1 -- 172.21.15.12:0/1064382632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdffc108ef0 msgr2=0x7fdffc077100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.384+0000 7fdff9e00700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdffc108ef0 0x7fdffc077100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.384+0000 7fdff9e00700 1 -- 172.21.15.12:0/1064382632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc1a9150 con 0x7fdffc1045c0 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.385+0000 7fdff9e00700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc076bc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdfec006010 tx=0x7fdfec0055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.385+0000 7fdff2a00700 1 -- 172.21.15.12:0/1064382632 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec026070 con 0x7fdffc1045c0 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.385+0000 7fdff2a00700 1 -- 172.21.15.12:0/1064382632 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfec00f460 con 0x7fdffc1045c0 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.385+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdffc1a93e0 con 0x7fdffc1045c0 2024-05-27T07:22:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.385+0000 7fdff2a00700 1 -- 172.21.15.12:0/1064382632 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec009e60 con 0x7fdffc1045c0 2024-05-27T07:22:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.385+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdffc1a9830 con 0x7fdffc1045c0 2024-05-27T07:22:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.387+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdffc0512b0 con 0x7fdffc1045c0 2024-05-27T07:22:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.389+0000 7fdff2a00700 1 -- 172.21.15.12:0/1064382632 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7fdfec029070 con 0x7fdffc1045c0 2024-05-27T07:22:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.391+0000 7fdff2a00700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 0x7fdfe00b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.391+0000 7fdff9400700 1 -- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 msgr2=0x7fdfe00b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.391+0000 7fdff2a00700 1 -- 172.21.15.12:0/1064382632 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7fdfec0e3470 con 0x7fdffc1045c0 2024-05-27T07:22:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.391+0000 7fdff9400700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 0x7fdfe00b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:22:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.397+0000 7fdff2a00700 1 -- 172.21.15.12:0/1064382632 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdfec030080 con 0x7fdffc1045c0 2024-05-27T07:22:15.485 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:15.114+0000 7f927e77d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:22:15.486 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:15.354+0000 7f927e77d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:22:15.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.592+0000 7fdff9400700 1 -- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 msgr2=0x7fdfe00b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:15.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.592+0000 7fdff9400700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 0x7fdfe00b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:22:15.658 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:15.438+0000 7f94a3b61200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:22:15.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.683+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdffc02e5e0 con 0x7fdffc1045c0 2024-05-27T07:22:15.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.684+0000 7fdff2a00700 1 -- 172.21.15.12:0/1064382632 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+88 (secure 0 0 0) 0x7fdfec013070 con 0x7fdffc1045c0 2024-05-27T07:22:15.674 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:22:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 msgr2=0x7fdfe00b4de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:22:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 0x7fdfe00b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 msgr2=0x7fdffc076bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:15.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc076bc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdfec006010 tx=0x7fdfec0055f0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 shutdown_connections 2024-05-27T07:22:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7fdfe00b28d0 0x7fdfe00b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdffc109840 0x7fdffc077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdffc1045c0 0x7fdffc076bc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 --2- 172.21.15.12:0/1064382632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdffc108ef0 0x7fdffc077100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:15.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 >> 172.21.15.12:0/1064382632 conn(0x7fdffc0ffe90 msgr2=0x7fdffc1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 shutdown_connections 2024-05-27T07:22:15.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:15.687+0000 7fe001000700 1 -- 172.21.15.12:0/1064382632 wait complete. 2024-05-27T07:22:15.723 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:15.409+0000 7fdc9de53200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:22:15.974 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:15.578+0000 7f927e77d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:22:15.974 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:15 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1064382632' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:22:16.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:15 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1064382632' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:22:16.090 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:15.667+0000 7f94a3b61200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:15 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1064382632' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:15.800+0000 7fdc9de53200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:15.925+0000 7fdc9de53200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:22:16.235 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:15.982+0000 7f927e77d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:22:16.236 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:16.112+0000 7f927e77d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:22:16.437 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:16.182+0000 7fdc9de53200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:22:16.437 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:16.445+0000 7fdc9de53200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:22:16.494 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:16.101+0000 7f94a3b61200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:22:16.494 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:16.234+0000 7f94a3b61200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:22:16.594 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:22:16.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:16.595 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-05-27T07:22:16.635 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:22:16.635 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:22:16.635 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:22:16.635 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:22:16.635 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-05-27T07:22:16.635 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:22:16.635 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 41535sec preferred_lft 41535sec 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:22:16.636 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:22:16.637 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:16.637 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-05-27T07:22:16.637 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:22:16.637 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:16.637 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:22:16.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:22:16.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-05-27T07:22:16.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-05-27T07:22:16.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:22:16.638 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:22:16.638 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:16.375+0000 7f927e77d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:22:16.770 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:16.502+0000 7f94a3b61200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:22:16.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:16.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:16.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:16.915 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:22:16.979 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-05-27T07:22:16.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:16.984 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-05-27T07:22:16.985 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:16.645+0000 7f927e77d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:22:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-05-27T07:22:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:22:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-27T07:22:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-27T07:22:17.032 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:22:17.033 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:17.033 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:22:17.033 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-27T07:22:17.033 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-27T07:22:17.033 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-27T07:22:17.033 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:22:17.089 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:16.779+0000 7f94a3b61200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:22:17.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:17.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:17.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:17.246 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-05-27T07:22:17.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:17.286 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-05-27T07:22:17.380 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:17.380 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:22:17.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:22:17.380 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-27T07:22:17.380 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:22:17.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:17.539 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:17.253+0000 7fdc9de53200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:22:17.540 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:17.548+0000 7fdc9de53200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:22:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:17.569 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-27T07:22:17.569 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:17.569 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:22:17.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-27T07:22:17.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:22:17.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-27T07:22:17.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-27T07:22:17.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-27T07:22:17.570 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:22:17.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:17.735 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:17.476+0000 7f927e77d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:22:17.764 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:17.659+0000 7f94a3b61200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:22:17.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:17.831 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:17.831 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-05-27T07:22:17.831 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-05-27T07:22:17.831 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:22:17.831 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-05-27T07:22:17.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:18.023 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-05-27T07:22:18.023 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:22:18.023 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:18.061 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:22:18.061 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:22:18.061 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:18.089 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:17.977+0000 7f94a3b61200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:22:18.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-05-27T07:22:18.235 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:17.781+0000 7f927e77d200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:22:18.248 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:22:18.292 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-05-27T07:22:18.293 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-05-27T07:22:18.293 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-05-27T07:22:18.293 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-05-27T07:22:18.293 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-05-27T07:22:18.293 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-05-27T07:22:18.293 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-05-27T07:22:18.294 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-05-27T07:22:18.294 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-05-27T07:22:18.294 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-05-27T07:22:18.294 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-05-27T07:22:18.294 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-05-27T07:22:18.295 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:18.295 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-05-27T07:22:18.295 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-05-27T07:22:18.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-05-27T07:22:18.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-05-27T07:22:18.295 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-05-27T07:22:18.296 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-05-27T07:22:18.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:22:18.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:22:18.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:22:18.296 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:22:18.296 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-05-27T07:22:18.297 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-05-27T07:22:18.297 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:22:18.297 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/7fbf726264480c421aca3218d2013f70788ff034dc144fceebe5ec0cf52bf1ea/merged 2024-05-27T07:22:18.297 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/60aebd048ed37d2028603da24217d71224cd3b5b0b0f883e05ded68b3cc907fc/merged 2024-05-27T07:22:18.297 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-05-27T07:22:18.298 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:18.298 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/c9e05f2c227477b28c8a4ec8fdd7d2f74da4ac11eb899b318ba9d13c11d9b36f/merged 2024-05-27T07:22:18.298 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/4e3cfc19c14a3fac132bd4f33d5fc2900f75567b100b01f5c8d277aef4cf6142/merged 2024-05-27T07:22:18.298 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/fb7cf68777b803513b45652e1a3fd26be65494cc1d3be62a4780623b2b3ea83f/merged 2024-05-27T07:22:18.298 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b5c7dd98ced0dddde9eefe8d543de5f5d6900d4fe9e6a5de2a13bbc6fd3cdba0/merged 2024-05-27T07:22:18.298 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/86ac1d463a1fee0b386559beedf9fdd66f0fc75c2cb77e088b477352a78b5695/merged 2024-05-27T07:22:18.299 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/453ea3f6e74c1b2ec3db112190abe48827bb5baa90b63bb79c724a315baf8761/merged 2024-05-27T07:22:18.299 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/87eafa0cfd1573804569dce33ed5df40a6693954e5f82037af861b94f0049c6d/merged 2024-05-27T07:22:18.299 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/0df1da80208360021a4377028bf943919a9988a7dfc79ce6406fa6fde4521ec0/merged 2024-05-27T07:22:18.299 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-05-27T07:22:18.299 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:22:18.299 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:22:18.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:18.301 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-05-27T07:22:18.395 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-27T07:22:18.395 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/9515fe3d-085a-4399-9da5-21c1d3d75134 --client_fs=cephfs) 2024-05-27T07:22:18.400 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-05-27T07:22:18.462 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:22:18.473 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:18.115+0000 7fdc9de53200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:22:18.473 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:18.244+0000 7fdc9de53200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-05-27T07:22:18.502 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-05-27T07:22:18.503 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/7fbf726264480c421aca3218d2013f70788ff034dc144fceebe5ec0cf52bf1ea/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/60aebd048ed37d2028603da24217d71224cd3b5b0b0f883e05ded68b3cc907fc/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/c9e05f2c227477b28c8a4ec8fdd7d2f74da4ac11eb899b318ba9d13c11d9b36f/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/4e3cfc19c14a3fac132bd4f33d5fc2900f75567b100b01f5c8d277aef4cf6142/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/fb7cf68777b803513b45652e1a3fd26be65494cc1d3be62a4780623b2b3ea83f/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b5c7dd98ced0dddde9eefe8d543de5f5d6900d4fe9e6a5de2a13bbc6fd3cdba0/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/86ac1d463a1fee0b386559beedf9fdd66f0fc75c2cb77e088b477352a78b5695/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/453ea3f6e74c1b2ec3db112190abe48827bb5baa90b63bb79c724a315baf8761/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/87eafa0cfd1573804569dce33ed5df40a6693954e5f82037af861b94f0049c6d/merged 2024-05-27T07:22:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/0df1da80208360021a4377028bf943919a9988a7dfc79ce6406fa6fde4521ec0/merged 2024-05-27T07:22:18.505 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-05-27T07:22:18.505 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:22:18.505 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:22:18.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:18.506 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-05-27T07:22:18.632 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:2024-05-27T07:22:18.641+0000 7fa7d5d3d480 -1 init, newargv = 0x7fa7b8004360 newargc=15 2024-05-27T07:22:18.632 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[90645]: starting ceph client 2024-05-27T07:22:18.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[90645]: starting fuse 2024-05-27T07:22:18.737 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:18.370+0000 7f927e77d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:22:18.737 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:18.505+0000 7f927e77d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:22:18.840 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:18.580+0000 7f94a3b61200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:22:18.840 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:18.727+0000 7f94a3b61200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:22:18.973 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:18.726+0000 7fdc9de53200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:22:19.466 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:18.998+0000 7f927e77d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:22:19.467 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:19.475+0000 7f927e77d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:22:19.473 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:19.190+0000 7fdc9de53200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:22:19.473 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:19.316+0000 7fdc9de53200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:22:19.590 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:19.252+0000 7f94a3b61200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:22:19.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo modprobe fuse 2024-05-27T07:22:19.735 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:19.608+0000 7f927e77d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:22:19.770 DEBUG:teuthology.orchestra.run.smithi012:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:22:19.820 INFO:teuthology.orchestra.run.smithi012.stdout:/proc 2024-05-27T07:22:19.820 INFO:teuthology.orchestra.run.smithi012.stdout:/sys 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/dev 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/security 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/shm 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/pts 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/run 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/cgroup 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/pstore 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/bpf 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/ 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/selinux 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/mqueue 2024-05-27T07:22:19.821 INFO:teuthology.orchestra.run.smithi012.stdout:/dev/hugepages 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/debug 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/tracing 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/kernel/config 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/sys/fs/fuse/connections 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/1000 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/ceph 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:22:19.822 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/7fbf726264480c421aca3218d2013f70788ff034dc144fceebe5ec0cf52bf1ea/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/60aebd048ed37d2028603da24217d71224cd3b5b0b0f883e05ded68b3cc907fc/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/run/user/0 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/c9e05f2c227477b28c8a4ec8fdd7d2f74da4ac11eb899b318ba9d13c11d9b36f/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/4e3cfc19c14a3fac132bd4f33d5fc2900f75567b100b01f5c8d277aef4cf6142/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/fb7cf68777b803513b45652e1a3fd26be65494cc1d3be62a4780623b2b3ea83f/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/b5c7dd98ced0dddde9eefe8d543de5f5d6900d4fe9e6a5de2a13bbc6fd3cdba0/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/86ac1d463a1fee0b386559beedf9fdd66f0fc75c2cb77e088b477352a78b5695/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/453ea3f6e74c1b2ec3db112190abe48827bb5baa90b63bb79c724a315baf8761/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/87eafa0cfd1573804569dce33ed5df40a6693954e5f82037af861b94f0049c6d/merged 2024-05-27T07:22:19.823 INFO:teuthology.orchestra.run.smithi012.stdout:/var/lib/containers/storage/overlay/0df1da80208360021a4377028bf943919a9988a7dfc79ce6406fa6fde4521ec0/merged 2024-05-27T07:22:19.824 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns 2024-05-27T07:22:19.824 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:22:19.824 INFO:teuthology.orchestra.run.smithi012.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:22:19.824 INFO:teuthology.orchestra.run.smithi012.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:19.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:19.825 DEBUG:teuthology.orchestra.run.smithi012:> ls /sys/fs/fuse/connections 2024-05-27T07:22:19.869 INFO:teuthology.orchestra.run.smithi012.stdout:119 2024-05-27T07:22:19.869 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-05-27T07:22:19.869 DEBUG:teuthology.orchestra.run.smithi012:> sudo stdin-killer -- python3 -c ' 2024-05-27T07:22:19.869 DEBUG:teuthology.orchestra.run.smithi012:> import glob 2024-05-27T07:22:19.869 DEBUG:teuthology.orchestra.run.smithi012:> import re 2024-05-27T07:22:19.869 DEBUG:teuthology.orchestra.run.smithi012:> import os 2024-05-27T07:22:19.869 DEBUG:teuthology.orchestra.run.smithi012:> import subprocess 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> def _find_admin_socket(client_name): 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> files = glob.glob(asok_path) 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> # Given a non-glob path, it better be there 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> if "*" not in asok_path: 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> assert(len(files) == 1) 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> return files[0] 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> for f in files: 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-27T07:22:19.870 DEBUG:teuthology.orchestra.run.smithi012:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> contents = proc_f.read() 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> if mountpoint in contents: 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> return f 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> print(_find_admin_socket("client.0")) 2024-05-27T07:22:19.871 DEBUG:teuthology.orchestra.run.smithi012:> ' 2024-05-27T07:22:19.953 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:19.646+0000 7fdc9de53200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:22:19.953 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:19.829+0000 7fdc9de53200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:22:19.981 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:19.738+0000 7f94a3b61200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:22:19.982 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:19.877+0000 7f94a3b61200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:22:20.074 INFO:teuthology.orchestra.run.smithi012.stdout:/var/run/ceph/ceph-client.0.90645.asok 2024-05-27T07:22:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:20.091 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.90645.asok 2024-05-27T07:22:20.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:20.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.90645.asok status 2024-05-27T07:22:20.223 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:19.959+0000 7fdc9de53200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:22:20.235 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:19.950+0000 7f927e77d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:22:20.235 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:20.139+0000 7f927e77d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:22:20.340 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:20.223+0000 7f94a3b61200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:22:20.389 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:22:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "metadata": { 2024-05-27T07:22:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_sha1": "38f9d900c83c63799fbdbe61acc9a11b0d3554a6", 2024-05-27T07:22:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:22:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "entity_id": "0", 2024-05-27T07:22:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": "smithi012", 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "pid": "90645", 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "root": "/volumes/qa/sv_0/9515fe3d-085a-4399-9da5-21c1d3d75134" 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_count": 0, 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "dentry_pinned_count": 0, 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 15258, 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "inst": { 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "name": { 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "client", 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 15258 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:22:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2124615107 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": { 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2124615107 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "inst_str": "client.15258 192.168.0.1:0/2124615107", 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "addr_str": "192.168.0.1:0/2124615107", 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "inode_count": 5, 2024-05-27T07:22:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "mds_epoch": 26, 2024-05-27T07:22:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch": 89, 2024-05-27T07:22:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "osd_epoch_barrier": 0, 2024-05-27T07:22:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "blocklisted": false, 2024-05-27T07:22:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs" 2024-05-27T07:22:20.392 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:22:20.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:20.410 DEBUG:teuthology.orchestra.run.smithi012:> stat --file-system '--printf=%T 2024-05-27T07:22:20.410 DEBUG:teuthology.orchestra.run.smithi012:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:20.459 INFO:teuthology.orchestra.run.smithi012.stdout:fuseblk 2024-05-27T07:22:20.459 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:20.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:20.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:20.536 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T07:22:20.546 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:22:20.546 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:22:20.546 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-05-27T07:22:20.682 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:20.275+0000 7f927e77d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:22:20.723 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:20.370+0000 7fdc9de53200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:22:20.723 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:20.496+0000 7fdc9de53200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 -- 172.21.15.12:0/3399227640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 msgr2=0x7f49281067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 --2- 172.21.15.12:0/3399227640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f49281067b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4918009900 tx=0x7f491802a3f0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 -- 172.21.15.12:0/3399227640 shutdown_connections 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 --2- 172.21.15.12:0/3399227640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49281098f0 0x7f492810bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 --2- 172.21.15.12:0/3399227640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4928106d80 0x7f4928109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 --2- 172.21.15.12:0/3399227640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f49281067b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 -- 172.21.15.12:0/3399227640 >> 172.21.15.12:0/3399227640 conn(0x7f49280fdd30 msgr2=0x7f49281001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:20.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 -- 172.21.15.12:0/3399227640 shutdown_connections 2024-05-27T07:22:20.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.825+0000 7f492f200640 1 -- 172.21.15.12:0/3399227640 wait complete. 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.826+0000 7f492f200640 1 Processor -- start 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492f200640 1 -- start start 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492f200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f492819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4928106d80 0x7f492819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492f200640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49281098f0 0x7f49281a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492f200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f492810f620 con 0x7f4928104390 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f492810f4a0 con 0x7f4928106d80 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492f200640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f492810f7a0 con 0x7f49281098f0 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f4927e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f492819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:20.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f4927e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f492819c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50482/0 (socket says 172.21.15.12:50482) 2024-05-27T07:22:20.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f4927e00640 1 -- 172.21.15.12:0/3918736482 learned_addr learned my addr 172.21.15.12:0/3918736482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:20.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.827+0000 7f492d200640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49281098f0 0x7f49281a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:20.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.828+0000 7f4927400640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4928106d80 0x7f492819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.828+0000 7f4927e00640 1 -- 172.21.15.12:0/3918736482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49281098f0 msgr2=0x7f49281a3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.828+0000 7f4927e00640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49281098f0 0x7f49281a3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.828+0000 7f4927e00640 1 -- 172.21.15.12:0/3918736482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4928106d80 msgr2=0x7f492819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.828+0000 7f4927e00640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4928106d80 0x7f492819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.828+0000 7f4927e00640 1 -- 172.21.15.12:0/3918736482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49281a3f90 con 0x7f4928104390 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.828+0000 7f4927e00640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f492819c450 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4918005760 tx=0x7f4918003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.829+0000 7f4924c00640 1 -- 172.21.15.12:0/3918736482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491803f070 con 0x7f4928104390 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.829+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49281a4220 con 0x7f4928104390 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.829+0000 7f4924c00640 1 -- 172.21.15.12:0/3918736482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4918031ba0 con 0x7f4928104390 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.829+0000 7f4924c00640 1 -- 172.21.15.12:0/3918736482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918035030 con 0x7f4928104390 2024-05-27T07:22:20.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.829+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49281a4760 con 0x7f4928104390 2024-05-27T07:22:20.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.830+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49281a4a80 con 0x7f4928104390 2024-05-27T07:22:20.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.833+0000 7f4924c00640 1 -- 172.21.15.12:0/3918736482 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f4918042070 con 0x7f4928104390 2024-05-27T07:22:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.835+0000 7f4924c00640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 0x7f490c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.835+0000 7f4927400640 1 -- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 msgr2=0x7f490c0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.836+0000 7f4924c00640 1 -- 172.21.15.12:0/3918736482 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f49181037c0 con 0x7f4928104390 2024-05-27T07:22:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.836+0000 7f4927400640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 0x7f490c0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:22:20.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:20.839+0000 7f4924c00640 1 -- 172.21.15.12:0/3918736482 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f49180cb110 con 0x7f4928104390 2024-05-27T07:22:20.839 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:20.419+0000 7f94a3b61200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:22:20.839 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:20.559+0000 7f94a3b61200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:22:20.985 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:20.690+0000 7f927e77d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:22:20.986 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:20.822+0000 7f927e77d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:22:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.037+0000 7f4927400640 1 -- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 msgr2=0x7f490c0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.037+0000 7f4927400640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 0x7f490c0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:22:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.094+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f49281a4c50 con 0x7f4928104390 2024-05-27T07:22:21.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.149+0000 7f4924c00640 1 -- 172.21.15.12:0/3918736482 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v27) ==== 117+0+0 (secure 0 0 0) 0x7f49180caa10 con 0x7f4928104390 2024-05-27T07:22:21.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 msgr2=0x7f490c0b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 0x7f490c0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 msgr2=0x7f492819c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f492819c450 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4918005760 tx=0x7f4918003c10 comp rx=0 tx=0).stop 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 shutdown_connections 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f490c0b2830 0x7f490c0b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49281098f0 0x7f49281a3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4928106d80 0x7f492819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 --2- 172.21.15.12:0/3918736482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4928104390 0x7f492819c450 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 >> 172.21.15.12:0/3918736482 conn(0x7f49280fdd30 msgr2=0x7f4928102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 shutdown_connections 2024-05-27T07:22:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.152+0000 7f492f200640 1 -- 172.21.15.12:0/3918736482 wait complete. 2024-05-27T07:22:21.168 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-27T07:22:21.179 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-27T07:22:21.210 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T07:22:21.275 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:21 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3918736482' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-27T07:22:21.276 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:20.983+0000 7f94a3b61200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:22:21.276 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:21.117+0000 7f94a3b61200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:22:21.473 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:21.270+0000 7fdc9de53200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:22:21.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:21 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3918736482' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 -- 172.21.15.12:0/2242780810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 msgr2=0x7f9b0c106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2242780810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c106930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009900 tx=0x7f9afc02a410 comp rx=0 tx=0).stop 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 -- 172.21.15.12:0/2242780810 shutdown_connections 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2242780810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b0c10bb70 0x7f9b0c10df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2242780810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b0c106e70 0x7f9b0c107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2242780810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c106930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 -- 172.21.15.12:0/2242780810 >> 172.21.15.12:0/2242780810 conn(0x7f9b0c100e40 msgr2=0x7f9b0c1032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 -- 172.21.15.12:0/2242780810 shutdown_connections 2024-05-27T07:22:21.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.485+0000 7f9b10e00640 1 -- 172.21.15.12:0/2242780810 wait complete. 2024-05-27T07:22:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.486+0000 7f9b10e00640 1 Processor -- start 2024-05-27T07:22:21.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.486+0000 7f9b10e00640 1 -- start start 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b10e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b10e00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b0c106e70 0x7f9b0c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b10e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b0c10bb70 0x7f9b0c1a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b10e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c1115c0 con 0x7f9b0c105c30 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b10e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c111440 con 0x7f9b0c10bb70 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b10e00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c111740 con 0x7f9b0c106e70 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b0b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b0b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50510/0 (socket says 172.21.15.12:50510) 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b0b400640 1 -- 172.21.15.12:0/2140685527 learned_addr learned my addr 172.21.15.12:0/2140685527 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b0aa00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b0c106e70 0x7f9b0c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:21.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.487+0000 7f9b0be00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b0c10bb70 0x7f9b0c1a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:21.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b0b400640 1 -- 172.21.15.12:0/2140685527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b0c106e70 msgr2=0x7f9b0c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:21.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b0b400640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b0c106e70 0x7f9b0c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b0b400640 1 -- 172.21.15.12:0/2140685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b0c10bb70 msgr2=0x7f9b0c1a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b0b400640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b0c10bb70 0x7f9b0c1a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b0b400640 1 -- 172.21.15.12:0/2140685527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0c1a3d70 con 0x7f9b0c105c30 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b0aa00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b0c106e70 0x7f9b0c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b0b400640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c19c280 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9afc005760 tx=0x7f9afc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b03e00640 1 -- 172.21.15.12:0/2140685527 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc03f030 con 0x7f9b0c105c30 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b03e00640 1 -- 172.21.15.12:0/2140685527 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9afc004440 con 0x7f9b0c105c30 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0c1a4000 con 0x7f9b0c105c30 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b03e00640 1 -- 172.21.15.12:0/2140685527 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc035070 con 0x7f9b0c105c30 2024-05-27T07:22:21.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.488+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b0c1a44f0 con 0x7f9b0c105c30 2024-05-27T07:22:21.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.490+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b0c10a980 con 0x7f9b0c105c30 2024-05-27T07:22:21.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.494+0000 7f9b03e00640 1 -- 172.21.15.12:0/2140685527 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f9afc042030 con 0x7f9b0c105c30 2024-05-27T07:22:21.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:21 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3918736482' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-27T07:22:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.495+0000 7f9b03e00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 0x7f9aec0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.495+0000 7f9b0aa00640 1 -- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 msgr2=0x7f9aec0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.496+0000 7f9b03e00640 1 -- 172.21.15.12:0/2140685527 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f9afc102ea0 con 0x7f9b0c105c30 2024-05-27T07:22:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.496+0000 7f9b0aa00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 0x7f9aec0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:22:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.498+0000 7f9b03e00640 1 -- 172.21.15.12:0/2140685527 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9afc0ca7f0 con 0x7f9b0c105c30 2024-05-27T07:22:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.697+0000 7f9b0aa00640 1 -- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 msgr2=0x7f9aec0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.697+0000 7f9b0aa00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 0x7f9aec0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:22:21.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.710+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b0c02ddf0 con 0x7f9b0c105c30 2024-05-27T07:22:21.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.713+0000 7f9b03e00640 1 -- 172.21.15.12:0/2140685527 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24806 (secure 0 0 0) 0x7f9afc0ca0f0 con 0x7f9b0c105c30 2024-05-27T07:22:21.704 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:22:21.705 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":89,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:18:47.037205+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 msgr2=0x7f9aec0b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 0x7f9aec0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 msgr2=0x7f9b0c19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c19c280 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9afc005760 tx=0x7f9afc004040 comp rx=0 tx=0).stop 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 shutdown_connections 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9aec0b2830 0x7f9aec0b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b0c10bb70 0x7f9b0c1a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b0c106e70 0x7f9b0c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 --2- 172.21.15.12:0/2140685527 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b0c105c30 0x7f9b0c19c280 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.716+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 >> 172.21.15.12:0/2140685527 conn(0x7f9b0c100e40 msgr2=0x7f9b0c107b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:21.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.717+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 shutdown_connections 2024-05-27T07:22:21.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:21.717+0000 7f9b10e00640 1 -- 172.21.15.12:0/2140685527 wait complete. 2024-05-27T07:22:21.733 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:10:07.718700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:17:01.212613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:17:05.645973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:17:19.864490+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T07:22:21.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T07:22:21.926 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:21.630+0000 7f927e77d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:22:21.973 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:22:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z[37555]: 2024-05-27T07:22:21.562+0000 7fdc9de53200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:22:22.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 -- 172.21.15.12:0/4110297463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c103b60 msgr2=0x7f9a5c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 --2- 172.21.15.12:0/4110297463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c103b60 0x7f9a5c103f70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009900 tx=0x7f9a4402a410 comp rx=0 tx=0).stop 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 -- 172.21.15.12:0/4110297463 shutdown_connections 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 --2- 172.21.15.12:0/4110297463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c108860 0x7f9a5c10ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 --2- 172.21.15.12:0/4110297463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c103b60 0x7f9a5c103f70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 --2- 172.21.15.12:0/4110297463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a5c102920 0x7f9a5c103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 -- 172.21.15.12:0/4110297463 >> 172.21.15.12:0/4110297463 conn(0x7f9a5c0fdb10 msgr2=0x7f9a5c0fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 -- 172.21.15.12:0/4110297463 shutdown_connections 2024-05-27T07:22:22.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.023+0000 7f9a5be00640 1 -- 172.21.15.12:0/4110297463 wait complete. 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.024+0000 7f9a5be00640 1 Processor -- start 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.024+0000 7f9a5be00640 1 -- start start 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.024+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c102920 0x7f9a5c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a5c103b60 0x7f9a5c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c108860 0x7f9a5c1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5be00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10e350 con 0x7f9a5c102920 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10e1d0 con 0x7f9a5c103b60 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5be00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10e4d0 con 0x7f9a5c108860 2024-05-27T07:22:22.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c102920 0x7f9a5c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c102920 0x7f9a5c19a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56018/0 (socket says 172.21.15.12:56018) 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 -- 172.21.15.12:0/3394584742 learned_addr learned my addr 172.21.15.12:0/3394584742 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5a000640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a5c103b60 0x7f9a5c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5b400640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c108860 0x7f9a5c1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 -- 172.21.15.12:0/3394584742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c108860 msgr2=0x7f9a5c1a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c108860 0x7f9a5c1a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 -- 172.21.15.12:0/3394584742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a5c103b60 msgr2=0x7f9a5c19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a5c103b60 0x7f9a5c19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5aa00640 1 -- 172.21.15.12:0/3394584742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5c1a1bd0 con 0x7f9a5c102920 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.025+0000 7f9a5b400640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c108860 0x7f9a5c1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:22.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.026+0000 7f9a5aa00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c102920 0x7f9a5c19a100 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00e9f0 tx=0x7f9a4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.026+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c00ccc0 con 0x7f9a5c102920 2024-05-27T07:22:22.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.026+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a4c0044b0 con 0x7f9a5c102920 2024-05-27T07:22:22.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.026+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5c1a1ec0 con 0x7f9a5c102920 2024-05-27T07:22:22.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.026+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c010430 con 0x7f9a5c102920 2024-05-27T07:22:22.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.026+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5c1a2320 con 0x7f9a5c102920 2024-05-27T07:22:22.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.028+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5c1a25b0 con 0x7f9a5c102920 2024-05-27T07:22:22.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.030+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f9a4c003f90 con 0x7f9a5c102920 2024-05-27T07:22:22.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.032+0000 7f9a53400640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 0x7f9a400b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.032+0000 7f9a5a000640 1 -- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 msgr2=0x7f9a400b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:22.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.033+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9208+0+0 (secure 0 0 0) 0x7f9a4c0e2a70 con 0x7f9a5c102920 2024-05-27T07:22:22.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.033+0000 7f9a5a000640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 0x7f9a400b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:22:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.037+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9a4c0aa360 con 0x7f9a5c102920 2024-05-27T07:22:22.089 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:21.943+0000 7f94a3b61200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:22:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.201+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 20) ==== 149669+0+0 (secure 0 0 0) 0x7f9a4c0a5ef0 con 0x7f9a5c102920 2024-05-27T07:22:22.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.234+0000 7f9a5a000640 1 -- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 msgr2=0x7f9a400b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2856894494 2024-05-27T07:22:22.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.234+0000 7f9a5a000640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 0x7f9a400b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:22:22.235 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:22:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y[36678]: 2024-05-27T07:22:21.934+0000 7f927e77d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:22:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3918736482' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-27T07:22:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:22:22.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: Standby manager daemon z restarted 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: Standby manager daemon z started 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2140685527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: Standby manager daemon y restarted 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: Standby manager daemon y started 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:22:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:22 smithi062 ceph-mon[35304]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:22:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.274+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a5c1a2780 con 0x7f9a5c102920 2024-05-27T07:22:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.275+0000 7f9a53400640 1 -- 172.21.15.12:0/3394584742 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f9a4c0dea50 con 0x7f9a5c102920 2024-05-27T07:22:22.265 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 msgr2=0x7f9a400b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 0x7f9a400b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c102920 msgr2=0x7f9a5c19a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c102920 0x7f9a5c19a100 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00e9f0 tx=0x7f9a4c00eeb0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 shutdown_connections 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:6800/2856894494,v1:172.21.15.12:6801/2856894494] conn(0x7f9a400b2830 0x7f9a400b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c108860 0x7f9a5c1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a5c103b60 0x7f9a5c19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 --2- 172.21.15.12:0/3394584742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9a5c102920 0x7f9a5c19a100 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.277+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 >> 172.21.15.12:0/3394584742 conn(0x7f9a5c0fdb10 msgr2=0x7f9a5c109de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.278+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 shutdown_connections 2024-05-27T07:22:22.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.278+0000 7f9a5be00640 1 -- 172.21.15.12:0/3394584742 wait complete. 2024-05-27T07:22:22.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T07:22:22.414 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.277+0000 7f94a3b61200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:22:22.415 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3918736482' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-27T07:22:22.415 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:22:22.415 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: Standby manager daemon z restarted 2024-05-27T07:22:22.415 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: Standby manager daemon z started 2024-05-27T07:22:22.415 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2140685527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: Standby manager daemon y restarted 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: Standby manager daemon y started 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:22:22.416 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:22 smithi012 ceph-mon[31473]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:22:22.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3918736482' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: Standby manager daemon z restarted 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: Standby manager daemon z started 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.162:0/4052865217' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2140685527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: Standby manager daemon y restarted 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: Standby manager daemon y started 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:22:22.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:22 smithi162 ceph-mon[35676]: from='mgr.? 172.21.15.62:0/3568853704' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:22:22.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.623+0000 7fc672000640 1 -- 172.21.15.12:0/1247324376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 msgr2=0x7fc66c10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.623+0000 7fc672000640 1 --2- 172.21.15.12:0/1247324376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 0x7fc66c10e010 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc664009ca0 tx=0x7fc66402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.624+0000 7fc672000640 1 -- 172.21.15.12:0/1247324376 shutdown_connections 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.624+0000 7fc672000640 1 --2- 172.21.15.12:0/1247324376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 0x7fc66c10e010 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.624+0000 7fc672000640 1 --2- 172.21.15.12:0/1247324376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 0x7fc66c10b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.624+0000 7fc672000640 1 --2- 172.21.15.12:0/1247324376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc66c075050 0x7fc66c075460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.624+0000 7fc672000640 1 -- 172.21.15.12:0/1247324376 >> 172.21.15.12:0/1247324376 conn(0x7fc66c0fee10 msgr2=0x7fc66c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.624+0000 7fc672000640 1 -- 172.21.15.12:0/1247324376 shutdown_connections 2024-05-27T07:22:22.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.624+0000 7fc672000640 1 -- 172.21.15.12:0/1247324376 wait complete. 2024-05-27T07:22:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.625+0000 7fc672000640 1 Processor -- start 2024-05-27T07:22:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.625+0000 7fc672000640 1 -- start start 2024-05-27T07:22:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc672000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc66c075050 0x7fc66c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc672000640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 0x7fc66c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc672000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 0x7fc66c1a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc672000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c111840 con 0x7fc66c10bbf0 2024-05-27T07:22:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc672000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c1116c0 con 0x7fc66c075050 2024-05-27T07:22:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc672000640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c1119c0 con 0x7fc66c0759a0 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc66aa00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 0x7fc66c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.626+0000 7fc66aa00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 0x7fc66c19c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:40390/0 (socket says 172.21.15.12:40390) 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66aa00640 1 -- 172.21.15.12:0/3216905020 learned_addr learned my addr 172.21.15.12:0/3216905020 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66b400640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc66c075050 0x7fc66c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66be00640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 0x7fc66c1a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66aa00640 1 -- 172.21.15.12:0/3216905020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc66c075050 msgr2=0x7fc66c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66aa00640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc66c075050 0x7fc66c19c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66aa00640 1 -- 172.21.15.12:0/3216905020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 msgr2=0x7fc66c1a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66aa00640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 0x7fc66c1a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.627+0000 7fc66aa00640 1 -- 172.21.15.12:0/3216905020 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66c1a3f50 con 0x7fc66c0759a0 2024-05-27T07:22:22.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.628+0000 7fc66aa00640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 0x7fc66c19c950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc65800e9f0 tx=0x7fc65800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.628+0000 7fc663e00640 1 -- 172.21.15.12:0/3216905020 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65800cd80 con 0x7fc66c0759a0 2024-05-27T07:22:22.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.628+0000 7fc663e00640 1 -- 172.21.15.12:0/3216905020 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc6580054b0 con 0x7fc66c0759a0 2024-05-27T07:22:22.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.628+0000 7fc663e00640 1 -- 172.21.15.12:0/3216905020 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658010430 con 0x7fc66c0759a0 2024-05-27T07:22:22.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.628+0000 7fc672000640 1 -- 172.21.15.12:0/3216905020 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66c1a4240 con 0x7fc66c0759a0 2024-05-27T07:22:22.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.628+0000 7fc672000640 1 -- 172.21.15.12:0/3216905020 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66c1a4740 con 0x7fc66c0759a0 2024-05-27T07:22:22.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.630+0000 7fc661600640 1 -- 172.21.15.12:0/3216905020 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66c0638c0 con 0x7fc66c0759a0 2024-05-27T07:22:22.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.633+0000 7fc663e00640 1 -- 172.21.15.12:0/3216905020 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 21) ==== 149337+0+0 (secure 0 0 0) 0x7fc6580046c0 con 0x7fc66c0759a0 2024-05-27T07:22:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.636+0000 7fc663e00640 1 -- 172.21.15.12:0/3216905020 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fc6580e2730 con 0x7fc66c0759a0 2024-05-27T07:22:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.637+0000 7fc663e00640 1 -- 172.21.15.12:0/3216905020 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc6580aa320 con 0x7fc66c0759a0 2024-05-27T07:22:22.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.878+0000 7fc661600640 1 -- 172.21.15.12:0/3216905020 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc66c063ae0 con 0x7fc66c0759a0 2024-05-27T07:22:22.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.879+0000 7fc663e00640 1 -- 172.21.15.12:0/3216905020 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fc6580a9c20 con 0x7fc66c0759a0 2024-05-27T07:22:22.870 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 -- 172.21.15.12:0/3216905020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 msgr2=0x7fc66c19c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 0x7fc66c19c950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc65800e9f0 tx=0x7fc65800edf0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 -- 172.21.15.12:0/3216905020 shutdown_connections 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc66c10bbf0 0x7fc66c1a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc66c0759a0 0x7fc66c19c950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 --2- 172.21.15.12:0/3216905020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc66c075050 0x7fc66c19c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 -- 172.21.15.12:0/3216905020 >> 172.21.15.12:0/3216905020 conn(0x7fc66c0fee10 msgr2=0x7fc66c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 -- 172.21.15.12:0/3216905020 shutdown_connections 2024-05-27T07:22:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:22.881+0000 7fc661600640 1 -- 172.21.15.12:0/3216905020 wait complete. 2024-05-27T07:22:22.901 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T07:22:22.959 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.915+0000 7f93f8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.959 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.915+0000 7f93f8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.959 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.915+0000 7f93f8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.960 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.915+0000 7f93f8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.960 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.915+0000 7f93f8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.960 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.917+0000 7f93f4e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.960 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.917+0000 7f93f4e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.960 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.917+0000 7f93f4e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.960 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.917+0000 7f93f4e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:22.960 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:22:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:22:22.917+0000 7f93f4e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 -- 172.21.15.12:0/1975411143 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 msgr2=0x7f3ed010fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/1975411143 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed010fc50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009900 tx=0x7f3ebc02a410 comp rx=0 tx=0).stop 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 -- 172.21.15.12:0/1975411143 shutdown_connections 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/1975411143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ed0110190 0x7f3ed01125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/1975411143 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed010fc50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/1975411143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ed00750b0 0x7f3ed00754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 -- 172.21.15.12:0/1975411143 >> 172.21.15.12:0/1975411143 conn(0x7f3ed00fec40 msgr2=0x7f3ed01010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 -- 172.21.15.12:0/1975411143 shutdown_connections 2024-05-27T07:22:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.173+0000 7f3ed5e00640 1 -- 172.21.15.12:0/1975411143 wait complete. 2024-05-27T07:22:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.174+0000 7f3ed5e00640 1 Processor -- start 2024-05-27T07:22:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.174+0000 7f3ed5e00640 1 -- start start 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed5e00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ed00750b0 0x7f3ed01a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed5e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed01a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ed0110190 0x7f3ed01a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed5e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed0115a70 con 0x7f3ed0075a00 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed5e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01158f0 con 0x7f3ed0110190 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed5e00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed0115bf0 con 0x7f3ed00750b0 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed01a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed01a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56160/0 (socket says 172.21.15.12:56160) 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ecfe00640 1 -- 172.21.15.12:0/2950054034 learned_addr learned my addr 172.21.15.12:0/2950054034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ed0110190 0x7f3ed01a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.175+0000 7f3ed4a00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ed00750b0 0x7f3ed01a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecfe00640 1 -- 172.21.15.12:0/2950054034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ed00750b0 msgr2=0x7f3ed01a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecfe00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ed00750b0 0x7f3ed01a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecfe00640 1 -- 172.21.15.12:0/2950054034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ed0110190 msgr2=0x7f3ed01a7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecfe00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ed0110190 0x7f3ed01a7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecfe00640 1 -- 172.21.15.12:0/2950054034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed01a8220 con 0x7f3ed0075a00 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecfe00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed01a0c20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005760 tx=0x7f3ebc003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc03f070 con 0x7f3ed0075a00 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ebc031da0 con 0x7f3ed0075a00 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed01a84b0 con 0x7f3ed0075a00 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.176+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc035030 con 0x7f3ed0075a00 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.177+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed01a89a0 con 0x7f3ed0075a00 2024-05-27T07:22:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.178+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed0105e20 con 0x7f3ed0075a00 2024-05-27T07:22:23.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.182+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 21) ==== 149337+0+0 (secure 0 0 0) 0x7f3ebc042070 con 0x7f3ed0075a00 2024-05-27T07:22:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.184+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f3ebc103210 con 0x7f3ed0075a00 2024-05-27T07:22:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.184+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3ebc0ca7b0 con 0x7f3ed0075a00 2024-05-27T07:22:23.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.337+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 22) ==== 149464+0+0 (secure 0 0 0) 0x7f3ebc0c6120 con 0x7f3ed0075a00 2024-05-27T07:22:23.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.337+0000 7f3ecd600640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3eb80bad00 0x7f3eb80bd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.340+0000 7f3ed4a00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3eb80bad00 0x7f3eb80bd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.343+0000 7f3ed4a00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3eb80bad00 0x7f3eb80bd120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005d00 tx=0x7f3ec400b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:23.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: mgrmap e20: x(active, since 15m), standbys: z, y 2024-05-27T07:22:23.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3394584742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:22:23.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: Active manager daemon x restarted 2024-05-27T07:22:23.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: Activating manager daemon x 2024-05-27T07:22:23.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:22:23.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: mgrmap e21: x(active, starting, since 0.0299441s), standbys: z, y 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:22:23.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:22:23.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: Manager daemon x is now available 2024-05-27T07:22:23.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:23.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:22:23.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:22:23.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:23 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3216905020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:22:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.409+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3ed00638c0 con 0x7f3ed0075a00 2024-05-27T07:22:23.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.410+0000 7f3ecd600640 1 -- 172.21.15.12:0/2950054034 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f3ebc0bedc0 con 0x7f3ed0075a00 2024-05-27T07:22:23.401 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-05-27T07:22:23.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.412+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3eb80bad00 msgr2=0x7f3eb80bd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.412+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3eb80bad00 0x7f3eb80bd120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005d00 tx=0x7f3ec400b410 comp rx=0 tx=0).stop 2024-05-27T07:22:23.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.412+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 msgr2=0x7f3ed01a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.412+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed01a0c20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005760 tx=0x7f3ebc003d50 comp rx=0 tx=0).stop 2024-05-27T07:22:23.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 shutdown_connections 2024-05-27T07:22:23.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3eb80bad00 0x7f3eb80bd120 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ed0110190 0x7f3ed01a7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ed0075a00 0x7f3ed01a0c20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 --2- 172.21.15.12:0/2950054034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ed00750b0 0x7f3ed01a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 >> 172.21.15.12:0/2950054034 conn(0x7f3ed00fec40 msgr2=0x7f3ed0102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:23.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 shutdown_connections 2024-05-27T07:22:23.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.413+0000 7f3ed5e00640 1 -- 172.21.15.12:0/2950054034 wait complete. 2024-05-27T07:22:23.428 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T07:22:23.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: mgrmap e20: x(active, since 15m), standbys: z, y 2024-05-27T07:22:23.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3394584742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:22:23.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: Active manager daemon x restarted 2024-05-27T07:22:23.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: Activating manager daemon x 2024-05-27T07:22:23.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: mgrmap e21: x(active, starting, since 0.0299441s), standbys: z, y 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:22:23.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:22:23.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:22:23.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: Manager daemon x is now available 2024-05-27T07:22:23.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:23.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:22:23.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:22:23.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:23 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3216905020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:22:23.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: mgrmap e20: x(active, since 15m), standbys: z, y 2024-05-27T07:22:23.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3394584742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:22:23.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: Active manager daemon x restarted 2024-05-27T07:22:23.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: Activating manager daemon x 2024-05-27T07:22:23.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:22:23.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: mgrmap e21: x(active, starting, since 0.0299441s), standbys: z, y 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:22:23.488 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:22:23.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:22:23.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:22:23.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:22:23.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:22:23.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: Manager daemon x is now available 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:23.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:22:23.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:22:23.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:23 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3216905020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:22:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.701+0000 7f1f20600640 1 -- 172.21.15.12:0/3116498686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 msgr2=0x7f1f18106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.701+0000 7f1f20600640 1 --2- 172.21.15.12:0/3116498686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 0x7f1f18106770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009900 tx=0x7f1f0c02a3f0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.701+0000 7f1f20600640 1 -- 172.21.15.12:0/3116498686 shutdown_connections 2024-05-27T07:22:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.701+0000 7f1f20600640 1 --2- 172.21.15.12:0/3116498686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 0x7f1f1810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.701+0000 7f1f20600640 1 --2- 172.21.15.12:0/3116498686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f18106d40 0x7f1f18109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.701+0000 7f1f20600640 1 --2- 172.21.15.12:0/3116498686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 0x7f1f18106770 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.701+0000 7f1f20600640 1 -- 172.21.15.12:0/3116498686 >> 172.21.15.12:0/3116498686 conn(0x7f1f180fdcf0 msgr2=0x7f1f181001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.702+0000 7f1f20600640 1 -- 172.21.15.12:0/3116498686 shutdown_connections 2024-05-27T07:22:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.702+0000 7f1f20600640 1 -- 172.21.15.12:0/3116498686 wait complete. 2024-05-27T07:22:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 Processor -- start 2024-05-27T07:22:23.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 -- start start 2024-05-27T07:22:23.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 0x7f1f1819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f18106d40 0x7f1f1819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 0x7f1f181a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1810f570 con 0x7f1f181098b0 2024-05-27T07:22:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1810f3f0 con 0x7f1f18106d40 2024-05-27T07:22:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f20600640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1810f6f0 con 0x7f1f18104350 2024-05-27T07:22:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f1e600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 0x7f1f181a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.703+0000 7f1f1dc00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 0x7f1f1819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 0x7f1f181a3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56182/0 (socket says 172.21.15.12:56182) 2024-05-27T07:22:23.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1dc00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 0x7f1f1819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:40490/0 (socket says 172.21.15.12:40490) 2024-05-27T07:22:23.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 -- 172.21.15.12:0/2671935758 learned_addr learned my addr 172.21.15.12:0/2671935758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:23.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1d200640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f18106d40 0x7f1f1819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 -- 172.21.15.12:0/2671935758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 msgr2=0x7f1f1819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 0x7f1f1819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 -- 172.21.15.12:0/2671935758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f18106d40 msgr2=0x7f1f1819cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f18106d40 0x7f1f1819cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 -- 172.21.15.12:0/2671935758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f181a42a0 con 0x7f1f181098b0 2024-05-27T07:22:23.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.704+0000 7f1f1e600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 0x7f1f181a3c30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000eaa0 tx=0x7f1f1000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:23.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.705+0000 7f1f16000640 1 -- 172.21.15.12:0/2671935758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000cdb0 con 0x7f1f181098b0 2024-05-27T07:22:23.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.705+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f181a4590 con 0x7f1f181098b0 2024-05-27T07:22:23.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.705+0000 7f1f16000640 1 -- 172.21.15.12:0/2671935758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f10005020 con 0x7f1f181098b0 2024-05-27T07:22:23.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.705+0000 7f1f16000640 1 -- 172.21.15.12:0/2671935758 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100116d0 con 0x7f1f181098b0 2024-05-27T07:22:23.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.705+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f181a4ae0 con 0x7f1f181098b0 2024-05-27T07:22:23.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.706+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f1806c3c0 con 0x7f1f181098b0 2024-05-27T07:22:23.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.709+0000 7f1f16000640 1 -- 172.21.15.12:0/2671935758 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) ==== 149464+0+0 (secure 0 0 0) 0x7f1f100051d0 con 0x7f1f181098b0 2024-05-27T07:22:23.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.711+0000 7f1f16000640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f000b25e0 0x7f1f000b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:23.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.712+0000 7f1f16000640 1 -- 172.21.15.12:0/2671935758 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f1f100e2820 con 0x7f1f181098b0 2024-05-27T07:22:23.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.712+0000 7f1f1dc00640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f000b25e0 0x7f1f000b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:23.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.713+0000 7f1f1dc00640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f000b25e0 0x7f1f000b4af0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0057c0 tx=0x7f1f0c000bc0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.715+0000 7f1f16000640 1 -- 172.21.15.12:0/2671935758 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1f10016030 con 0x7f1f181098b0 2024-05-27T07:22:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.943+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1f181a4da0 con 0x7f1f181098b0 2024-05-27T07:22:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.943+0000 7f1f16000640 1 -- 172.21.15.12:0/2671935758 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f1f100b23a0 con 0x7f1f181098b0 2024-05-27T07:22:23.934 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-05-27T07:22:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f000b25e0 msgr2=0x7f1f000b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f000b25e0 0x7f1f000b4af0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0057c0 tx=0x7f1f0c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 msgr2=0x7f1f181a3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 0x7f1f181a3c30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000eaa0 tx=0x7f1f1000ef60 comp rx=0 tx=0).stop 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 shutdown_connections 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f181098b0 0x7f1f181a3c30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f000b25e0 0x7f1f000b4af0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f18106d40 0x7f1f1819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 --2- 172.21.15.12:0/2671935758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f18104350 0x7f1f1819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 >> 172.21.15.12:0/2671935758 conn(0x7f1f180fdcf0 msgr2=0x7f1f18103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 shutdown_connections 2024-05-27T07:22:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:23.946+0000 7f1f20600640 1 -- 172.21.15.12:0/2671935758 wait complete. 2024-05-27T07:22:23.964 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-27T07:22:23.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:22:24.585 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:24 smithi062 ceph-mon[35304]: [27/May/2024:07:22:22] ENGINE Bus STARTING 2024-05-27T07:22:24.585 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:24 smithi062 ceph-mon[35304]: [27/May/2024:07:22:23] ENGINE Serving on https://172.21.15.12:7150 2024-05-27T07:22:24.585 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:24 smithi062 ceph-mon[35304]: [27/May/2024:07:22:23] ENGINE Serving on http://172.21.15.12:8765 2024-05-27T07:22:24.585 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:24 smithi062 ceph-mon[35304]: [27/May/2024:07:22:23] ENGINE Bus STARTED 2024-05-27T07:22:24.585 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:24 smithi062 ceph-mon[35304]: mgrmap e22: x(active, since 1.05375s), standbys: z, y 2024-05-27T07:22:24.585 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:24 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2950054034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:22:24.586 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:24 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2671935758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:24 smithi012 ceph-mon[31473]: [27/May/2024:07:22:22] ENGINE Bus STARTING 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:24 smithi012 ceph-mon[31473]: [27/May/2024:07:22:23] ENGINE Serving on https://172.21.15.12:7150 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:24 smithi012 ceph-mon[31473]: [27/May/2024:07:22:23] ENGINE Serving on http://172.21.15.12:8765 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:24 smithi012 ceph-mon[31473]: [27/May/2024:07:22:23] ENGINE Bus STARTED 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:24 smithi012 ceph-mon[31473]: mgrmap e22: x(active, since 1.05375s), standbys: z, y 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:24 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2950054034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:24 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2671935758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:22:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:24 smithi162 ceph-mon[35676]: [27/May/2024:07:22:22] ENGINE Bus STARTING 2024-05-27T07:22:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:24 smithi162 ceph-mon[35676]: [27/May/2024:07:22:23] ENGINE Serving on https://172.21.15.12:7150 2024-05-27T07:22:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:24 smithi162 ceph-mon[35676]: [27/May/2024:07:22:23] ENGINE Serving on http://172.21.15.12:8765 2024-05-27T07:22:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:24 smithi162 ceph-mon[35676]: [27/May/2024:07:22:23] ENGINE Bus STARTED 2024-05-27T07:22:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:24 smithi162 ceph-mon[35676]: mgrmap e22: x(active, since 1.05375s), standbys: z, y 2024-05-27T07:22:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:24 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2950054034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:22:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:24 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2671935758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:22:24.817 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:25.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:25 smithi012 ceph-mon[31473]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:25 smithi012 ceph-mon[31473]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:25 smithi012 ceph-mon[31473]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:22:25.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:25 smithi162 ceph-mon[35676]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:25 smithi162 ceph-mon[35676]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:25 smithi162 ceph-mon[35676]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:22:25.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:25 smithi062 ceph-mon[35304]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:25 smithi062 ceph-mon[35304]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:25 smithi062 ceph-mon[35304]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:22:27.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:26 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:26 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:26 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:26 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:26 smithi162 ceph-mon[35676]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:27.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:26 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:26 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:26 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:26 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:26 smithi062 ceph-mon[35304]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:27.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 -- 172.21.15.12:0/3357890947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f4107de0 msgr2=0x7f13f41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 --2- 172.21.15.12:0/3357890947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f4107de0 0x7f13f41081f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f13e00099c0 tx=0x7f13e0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.258 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:26 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.259 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:26 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.259 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:26 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.259 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:26 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:27.259 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:26 smithi012 ceph-mon[31473]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:27.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 -- 172.21.15.12:0/3357890947 shutdown_connections 2024-05-27T07:22:27.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 --2- 172.21.15.12:0/3357890947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13f41034b0 0x7f13f41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 --2- 172.21.15.12:0/3357890947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13f4108730 0x7f13f4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 --2- 172.21.15.12:0/3357890947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f4107de0 0x7f13f41081f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 -- 172.21.15.12:0/3357890947 >> 172.21.15.12:0/3357890947 conn(0x7f13f40fedb0 msgr2=0x7f13f41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 -- 172.21.15.12:0/3357890947 shutdown_connections 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.267+0000 7f13f8e00700 1 -- 172.21.15.12:0/3357890947 wait complete. 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.269+0000 7f13f8e00700 1 Processor -- start 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.269+0000 7f13f8e00700 1 -- start start 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f8e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f41034b0 0x7f13f41a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f8e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13f4107de0 0x7f13f41a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13f4108730 0x7f13f419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f8e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41131f0 con 0x7f13f41034b0 2024-05-27T07:22:27.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f8e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4113070 con 0x7f13f4108730 2024-05-27T07:22:27.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f8e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4113370 con 0x7f13f4107de0 2024-05-27T07:22:27.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f41034b0 0x7f13f41a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f41034b0 0x7f13f41a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56222/0 (socket says 172.21.15.12:56222) 2024-05-27T07:22:27.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f1e00700 1 -- 172.21.15.12:0/2990391600 learned_addr learned my addr 172.21.15.12:0/2990391600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:27.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f1400700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13f4107de0 0x7f13f41a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.270+0000 7f13f2800700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13f4108730 0x7f13f419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.271+0000 7f13f1e00700 1 -- 172.21.15.12:0/2990391600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13f4107de0 msgr2=0x7f13f41a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.271+0000 7f13f1e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13f4107de0 0x7f13f41a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.271+0000 7f13f1e00700 1 -- 172.21.15.12:0/2990391600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13f4108730 msgr2=0x7f13f419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.271+0000 7f13f1e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13f4108730 0x7f13f419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.271+0000 7f13f1e00700 1 -- 172.21.15.12:0/2990391600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f419b1c0 con 0x7f13f41034b0 2024-05-27T07:22:27.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.271+0000 7f13f1e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f41034b0 0x7f13f41a08a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f13e400e9a0 tx=0x7f13e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.272+0000 7f13eaa00700 1 -- 172.21.15.12:0/2990391600 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e400cbd0 con 0x7f13f41034b0 2024-05-27T07:22:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.272+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f419b4b0 con 0x7f13f41034b0 2024-05-27T07:22:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.272+0000 7f13eaa00700 1 -- 172.21.15.12:0/2990391600 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13e400cd70 con 0x7f13f41034b0 2024-05-27T07:22:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.272+0000 7f13eaa00700 1 -- 172.21.15.12:0/2990391600 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4018680 con 0x7f13f41034b0 2024-05-27T07:22:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.272+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f41a7750 con 0x7f13f41034b0 2024-05-27T07:22:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.274+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f40512b0 con 0x7f13f41034b0 2024-05-27T07:22:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.278+0000 7f13eaa00700 1 -- 172.21.15.12:0/2990391600 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f13e4010430 con 0x7f13f41034b0 2024-05-27T07:22:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.280+0000 7f13eaa00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f13d80b2b00 0x7f13d80b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.280+0000 7f13eaa00700 1 -- 172.21.15.12:0/2990391600 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f13e40e2fe0 con 0x7f13f41034b0 2024-05-27T07:22:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.280+0000 7f13f1400700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f13d80b2b00 0x7f13d80b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.282+0000 7f13eaa00700 1 -- 172.21.15.12:0/2990391600 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f13e40aa570 con 0x7f13f41034b0 2024-05-27T07:22:27.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.282+0000 7f13f1400700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f13d80b2b00 0x7f13d80b5010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f13e0000c00 tx=0x7f13e001a040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.580+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13f402e5e0 con 0x7f13f41034b0 2024-05-27T07:22:27.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.581+0000 7f13eaa00700 1 -- 172.21.15.12:0/2990391600 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11867 (secure 0 0 0) 0x7f13e40a9e70 con 0x7f13f41034b0 2024-05-27T07:22:27.572 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:22:27.572 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:22:21.096387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.586+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f13d80b2b00 msgr2=0x7f13d80b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.586+0000 7f13f8e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f13d80b2b00 0x7f13d80b5010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f13e0000c00 tx=0x7f13e001a040 comp rx=0 tx=0).stop 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.586+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f41034b0 msgr2=0x7f13f41a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.586+0000 7f13f8e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f41034b0 0x7f13f41a08a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f13e400e9a0 tx=0x7f13e400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 shutdown_connections 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f13d80b2b00 0x7f13d80b5010 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13f4108730 0x7f13f419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13f41034b0 0x7f13f41a08a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 --2- 172.21.15.12:0/2990391600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13f4107de0 0x7f13f41a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 >> 172.21.15.12:0/2990391600 conn(0x7f13f40fedb0 msgr2=0x7f13f406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 shutdown_connections 2024-05-27T07:22:27.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:27.587+0000 7f13f8e00700 1 -- 172.21.15.12:0/2990391600 wait complete. 2024-05-27T07:22:27.579 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 27 2024-05-27T07:22:28.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:27 smithi162 ceph-mon[35676]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:22:28.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:27 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:28.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:27 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:28.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:27 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2990391600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:28.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:27 smithi062 ceph-mon[35304]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:22:28.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:27 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:28.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:27 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:28.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:27 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2990391600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:28.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:27 smithi012 ceph-mon[31473]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:22:28.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:27 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:28.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:27 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:28.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:27 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2990391600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:28.494 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-27T07:22:28.494 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-27T07:22:28.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 -- 172.21.15.12:0/445211148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c075a00 msgr2=0x7fe71c10fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 --2- 172.21.15.12:0/445211148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c075a00 0x7fe71c10fc20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe708009900 tx=0x7fe70802a410 comp rx=0 tx=0).stop 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 -- 172.21.15.12:0/445211148 shutdown_connections 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 --2- 172.21.15.12:0/445211148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe71c110160 0x7fe71c112580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 --2- 172.21.15.12:0/445211148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c075a00 0x7fe71c10fc20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 --2- 172.21.15.12:0/445211148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe71c0750b0 0x7fe71c0754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 -- 172.21.15.12:0/445211148 >> 172.21.15.12:0/445211148 conn(0x7fe71c0febd0 msgr2=0x7fe71c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 -- 172.21.15.12:0/445211148 shutdown_connections 2024-05-27T07:22:28.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.801+0000 7fe71be00640 1 -- 172.21.15.12:0/445211148 wait complete. 2024-05-27T07:22:28.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.802+0000 7fe71be00640 1 Processor -- start 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71be00640 1 -- start start 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71be00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c0750b0 0x7fe71c1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe71c075a00 0x7fe71c1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71be00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe71c110160 0x7fe71c1a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71be00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe71c1159f0 con 0x7fe71c0750b0 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe71c115870 con 0x7fe71c075a00 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71be00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe71c115b70 con 0x7fe71c110160 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c0750b0 0x7fe71c1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71aa00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c0750b0 0x7fe71c1a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56234/0 (socket says 172.21.15.12:56234) 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.803+0000 7fe71aa00640 1 -- 172.21.15.12:0/1374654477 learned_addr learned my addr 172.21.15.12:0/1374654477 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71a000640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe71c075a00 0x7fe71c1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:28.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71b400640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe71c110160 0x7fe71c1a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:28.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71aa00640 1 -- 172.21.15.12:0/1374654477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe71c110160 msgr2=0x7fe71c1a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:28.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71aa00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe71c110160 0x7fe71c1a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:28.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71aa00640 1 -- 172.21.15.12:0/1374654477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe71c075a00 msgr2=0x7fe71c1a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:28.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71aa00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe71c075a00 0x7fe71c1a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:28.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71aa00640 1 -- 172.21.15.12:0/1374654477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71c1a8200 con 0x7fe71c0750b0 2024-05-27T07:22:28.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71b400640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe71c110160 0x7fe71c1a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:28.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.804+0000 7fe71aa00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c0750b0 0x7fe71c1a06c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00d740 tx=0x7fe70c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:28.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.805+0000 7fe713400640 1 -- 172.21.15.12:0/1374654477 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c017070 con 0x7fe71c0750b0 2024-05-27T07:22:28.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.805+0000 7fe713400640 1 -- 172.21.15.12:0/1374654477 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe70c00de50 con 0x7fe71c0750b0 2024-05-27T07:22:28.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.805+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe71c1a84f0 con 0x7fe71c0750b0 2024-05-27T07:22:28.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.805+0000 7fe713400640 1 -- 172.21.15.12:0/1374654477 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c004c60 con 0x7fe71c0750b0 2024-05-27T07:22:28.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.805+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe71c1a8a40 con 0x7fe71c0750b0 2024-05-27T07:22:28.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.806+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe71c105df0 con 0x7fe71c0750b0 2024-05-27T07:22:28.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.809+0000 7fe713400640 1 -- 172.21.15.12:0/1374654477 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7fe70c004e00 con 0x7fe71c0750b0 2024-05-27T07:22:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.811+0000 7fe713400640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7000b2a60 0x7fe7000b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.811+0000 7fe713400640 1 -- 172.21.15.12:0/1374654477 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe70c0e29e0 con 0x7fe71c0750b0 2024-05-27T07:22:28.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.811+0000 7fe71a000640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7000b2a60 0x7fe7000b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:28.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.812+0000 7fe71a000640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7000b2a60 0x7fe7000b4f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe708006010 tx=0x7fe708000bc0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:28.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:28.818+0000 7fe713400640 1 -- 172.21.15.12:0/1374654477 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe70c0a9ec0 con 0x7fe71c0750b0 2024-05-27T07:22:29.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.076+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe71c063100 con 0x7fe7000b2a60 2024-05-27T07:22:29.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.089+0000 7fe713400640 1 -- 172.21.15.12:0/1374654477 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339472 (secure 0 0 0) 0x7fe71c063100 con 0x7fe7000b2a60 2024-05-27T07:22:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:22:29.086 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-05-27T07:22:29.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7000b2a60 msgr2=0x7fe7000b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7000b2a60 0x7fe7000b4f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe708006010 tx=0x7fe708000bc0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c0750b0 msgr2=0x7fe71c1a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c0750b0 0x7fe71c1a06c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00d740 tx=0x7fe70c00dc00 comp rx=0 tx=0).stop 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 shutdown_connections 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe71c110160 0x7fe71c1a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7000b2a60 0x7fe7000b4f70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe71c075a00 0x7fe71c1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 --2- 172.21.15.12:0/1374654477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe71c0750b0 0x7fe71c1a06c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.094+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 >> 172.21.15.12:0/1374654477 conn(0x7fe71c0febd0 msgr2=0x7fe71c102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.096+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 shutdown_connections 2024-05-27T07:22:29.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.096+0000 7fe71be00640 1 -- 172.21.15.12:0/1374654477 wait complete. 2024-05-27T07:22:29.111 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:22:28.329802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189379,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1553,"num_read_kb":916,"num_write":428,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":832,"ondisk_log_size":832,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377364,"kb_used_data":15332,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124564460,"statfs":{"total":1151940427776,"available":1151554007040,"internally_reserved":0,"allocated":15699968,"data_stored":9319518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.974778"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377753+0000","last_change":"2024-05-27T07:17:06.763882+0000","last_active":"2024-05-27T07:22:22.377753+0000","last_peered":"2024-05-27T07:22:22.377753+0000","last_clean":"2024-05-27T07:22:22.377753+0000","last_became_active":"2024-05-27T07:17:06.759625+0000","last_became_peered":"2024-05-27T07:17:06.759625+0000","last_unstale":"2024-05-27T07:22:22.377753+0000","last_undegraded":"2024-05-27T07:22:22.377753+0000","last_fullsized":"2024-05-27T07:22:22.377753+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:24.105867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595140+0000","last_change":"2024-05-27T07:17:06.755737+0000","last_active":"2024-05-27T07:22:22.595140+0000","last_peered":"2024-05-27T07:22:22.595140+0000","last_clean":"2024-05-27T07:22:22.595140+0000","last_became_active":"2024-05-27T07:17:06.754588+0000","last_became_peered":"2024-05-27T07:17:06.754588+0000","last_unstale":"2024-05-27T07:22:22.595140+0000","last_undegraded":"2024-05-27T07:22:22.595140+0000","last_fullsized":"2024-05-27T07:22:22.595140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:02.417140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595016+0000","last_change":"2024-05-27T07:17:06.772057+0000","last_active":"2024-05-27T07:22:22.595016+0000","last_peered":"2024-05-27T07:22:22.595016+0000","last_clean":"2024-05-27T07:22:22.595016+0000","last_became_active":"2024-05-27T07:17:06.771636+0000","last_became_peered":"2024-05-27T07:17:06.771636+0000","last_unstale":"2024-05-27T07:22:22.595016+0000","last_undegraded":"2024-05-27T07:22:22.595016+0000","last_fullsized":"2024-05-27T07:22:22.595016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:45.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373701+0000","last_change":"2024-05-27T07:17:06.763599+0000","last_active":"2024-05-27T07:22:22.373701+0000","last_peered":"2024-05-27T07:22:22.373701+0000","last_clean":"2024-05-27T07:22:22.373701+0000","last_became_active":"2024-05-27T07:17:06.763125+0000","last_became_peered":"2024-05-27T07:17:06.763125+0000","last_unstale":"2024-05-27T07:22:22.373701+0000","last_undegraded":"2024-05-27T07:22:22.373701+0000","last_fullsized":"2024-05-27T07:22:22.373701+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:43.359673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372016+0000","last_change":"2024-05-27T07:17:06.755896+0000","last_active":"2024-05-27T07:22:22.372016+0000","last_peered":"2024-05-27T07:22:22.372016+0000","last_clean":"2024-05-27T07:22:22.372016+0000","last_became_active":"2024-05-27T07:17:06.755495+0000","last_became_peered":"2024-05-27T07:17:06.755495+0000","last_unstale":"2024-05-27T07:22:22.372016+0000","last_undegraded":"2024-05-27T07:22:22.372016+0000","last_fullsized":"2024-05-27T07:22:22.372016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:41:37.632012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377962+0000","last_change":"2024-05-27T07:17:46.699881+0000","last_active":"2024-05-27T07:22:22.377962+0000","last_peered":"2024-05-27T07:22:22.377962+0000","last_clean":"2024-05-27T07:22:22.377962+0000","last_became_active":"2024-05-27T07:17:46.699080+0000","last_became_peered":"2024-05-27T07:17:46.699080+0000","last_unstale":"2024-05-27T07:22:22.377962+0000","last_undegraded":"2024-05-27T07:22:22.377962+0000","last_fullsized":"2024-05-27T07:22:22.377962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:43.580762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300909+0000","last_change":"2024-05-27T07:17:06.770468+0000","last_active":"2024-05-27T07:22:22.300909+0000","last_peered":"2024-05-27T07:22:22.300909+0000","last_clean":"2024-05-27T07:22:22.300909+0000","last_became_active":"2024-05-27T07:17:06.768345+0000","last_became_peered":"2024-05-27T07:17:06.768345+0000","last_unstale":"2024-05-27T07:22:22.300909+0000","last_undegraded":"2024-05-27T07:22:22.300909+0000","last_fullsized":"2024-05-27T07:22:22.300909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:12:27.646654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366997+0000","last_change":"2024-05-27T07:17:06.758769+0000","last_active":"2024-05-27T07:22:22.366997+0000","last_peered":"2024-05-27T07:22:22.366997+0000","last_clean":"2024-05-27T07:22:22.366997+0000","last_became_active":"2024-05-27T07:17:06.758393+0000","last_became_peered":"2024-05-27T07:17:06.758393+0000","last_unstale":"2024-05-27T07:22:22.366997+0000","last_undegraded":"2024-05-27T07:22:22.366997+0000","last_fullsized":"2024-05-27T07:22:22.366997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:56.450569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377841+0000","last_change":"2024-05-27T07:17:06.770681+0000","last_active":"2024-05-27T07:22:22.377841+0000","last_peered":"2024-05-27T07:22:22.377841+0000","last_clean":"2024-05-27T07:22:22.377841+0000","last_became_active":"2024-05-27T07:17:06.766520+0000","last_became_peered":"2024-05-27T07:17:06.766520+0000","last_unstale":"2024-05-27T07:22:22.377841+0000","last_undegraded":"2024-05-27T07:22:22.377841+0000","last_fullsized":"2024-05-27T07:22:22.377841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:56.931743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376580+0000","last_change":"2024-05-27T07:17:06.761462+0000","last_active":"2024-05-27T07:22:22.376580+0000","last_peered":"2024-05-27T07:22:22.376580+0000","last_clean":"2024-05-27T07:22:22.376580+0000","last_became_active":"2024-05-27T07:17:06.759344+0000","last_became_peered":"2024-05-27T07:17:06.759344+0000","last_unstale":"2024-05-27T07:22:22.376580+0000","last_undegraded":"2024-05-27T07:22:22.376580+0000","last_fullsized":"2024-05-27T07:22:22.376580+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:12:22.688477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303430+0000","last_change":"2024-05-27T07:17:06.761947+0000","last_active":"2024-05-27T07:22:22.303430+0000","last_peered":"2024-05-27T07:22:22.303430+0000","last_clean":"2024-05-27T07:22:22.303430+0000","last_became_active":"2024-05-27T07:17:06.761464+0000","last_became_peered":"2024-05-27T07:17:06.761464+0000","last_unstale":"2024-05-27T07:22:22.303430+0000","last_undegraded":"2024-05-27T07:22:22.303430+0000","last_fullsized":"2024-05-27T07:22:22.303430+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:08:19.939311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374234+0000","last_change":"2024-05-27T07:17:06.772039+0000","last_active":"2024-05-27T07:22:22.374234+0000","last_peered":"2024-05-27T07:22:22.374234+0000","last_clean":"2024-05-27T07:22:22.374234+0000","last_became_active":"2024-05-27T07:17:06.771303+0000","last_became_peered":"2024-05-27T07:17:06.771303+0000","last_unstale":"2024-05-27T07:22:22.374234+0000","last_undegraded":"2024-05-27T07:22:22.374234+0000","last_fullsized":"2024-05-27T07:22:22.374234+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:30.254848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297550+0000","last_change":"2024-05-27T07:17:06.774228+0000","last_active":"2024-05-27T07:22:22.297550+0000","last_peered":"2024-05-27T07:22:22.297550+0000","last_clean":"2024-05-27T07:22:22.297550+0000","last_became_active":"2024-05-27T07:17:06.771436+0000","last_became_peered":"2024-05-27T07:17:06.771436+0000","last_unstale":"2024-05-27T07:22:22.297550+0000","last_undegraded":"2024-05-27T07:22:22.297550+0000","last_fullsized":"2024-05-27T07:22:22.297550+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:25.749143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.593630+0000","last_change":"2024-05-27T07:17:06.755558+0000","last_active":"2024-05-27T07:22:22.593630+0000","last_peered":"2024-05-27T07:22:22.593630+0000","last_clean":"2024-05-27T07:22:22.593630+0000","last_became_active":"2024-05-27T07:17:06.754858+0000","last_became_peered":"2024-05-27T07:17:06.754858+0000","last_unstale":"2024-05-27T07:22:22.593630+0000","last_undegraded":"2024-05-27T07:22:22.593630+0000","last_fullsized":"2024-05-27T07:22:22.593630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:09.853982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595599+0000","last_change":"2024-05-27T07:17:06.770731+0000","last_active":"2024-05-27T07:22:22.595599+0000","last_peered":"2024-05-27T07:22:22.595599+0000","last_clean":"2024-05-27T07:22:22.595599+0000","last_became_active":"2024-05-27T07:17:06.766923+0000","last_became_peered":"2024-05-27T07:17:06.766923+0000","last_unstale":"2024-05-27T07:22:22.595599+0000","last_undegraded":"2024-05-27T07:22:22.595599+0000","last_fullsized":"2024-05-27T07:22:22.595599+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.110121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369807+0000","last_change":"2024-05-27T07:17:06.758928+0000","last_active":"2024-05-27T07:22:22.369807+0000","last_peered":"2024-05-27T07:22:22.369807+0000","last_clean":"2024-05-27T07:22:22.369807+0000","last_became_active":"2024-05-27T07:17:06.758328+0000","last_became_peered":"2024-05-27T07:17:06.758328+0000","last_unstale":"2024-05-27T07:22:22.369807+0000","last_undegraded":"2024-05-27T07:22:22.369807+0000","last_fullsized":"2024-05-27T07:22:22.369807+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:36.284395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377323+0000","last_change":"2024-05-27T07:17:06.760006+0000","last_active":"2024-05-27T07:22:22.377323+0000","last_peered":"2024-05-27T07:22:22.377323+0000","last_clean":"2024-05-27T07:22:22.377323+0000","last_became_active":"2024-05-27T07:17:06.758177+0000","last_became_peered":"2024-05-27T07:17:06.758177+0000","last_unstale":"2024-05-27T07:22:22.377323+0000","last_undegraded":"2024-05-27T07:22:22.377323+0000","last_fullsized":"2024-05-27T07:22:22.377323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.516289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370659+0000","last_change":"2024-05-27T07:17:06.768484+0000","last_active":"2024-05-27T07:22:22.370659+0000","last_peered":"2024-05-27T07:22:22.370659+0000","last_clean":"2024-05-27T07:22:22.370659+0000","last_became_active":"2024-05-27T07:17:06.768105+0000","last_became_peered":"2024-05-27T07:17:06.768105+0000","last_unstale":"2024-05-27T07:22:22.370659+0000","last_undegraded":"2024-05-27T07:22:22.370659+0000","last_fullsized":"2024-05-27T07:22:22.370659+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:54:48.751678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373306+0000","last_change":"2024-05-27T07:17:06.762782+0000","last_active":"2024-05-27T07:22:22.373306+0000","last_peered":"2024-05-27T07:22:22.373306+0000","last_clean":"2024-05-27T07:22:22.373306+0000","last_became_active":"2024-05-27T07:17:06.760129+0000","last_became_peered":"2024-05-27T07:17:06.760129+0000","last_unstale":"2024-05-27T07:22:22.373306+0000","last_undegraded":"2024-05-27T07:22:22.373306+0000","last_fullsized":"2024-05-27T07:22:22.373306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:47:19.360662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370479+0000","last_change":"2024-05-27T07:17:21.899772+0000","last_active":"2024-05-27T07:22:22.370479+0000","last_peered":"2024-05-27T07:22:22.370479+0000","last_clean":"2024-05-27T07:22:22.370479+0000","last_became_active":"2024-05-27T07:17:21.899038+0000","last_became_peered":"2024-05-27T07:17:21.899038+0000","last_unstale":"2024-05-27T07:22:22.370479+0000","last_undegraded":"2024-05-27T07:22:22.370479+0000","last_fullsized":"2024-05-27T07:22:22.370479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:38.784433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369427+0000","last_change":"2024-05-27T07:17:46.708798+0000","last_active":"2024-05-27T07:22:22.369427+0000","last_peered":"2024-05-27T07:22:22.369427+0000","last_clean":"2024-05-27T07:22:22.369427+0000","last_became_active":"2024-05-27T07:17:46.696433+0000","last_became_peered":"2024-05-27T07:17:46.696433+0000","last_unstale":"2024-05-27T07:22:22.369427+0000","last_undegraded":"2024-05-27T07:22:22.369427+0000","last_fullsized":"2024-05-27T07:22:22.369427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:40.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368466+0000","last_change":"2024-05-27T07:17:06.760865+0000","last_active":"2024-05-27T07:22:22.368466+0000","last_peered":"2024-05-27T07:22:22.368466+0000","last_clean":"2024-05-27T07:22:22.368466+0000","last_became_active":"2024-05-27T07:17:06.759491+0000","last_became_peered":"2024-05-27T07:17:06.759491+0000","last_unstale":"2024-05-27T07:22:22.368466+0000","last_undegraded":"2024-05-27T07:22:22.368466+0000","last_fullsized":"2024-05-27T07:22:22.368466+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:31.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372496+0000","last_change":"2024-05-27T07:17:21.898583+0000","last_active":"2024-05-27T07:22:22.372496+0000","last_peered":"2024-05-27T07:22:22.372496+0000","last_clean":"2024-05-27T07:22:22.372496+0000","last_became_active":"2024-05-27T07:17:21.897937+0000","last_became_peered":"2024-05-27T07:17:21.897937+0000","last_unstale":"2024-05-27T07:22:22.372496+0000","last_undegraded":"2024-05-27T07:22:22.372496+0000","last_fullsized":"2024-05-27T07:22:22.372496+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:45:00.412914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368660+0000","last_change":"2024-05-27T07:17:02.432465+0000","last_active":"2024-05-27T07:22:22.368660+0000","last_peered":"2024-05-27T07:22:22.368660+0000","last_clean":"2024-05-27T07:22:22.368660+0000","last_became_active":"2024-05-27T07:17:02.431638+0000","last_became_peered":"2024-05-27T07:17:02.431638+0000","last_unstale":"2024-05-27T07:22:22.368660+0000","last_undegraded":"2024-05-27T07:22:22.368660+0000","last_fullsized":"2024-05-27T07:22:22.368660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:03:13.167952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595303+0000","last_change":"2024-05-27T07:17:06.763133+0000","last_active":"2024-05-27T07:22:22.595303+0000","last_peered":"2024-05-27T07:22:22.595303+0000","last_clean":"2024-05-27T07:22:22.595303+0000","last_became_active":"2024-05-27T07:17:06.762770+0000","last_became_peered":"2024-05-27T07:17:06.762770+0000","last_unstale":"2024-05-27T07:22:22.595303+0000","last_undegraded":"2024-05-27T07:22:22.595303+0000","last_fullsized":"2024-05-27T07:22:22.595303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:33.223722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373220+0000","last_change":"2024-05-27T07:17:21.900369+0000","last_active":"2024-05-27T07:22:22.373220+0000","last_peered":"2024-05-27T07:22:22.373220+0000","last_clean":"2024-05-27T07:22:22.373220+0000","last_became_active":"2024-05-27T07:17:21.899758+0000","last_became_peered":"2024-05-27T07:17:21.899758+0000","last_unstale":"2024-05-27T07:22:22.373220+0000","last_undegraded":"2024-05-27T07:22:22.373220+0000","last_fullsized":"2024-05-27T07:22:22.373220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:42.116302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372292+0000","last_change":"2024-05-27T07:17:02.347047+0000","last_active":"2024-05-27T07:22:22.372292+0000","last_peered":"2024-05-27T07:22:22.372292+0000","last_clean":"2024-05-27T07:22:22.372292+0000","last_became_active":"2024-05-27T07:17:02.346562+0000","last_became_peered":"2024-05-27T07:17:02.346562+0000","last_unstale":"2024-05-27T07:22:22.372292+0000","last_undegraded":"2024-05-27T07:22:22.372292+0000","last_fullsized":"2024-05-27T07:22:22.372292+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:25:50.133089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372097+0000","last_change":"2024-05-27T07:17:06.767930+0000","last_active":"2024-05-27T07:22:22.372097+0000","last_peered":"2024-05-27T07:22:22.372097+0000","last_clean":"2024-05-27T07:22:22.372097+0000","last_became_active":"2024-05-27T07:17:06.767439+0000","last_became_peered":"2024-05-27T07:17:06.767439+0000","last_unstale":"2024-05-27T07:22:22.372097+0000","last_undegraded":"2024-05-27T07:22:22.372097+0000","last_fullsized":"2024-05-27T07:22:22.372097+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:18:47.464602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372182+0000","last_change":"2024-05-27T07:17:21.889199+0000","last_active":"2024-05-27T07:22:22.372182+0000","last_peered":"2024-05-27T07:22:22.372182+0000","last_clean":"2024-05-27T07:22:22.372182+0000","last_became_active":"2024-05-27T07:17:21.888667+0000","last_became_peered":"2024-05-27T07:17:21.888667+0000","last_unstale":"2024-05-27T07:22:22.372182+0000","last_undegraded":"2024-05-27T07:22:22.372182+0000","last_fullsized":"2024-05-27T07:22:22.372182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.029820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371656+0000","last_change":"2024-05-27T07:17:02.339872+0000","last_active":"2024-05-27T07:22:22.371656+0000","last_peered":"2024-05-27T07:22:22.371656+0000","last_clean":"2024-05-27T07:22:22.371656+0000","last_became_active":"2024-05-27T07:17:02.338975+0000","last_became_peered":"2024-05-27T07:17:02.338975+0000","last_unstale":"2024-05-27T07:22:22.371656+0000","last_undegraded":"2024-05-27T07:22:22.371656+0000","last_fullsized":"2024-05-27T07:22:22.371656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:14.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301875+0000","last_change":"2024-05-27T07:17:06.758234+0000","last_active":"2024-05-27T07:22:22.301875+0000","last_peered":"2024-05-27T07:22:22.301875+0000","last_clean":"2024-05-27T07:22:22.301875+0000","last_became_active":"2024-05-27T07:17:06.757827+0000","last_became_peered":"2024-05-27T07:17:06.757827+0000","last_unstale":"2024-05-27T07:22:22.301875+0000","last_undegraded":"2024-05-27T07:22:22.301875+0000","last_fullsized":"2024-05-27T07:22:22.301875+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:21.688435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372775+0000","last_change":"2024-05-27T07:17:21.882540+0000","last_active":"2024-05-27T07:22:22.372775+0000","last_peered":"2024-05-27T07:22:22.372775+0000","last_clean":"2024-05-27T07:22:22.372775+0000","last_became_active":"2024-05-27T07:17:21.882087+0000","last_became_peered":"2024-05-27T07:17:21.882087+0000","last_unstale":"2024-05-27T07:22:22.372775+0000","last_undegraded":"2024-05-27T07:22:22.372775+0000","last_fullsized":"2024-05-27T07:22:22.372775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:53:04.363521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'9","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.712554+0000","last_change":"2024-05-27T07:17:46.704997+0000","last_active":"2024-05-27T07:22:22.712554+0000","last_peered":"2024-05-27T07:22:22.712554+0000","last_clean":"2024-05-27T07:22:22.712554+0000","last_became_active":"2024-05-27T07:17:46.704355+0000","last_became_peered":"2024-05-27T07:17:46.704355+0000","last_unstale":"2024-05-27T07:22:22.712554+0000","last_undegraded":"2024-05-27T07:22:22.712554+0000","last_fullsized":"2024-05-27T07:22:22.712554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:43:02.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":229,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372140+0000","last_change":"2024-05-27T07:17:06.765809+0000","last_active":"2024-05-27T07:22:22.372140+0000","last_peered":"2024-05-27T07:22:22.372140+0000","last_clean":"2024-05-27T07:22:22.372140+0000","last_became_active":"2024-05-27T07:17:06.763845+0000","last_became_peered":"2024-05-27T07:17:06.763845+0000","last_unstale":"2024-05-27T07:22:22.372140+0000","last_undegraded":"2024-05-27T07:22:22.372140+0000","last_fullsized":"2024-05-27T07:22:22.372140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:25.399181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372567+0000","last_change":"2024-05-27T07:17:21.886316+0000","last_active":"2024-05-27T07:22:22.372567+0000","last_peered":"2024-05-27T07:22:22.372567+0000","last_clean":"2024-05-27T07:22:22.372567+0000","last_became_active":"2024-05-27T07:17:21.885129+0000","last_became_peered":"2024-05-27T07:17:21.885129+0000","last_unstale":"2024-05-27T07:22:22.372567+0000","last_undegraded":"2024-05-27T07:22:22.372567+0000","last_fullsized":"2024-05-27T07:22:22.372567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:10.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377109+0000","last_change":"2024-05-27T07:17:02.338765+0000","last_active":"2024-05-27T07:22:22.377109+0000","last_peered":"2024-05-27T07:22:22.377109+0000","last_clean":"2024-05-27T07:22:22.377109+0000","last_became_active":"2024-05-27T07:17:02.337103+0000","last_became_peered":"2024-05-27T07:17:02.337103+0000","last_unstale":"2024-05-27T07:22:22.377109+0000","last_undegraded":"2024-05-27T07:22:22.377109+0000","last_fullsized":"2024-05-27T07:22:22.377109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:15:07.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301489+0000","last_change":"2024-05-27T07:17:06.766839+0000","last_active":"2024-05-27T07:22:22.301489+0000","last_peered":"2024-05-27T07:22:22.301489+0000","last_clean":"2024-05-27T07:22:22.301489+0000","last_became_active":"2024-05-27T07:17:06.766373+0000","last_became_peered":"2024-05-27T07:17:06.766373+0000","last_unstale":"2024-05-27T07:22:22.301489+0000","last_undegraded":"2024-05-27T07:22:22.301489+0000","last_fullsized":"2024-05-27T07:22:22.301489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:09:55.064045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376814+0000","last_change":"2024-05-27T07:18:47.133692+0000","last_active":"2024-05-27T07:22:22.376814+0000","last_peered":"2024-05-27T07:22:22.376814+0000","last_clean":"2024-05-27T07:22:22.376814+0000","last_became_active":"2024-05-27T07:18:47.132122+0000","last_became_peered":"2024-05-27T07:18:47.132122+0000","last_unstale":"2024-05-27T07:22:22.376814+0000","last_undegraded":"2024-05-27T07:22:22.376814+0000","last_fullsized":"2024-05-27T07:22:22.376814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:57.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296700+0000","last_change":"2024-05-27T07:17:02.329809+0000","last_active":"2024-05-27T07:22:22.296700+0000","last_peered":"2024-05-27T07:22:22.296700+0000","last_clean":"2024-05-27T07:22:22.296700+0000","last_became_active":"2024-05-27T07:17:02.328758+0000","last_became_peered":"2024-05-27T07:17:02.328758+0000","last_unstale":"2024-05-27T07:22:22.296700+0000","last_undegraded":"2024-05-27T07:22:22.296700+0000","last_fullsized":"2024-05-27T07:22:22.296700+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:52.479444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366880+0000","last_change":"2024-05-27T07:17:06.749836+0000","last_active":"2024-05-27T07:22:22.366880+0000","last_peered":"2024-05-27T07:22:22.366880+0000","last_clean":"2024-05-27T07:22:22.366880+0000","last_became_active":"2024-05-27T07:17:06.749512+0000","last_became_peered":"2024-05-27T07:17:06.749512+0000","last_unstale":"2024-05-27T07:22:22.366880+0000","last_undegraded":"2024-05-27T07:22:22.366880+0000","last_fullsized":"2024-05-27T07:22:22.366880+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:41:51.000415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595128+0000","last_change":"2024-05-27T07:17:21.888376+0000","last_active":"2024-05-27T07:22:22.595128+0000","last_peered":"2024-05-27T07:22:22.595128+0000","last_clean":"2024-05-27T07:22:22.595128+0000","last_became_active":"2024-05-27T07:17:21.886183+0000","last_became_peered":"2024-05-27T07:17:21.886183+0000","last_unstale":"2024-05-27T07:22:22.595128+0000","last_undegraded":"2024-05-27T07:22:22.595128+0000","last_fullsized":"2024-05-27T07:22:22.595128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:34.541338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300149+0000","last_change":"2024-05-27T07:17:02.334113+0000","last_active":"2024-05-27T07:22:22.300149+0000","last_peered":"2024-05-27T07:22:22.300149+0000","last_clean":"2024-05-27T07:22:22.300149+0000","last_became_active":"2024-05-27T07:17:02.333201+0000","last_became_peered":"2024-05-27T07:17:02.333201+0000","last_unstale":"2024-05-27T07:22:22.300149+0000","last_undegraded":"2024-05-27T07:22:22.300149+0000","last_fullsized":"2024-05-27T07:22:22.300149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:13.484654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369241+0000","last_change":"2024-05-27T07:17:06.751676+0000","last_active":"2024-05-27T07:22:22.369241+0000","last_peered":"2024-05-27T07:22:22.369241+0000","last_clean":"2024-05-27T07:22:22.369241+0000","last_became_active":"2024-05-27T07:17:06.751221+0000","last_became_peered":"2024-05-27T07:17:06.751221+0000","last_unstale":"2024-05-27T07:22:22.369241+0000","last_undegraded":"2024-05-27T07:22:22.369241+0000","last_fullsized":"2024-05-27T07:22:22.369241+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:40.012561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367255+0000","last_change":"2024-05-27T07:17:21.897640+0000","last_active":"2024-05-27T07:22:22.367255+0000","last_peered":"2024-05-27T07:22:22.367255+0000","last_clean":"2024-05-27T07:22:22.367255+0000","last_became_active":"2024-05-27T07:17:21.895355+0000","last_became_peered":"2024-05-27T07:17:21.895355+0000","last_unstale":"2024-05-27T07:22:22.367255+0000","last_undegraded":"2024-05-27T07:22:22.367255+0000","last_fullsized":"2024-05-27T07:22:22.367255+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:51:12.030069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'5","reported_seq":277,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375588+0000","last_change":"2024-05-27T07:17:02.339464+0000","last_active":"2024-05-27T07:22:22.375588+0000","last_peered":"2024-05-27T07:22:22.375588+0000","last_clean":"2024-05-27T07:22:22.375588+0000","last_became_active":"2024-05-27T07:17:02.338577+0000","last_became_peered":"2024-05-27T07:17:02.338577+0000","last_unstale":"2024-05-27T07:22:22.375588+0000","last_undegraded":"2024-05-27T07:22:22.375588+0000","last_fullsized":"2024-05-27T07:22:22.375588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:29:59.210157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376770+0000","last_change":"2024-05-27T07:17:06.766883+0000","last_active":"2024-05-27T07:22:22.376770+0000","last_peered":"2024-05-27T07:22:22.376770+0000","last_clean":"2024-05-27T07:22:22.376770+0000","last_became_active":"2024-05-27T07:17:06.766166+0000","last_became_peered":"2024-05-27T07:17:06.766166+0000","last_unstale":"2024-05-27T07:22:22.376770+0000","last_undegraded":"2024-05-27T07:22:22.376770+0000","last_fullsized":"2024-05-27T07:22:22.376770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:28.936499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369022+0000","last_change":"2024-05-27T07:17:21.893930+0000","last_active":"2024-05-27T07:22:22.369022+0000","last_peered":"2024-05-27T07:22:22.369022+0000","last_clean":"2024-05-27T07:22:22.369022+0000","last_became_active":"2024-05-27T07:17:21.893029+0000","last_became_peered":"2024-05-27T07:17:21.893029+0000","last_unstale":"2024-05-27T07:22:22.369022+0000","last_undegraded":"2024-05-27T07:22:22.369022+0000","last_fullsized":"2024-05-27T07:22:22.369022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:32.453078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303681+0000","last_change":"2024-05-27T07:17:02.346531+0000","last_active":"2024-05-27T07:22:22.303681+0000","last_peered":"2024-05-27T07:22:22.303681+0000","last_clean":"2024-05-27T07:22:22.303681+0000","last_became_active":"2024-05-27T07:17:02.345844+0000","last_became_peered":"2024-05-27T07:17:02.345844+0000","last_unstale":"2024-05-27T07:22:22.303681+0000","last_undegraded":"2024-05-27T07:22:22.303681+0000","last_fullsized":"2024-05-27T07:22:22.303681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:28.544616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595299+0000","last_change":"2024-05-27T07:17:06.769845+0000","last_active":"2024-05-27T07:22:22.595299+0000","last_peered":"2024-05-27T07:22:22.595299+0000","last_clean":"2024-05-27T07:22:22.595299+0000","last_became_active":"2024-05-27T07:17:06.768467+0000","last_became_peered":"2024-05-27T07:17:06.768467+0000","last_unstale":"2024-05-27T07:22:22.595299+0000","last_undegraded":"2024-05-27T07:22:22.595299+0000","last_fullsized":"2024-05-27T07:22:22.595299+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:30.311824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374196+0000","last_change":"2024-05-27T07:17:21.899885+0000","last_active":"2024-05-27T07:22:22.374196+0000","last_peered":"2024-05-27T07:22:22.374196+0000","last_clean":"2024-05-27T07:22:22.374196+0000","last_became_active":"2024-05-27T07:17:21.899340+0000","last_became_peered":"2024-05-27T07:17:21.899340+0000","last_unstale":"2024-05-27T07:22:22.374196+0000","last_undegraded":"2024-05-27T07:22:22.374196+0000","last_fullsized":"2024-05-27T07:22:22.374196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:12.665409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301032+0000","last_change":"2024-05-27T07:17:02.435770+0000","last_active":"2024-05-27T07:22:22.301032+0000","last_peered":"2024-05-27T07:22:22.301032+0000","last_clean":"2024-05-27T07:22:22.301032+0000","last_became_active":"2024-05-27T07:17:02.434535+0000","last_became_peered":"2024-05-27T07:17:02.434535+0000","last_unstale":"2024-05-27T07:22:22.301032+0000","last_undegraded":"2024-05-27T07:22:22.301032+0000","last_fullsized":"2024-05-27T07:22:22.301032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:53.745583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372886+0000","last_change":"2024-05-27T07:17:46.704100+0000","last_active":"2024-05-27T07:22:22.372886+0000","last_peered":"2024-05-27T07:22:22.372886+0000","last_clean":"2024-05-27T07:22:22.372886+0000","last_became_active":"2024-05-27T07:17:46.703475+0000","last_became_peered":"2024-05-27T07:17:46.703475+0000","last_unstale":"2024-05-27T07:22:22.372886+0000","last_undegraded":"2024-05-27T07:22:22.372886+0000","last_fullsized":"2024-05-27T07:22:22.372886+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:20:29.173923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377473+0000","last_change":"2024-05-27T07:18:47.131521+0000","last_active":"2024-05-27T07:22:22.377473+0000","last_peered":"2024-05-27T07:22:22.377473+0000","last_clean":"2024-05-27T07:22:22.377473+0000","last_became_active":"2024-05-27T07:18:47.129995+0000","last_became_peered":"2024-05-27T07:18:47.129995+0000","last_unstale":"2024-05-27T07:22:22.377473+0000","last_undegraded":"2024-05-27T07:22:22.377473+0000","last_fullsized":"2024-05-27T07:22:22.377473+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:45:27.268768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":565,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.933547+0000","last_change":"2024-05-27T07:17:46.719293+0000","last_active":"2024-05-27T07:22:22.933547+0000","last_peered":"2024-05-27T07:22:22.933547+0000","last_clean":"2024-05-27T07:22:22.933547+0000","last_became_active":"2024-05-27T07:17:46.703496+0000","last_became_peered":"2024-05-27T07:17:46.703496+0000","last_unstale":"2024-05-27T07:22:22.933547+0000","last_undegraded":"2024-05-27T07:22:22.933547+0000","last_fullsized":"2024-05-27T07:22:22.933547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:26.310524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67910,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373655+0000","last_change":"2024-05-27T07:17:06.763141+0000","last_active":"2024-05-27T07:22:22.373655+0000","last_peered":"2024-05-27T07:22:22.373655+0000","last_clean":"2024-05-27T07:22:22.373655+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:22:22.373655+0000","last_undegraded":"2024-05-27T07:22:22.373655+0000","last_fullsized":"2024-05-27T07:22:22.373655+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:47.072552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302824+0000","last_change":"2024-05-27T07:17:21.895342+0000","last_active":"2024-05-27T07:22:22.302824+0000","last_peered":"2024-05-27T07:22:22.302824+0000","last_clean":"2024-05-27T07:22:22.302824+0000","last_became_active":"2024-05-27T07:17:21.894630+0000","last_became_peered":"2024-05-27T07:17:21.894630+0000","last_unstale":"2024-05-27T07:22:22.302824+0000","last_undegraded":"2024-05-27T07:22:22.302824+0000","last_fullsized":"2024-05-27T07:22:22.302824+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:50.501683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371791+0000","last_change":"2024-05-27T07:17:02.341848+0000","last_active":"2024-05-27T07:22:22.371791+0000","last_peered":"2024-05-27T07:22:22.371791+0000","last_clean":"2024-05-27T07:22:22.371791+0000","last_became_active":"2024-05-27T07:17:02.339477+0000","last_became_peered":"2024-05-27T07:17:02.339477+0000","last_unstale":"2024-05-27T07:22:22.371791+0000","last_undegraded":"2024-05-27T07:22:22.371791+0000","last_fullsized":"2024-05-27T07:22:22.371791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:23.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297148+0000","last_change":"2024-05-27T07:17:06.761340+0000","last_active":"2024-05-27T07:22:22.297148+0000","last_peered":"2024-05-27T07:22:22.297148+0000","last_clean":"2024-05-27T07:22:22.297148+0000","last_became_active":"2024-05-27T07:17:06.759103+0000","last_became_peered":"2024-05-27T07:17:06.759103+0000","last_unstale":"2024-05-27T07:22:22.297148+0000","last_undegraded":"2024-05-27T07:22:22.297148+0000","last_fullsized":"2024-05-27T07:22:22.297148+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:42.263106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375699+0000","last_change":"2024-05-27T07:17:21.883252+0000","last_active":"2024-05-27T07:22:22.375699+0000","last_peered":"2024-05-27T07:22:22.375699+0000","last_clean":"2024-05-27T07:22:22.375699+0000","last_became_active":"2024-05-27T07:17:21.882468+0000","last_became_peered":"2024-05-27T07:17:21.882468+0000","last_unstale":"2024-05-27T07:22:22.375699+0000","last_undegraded":"2024-05-27T07:22:22.375699+0000","last_fullsized":"2024-05-27T07:22:22.375699+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:59.250357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301049+0000","last_change":"2024-05-27T07:17:02.343343+0000","last_active":"2024-05-27T07:22:22.301049+0000","last_peered":"2024-05-27T07:22:22.301049+0000","last_clean":"2024-05-27T07:22:22.301049+0000","last_became_active":"2024-05-27T07:17:02.342741+0000","last_became_peered":"2024-05-27T07:17:02.342741+0000","last_unstale":"2024-05-27T07:22:22.301049+0000","last_undegraded":"2024-05-27T07:22:22.301049+0000","last_fullsized":"2024-05-27T07:22:22.301049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:46.006865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301543+0000","last_change":"2024-05-27T07:17:06.771473+0000","last_active":"2024-05-27T07:22:22.301543+0000","last_peered":"2024-05-27T07:22:22.301543+0000","last_clean":"2024-05-27T07:22:22.301543+0000","last_became_active":"2024-05-27T07:17:06.771082+0000","last_became_peered":"2024-05-27T07:17:06.771082+0000","last_unstale":"2024-05-27T07:22:22.301543+0000","last_undegraded":"2024-05-27T07:22:22.301543+0000","last_fullsized":"2024-05-27T07:22:22.301543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:01.371182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369606+0000","last_change":"2024-05-27T07:17:21.897691+0000","last_active":"2024-05-27T07:22:22.369606+0000","last_peered":"2024-05-27T07:22:22.369606+0000","last_clean":"2024-05-27T07:22:22.369606+0000","last_became_active":"2024-05-27T07:17:21.895221+0000","last_became_peered":"2024-05-27T07:17:21.895221+0000","last_unstale":"2024-05-27T07:22:22.369606+0000","last_undegraded":"2024-05-27T07:22:22.369606+0000","last_fullsized":"2024-05-27T07:22:22.369606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:14.626947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":283,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371332+0000","last_change":"2024-05-27T07:17:02.343375+0000","last_active":"2024-05-27T07:22:22.371332+0000","last_peered":"2024-05-27T07:22:22.371332+0000","last_clean":"2024-05-27T07:22:22.371332+0000","last_became_active":"2024-05-27T07:17:02.341315+0000","last_became_peered":"2024-05-27T07:17:02.341315+0000","last_unstale":"2024-05-27T07:22:22.371332+0000","last_undegraded":"2024-05-27T07:22:22.371332+0000","last_fullsized":"2024-05-27T07:22:22.371332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:41.927265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301159+0000","last_change":"2024-05-27T07:17:06.765862+0000","last_active":"2024-05-27T07:22:22.301159+0000","last_peered":"2024-05-27T07:22:22.301159+0000","last_clean":"2024-05-27T07:22:22.301159+0000","last_became_active":"2024-05-27T07:17:06.765439+0000","last_became_peered":"2024-05-27T07:17:06.765439+0000","last_unstale":"2024-05-27T07:22:22.301159+0000","last_undegraded":"2024-05-27T07:22:22.301159+0000","last_fullsized":"2024-05-27T07:22:22.301159+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:47.586833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.298125+0000","last_change":"2024-05-27T07:17:21.891810+0000","last_active":"2024-05-27T07:22:22.298125+0000","last_peered":"2024-05-27T07:22:22.298125+0000","last_clean":"2024-05-27T07:22:22.298125+0000","last_became_active":"2024-05-27T07:17:21.890473+0000","last_became_peered":"2024-05-27T07:17:21.890473+0000","last_unstale":"2024-05-27T07:22:22.298125+0000","last_undegraded":"2024-05-27T07:22:22.298125+0000","last_fullsized":"2024-05-27T07:22:22.298125+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:12.731189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375448+0000","last_change":"2024-05-27T07:17:46.715870+0000","last_active":"2024-05-27T07:22:22.375448+0000","last_peered":"2024-05-27T07:22:22.375448+0000","last_clean":"2024-05-27T07:22:22.375448+0000","last_became_active":"2024-05-27T07:17:46.701248+0000","last_became_peered":"2024-05-27T07:17:46.701248+0000","last_unstale":"2024-05-27T07:22:22.375448+0000","last_undegraded":"2024-05-27T07:22:22.375448+0000","last_fullsized":"2024-05-27T07:22:22.375448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:18.693752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301040+0000","last_change":"2024-05-27T07:17:06.759937+0000","last_active":"2024-05-27T07:22:22.301040+0000","last_peered":"2024-05-27T07:22:22.301040+0000","last_clean":"2024-05-27T07:22:22.301040+0000","last_became_active":"2024-05-27T07:17:06.759526+0000","last_became_peered":"2024-05-27T07:17:06.759526+0000","last_unstale":"2024-05-27T07:22:22.301040+0000","last_undegraded":"2024-05-27T07:22:22.301040+0000","last_fullsized":"2024-05-27T07:22:22.301040+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:22.934317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300660+0000","last_change":"2024-05-27T07:17:21.898331+0000","last_active":"2024-05-27T07:22:22.300660+0000","last_peered":"2024-05-27T07:22:22.300660+0000","last_clean":"2024-05-27T07:22:22.300660+0000","last_became_active":"2024-05-27T07:17:21.897965+0000","last_became_peered":"2024-05-27T07:17:21.897965+0000","last_unstale":"2024-05-27T07:22:22.300660+0000","last_undegraded":"2024-05-27T07:22:22.300660+0000","last_fullsized":"2024-05-27T07:22:22.300660+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:20.584175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373702+0000","last_change":"2024-05-27T07:17:02.346029+0000","last_active":"2024-05-27T07:22:22.373702+0000","last_peered":"2024-05-27T07:22:22.373702+0000","last_clean":"2024-05-27T07:22:22.373702+0000","last_became_active":"2024-05-27T07:17:02.341292+0000","last_became_peered":"2024-05-27T07:17:02.341292+0000","last_unstale":"2024-05-27T07:22:22.373702+0000","last_undegraded":"2024-05-27T07:22:22.373702+0000","last_fullsized":"2024-05-27T07:22:22.373702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:31.705741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376883+0000","last_change":"2024-05-27T07:17:46.691785+0000","last_active":"2024-05-27T07:22:22.376883+0000","last_peered":"2024-05-27T07:22:22.376883+0000","last_clean":"2024-05-27T07:22:22.376883+0000","last_became_active":"2024-05-27T07:17:46.690445+0000","last_became_peered":"2024-05-27T07:17:46.690445+0000","last_unstale":"2024-05-27T07:22:22.376883+0000","last_undegraded":"2024-05-27T07:22:22.376883+0000","last_fullsized":"2024-05-27T07:22:22.376883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:59:27.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372885+0000","last_change":"2024-05-27T07:17:21.892434+0000","last_active":"2024-05-27T07:22:22.372885+0000","last_peered":"2024-05-27T07:22:22.372885+0000","last_clean":"2024-05-27T07:22:22.372885+0000","last_became_active":"2024-05-27T07:17:21.891736+0000","last_became_peered":"2024-05-27T07:17:21.891736+0000","last_unstale":"2024-05-27T07:22:22.372885+0000","last_undegraded":"2024-05-27T07:22:22.372885+0000","last_fullsized":"2024-05-27T07:22:22.372885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:48.458819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370417+0000","last_change":"2024-05-27T07:17:02.436323+0000","last_active":"2024-05-27T07:22:22.370417+0000","last_peered":"2024-05-27T07:22:22.370417+0000","last_clean":"2024-05-27T07:22:22.370417+0000","last_became_active":"2024-05-27T07:17:02.435000+0000","last_became_peered":"2024-05-27T07:17:02.435000+0000","last_unstale":"2024-05-27T07:22:22.370417+0000","last_undegraded":"2024-05-27T07:22:22.370417+0000","last_fullsized":"2024-05-27T07:22:22.370417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:08.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374831+0000","last_change":"2024-05-27T07:17:06.760418+0000","last_active":"2024-05-27T07:22:22.374831+0000","last_peered":"2024-05-27T07:22:22.374831+0000","last_clean":"2024-05-27T07:22:22.374831+0000","last_became_active":"2024-05-27T07:17:06.759875+0000","last_became_peered":"2024-05-27T07:17:06.759875+0000","last_unstale":"2024-05-27T07:22:22.374831+0000","last_undegraded":"2024-05-27T07:22:22.374831+0000","last_fullsized":"2024-05-27T07:22:22.374831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:12:54.324323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297331+0000","last_change":"2024-05-27T07:17:21.888048+0000","last_active":"2024-05-27T07:22:22.297331+0000","last_peered":"2024-05-27T07:22:22.297331+0000","last_clean":"2024-05-27T07:22:22.297331+0000","last_became_active":"2024-05-27T07:17:21.887579+0000","last_became_peered":"2024-05-27T07:17:21.887579+0000","last_unstale":"2024-05-27T07:22:22.297331+0000","last_undegraded":"2024-05-27T07:22:22.297331+0000","last_fullsized":"2024-05-27T07:22:22.297331+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:34:44.569959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301559+0000","last_change":"2024-05-27T07:17:02.337655+0000","last_active":"2024-05-27T07:22:22.301559+0000","last_peered":"2024-05-27T07:22:22.301559+0000","last_clean":"2024-05-27T07:22:22.301559+0000","last_became_active":"2024-05-27T07:17:02.337252+0000","last_became_peered":"2024-05-27T07:17:02.337252+0000","last_unstale":"2024-05-27T07:22:22.301559+0000","last_undegraded":"2024-05-27T07:22:22.301559+0000","last_fullsized":"2024-05-27T07:22:22.301559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:11:25.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376338+0000","last_change":"2024-05-27T07:17:06.762934+0000","last_active":"2024-05-27T07:22:22.376338+0000","last_peered":"2024-05-27T07:22:22.376338+0000","last_clean":"2024-05-27T07:22:22.376338+0000","last_became_active":"2024-05-27T07:17:06.762507+0000","last_became_peered":"2024-05-27T07:17:06.762507+0000","last_unstale":"2024-05-27T07:22:22.376338+0000","last_undegraded":"2024-05-27T07:22:22.376338+0000","last_fullsized":"2024-05-27T07:22:22.376338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:23:35.164892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296768+0000","last_change":"2024-05-27T07:17:21.886991+0000","last_active":"2024-05-27T07:22:22.296768+0000","last_peered":"2024-05-27T07:22:22.296768+0000","last_clean":"2024-05-27T07:22:22.296768+0000","last_became_active":"2024-05-27T07:17:21.885945+0000","last_became_peered":"2024-05-27T07:17:21.885945+0000","last_unstale":"2024-05-27T07:22:22.296768+0000","last_undegraded":"2024-05-27T07:22:22.296768+0000","last_fullsized":"2024-05-27T07:22:22.296768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:23:01.853601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300810+0000","last_change":"2024-05-27T07:17:46.864130+0000","last_active":"2024-05-27T07:22:22.300810+0000","last_peered":"2024-05-27T07:22:22.300810+0000","last_clean":"2024-05-27T07:22:22.300810+0000","last_became_active":"2024-05-27T07:17:46.862997+0000","last_became_peered":"2024-05-27T07:17:46.862997+0000","last_unstale":"2024-05-27T07:22:22.300810+0000","last_undegraded":"2024-05-27T07:22:22.300810+0000","last_fullsized":"2024-05-27T07:22:22.300810+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:10.594627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":278,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296813+0000","last_change":"2024-05-27T07:17:02.435463+0000","last_active":"2024-05-27T07:22:22.296813+0000","last_peered":"2024-05-27T07:22:22.296813+0000","last_clean":"2024-05-27T07:22:22.296813+0000","last_became_active":"2024-05-27T07:17:02.432131+0000","last_became_peered":"2024-05-27T07:17:02.432131+0000","last_unstale":"2024-05-27T07:22:22.296813+0000","last_undegraded":"2024-05-27T07:22:22.296813+0000","last_fullsized":"2024-05-27T07:22:22.296813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:41.874577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302768+0000","last_change":"2024-05-27T07:18:47.134043+0000","last_active":"2024-05-27T07:22:22.302768+0000","last_peered":"2024-05-27T07:22:22.302768+0000","last_clean":"2024-05-27T07:22:22.302768+0000","last_became_active":"2024-05-27T07:18:47.132050+0000","last_became_peered":"2024-05-27T07:18:47.132050+0000","last_unstale":"2024-05-27T07:22:22.302768+0000","last_undegraded":"2024-05-27T07:22:22.302768+0000","last_fullsized":"2024-05-27T07:22:22.302768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:12.170883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367929+0000","last_change":"2024-05-27T07:17:06.751168+0000","last_active":"2024-05-27T07:22:22.367929+0000","last_peered":"2024-05-27T07:22:22.367929+0000","last_clean":"2024-05-27T07:22:22.367929+0000","last_became_active":"2024-05-27T07:17:06.750816+0000","last_became_peered":"2024-05-27T07:17:06.750816+0000","last_unstale":"2024-05-27T07:22:22.367929+0000","last_undegraded":"2024-05-27T07:22:22.367929+0000","last_fullsized":"2024-05-27T07:22:22.367929+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:33:45.607426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373763+0000","last_change":"2024-05-27T07:17:02.337490+0000","last_active":"2024-05-27T07:22:22.373763+0000","last_peered":"2024-05-27T07:22:22.373763+0000","last_clean":"2024-05-27T07:22:22.373763+0000","last_became_active":"2024-05-27T07:17:02.335317+0000","last_became_peered":"2024-05-27T07:17:02.335317+0000","last_unstale":"2024-05-27T07:22:22.373763+0000","last_undegraded":"2024-05-27T07:22:22.373763+0000","last_fullsized":"2024-05-27T07:22:22.373763+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:33:46.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":123,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300719+0000","last_change":"2024-05-27T07:15:10.825312+0000","last_active":"2024-05-27T07:22:22.300719+0000","last_peered":"2024-05-27T07:22:22.300719+0000","last_clean":"2024-05-27T07:22:22.300719+0000","last_became_active":"2024-05-27T07:15:10.809245+0000","last_became_peered":"2024-05-27T07:15:10.809245+0000","last_unstale":"2024-05-27T07:22:22.300719+0000","last_undegraded":"2024-05-27T07:22:22.300719+0000","last_fullsized":"2024-05-27T07:22:22.300719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_clean_scrub_stamp":"2024-05-27T07:10:08.008565+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:32.769372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302265+0000","last_change":"2024-05-27T07:18:47.133948+0000","last_active":"2024-05-27T07:22:22.302265+0000","last_peered":"2024-05-27T07:22:22.302265+0000","last_clean":"2024-05-27T07:22:22.302265+0000","last_became_active":"2024-05-27T07:18:47.132566+0000","last_became_peered":"2024-05-27T07:18:47.132566+0000","last_unstale":"2024-05-27T07:22:22.302265+0000","last_undegraded":"2024-05-27T07:22:22.302265+0000","last_fullsized":"2024-05-27T07:22:22.302265+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:04.463121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296515+0000","last_change":"2024-05-27T07:17:06.766187+0000","last_active":"2024-05-27T07:22:22.296515+0000","last_peered":"2024-05-27T07:22:22.296515+0000","last_clean":"2024-05-27T07:22:22.296515+0000","last_became_active":"2024-05-27T07:17:06.763365+0000","last_became_peered":"2024-05-27T07:17:06.763365+0000","last_unstale":"2024-05-27T07:22:22.296515+0000","last_undegraded":"2024-05-27T07:22:22.296515+0000","last_fullsized":"2024-05-27T07:22:22.296515+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:13.668426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301091+0000","last_change":"2024-05-27T07:17:02.344452+0000","last_active":"2024-05-27T07:22:22.301091+0000","last_peered":"2024-05-27T07:22:22.301091+0000","last_clean":"2024-05-27T07:22:22.301091+0000","last_became_active":"2024-05-27T07:17:02.343885+0000","last_became_peered":"2024-05-27T07:17:02.343885+0000","last_unstale":"2024-05-27T07:22:22.301091+0000","last_undegraded":"2024-05-27T07:22:22.301091+0000","last_fullsized":"2024-05-27T07:22:22.301091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:26.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375242+0000","last_change":"2024-05-27T07:17:21.883024+0000","last_active":"2024-05-27T07:22:22.375242+0000","last_peered":"2024-05-27T07:22:22.375242+0000","last_clean":"2024-05-27T07:22:22.375242+0000","last_became_active":"2024-05-27T07:17:21.882429+0000","last_became_peered":"2024-05-27T07:17:21.882429+0000","last_unstale":"2024-05-27T07:22:22.375242+0000","last_undegraded":"2024-05-27T07:22:22.375242+0000","last_fullsized":"2024-05-27T07:22:22.375242+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:36:40.443537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373282+0000","last_change":"2024-05-27T07:17:02.432768+0000","last_active":"2024-05-27T07:22:22.373282+0000","last_peered":"2024-05-27T07:22:22.373282+0000","last_clean":"2024-05-27T07:22:22.373282+0000","last_became_active":"2024-05-27T07:17:02.431551+0000","last_became_peered":"2024-05-27T07:17:02.431551+0000","last_unstale":"2024-05-27T07:22:22.373282+0000","last_undegraded":"2024-05-27T07:22:22.373282+0000","last_fullsized":"2024-05-27T07:22:22.373282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:39.759512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297928+0000","last_change":"2024-05-27T07:17:06.770992+0000","last_active":"2024-05-27T07:22:22.297928+0000","last_peered":"2024-05-27T07:22:22.297928+0000","last_clean":"2024-05-27T07:22:22.297928+0000","last_became_active":"2024-05-27T07:17:06.770560+0000","last_became_peered":"2024-05-27T07:17:06.770560+0000","last_unstale":"2024-05-27T07:22:22.297928+0000","last_undegraded":"2024-05-27T07:22:22.297928+0000","last_fullsized":"2024-05-27T07:22:22.297928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:07:34.443551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366856+0000","last_change":"2024-05-27T07:17:21.895362+0000","last_active":"2024-05-27T07:22:22.366856+0000","last_peered":"2024-05-27T07:22:22.366856+0000","last_clean":"2024-05-27T07:22:22.366856+0000","last_became_active":"2024-05-27T07:17:21.894563+0000","last_became_peered":"2024-05-27T07:17:21.894563+0000","last_unstale":"2024-05-27T07:22:22.366856+0000","last_undegraded":"2024-05-27T07:22:22.366856+0000","last_fullsized":"2024-05-27T07:22:22.366856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:44.394301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'49","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.689696+0000","last_change":"2024-05-27T07:17:02.338203+0000","last_active":"2024-05-27T07:22:22.689696+0000","last_peered":"2024-05-27T07:22:22.689696+0000","last_clean":"2024-05-27T07:22:22.689696+0000","last_became_active":"2024-05-27T07:17:02.336192+0000","last_became_peered":"2024-05-27T07:17:02.336192+0000","last_unstale":"2024-05-27T07:22:22.689696+0000","last_undegraded":"2024-05-27T07:22:22.689696+0000","last_fullsized":"2024-05-27T07:22:22.689696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:48.187517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368696+0000","last_change":"2024-05-27T07:17:06.759729+0000","last_active":"2024-05-27T07:22:22.368696+0000","last_peered":"2024-05-27T07:22:22.368696+0000","last_clean":"2024-05-27T07:22:22.368696+0000","last_became_active":"2024-05-27T07:17:06.756766+0000","last_became_peered":"2024-05-27T07:17:06.756766+0000","last_unstale":"2024-05-27T07:22:22.368696+0000","last_undegraded":"2024-05-27T07:22:22.368696+0000","last_fullsized":"2024-05-27T07:22:22.368696+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:01.090586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369921+0000","last_change":"2024-05-27T07:17:21.878487+0000","last_active":"2024-05-27T07:22:22.369921+0000","last_peered":"2024-05-27T07:22:22.369921+0000","last_clean":"2024-05-27T07:22:22.369921+0000","last_became_active":"2024-05-27T07:17:21.877623+0000","last_became_peered":"2024-05-27T07:17:21.877623+0000","last_unstale":"2024-05-27T07:22:22.369921+0000","last_undegraded":"2024-05-27T07:22:22.369921+0000","last_fullsized":"2024-05-27T07:22:22.369921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:28:52.618738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375223+0000","last_change":"2024-05-27T07:17:02.486495+0000","last_active":"2024-05-27T07:22:22.375223+0000","last_peered":"2024-05-27T07:22:22.375223+0000","last_clean":"2024-05-27T07:22:22.375223+0000","last_became_active":"2024-05-27T07:17:02.485493+0000","last_became_peered":"2024-05-27T07:17:02.485493+0000","last_unstale":"2024-05-27T07:22:22.375223+0000","last_undegraded":"2024-05-27T07:22:22.375223+0000","last_fullsized":"2024-05-27T07:22:22.375223+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:31.977210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595283+0000","last_change":"2024-05-27T07:17:06.760937+0000","last_active":"2024-05-27T07:22:22.595283+0000","last_peered":"2024-05-27T07:22:22.595283+0000","last_clean":"2024-05-27T07:22:22.595283+0000","last_became_active":"2024-05-27T07:17:06.760254+0000","last_became_peered":"2024-05-27T07:17:06.760254+0000","last_unstale":"2024-05-27T07:22:22.595283+0000","last_undegraded":"2024-05-27T07:22:22.595283+0000","last_fullsized":"2024-05-27T07:22:22.595283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:50.147319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373319+0000","last_change":"2024-05-27T07:17:21.901511+0000","last_active":"2024-05-27T07:22:22.373319+0000","last_peered":"2024-05-27T07:22:22.373319+0000","last_clean":"2024-05-27T07:22:22.373319+0000","last_became_active":"2024-05-27T07:17:21.899847+0000","last_became_peered":"2024-05-27T07:17:21.899847+0000","last_unstale":"2024-05-27T07:22:22.373319+0000","last_undegraded":"2024-05-27T07:22:22.373319+0000","last_fullsized":"2024-05-27T07:22:22.373319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:28.133509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370105+0000","last_change":"2024-05-27T07:17:02.432520+0000","last_active":"2024-05-27T07:22:22.370105+0000","last_peered":"2024-05-27T07:22:22.370105+0000","last_clean":"2024-05-27T07:22:22.370105+0000","last_became_active":"2024-05-27T07:17:02.431563+0000","last_became_peered":"2024-05-27T07:17:02.431563+0000","last_unstale":"2024-05-27T07:22:22.370105+0000","last_undegraded":"2024-05-27T07:22:22.370105+0000","last_fullsized":"2024-05-27T07:22:22.370105+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:17:28.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372653+0000","last_change":"2024-05-27T07:17:06.754408+0000","last_active":"2024-05-27T07:22:22.372653+0000","last_peered":"2024-05-27T07:22:22.372653+0000","last_clean":"2024-05-27T07:22:22.372653+0000","last_became_active":"2024-05-27T07:17:06.753787+0000","last_became_peered":"2024-05-27T07:17:06.753787+0000","last_unstale":"2024-05-27T07:22:22.372653+0000","last_undegraded":"2024-05-27T07:22:22.372653+0000","last_fullsized":"2024-05-27T07:22:22.372653+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:21:50.461296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369512+0000","last_change":"2024-05-27T07:17:21.895277+0000","last_active":"2024-05-27T07:22:22.369512+0000","last_peered":"2024-05-27T07:22:22.369512+0000","last_clean":"2024-05-27T07:22:22.369512+0000","last_became_active":"2024-05-27T07:17:21.894518+0000","last_became_peered":"2024-05-27T07:17:21.894518+0000","last_unstale":"2024-05-27T07:22:22.369512+0000","last_undegraded":"2024-05-27T07:22:22.369512+0000","last_fullsized":"2024-05-27T07:22:22.369512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:02.008209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367305+0000","last_change":"2024-05-27T07:17:02.343604+0000","last_active":"2024-05-27T07:22:22.367305+0000","last_peered":"2024-05-27T07:22:22.367305+0000","last_clean":"2024-05-27T07:22:22.367305+0000","last_became_active":"2024-05-27T07:17:02.341890+0000","last_became_peered":"2024-05-27T07:17:02.341890+0000","last_unstale":"2024-05-27T07:22:22.367305+0000","last_undegraded":"2024-05-27T07:22:22.367305+0000","last_fullsized":"2024-05-27T07:22:22.367305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:10:21.670193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375088+0000","last_change":"2024-05-27T07:17:06.754767+0000","last_active":"2024-05-27T07:22:22.375088+0000","last_peered":"2024-05-27T07:22:22.375088+0000","last_clean":"2024-05-27T07:22:22.375088+0000","last_became_active":"2024-05-27T07:17:06.754291+0000","last_became_peered":"2024-05-27T07:17:06.754291+0000","last_unstale":"2024-05-27T07:22:22.375088+0000","last_undegraded":"2024-05-27T07:22:22.375088+0000","last_fullsized":"2024-05-27T07:22:22.375088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:46:48.145874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377303+0000","last_change":"2024-05-27T07:17:21.892342+0000","last_active":"2024-05-27T07:22:22.377303+0000","last_peered":"2024-05-27T07:22:22.377303+0000","last_clean":"2024-05-27T07:22:22.377303+0000","last_became_active":"2024-05-27T07:17:21.891393+0000","last_became_peered":"2024-05-27T07:17:21.891393+0000","last_unstale":"2024-05-27T07:22:22.377303+0000","last_undegraded":"2024-05-27T07:22:22.377303+0000","last_fullsized":"2024-05-27T07:22:22.377303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:30.510616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370938+0000","last_change":"2024-05-27T07:17:02.340238+0000","last_active":"2024-05-27T07:22:22.370938+0000","last_peered":"2024-05-27T07:22:22.370938+0000","last_clean":"2024-05-27T07:22:22.370938+0000","last_became_active":"2024-05-27T07:17:02.339561+0000","last_became_peered":"2024-05-27T07:17:02.339561+0000","last_unstale":"2024-05-27T07:22:22.370938+0000","last_undegraded":"2024-05-27T07:22:22.370938+0000","last_fullsized":"2024-05-27T07:22:22.370938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:32.451397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368236+0000","last_change":"2024-05-27T07:17:06.769397+0000","last_active":"2024-05-27T07:22:22.368236+0000","last_peered":"2024-05-27T07:22:22.368236+0000","last_clean":"2024-05-27T07:22:22.368236+0000","last_became_active":"2024-05-27T07:17:06.768936+0000","last_became_peered":"2024-05-27T07:17:06.768936+0000","last_unstale":"2024-05-27T07:22:22.368236+0000","last_undegraded":"2024-05-27T07:22:22.368236+0000","last_fullsized":"2024-05-27T07:22:22.368236+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:52.767902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374048+0000","last_change":"2024-05-27T07:17:21.894225+0000","last_active":"2024-05-27T07:22:22.374048+0000","last_peered":"2024-05-27T07:22:22.374048+0000","last_clean":"2024-05-27T07:22:22.374048+0000","last_became_active":"2024-05-27T07:17:21.893499+0000","last_became_peered":"2024-05-27T07:17:21.893499+0000","last_unstale":"2024-05-27T07:22:22.374048+0000","last_undegraded":"2024-05-27T07:22:22.374048+0000","last_fullsized":"2024-05-27T07:22:22.374048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:54.985290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371433+0000","last_change":"2024-05-27T07:17:02.347736+0000","last_active":"2024-05-27T07:22:22.371433+0000","last_peered":"2024-05-27T07:22:22.371433+0000","last_clean":"2024-05-27T07:22:22.371433+0000","last_became_active":"2024-05-27T07:17:02.347237+0000","last_became_peered":"2024-05-27T07:17:02.347237+0000","last_unstale":"2024-05-27T07:22:22.371433+0000","last_undegraded":"2024-05-27T07:22:22.371433+0000","last_fullsized":"2024-05-27T07:22:22.371433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:36:15.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301779+0000","last_change":"2024-05-27T07:17:06.759348+0000","last_active":"2024-05-27T07:22:22.301779+0000","last_peered":"2024-05-27T07:22:22.301779+0000","last_clean":"2024-05-27T07:22:22.301779+0000","last_became_active":"2024-05-27T07:17:06.758967+0000","last_became_peered":"2024-05-27T07:17:06.758967+0000","last_unstale":"2024-05-27T07:22:22.301779+0000","last_undegraded":"2024-05-27T07:22:22.301779+0000","last_fullsized":"2024-05-27T07:22:22.301779+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:45.430719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371200+0000","last_change":"2024-05-27T07:17:21.882919+0000","last_active":"2024-05-27T07:22:22.371200+0000","last_peered":"2024-05-27T07:22:22.371200+0000","last_clean":"2024-05-27T07:22:22.371200+0000","last_became_active":"2024-05-27T07:17:21.882330+0000","last_became_peered":"2024-05-27T07:17:21.882330+0000","last_unstale":"2024-05-27T07:22:22.371200+0000","last_undegraded":"2024-05-27T07:22:22.371200+0000","last_fullsized":"2024-05-27T07:22:22.371200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:51.409079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373120+0000","last_change":"2024-05-27T07:17:02.486099+0000","last_active":"2024-05-27T07:22:22.373120+0000","last_peered":"2024-05-27T07:22:22.373120+0000","last_clean":"2024-05-27T07:22:22.373120+0000","last_became_active":"2024-05-27T07:17:02.485116+0000","last_became_peered":"2024-05-27T07:17:02.485116+0000","last_unstale":"2024-05-27T07:22:22.373120+0000","last_undegraded":"2024-05-27T07:22:22.373120+0000","last_fullsized":"2024-05-27T07:22:22.373120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:36.338287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303614+0000","last_change":"2024-05-27T07:17:06.773586+0000","last_active":"2024-05-27T07:22:22.303614+0000","last_peered":"2024-05-27T07:22:22.303614+0000","last_clean":"2024-05-27T07:22:22.303614+0000","last_became_active":"2024-05-27T07:17:06.773139+0000","last_became_peered":"2024-05-27T07:17:06.773139+0000","last_unstale":"2024-05-27T07:22:22.303614+0000","last_undegraded":"2024-05-27T07:22:22.303614+0000","last_fullsized":"2024-05-27T07:22:22.303614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:19:36.373091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.594249+0000","last_change":"2024-05-27T07:17:21.892351+0000","last_active":"2024-05-27T07:22:22.594249+0000","last_peered":"2024-05-27T07:22:22.594249+0000","last_clean":"2024-05-27T07:22:22.594249+0000","last_became_active":"2024-05-27T07:17:21.891862+0000","last_became_peered":"2024-05-27T07:17:21.891862+0000","last_unstale":"2024-05-27T07:22:22.594249+0000","last_undegraded":"2024-05-27T07:22:22.594249+0000","last_fullsized":"2024-05-27T07:22:22.594249+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:50:18.193149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.677139+0000","last_change":"2024-05-27T07:17:02.336917+0000","last_active":"2024-05-27T07:22:22.677139+0000","last_peered":"2024-05-27T07:22:22.677139+0000","last_clean":"2024-05-27T07:22:22.677139+0000","last_became_active":"2024-05-27T07:17:02.335836+0000","last_became_peered":"2024-05-27T07:17:02.335836+0000","last_unstale":"2024-05-27T07:22:22.677139+0000","last_undegraded":"2024-05-27T07:22:22.677139+0000","last_fullsized":"2024-05-27T07:22:22.677139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:54.109480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302465+0000","last_change":"2024-05-27T07:17:46.705826+0000","last_active":"2024-05-27T07:22:22.302465+0000","last_peered":"2024-05-27T07:22:22.302465+0000","last_clean":"2024-05-27T07:22:22.302465+0000","last_became_active":"2024-05-27T07:17:46.704749+0000","last_became_peered":"2024-05-27T07:17:46.704749+0000","last_unstale":"2024-05-27T07:22:22.302465+0000","last_undegraded":"2024-05-27T07:22:22.302465+0000","last_fullsized":"2024-05-27T07:22:22.302465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:26.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.299838+0000","last_change":"2024-05-27T07:17:21.885535+0000","last_active":"2024-05-27T07:22:22.299838+0000","last_peered":"2024-05-27T07:22:22.299838+0000","last_clean":"2024-05-27T07:22:22.299838+0000","last_became_active":"2024-05-27T07:17:21.884118+0000","last_became_peered":"2024-05-27T07:17:21.884118+0000","last_unstale":"2024-05-27T07:22:22.299838+0000","last_undegraded":"2024-05-27T07:22:22.299838+0000","last_fullsized":"2024-05-27T07:22:22.299838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:26.600199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297957+0000","last_change":"2024-05-27T07:17:02.340037+0000","last_active":"2024-05-27T07:22:22.297957+0000","last_peered":"2024-05-27T07:22:22.297957+0000","last_clean":"2024-05-27T07:22:22.297957+0000","last_became_active":"2024-05-27T07:17:02.339378+0000","last_became_peered":"2024-05-27T07:17:02.339378+0000","last_unstale":"2024-05-27T07:22:22.297957+0000","last_undegraded":"2024-05-27T07:22:22.297957+0000","last_fullsized":"2024-05-27T07:22:22.297957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:33.774917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.304323+0000","last_change":"2024-05-27T07:17:06.763277+0000","last_active":"2024-05-27T07:22:22.304323+0000","last_peered":"2024-05-27T07:22:22.304323+0000","last_clean":"2024-05-27T07:22:22.304323+0000","last_became_active":"2024-05-27T07:17:06.762837+0000","last_became_peered":"2024-05-27T07:17:06.762837+0000","last_unstale":"2024-05-27T07:22:22.304323+0000","last_undegraded":"2024-05-27T07:22:22.304323+0000","last_fullsized":"2024-05-27T07:22:22.304323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:52.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375646+0000","last_change":"2024-05-27T07:17:06.752556+0000","last_active":"2024-05-27T07:22:22.375646+0000","last_peered":"2024-05-27T07:22:22.375646+0000","last_clean":"2024-05-27T07:22:22.375646+0000","last_became_active":"2024-05-27T07:17:06.751085+0000","last_became_peered":"2024-05-27T07:17:06.751085+0000","last_unstale":"2024-05-27T07:22:22.375646+0000","last_undegraded":"2024-05-27T07:22:22.375646+0000","last_fullsized":"2024-05-27T07:22:22.375646+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:13:53.883845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370932+0000","last_change":"2024-05-27T07:17:06.753674+0000","last_active":"2024-05-27T07:22:22.370932+0000","last_peered":"2024-05-27T07:22:22.370932+0000","last_clean":"2024-05-27T07:22:22.370932+0000","last_became_active":"2024-05-27T07:17:06.752373+0000","last_became_peered":"2024-05-27T07:17:06.752373+0000","last_unstale":"2024-05-27T07:22:22.370932+0000","last_undegraded":"2024-05-27T07:22:22.370932+0000","last_fullsized":"2024-05-27T07:22:22.370932+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:40:17.474462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370771+0000","last_change":"2024-05-27T07:17:06.766796+0000","last_active":"2024-05-27T07:22:22.370771+0000","last_peered":"2024-05-27T07:22:22.370771+0000","last_clean":"2024-05-27T07:22:22.370771+0000","last_became_active":"2024-05-27T07:17:06.766435+0000","last_became_peered":"2024-05-27T07:17:06.766435+0000","last_unstale":"2024-05-27T07:22:22.370771+0000","last_undegraded":"2024-05-27T07:22:22.370771+0000","last_fullsized":"2024-05-27T07:22:22.370771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:38.747810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.594555+0000","last_change":"2024-05-27T07:17:06.762804+0000","last_active":"2024-05-27T07:22:22.594555+0000","last_peered":"2024-05-27T07:22:22.594555+0000","last_clean":"2024-05-27T07:22:22.594555+0000","last_became_active":"2024-05-27T07:17:06.762490+0000","last_became_peered":"2024-05-27T07:17:06.762490+0000","last_unstale":"2024-05-27T07:22:22.594555+0000","last_undegraded":"2024-05-27T07:22:22.594555+0000","last_fullsized":"2024-05-27T07:22:22.594555+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:59:33.584848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.298389+0000","last_change":"2024-05-27T07:17:06.765640+0000","last_active":"2024-05-27T07:22:22.298389+0000","last_peered":"2024-05-27T07:22:22.298389+0000","last_clean":"2024-05-27T07:22:22.298389+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:22:22.298389+0000","last_undegraded":"2024-05-27T07:22:22.298389+0000","last_fullsized":"2024-05-27T07:22:22.298389+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:19.695007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376739+0000","last_change":"2024-05-27T07:17:06.759982+0000","last_active":"2024-05-27T07:22:22.376739+0000","last_peered":"2024-05-27T07:22:22.376739+0000","last_clean":"2024-05-27T07:22:22.376739+0000","last_became_active":"2024-05-27T07:17:06.759367+0000","last_became_peered":"2024-05-27T07:17:06.759367+0000","last_unstale":"2024-05-27T07:22:22.376739+0000","last_undegraded":"2024-05-27T07:22:22.376739+0000","last_fullsized":"2024-05-27T07:22:22.376739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:35:04.686989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373962+0000","last_change":"2024-05-27T07:17:06.768264+0000","last_active":"2024-05-27T07:22:22.373962+0000","last_peered":"2024-05-27T07:22:22.373962+0000","last_clean":"2024-05-27T07:22:22.373962+0000","last_became_active":"2024-05-27T07:17:06.767952+0000","last_became_peered":"2024-05-27T07:17:06.767952+0000","last_unstale":"2024-05-27T07:22:22.373962+0000","last_undegraded":"2024-05-27T07:22:22.373962+0000","last_fullsized":"2024-05-27T07:22:22.373962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:34.867113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369176+0000","last_change":"2024-05-27T07:17:46.704705+0000","last_active":"2024-05-27T07:22:22.369176+0000","last_peered":"2024-05-27T07:22:22.369176+0000","last_clean":"2024-05-27T07:22:22.369176+0000","last_became_active":"2024-05-27T07:17:46.703503+0000","last_became_peered":"2024-05-27T07:17:46.703503+0000","last_unstale":"2024-05-27T07:22:22.369176+0000","last_undegraded":"2024-05-27T07:22:22.369176+0000","last_fullsized":"2024-05-27T07:22:22.369176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:39.975009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300202+0000","last_change":"2024-05-27T07:17:06.752080+0000","last_active":"2024-05-27T07:22:22.300202+0000","last_peered":"2024-05-27T07:22:22.300202+0000","last_clean":"2024-05-27T07:22:22.300202+0000","last_became_active":"2024-05-27T07:17:06.750486+0000","last_became_peered":"2024-05-27T07:17:06.750486+0000","last_unstale":"2024-05-27T07:22:22.300202+0000","last_undegraded":"2024-05-27T07:22:22.300202+0000","last_fullsized":"2024-05-27T07:22:22.300202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:12.461727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367380+0000","last_change":"2024-05-27T07:17:06.759200+0000","last_active":"2024-05-27T07:22:22.367380+0000","last_peered":"2024-05-27T07:22:22.367380+0000","last_clean":"2024-05-27T07:22:22.367380+0000","last_became_active":"2024-05-27T07:17:06.758819+0000","last_became_peered":"2024-05-27T07:17:06.758819+0000","last_unstale":"2024-05-27T07:22:22.367380+0000","last_undegraded":"2024-05-27T07:22:22.367380+0000","last_fullsized":"2024-05-27T07:22:22.367380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:00.274229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371730+0000","last_change":"2024-05-27T07:17:06.760412+0000","last_active":"2024-05-27T07:22:22.371730+0000","last_peered":"2024-05-27T07:22:22.371730+0000","last_clean":"2024-05-27T07:22:22.371730+0000","last_became_active":"2024-05-27T07:17:06.759949+0000","last_became_peered":"2024-05-27T07:17:06.759949+0000","last_unstale":"2024-05-27T07:22:22.371730+0000","last_undegraded":"2024-05-27T07:22:22.371730+0000","last_fullsized":"2024-05-27T07:22:22.371730+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.235488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297363+0000","last_change":"2024-05-27T07:17:06.775371+0000","last_active":"2024-05-27T07:22:22.297363+0000","last_peered":"2024-05-27T07:22:22.297363+0000","last_clean":"2024-05-27T07:22:22.297363+0000","last_became_active":"2024-05-27T07:17:06.774904+0000","last_became_peered":"2024-05-27T07:17:06.774904+0000","last_unstale":"2024-05-27T07:22:22.297363+0000","last_undegraded":"2024-05-27T07:22:22.297363+0000","last_fullsized":"2024-05-27T07:22:22.297363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:22:13.484733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376311+0000","last_change":"2024-05-27T07:17:06.757066+0000","last_active":"2024-05-27T07:22:22.376311+0000","last_peered":"2024-05-27T07:22:22.376311+0000","last_clean":"2024-05-27T07:22:22.376311+0000","last_became_active":"2024-05-27T07:17:06.756316+0000","last_became_peered":"2024-05-27T07:17:06.756316+0000","last_unstale":"2024-05-27T07:22:22.376311+0000","last_undegraded":"2024-05-27T07:22:22.376311+0000","last_fullsized":"2024-05-27T07:22:22.376311+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:15:12.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598777,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1472,"num_read_kb":847,"num_write":303,"num_write_kb":1448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1796331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710802,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":525814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907031,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33184,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711968,"statfs":{"total":95995035648,"available":95961055232,"internally_reserved":0,"allocated":1871872,"data_stored":1312141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070562,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1454080,"data_stored":922668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266792,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1445888,"data_stored":922476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463023,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1646592,"data_stored":1122653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626549,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":520532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":588352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":521920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":521170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182542,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":520726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986309,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":596155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822782,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28944,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716208,"statfs":{"total":95995035648,"available":95965396992,"internally_reserved":0,"allocated":1789952,"data_stored":1244911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:22:29.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:22:29.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:22:29.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-05-27T07:22:29.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:22:29.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:22:29.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-05-27T07:22:29.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 -- 172.21.15.12:0/2377713404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b700750b0 msgr2=0x7f6b700754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 --2- 172.21.15.12:0/2377713404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b700750b0 0x7f6b700754c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009900 tx=0x7f6b6002a410 comp rx=0 tx=0).stop 2024-05-27T07:22:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 -- 172.21.15.12:0/2377713404 shutdown_connections 2024-05-27T07:22:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 --2- 172.21.15.12:0/2377713404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7010b970 0x7f6b7010ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 --2- 172.21.15.12:0/2377713404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b70075a00 0x7f6b7010b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 --2- 172.21.15.12:0/2377713404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b700750b0 0x7f6b700754c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 -- 172.21.15.12:0/2377713404 >> 172.21.15.12:0/2377713404 conn(0x7f6b700febd0 msgr2=0x7f6b701010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:29.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 -- 172.21.15.12:0/2377713404 shutdown_connections 2024-05-27T07:22:29.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.411+0000 7f6b74a00640 1 -- 172.21.15.12:0/2377713404 wait complete. 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 Processor -- start 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 -- start start 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b700750b0 0x7f6b7019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b70075a00 0x7f6b7019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b7010b970 0x7f6b701a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b70111610 con 0x7f6b7010b970 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6b70111490 con 0x7f6b700750b0 2024-05-27T07:22:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.412+0000 7f6b74a00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b70111790 con 0x7f6b70075a00 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b7010b970 0x7f6b701a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b7010b970 0x7f6b701a3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56250/0 (socket says 172.21.15.12:56250) 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 -- 172.21.15.12:0/441648402 learned_addr learned my addr 172.21.15.12:0/441648402 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6f400640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b700750b0 0x7f6b7019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6ea00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b70075a00 0x7f6b7019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 -- 172.21.15.12:0/441648402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b70075a00 msgr2=0x7f6b7019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b70075a00 0x7f6b7019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 -- 172.21.15.12:0/441648402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b700750b0 msgr2=0x7f6b7019c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b700750b0 0x7f6b7019c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.413+0000 7f6b6fe00640 1 -- 172.21.15.12:0/441648402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b701a3df0 con 0x7f6b7010b970 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.414+0000 7f6b6fe00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b7010b970 0x7f6b701a3780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800e9d0 tx=0x7f6b6800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.414+0000 7f6b67e00640 1 -- 172.21.15.12:0/441648402 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800cd80 con 0x7f6b7010b970 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.414+0000 7f6b67e00640 1 -- 172.21.15.12:0/441648402 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b68005020 con 0x7f6b7010b970 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.414+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b701a40e0 con 0x7f6b7010b970 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.414+0000 7f6b67e00640 1 -- 172.21.15.12:0/441648402 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b680116d0 con 0x7f6b7010b970 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.414+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b701a4630 con 0x7f6b7010b970 2024-05-27T07:22:29.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.415+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b7010ab30 con 0x7f6b7010b970 2024-05-27T07:22:29.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.418+0000 7f6b67e00640 1 -- 172.21.15.12:0/441648402 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f6b68004a40 con 0x7f6b7010b970 2024-05-27T07:22:29.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.419+0000 7f6b67e00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6b540b2a60 0x7f6b540b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.419+0000 7f6b67e00640 1 -- 172.21.15.12:0/441648402 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f6b680e30c0 con 0x7f6b7010b970 2024-05-27T07:22:29.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.420+0000 7f6b6f400640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6b540b2a60 0x7f6b540b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.421+0000 7f6b6f400640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6b540b2a60 0x7f6b540b4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6b600057c0 tx=0x7f6b60005720 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:29.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.422+0000 7f6b67e00640 1 -- 172.21.15.12:0/441648402 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6b68016030 con 0x7f6b7010b970 2024-05-27T07:22:29.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.633+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6b70063100 con 0x7f6b540b2a60 2024-05-27T07:22:29.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.645+0000 7f6b67e00640 1 -- 172.21.15.12:0/441648402 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339472 (secure 0 0 0) 0x7f6b70063100 con 0x7f6b540b2a60 2024-05-27T07:22:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:22:29.641 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.651+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6b540b2a60 msgr2=0x7f6b540b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.651+0000 7f6b74a00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6b540b2a60 0x7f6b540b4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6b600057c0 tx=0x7f6b60005720 comp rx=0 tx=0).stop 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.651+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b7010b970 msgr2=0x7f6b701a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.651+0000 7f6b74a00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b7010b970 0x7f6b701a3780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800e9d0 tx=0x7f6b6800ee90 comp rx=0 tx=0).stop 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 shutdown_connections 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b7010b970 0x7f6b701a3780 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b70075a00 0x7f6b7019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6b540b2a60 0x7f6b540b4f70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 --2- 172.21.15.12:0/441648402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b700750b0 0x7f6b7019c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 >> 172.21.15.12:0/441648402 conn(0x7f6b700febd0 msgr2=0x7f6b700ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:29.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 shutdown_connections 2024-05-27T07:22:29.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.652+0000 7f6b74a00640 1 -- 172.21.15.12:0/441648402 wait complete. 2024-05-27T07:22:29.669 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:22:28.329802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189379,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1553,"num_read_kb":916,"num_write":428,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":832,"ondisk_log_size":832,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377364,"kb_used_data":15332,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124564460,"statfs":{"total":1151940427776,"available":1151554007040,"internally_reserved":0,"allocated":15699968,"data_stored":9319518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.974778"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377753+0000","last_change":"2024-05-27T07:17:06.763882+0000","last_active":"2024-05-27T07:22:22.377753+0000","last_peered":"2024-05-27T07:22:22.377753+0000","last_clean":"2024-05-27T07:22:22.377753+0000","last_became_active":"2024-05-27T07:17:06.759625+0000","last_became_peered":"2024-05-27T07:17:06.759625+0000","last_unstale":"2024-05-27T07:22:22.377753+0000","last_undegraded":"2024-05-27T07:22:22.377753+0000","last_fullsized":"2024-05-27T07:22:22.377753+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:24.105867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595140+0000","last_change":"2024-05-27T07:17:06.755737+0000","last_active":"2024-05-27T07:22:22.595140+0000","last_peered":"2024-05-27T07:22:22.595140+0000","last_clean":"2024-05-27T07:22:22.595140+0000","last_became_active":"2024-05-27T07:17:06.754588+0000","last_became_peered":"2024-05-27T07:17:06.754588+0000","last_unstale":"2024-05-27T07:22:22.595140+0000","last_undegraded":"2024-05-27T07:22:22.595140+0000","last_fullsized":"2024-05-27T07:22:22.595140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:02.417140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595016+0000","last_change":"2024-05-27T07:17:06.772057+0000","last_active":"2024-05-27T07:22:22.595016+0000","last_peered":"2024-05-27T07:22:22.595016+0000","last_clean":"2024-05-27T07:22:22.595016+0000","last_became_active":"2024-05-27T07:17:06.771636+0000","last_became_peered":"2024-05-27T07:17:06.771636+0000","last_unstale":"2024-05-27T07:22:22.595016+0000","last_undegraded":"2024-05-27T07:22:22.595016+0000","last_fullsized":"2024-05-27T07:22:22.595016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:45.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373701+0000","last_change":"2024-05-27T07:17:06.763599+0000","last_active":"2024-05-27T07:22:22.373701+0000","last_peered":"2024-05-27T07:22:22.373701+0000","last_clean":"2024-05-27T07:22:22.373701+0000","last_became_active":"2024-05-27T07:17:06.763125+0000","last_became_peered":"2024-05-27T07:17:06.763125+0000","last_unstale":"2024-05-27T07:22:22.373701+0000","last_undegraded":"2024-05-27T07:22:22.373701+0000","last_fullsized":"2024-05-27T07:22:22.373701+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:43.359673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372016+0000","last_change":"2024-05-27T07:17:06.755896+0000","last_active":"2024-05-27T07:22:22.372016+0000","last_peered":"2024-05-27T07:22:22.372016+0000","last_clean":"2024-05-27T07:22:22.372016+0000","last_became_active":"2024-05-27T07:17:06.755495+0000","last_became_peered":"2024-05-27T07:17:06.755495+0000","last_unstale":"2024-05-27T07:22:22.372016+0000","last_undegraded":"2024-05-27T07:22:22.372016+0000","last_fullsized":"2024-05-27T07:22:22.372016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:41:37.632012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377962+0000","last_change":"2024-05-27T07:17:46.699881+0000","last_active":"2024-05-27T07:22:22.377962+0000","last_peered":"2024-05-27T07:22:22.377962+0000","last_clean":"2024-05-27T07:22:22.377962+0000","last_became_active":"2024-05-27T07:17:46.699080+0000","last_became_peered":"2024-05-27T07:17:46.699080+0000","last_unstale":"2024-05-27T07:22:22.377962+0000","last_undegraded":"2024-05-27T07:22:22.377962+0000","last_fullsized":"2024-05-27T07:22:22.377962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:43.580762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300909+0000","last_change":"2024-05-27T07:17:06.770468+0000","last_active":"2024-05-27T07:22:22.300909+0000","last_peered":"2024-05-27T07:22:22.300909+0000","last_clean":"2024-05-27T07:22:22.300909+0000","last_became_active":"2024-05-27T07:17:06.768345+0000","last_became_peered":"2024-05-27T07:17:06.768345+0000","last_unstale":"2024-05-27T07:22:22.300909+0000","last_undegraded":"2024-05-27T07:22:22.300909+0000","last_fullsized":"2024-05-27T07:22:22.300909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:12:27.646654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366997+0000","last_change":"2024-05-27T07:17:06.758769+0000","last_active":"2024-05-27T07:22:22.366997+0000","last_peered":"2024-05-27T07:22:22.366997+0000","last_clean":"2024-05-27T07:22:22.366997+0000","last_became_active":"2024-05-27T07:17:06.758393+0000","last_became_peered":"2024-05-27T07:17:06.758393+0000","last_unstale":"2024-05-27T07:22:22.366997+0000","last_undegraded":"2024-05-27T07:22:22.366997+0000","last_fullsized":"2024-05-27T07:22:22.366997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:56.450569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377841+0000","last_change":"2024-05-27T07:17:06.770681+0000","last_active":"2024-05-27T07:22:22.377841+0000","last_peered":"2024-05-27T07:22:22.377841+0000","last_clean":"2024-05-27T07:22:22.377841+0000","last_became_active":"2024-05-27T07:17:06.766520+0000","last_became_peered":"2024-05-27T07:17:06.766520+0000","last_unstale":"2024-05-27T07:22:22.377841+0000","last_undegraded":"2024-05-27T07:22:22.377841+0000","last_fullsized":"2024-05-27T07:22:22.377841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:56.931743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376580+0000","last_change":"2024-05-27T07:17:06.761462+0000","last_active":"2024-05-27T07:22:22.376580+0000","last_peered":"2024-05-27T07:22:22.376580+0000","last_clean":"2024-05-27T07:22:22.376580+0000","last_became_active":"2024-05-27T07:17:06.759344+0000","last_became_peered":"2024-05-27T07:17:06.759344+0000","last_unstale":"2024-05-27T07:22:22.376580+0000","last_undegraded":"2024-05-27T07:22:22.376580+0000","last_fullsized":"2024-05-27T07:22:22.376580+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:12:22.688477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303430+0000","last_change":"2024-05-27T07:17:06.761947+0000","last_active":"2024-05-27T07:22:22.303430+0000","last_peered":"2024-05-27T07:22:22.303430+0000","last_clean":"2024-05-27T07:22:22.303430+0000","last_became_active":"2024-05-27T07:17:06.761464+0000","last_became_peered":"2024-05-27T07:17:06.761464+0000","last_unstale":"2024-05-27T07:22:22.303430+0000","last_undegraded":"2024-05-27T07:22:22.303430+0000","last_fullsized":"2024-05-27T07:22:22.303430+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:08:19.939311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374234+0000","last_change":"2024-05-27T07:17:06.772039+0000","last_active":"2024-05-27T07:22:22.374234+0000","last_peered":"2024-05-27T07:22:22.374234+0000","last_clean":"2024-05-27T07:22:22.374234+0000","last_became_active":"2024-05-27T07:17:06.771303+0000","last_became_peered":"2024-05-27T07:17:06.771303+0000","last_unstale":"2024-05-27T07:22:22.374234+0000","last_undegraded":"2024-05-27T07:22:22.374234+0000","last_fullsized":"2024-05-27T07:22:22.374234+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:30.254848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297550+0000","last_change":"2024-05-27T07:17:06.774228+0000","last_active":"2024-05-27T07:22:22.297550+0000","last_peered":"2024-05-27T07:22:22.297550+0000","last_clean":"2024-05-27T07:22:22.297550+0000","last_became_active":"2024-05-27T07:17:06.771436+0000","last_became_peered":"2024-05-27T07:17:06.771436+0000","last_unstale":"2024-05-27T07:22:22.297550+0000","last_undegraded":"2024-05-27T07:22:22.297550+0000","last_fullsized":"2024-05-27T07:22:22.297550+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:25.749143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.593630+0000","last_change":"2024-05-27T07:17:06.755558+0000","last_active":"2024-05-27T07:22:22.593630+0000","last_peered":"2024-05-27T07:22:22.593630+0000","last_clean":"2024-05-27T07:22:22.593630+0000","last_became_active":"2024-05-27T07:17:06.754858+0000","last_became_peered":"2024-05-27T07:17:06.754858+0000","last_unstale":"2024-05-27T07:22:22.593630+0000","last_undegraded":"2024-05-27T07:22:22.593630+0000","last_fullsized":"2024-05-27T07:22:22.593630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:09.853982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595599+0000","last_change":"2024-05-27T07:17:06.770731+0000","last_active":"2024-05-27T07:22:22.595599+0000","last_peered":"2024-05-27T07:22:22.595599+0000","last_clean":"2024-05-27T07:22:22.595599+0000","last_became_active":"2024-05-27T07:17:06.766923+0000","last_became_peered":"2024-05-27T07:17:06.766923+0000","last_unstale":"2024-05-27T07:22:22.595599+0000","last_undegraded":"2024-05-27T07:22:22.595599+0000","last_fullsized":"2024-05-27T07:22:22.595599+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.110121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369807+0000","last_change":"2024-05-27T07:17:06.758928+0000","last_active":"2024-05-27T07:22:22.369807+0000","last_peered":"2024-05-27T07:22:22.369807+0000","last_clean":"2024-05-27T07:22:22.369807+0000","last_became_active":"2024-05-27T07:17:06.758328+0000","last_became_peered":"2024-05-27T07:17:06.758328+0000","last_unstale":"2024-05-27T07:22:22.369807+0000","last_undegraded":"2024-05-27T07:22:22.369807+0000","last_fullsized":"2024-05-27T07:22:22.369807+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:36.284395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377323+0000","last_change":"2024-05-27T07:17:06.760006+0000","last_active":"2024-05-27T07:22:22.377323+0000","last_peered":"2024-05-27T07:22:22.377323+0000","last_clean":"2024-05-27T07:22:22.377323+0000","last_became_active":"2024-05-27T07:17:06.758177+0000","last_became_peered":"2024-05-27T07:17:06.758177+0000","last_unstale":"2024-05-27T07:22:22.377323+0000","last_undegraded":"2024-05-27T07:22:22.377323+0000","last_fullsized":"2024-05-27T07:22:22.377323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.516289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370659+0000","last_change":"2024-05-27T07:17:06.768484+0000","last_active":"2024-05-27T07:22:22.370659+0000","last_peered":"2024-05-27T07:22:22.370659+0000","last_clean":"2024-05-27T07:22:22.370659+0000","last_became_active":"2024-05-27T07:17:06.768105+0000","last_became_peered":"2024-05-27T07:17:06.768105+0000","last_unstale":"2024-05-27T07:22:22.370659+0000","last_undegraded":"2024-05-27T07:22:22.370659+0000","last_fullsized":"2024-05-27T07:22:22.370659+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:54:48.751678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373306+0000","last_change":"2024-05-27T07:17:06.762782+0000","last_active":"2024-05-27T07:22:22.373306+0000","last_peered":"2024-05-27T07:22:22.373306+0000","last_clean":"2024-05-27T07:22:22.373306+0000","last_became_active":"2024-05-27T07:17:06.760129+0000","last_became_peered":"2024-05-27T07:17:06.760129+0000","last_unstale":"2024-05-27T07:22:22.373306+0000","last_undegraded":"2024-05-27T07:22:22.373306+0000","last_fullsized":"2024-05-27T07:22:22.373306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:47:19.360662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370479+0000","last_change":"2024-05-27T07:17:21.899772+0000","last_active":"2024-05-27T07:22:22.370479+0000","last_peered":"2024-05-27T07:22:22.370479+0000","last_clean":"2024-05-27T07:22:22.370479+0000","last_became_active":"2024-05-27T07:17:21.899038+0000","last_became_peered":"2024-05-27T07:17:21.899038+0000","last_unstale":"2024-05-27T07:22:22.370479+0000","last_undegraded":"2024-05-27T07:22:22.370479+0000","last_fullsized":"2024-05-27T07:22:22.370479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:38.784433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369427+0000","last_change":"2024-05-27T07:17:46.708798+0000","last_active":"2024-05-27T07:22:22.369427+0000","last_peered":"2024-05-27T07:22:22.369427+0000","last_clean":"2024-05-27T07:22:22.369427+0000","last_became_active":"2024-05-27T07:17:46.696433+0000","last_became_peered":"2024-05-27T07:17:46.696433+0000","last_unstale":"2024-05-27T07:22:22.369427+0000","last_undegraded":"2024-05-27T07:22:22.369427+0000","last_fullsized":"2024-05-27T07:22:22.369427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:40.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368466+0000","last_change":"2024-05-27T07:17:06.760865+0000","last_active":"2024-05-27T07:22:22.368466+0000","last_peered":"2024-05-27T07:22:22.368466+0000","last_clean":"2024-05-27T07:22:22.368466+0000","last_became_active":"2024-05-27T07:17:06.759491+0000","last_became_peered":"2024-05-27T07:17:06.759491+0000","last_unstale":"2024-05-27T07:22:22.368466+0000","last_undegraded":"2024-05-27T07:22:22.368466+0000","last_fullsized":"2024-05-27T07:22:22.368466+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:31.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372496+0000","last_change":"2024-05-27T07:17:21.898583+0000","last_active":"2024-05-27T07:22:22.372496+0000","last_peered":"2024-05-27T07:22:22.372496+0000","last_clean":"2024-05-27T07:22:22.372496+0000","last_became_active":"2024-05-27T07:17:21.897937+0000","last_became_peered":"2024-05-27T07:17:21.897937+0000","last_unstale":"2024-05-27T07:22:22.372496+0000","last_undegraded":"2024-05-27T07:22:22.372496+0000","last_fullsized":"2024-05-27T07:22:22.372496+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:45:00.412914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368660+0000","last_change":"2024-05-27T07:17:02.432465+0000","last_active":"2024-05-27T07:22:22.368660+0000","last_peered":"2024-05-27T07:22:22.368660+0000","last_clean":"2024-05-27T07:22:22.368660+0000","last_became_active":"2024-05-27T07:17:02.431638+0000","last_became_peered":"2024-05-27T07:17:02.431638+0000","last_unstale":"2024-05-27T07:22:22.368660+0000","last_undegraded":"2024-05-27T07:22:22.368660+0000","last_fullsized":"2024-05-27T07:22:22.368660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:03:13.167952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595303+0000","last_change":"2024-05-27T07:17:06.763133+0000","last_active":"2024-05-27T07:22:22.595303+0000","last_peered":"2024-05-27T07:22:22.595303+0000","last_clean":"2024-05-27T07:22:22.595303+0000","last_became_active":"2024-05-27T07:17:06.762770+0000","last_became_peered":"2024-05-27T07:17:06.762770+0000","last_unstale":"2024-05-27T07:22:22.595303+0000","last_undegraded":"2024-05-27T07:22:22.595303+0000","last_fullsized":"2024-05-27T07:22:22.595303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:33.223722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373220+0000","last_change":"2024-05-27T07:17:21.900369+0000","last_active":"2024-05-27T07:22:22.373220+0000","last_peered":"2024-05-27T07:22:22.373220+0000","last_clean":"2024-05-27T07:22:22.373220+0000","last_became_active":"2024-05-27T07:17:21.899758+0000","last_became_peered":"2024-05-27T07:17:21.899758+0000","last_unstale":"2024-05-27T07:22:22.373220+0000","last_undegraded":"2024-05-27T07:22:22.373220+0000","last_fullsized":"2024-05-27T07:22:22.373220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:42.116302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372292+0000","last_change":"2024-05-27T07:17:02.347047+0000","last_active":"2024-05-27T07:22:22.372292+0000","last_peered":"2024-05-27T07:22:22.372292+0000","last_clean":"2024-05-27T07:22:22.372292+0000","last_became_active":"2024-05-27T07:17:02.346562+0000","last_became_peered":"2024-05-27T07:17:02.346562+0000","last_unstale":"2024-05-27T07:22:22.372292+0000","last_undegraded":"2024-05-27T07:22:22.372292+0000","last_fullsized":"2024-05-27T07:22:22.372292+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:25:50.133089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372097+0000","last_change":"2024-05-27T07:17:06.767930+0000","last_active":"2024-05-27T07:22:22.372097+0000","last_peered":"2024-05-27T07:22:22.372097+0000","last_clean":"2024-05-27T07:22:22.372097+0000","last_became_active":"2024-05-27T07:17:06.767439+0000","last_became_peered":"2024-05-27T07:17:06.767439+0000","last_unstale":"2024-05-27T07:22:22.372097+0000","last_undegraded":"2024-05-27T07:22:22.372097+0000","last_fullsized":"2024-05-27T07:22:22.372097+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:18:47.464602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372182+0000","last_change":"2024-05-27T07:17:21.889199+0000","last_active":"2024-05-27T07:22:22.372182+0000","last_peered":"2024-05-27T07:22:22.372182+0000","last_clean":"2024-05-27T07:22:22.372182+0000","last_became_active":"2024-05-27T07:17:21.888667+0000","last_became_peered":"2024-05-27T07:17:21.888667+0000","last_unstale":"2024-05-27T07:22:22.372182+0000","last_undegraded":"2024-05-27T07:22:22.372182+0000","last_fullsized":"2024-05-27T07:22:22.372182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.029820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371656+0000","last_change":"2024-05-27T07:17:02.339872+0000","last_active":"2024-05-27T07:22:22.371656+0000","last_peered":"2024-05-27T07:22:22.371656+0000","last_clean":"2024-05-27T07:22:22.371656+0000","last_became_active":"2024-05-27T07:17:02.338975+0000","last_became_peered":"2024-05-27T07:17:02.338975+0000","last_unstale":"2024-05-27T07:22:22.371656+0000","last_undegraded":"2024-05-27T07:22:22.371656+0000","last_fullsized":"2024-05-27T07:22:22.371656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:14.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301875+0000","last_change":"2024-05-27T07:17:06.758234+0000","last_active":"2024-05-27T07:22:22.301875+0000","last_peered":"2024-05-27T07:22:22.301875+0000","last_clean":"2024-05-27T07:22:22.301875+0000","last_became_active":"2024-05-27T07:17:06.757827+0000","last_became_peered":"2024-05-27T07:17:06.757827+0000","last_unstale":"2024-05-27T07:22:22.301875+0000","last_undegraded":"2024-05-27T07:22:22.301875+0000","last_fullsized":"2024-05-27T07:22:22.301875+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:21.688435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372775+0000","last_change":"2024-05-27T07:17:21.882540+0000","last_active":"2024-05-27T07:22:22.372775+0000","last_peered":"2024-05-27T07:22:22.372775+0000","last_clean":"2024-05-27T07:22:22.372775+0000","last_became_active":"2024-05-27T07:17:21.882087+0000","last_became_peered":"2024-05-27T07:17:21.882087+0000","last_unstale":"2024-05-27T07:22:22.372775+0000","last_undegraded":"2024-05-27T07:22:22.372775+0000","last_fullsized":"2024-05-27T07:22:22.372775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:53:04.363521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'9","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.712554+0000","last_change":"2024-05-27T07:17:46.704997+0000","last_active":"2024-05-27T07:22:22.712554+0000","last_peered":"2024-05-27T07:22:22.712554+0000","last_clean":"2024-05-27T07:22:22.712554+0000","last_became_active":"2024-05-27T07:17:46.704355+0000","last_became_peered":"2024-05-27T07:17:46.704355+0000","last_unstale":"2024-05-27T07:22:22.712554+0000","last_undegraded":"2024-05-27T07:22:22.712554+0000","last_fullsized":"2024-05-27T07:22:22.712554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:43:02.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":229,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372140+0000","last_change":"2024-05-27T07:17:06.765809+0000","last_active":"2024-05-27T07:22:22.372140+0000","last_peered":"2024-05-27T07:22:22.372140+0000","last_clean":"2024-05-27T07:22:22.372140+0000","last_became_active":"2024-05-27T07:17:06.763845+0000","last_became_peered":"2024-05-27T07:17:06.763845+0000","last_unstale":"2024-05-27T07:22:22.372140+0000","last_undegraded":"2024-05-27T07:22:22.372140+0000","last_fullsized":"2024-05-27T07:22:22.372140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:25.399181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372567+0000","last_change":"2024-05-27T07:17:21.886316+0000","last_active":"2024-05-27T07:22:22.372567+0000","last_peered":"2024-05-27T07:22:22.372567+0000","last_clean":"2024-05-27T07:22:22.372567+0000","last_became_active":"2024-05-27T07:17:21.885129+0000","last_became_peered":"2024-05-27T07:17:21.885129+0000","last_unstale":"2024-05-27T07:22:22.372567+0000","last_undegraded":"2024-05-27T07:22:22.372567+0000","last_fullsized":"2024-05-27T07:22:22.372567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:10.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377109+0000","last_change":"2024-05-27T07:17:02.338765+0000","last_active":"2024-05-27T07:22:22.377109+0000","last_peered":"2024-05-27T07:22:22.377109+0000","last_clean":"2024-05-27T07:22:22.377109+0000","last_became_active":"2024-05-27T07:17:02.337103+0000","last_became_peered":"2024-05-27T07:17:02.337103+0000","last_unstale":"2024-05-27T07:22:22.377109+0000","last_undegraded":"2024-05-27T07:22:22.377109+0000","last_fullsized":"2024-05-27T07:22:22.377109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:15:07.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301489+0000","last_change":"2024-05-27T07:17:06.766839+0000","last_active":"2024-05-27T07:22:22.301489+0000","last_peered":"2024-05-27T07:22:22.301489+0000","last_clean":"2024-05-27T07:22:22.301489+0000","last_became_active":"2024-05-27T07:17:06.766373+0000","last_became_peered":"2024-05-27T07:17:06.766373+0000","last_unstale":"2024-05-27T07:22:22.301489+0000","last_undegraded":"2024-05-27T07:22:22.301489+0000","last_fullsized":"2024-05-27T07:22:22.301489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:09:55.064045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376814+0000","last_change":"2024-05-27T07:18:47.133692+0000","last_active":"2024-05-27T07:22:22.376814+0000","last_peered":"2024-05-27T07:22:22.376814+0000","last_clean":"2024-05-27T07:22:22.376814+0000","last_became_active":"2024-05-27T07:18:47.132122+0000","last_became_peered":"2024-05-27T07:18:47.132122+0000","last_unstale":"2024-05-27T07:22:22.376814+0000","last_undegraded":"2024-05-27T07:22:22.376814+0000","last_fullsized":"2024-05-27T07:22:22.376814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:57.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296700+0000","last_change":"2024-05-27T07:17:02.329809+0000","last_active":"2024-05-27T07:22:22.296700+0000","last_peered":"2024-05-27T07:22:22.296700+0000","last_clean":"2024-05-27T07:22:22.296700+0000","last_became_active":"2024-05-27T07:17:02.328758+0000","last_became_peered":"2024-05-27T07:17:02.328758+0000","last_unstale":"2024-05-27T07:22:22.296700+0000","last_undegraded":"2024-05-27T07:22:22.296700+0000","last_fullsized":"2024-05-27T07:22:22.296700+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:52.479444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366880+0000","last_change":"2024-05-27T07:17:06.749836+0000","last_active":"2024-05-27T07:22:22.366880+0000","last_peered":"2024-05-27T07:22:22.366880+0000","last_clean":"2024-05-27T07:22:22.366880+0000","last_became_active":"2024-05-27T07:17:06.749512+0000","last_became_peered":"2024-05-27T07:17:06.749512+0000","last_unstale":"2024-05-27T07:22:22.366880+0000","last_undegraded":"2024-05-27T07:22:22.366880+0000","last_fullsized":"2024-05-27T07:22:22.366880+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:41:51.000415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595128+0000","last_change":"2024-05-27T07:17:21.888376+0000","last_active":"2024-05-27T07:22:22.595128+0000","last_peered":"2024-05-27T07:22:22.595128+0000","last_clean":"2024-05-27T07:22:22.595128+0000","last_became_active":"2024-05-27T07:17:21.886183+0000","last_became_peered":"2024-05-27T07:17:21.886183+0000","last_unstale":"2024-05-27T07:22:22.595128+0000","last_undegraded":"2024-05-27T07:22:22.595128+0000","last_fullsized":"2024-05-27T07:22:22.595128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:34.541338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300149+0000","last_change":"2024-05-27T07:17:02.334113+0000","last_active":"2024-05-27T07:22:22.300149+0000","last_peered":"2024-05-27T07:22:22.300149+0000","last_clean":"2024-05-27T07:22:22.300149+0000","last_became_active":"2024-05-27T07:17:02.333201+0000","last_became_peered":"2024-05-27T07:17:02.333201+0000","last_unstale":"2024-05-27T07:22:22.300149+0000","last_undegraded":"2024-05-27T07:22:22.300149+0000","last_fullsized":"2024-05-27T07:22:22.300149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:13.484654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369241+0000","last_change":"2024-05-27T07:17:06.751676+0000","last_active":"2024-05-27T07:22:22.369241+0000","last_peered":"2024-05-27T07:22:22.369241+0000","last_clean":"2024-05-27T07:22:22.369241+0000","last_became_active":"2024-05-27T07:17:06.751221+0000","last_became_peered":"2024-05-27T07:17:06.751221+0000","last_unstale":"2024-05-27T07:22:22.369241+0000","last_undegraded":"2024-05-27T07:22:22.369241+0000","last_fullsized":"2024-05-27T07:22:22.369241+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:40.012561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367255+0000","last_change":"2024-05-27T07:17:21.897640+0000","last_active":"2024-05-27T07:22:22.367255+0000","last_peered":"2024-05-27T07:22:22.367255+0000","last_clean":"2024-05-27T07:22:22.367255+0000","last_became_active":"2024-05-27T07:17:21.895355+0000","last_became_peered":"2024-05-27T07:17:21.895355+0000","last_unstale":"2024-05-27T07:22:22.367255+0000","last_undegraded":"2024-05-27T07:22:22.367255+0000","last_fullsized":"2024-05-27T07:22:22.367255+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:51:12.030069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'5","reported_seq":277,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375588+0000","last_change":"2024-05-27T07:17:02.339464+0000","last_active":"2024-05-27T07:22:22.375588+0000","last_peered":"2024-05-27T07:22:22.375588+0000","last_clean":"2024-05-27T07:22:22.375588+0000","last_became_active":"2024-05-27T07:17:02.338577+0000","last_became_peered":"2024-05-27T07:17:02.338577+0000","last_unstale":"2024-05-27T07:22:22.375588+0000","last_undegraded":"2024-05-27T07:22:22.375588+0000","last_fullsized":"2024-05-27T07:22:22.375588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:29:59.210157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376770+0000","last_change":"2024-05-27T07:17:06.766883+0000","last_active":"2024-05-27T07:22:22.376770+0000","last_peered":"2024-05-27T07:22:22.376770+0000","last_clean":"2024-05-27T07:22:22.376770+0000","last_became_active":"2024-05-27T07:17:06.766166+0000","last_became_peered":"2024-05-27T07:17:06.766166+0000","last_unstale":"2024-05-27T07:22:22.376770+0000","last_undegraded":"2024-05-27T07:22:22.376770+0000","last_fullsized":"2024-05-27T07:22:22.376770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:28.936499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369022+0000","last_change":"2024-05-27T07:17:21.893930+0000","last_active":"2024-05-27T07:22:22.369022+0000","last_peered":"2024-05-27T07:22:22.369022+0000","last_clean":"2024-05-27T07:22:22.369022+0000","last_became_active":"2024-05-27T07:17:21.893029+0000","last_became_peered":"2024-05-27T07:17:21.893029+0000","last_unstale":"2024-05-27T07:22:22.369022+0000","last_undegraded":"2024-05-27T07:22:22.369022+0000","last_fullsized":"2024-05-27T07:22:22.369022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:32.453078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303681+0000","last_change":"2024-05-27T07:17:02.346531+0000","last_active":"2024-05-27T07:22:22.303681+0000","last_peered":"2024-05-27T07:22:22.303681+0000","last_clean":"2024-05-27T07:22:22.303681+0000","last_became_active":"2024-05-27T07:17:02.345844+0000","last_became_peered":"2024-05-27T07:17:02.345844+0000","last_unstale":"2024-05-27T07:22:22.303681+0000","last_undegraded":"2024-05-27T07:22:22.303681+0000","last_fullsized":"2024-05-27T07:22:22.303681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:28.544616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595299+0000","last_change":"2024-05-27T07:17:06.769845+0000","last_active":"2024-05-27T07:22:22.595299+0000","last_peered":"2024-05-27T07:22:22.595299+0000","last_clean":"2024-05-27T07:22:22.595299+0000","last_became_active":"2024-05-27T07:17:06.768467+0000","last_became_peered":"2024-05-27T07:17:06.768467+0000","last_unstale":"2024-05-27T07:22:22.595299+0000","last_undegraded":"2024-05-27T07:22:22.595299+0000","last_fullsized":"2024-05-27T07:22:22.595299+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:30.311824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374196+0000","last_change":"2024-05-27T07:17:21.899885+0000","last_active":"2024-05-27T07:22:22.374196+0000","last_peered":"2024-05-27T07:22:22.374196+0000","last_clean":"2024-05-27T07:22:22.374196+0000","last_became_active":"2024-05-27T07:17:21.899340+0000","last_became_peered":"2024-05-27T07:17:21.899340+0000","last_unstale":"2024-05-27T07:22:22.374196+0000","last_undegraded":"2024-05-27T07:22:22.374196+0000","last_fullsized":"2024-05-27T07:22:22.374196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:12.665409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301032+0000","last_change":"2024-05-27T07:17:02.435770+0000","last_active":"2024-05-27T07:22:22.301032+0000","last_peered":"2024-05-27T07:22:22.301032+0000","last_clean":"2024-05-27T07:22:22.301032+0000","last_became_active":"2024-05-27T07:17:02.434535+0000","last_became_peered":"2024-05-27T07:17:02.434535+0000","last_unstale":"2024-05-27T07:22:22.301032+0000","last_undegraded":"2024-05-27T07:22:22.301032+0000","last_fullsized":"2024-05-27T07:22:22.301032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:53.745583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372886+0000","last_change":"2024-05-27T07:17:46.704100+0000","last_active":"2024-05-27T07:22:22.372886+0000","last_peered":"2024-05-27T07:22:22.372886+0000","last_clean":"2024-05-27T07:22:22.372886+0000","last_became_active":"2024-05-27T07:17:46.703475+0000","last_became_peered":"2024-05-27T07:17:46.703475+0000","last_unstale":"2024-05-27T07:22:22.372886+0000","last_undegraded":"2024-05-27T07:22:22.372886+0000","last_fullsized":"2024-05-27T07:22:22.372886+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:20:29.173923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377473+0000","last_change":"2024-05-27T07:18:47.131521+0000","last_active":"2024-05-27T07:22:22.377473+0000","last_peered":"2024-05-27T07:22:22.377473+0000","last_clean":"2024-05-27T07:22:22.377473+0000","last_became_active":"2024-05-27T07:18:47.129995+0000","last_became_peered":"2024-05-27T07:18:47.129995+0000","last_unstale":"2024-05-27T07:22:22.377473+0000","last_undegraded":"2024-05-27T07:22:22.377473+0000","last_fullsized":"2024-05-27T07:22:22.377473+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:45:27.268768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":565,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.933547+0000","last_change":"2024-05-27T07:17:46.719293+0000","last_active":"2024-05-27T07:22:22.933547+0000","last_peered":"2024-05-27T07:22:22.933547+0000","last_clean":"2024-05-27T07:22:22.933547+0000","last_became_active":"2024-05-27T07:17:46.703496+0000","last_became_peered":"2024-05-27T07:17:46.703496+0000","last_unstale":"2024-05-27T07:22:22.933547+0000","last_undegraded":"2024-05-27T07:22:22.933547+0000","last_fullsized":"2024-05-27T07:22:22.933547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:26.310524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67910,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373655+0000","last_change":"2024-05-27T07:17:06.763141+0000","last_active":"2024-05-27T07:22:22.373655+0000","last_peered":"2024-05-27T07:22:22.373655+0000","last_clean":"2024-05-27T07:22:22.373655+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:22:22.373655+0000","last_undegraded":"2024-05-27T07:22:22.373655+0000","last_fullsized":"2024-05-27T07:22:22.373655+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:47.072552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302824+0000","last_change":"2024-05-27T07:17:21.895342+0000","last_active":"2024-05-27T07:22:22.302824+0000","last_peered":"2024-05-27T07:22:22.302824+0000","last_clean":"2024-05-27T07:22:22.302824+0000","last_became_active":"2024-05-27T07:17:21.894630+0000","last_became_peered":"2024-05-27T07:17:21.894630+0000","last_unstale":"2024-05-27T07:22:22.302824+0000","last_undegraded":"2024-05-27T07:22:22.302824+0000","last_fullsized":"2024-05-27T07:22:22.302824+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:50.501683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371791+0000","last_change":"2024-05-27T07:17:02.341848+0000","last_active":"2024-05-27T07:22:22.371791+0000","last_peered":"2024-05-27T07:22:22.371791+0000","last_clean":"2024-05-27T07:22:22.371791+0000","last_became_active":"2024-05-27T07:17:02.339477+0000","last_became_peered":"2024-05-27T07:17:02.339477+0000","last_unstale":"2024-05-27T07:22:22.371791+0000","last_undegraded":"2024-05-27T07:22:22.371791+0000","last_fullsized":"2024-05-27T07:22:22.371791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:23.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297148+0000","last_change":"2024-05-27T07:17:06.761340+0000","last_active":"2024-05-27T07:22:22.297148+0000","last_peered":"2024-05-27T07:22:22.297148+0000","last_clean":"2024-05-27T07:22:22.297148+0000","last_became_active":"2024-05-27T07:17:06.759103+0000","last_became_peered":"2024-05-27T07:17:06.759103+0000","last_unstale":"2024-05-27T07:22:22.297148+0000","last_undegraded":"2024-05-27T07:22:22.297148+0000","last_fullsized":"2024-05-27T07:22:22.297148+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:42.263106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375699+0000","last_change":"2024-05-27T07:17:21.883252+0000","last_active":"2024-05-27T07:22:22.375699+0000","last_peered":"2024-05-27T07:22:22.375699+0000","last_clean":"2024-05-27T07:22:22.375699+0000","last_became_active":"2024-05-27T07:17:21.882468+0000","last_became_peered":"2024-05-27T07:17:21.882468+0000","last_unstale":"2024-05-27T07:22:22.375699+0000","last_undegraded":"2024-05-27T07:22:22.375699+0000","last_fullsized":"2024-05-27T07:22:22.375699+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:59.250357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301049+0000","last_change":"2024-05-27T07:17:02.343343+0000","last_active":"2024-05-27T07:22:22.301049+0000","last_peered":"2024-05-27T07:22:22.301049+0000","last_clean":"2024-05-27T07:22:22.301049+0000","last_became_active":"2024-05-27T07:17:02.342741+0000","last_became_peered":"2024-05-27T07:17:02.342741+0000","last_unstale":"2024-05-27T07:22:22.301049+0000","last_undegraded":"2024-05-27T07:22:22.301049+0000","last_fullsized":"2024-05-27T07:22:22.301049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:46.006865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301543+0000","last_change":"2024-05-27T07:17:06.771473+0000","last_active":"2024-05-27T07:22:22.301543+0000","last_peered":"2024-05-27T07:22:22.301543+0000","last_clean":"2024-05-27T07:22:22.301543+0000","last_became_active":"2024-05-27T07:17:06.771082+0000","last_became_peered":"2024-05-27T07:17:06.771082+0000","last_unstale":"2024-05-27T07:22:22.301543+0000","last_undegraded":"2024-05-27T07:22:22.301543+0000","last_fullsized":"2024-05-27T07:22:22.301543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:01.371182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369606+0000","last_change":"2024-05-27T07:17:21.897691+0000","last_active":"2024-05-27T07:22:22.369606+0000","last_peered":"2024-05-27T07:22:22.369606+0000","last_clean":"2024-05-27T07:22:22.369606+0000","last_became_active":"2024-05-27T07:17:21.895221+0000","last_became_peered":"2024-05-27T07:17:21.895221+0000","last_unstale":"2024-05-27T07:22:22.369606+0000","last_undegraded":"2024-05-27T07:22:22.369606+0000","last_fullsized":"2024-05-27T07:22:22.369606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:14.626947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":283,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371332+0000","last_change":"2024-05-27T07:17:02.343375+0000","last_active":"2024-05-27T07:22:22.371332+0000","last_peered":"2024-05-27T07:22:22.371332+0000","last_clean":"2024-05-27T07:22:22.371332+0000","last_became_active":"2024-05-27T07:17:02.341315+0000","last_became_peered":"2024-05-27T07:17:02.341315+0000","last_unstale":"2024-05-27T07:22:22.371332+0000","last_undegraded":"2024-05-27T07:22:22.371332+0000","last_fullsized":"2024-05-27T07:22:22.371332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:41.927265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301159+0000","last_change":"2024-05-27T07:17:06.765862+0000","last_active":"2024-05-27T07:22:22.301159+0000","last_peered":"2024-05-27T07:22:22.301159+0000","last_clean":"2024-05-27T07:22:22.301159+0000","last_became_active":"2024-05-27T07:17:06.765439+0000","last_became_peered":"2024-05-27T07:17:06.765439+0000","last_unstale":"2024-05-27T07:22:22.301159+0000","last_undegraded":"2024-05-27T07:22:22.301159+0000","last_fullsized":"2024-05-27T07:22:22.301159+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:47.586833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.298125+0000","last_change":"2024-05-27T07:17:21.891810+0000","last_active":"2024-05-27T07:22:22.298125+0000","last_peered":"2024-05-27T07:22:22.298125+0000","last_clean":"2024-05-27T07:22:22.298125+0000","last_became_active":"2024-05-27T07:17:21.890473+0000","last_became_peered":"2024-05-27T07:17:21.890473+0000","last_unstale":"2024-05-27T07:22:22.298125+0000","last_undegraded":"2024-05-27T07:22:22.298125+0000","last_fullsized":"2024-05-27T07:22:22.298125+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:12.731189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375448+0000","last_change":"2024-05-27T07:17:46.715870+0000","last_active":"2024-05-27T07:22:22.375448+0000","last_peered":"2024-05-27T07:22:22.375448+0000","last_clean":"2024-05-27T07:22:22.375448+0000","last_became_active":"2024-05-27T07:17:46.701248+0000","last_became_peered":"2024-05-27T07:17:46.701248+0000","last_unstale":"2024-05-27T07:22:22.375448+0000","last_undegraded":"2024-05-27T07:22:22.375448+0000","last_fullsized":"2024-05-27T07:22:22.375448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:18.693752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301040+0000","last_change":"2024-05-27T07:17:06.759937+0000","last_active":"2024-05-27T07:22:22.301040+0000","last_peered":"2024-05-27T07:22:22.301040+0000","last_clean":"2024-05-27T07:22:22.301040+0000","last_became_active":"2024-05-27T07:17:06.759526+0000","last_became_peered":"2024-05-27T07:17:06.759526+0000","last_unstale":"2024-05-27T07:22:22.301040+0000","last_undegraded":"2024-05-27T07:22:22.301040+0000","last_fullsized":"2024-05-27T07:22:22.301040+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:22.934317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300660+0000","last_change":"2024-05-27T07:17:21.898331+0000","last_active":"2024-05-27T07:22:22.300660+0000","last_peered":"2024-05-27T07:22:22.300660+0000","last_clean":"2024-05-27T07:22:22.300660+0000","last_became_active":"2024-05-27T07:17:21.897965+0000","last_became_peered":"2024-05-27T07:17:21.897965+0000","last_unstale":"2024-05-27T07:22:22.300660+0000","last_undegraded":"2024-05-27T07:22:22.300660+0000","last_fullsized":"2024-05-27T07:22:22.300660+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:20.584175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373702+0000","last_change":"2024-05-27T07:17:02.346029+0000","last_active":"2024-05-27T07:22:22.373702+0000","last_peered":"2024-05-27T07:22:22.373702+0000","last_clean":"2024-05-27T07:22:22.373702+0000","last_became_active":"2024-05-27T07:17:02.341292+0000","last_became_peered":"2024-05-27T07:17:02.341292+0000","last_unstale":"2024-05-27T07:22:22.373702+0000","last_undegraded":"2024-05-27T07:22:22.373702+0000","last_fullsized":"2024-05-27T07:22:22.373702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:31.705741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376883+0000","last_change":"2024-05-27T07:17:46.691785+0000","last_active":"2024-05-27T07:22:22.376883+0000","last_peered":"2024-05-27T07:22:22.376883+0000","last_clean":"2024-05-27T07:22:22.376883+0000","last_became_active":"2024-05-27T07:17:46.690445+0000","last_became_peered":"2024-05-27T07:17:46.690445+0000","last_unstale":"2024-05-27T07:22:22.376883+0000","last_undegraded":"2024-05-27T07:22:22.376883+0000","last_fullsized":"2024-05-27T07:22:22.376883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:59:27.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372885+0000","last_change":"2024-05-27T07:17:21.892434+0000","last_active":"2024-05-27T07:22:22.372885+0000","last_peered":"2024-05-27T07:22:22.372885+0000","last_clean":"2024-05-27T07:22:22.372885+0000","last_became_active":"2024-05-27T07:17:21.891736+0000","last_became_peered":"2024-05-27T07:17:21.891736+0000","last_unstale":"2024-05-27T07:22:22.372885+0000","last_undegraded":"2024-05-27T07:22:22.372885+0000","last_fullsized":"2024-05-27T07:22:22.372885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:48.458819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370417+0000","last_change":"2024-05-27T07:17:02.436323+0000","last_active":"2024-05-27T07:22:22.370417+0000","last_peered":"2024-05-27T07:22:22.370417+0000","last_clean":"2024-05-27T07:22:22.370417+0000","last_became_active":"2024-05-27T07:17:02.435000+0000","last_became_peered":"2024-05-27T07:17:02.435000+0000","last_unstale":"2024-05-27T07:22:22.370417+0000","last_undegraded":"2024-05-27T07:22:22.370417+0000","last_fullsized":"2024-05-27T07:22:22.370417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:08.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374831+0000","last_change":"2024-05-27T07:17:06.760418+0000","last_active":"2024-05-27T07:22:22.374831+0000","last_peered":"2024-05-27T07:22:22.374831+0000","last_clean":"2024-05-27T07:22:22.374831+0000","last_became_active":"2024-05-27T07:17:06.759875+0000","last_became_peered":"2024-05-27T07:17:06.759875+0000","last_unstale":"2024-05-27T07:22:22.374831+0000","last_undegraded":"2024-05-27T07:22:22.374831+0000","last_fullsized":"2024-05-27T07:22:22.374831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:12:54.324323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297331+0000","last_change":"2024-05-27T07:17:21.888048+0000","last_active":"2024-05-27T07:22:22.297331+0000","last_peered":"2024-05-27T07:22:22.297331+0000","last_clean":"2024-05-27T07:22:22.297331+0000","last_became_active":"2024-05-27T07:17:21.887579+0000","last_became_peered":"2024-05-27T07:17:21.887579+0000","last_unstale":"2024-05-27T07:22:22.297331+0000","last_undegraded":"2024-05-27T07:22:22.297331+0000","last_fullsized":"2024-05-27T07:22:22.297331+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:34:44.569959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301559+0000","last_change":"2024-05-27T07:17:02.337655+0000","last_active":"2024-05-27T07:22:22.301559+0000","last_peered":"2024-05-27T07:22:22.301559+0000","last_clean":"2024-05-27T07:22:22.301559+0000","last_became_active":"2024-05-27T07:17:02.337252+0000","last_became_peered":"2024-05-27T07:17:02.337252+0000","last_unstale":"2024-05-27T07:22:22.301559+0000","last_undegraded":"2024-05-27T07:22:22.301559+0000","last_fullsized":"2024-05-27T07:22:22.301559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:11:25.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376338+0000","last_change":"2024-05-27T07:17:06.762934+0000","last_active":"2024-05-27T07:22:22.376338+0000","last_peered":"2024-05-27T07:22:22.376338+0000","last_clean":"2024-05-27T07:22:22.376338+0000","last_became_active":"2024-05-27T07:17:06.762507+0000","last_became_peered":"2024-05-27T07:17:06.762507+0000","last_unstale":"2024-05-27T07:22:22.376338+0000","last_undegraded":"2024-05-27T07:22:22.376338+0000","last_fullsized":"2024-05-27T07:22:22.376338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:23:35.164892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296768+0000","last_change":"2024-05-27T07:17:21.886991+0000","last_active":"2024-05-27T07:22:22.296768+0000","last_peered":"2024-05-27T07:22:22.296768+0000","last_clean":"2024-05-27T07:22:22.296768+0000","last_became_active":"2024-05-27T07:17:21.885945+0000","last_became_peered":"2024-05-27T07:17:21.885945+0000","last_unstale":"2024-05-27T07:22:22.296768+0000","last_undegraded":"2024-05-27T07:22:22.296768+0000","last_fullsized":"2024-05-27T07:22:22.296768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:23:01.853601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300810+0000","last_change":"2024-05-27T07:17:46.864130+0000","last_active":"2024-05-27T07:22:22.300810+0000","last_peered":"2024-05-27T07:22:22.300810+0000","last_clean":"2024-05-27T07:22:22.300810+0000","last_became_active":"2024-05-27T07:17:46.862997+0000","last_became_peered":"2024-05-27T07:17:46.862997+0000","last_unstale":"2024-05-27T07:22:22.300810+0000","last_undegraded":"2024-05-27T07:22:22.300810+0000","last_fullsized":"2024-05-27T07:22:22.300810+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:10.594627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":278,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296813+0000","last_change":"2024-05-27T07:17:02.435463+0000","last_active":"2024-05-27T07:22:22.296813+0000","last_peered":"2024-05-27T07:22:22.296813+0000","last_clean":"2024-05-27T07:22:22.296813+0000","last_became_active":"2024-05-27T07:17:02.432131+0000","last_became_peered":"2024-05-27T07:17:02.432131+0000","last_unstale":"2024-05-27T07:22:22.296813+0000","last_undegraded":"2024-05-27T07:22:22.296813+0000","last_fullsized":"2024-05-27T07:22:22.296813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:41.874577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302768+0000","last_change":"2024-05-27T07:18:47.134043+0000","last_active":"2024-05-27T07:22:22.302768+0000","last_peered":"2024-05-27T07:22:22.302768+0000","last_clean":"2024-05-27T07:22:22.302768+0000","last_became_active":"2024-05-27T07:18:47.132050+0000","last_became_peered":"2024-05-27T07:18:47.132050+0000","last_unstale":"2024-05-27T07:22:22.302768+0000","last_undegraded":"2024-05-27T07:22:22.302768+0000","last_fullsized":"2024-05-27T07:22:22.302768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:12.170883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367929+0000","last_change":"2024-05-27T07:17:06.751168+0000","last_active":"2024-05-27T07:22:22.367929+0000","last_peered":"2024-05-27T07:22:22.367929+0000","last_clean":"2024-05-27T07:22:22.367929+0000","last_became_active":"2024-05-27T07:17:06.750816+0000","last_became_peered":"2024-05-27T07:17:06.750816+0000","last_unstale":"2024-05-27T07:22:22.367929+0000","last_undegraded":"2024-05-27T07:22:22.367929+0000","last_fullsized":"2024-05-27T07:22:22.367929+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:33:45.607426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373763+0000","last_change":"2024-05-27T07:17:02.337490+0000","last_active":"2024-05-27T07:22:22.373763+0000","last_peered":"2024-05-27T07:22:22.373763+0000","last_clean":"2024-05-27T07:22:22.373763+0000","last_became_active":"2024-05-27T07:17:02.335317+0000","last_became_peered":"2024-05-27T07:17:02.335317+0000","last_unstale":"2024-05-27T07:22:22.373763+0000","last_undegraded":"2024-05-27T07:22:22.373763+0000","last_fullsized":"2024-05-27T07:22:22.373763+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:33:46.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":123,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300719+0000","last_change":"2024-05-27T07:15:10.825312+0000","last_active":"2024-05-27T07:22:22.300719+0000","last_peered":"2024-05-27T07:22:22.300719+0000","last_clean":"2024-05-27T07:22:22.300719+0000","last_became_active":"2024-05-27T07:15:10.809245+0000","last_became_peered":"2024-05-27T07:15:10.809245+0000","last_unstale":"2024-05-27T07:22:22.300719+0000","last_undegraded":"2024-05-27T07:22:22.300719+0000","last_fullsized":"2024-05-27T07:22:22.300719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_clean_scrub_stamp":"2024-05-27T07:10:08.008565+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:32.769372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302265+0000","last_change":"2024-05-27T07:18:47.133948+0000","last_active":"2024-05-27T07:22:22.302265+0000","last_peered":"2024-05-27T07:22:22.302265+0000","last_clean":"2024-05-27T07:22:22.302265+0000","last_became_active":"2024-05-27T07:18:47.132566+0000","last_became_peered":"2024-05-27T07:18:47.132566+0000","last_unstale":"2024-05-27T07:22:22.302265+0000","last_undegraded":"2024-05-27T07:22:22.302265+0000","last_fullsized":"2024-05-27T07:22:22.302265+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:04.463121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296515+0000","last_change":"2024-05-27T07:17:06.766187+0000","last_active":"2024-05-27T07:22:22.296515+0000","last_peered":"2024-05-27T07:22:22.296515+0000","last_clean":"2024-05-27T07:22:22.296515+0000","last_became_active":"2024-05-27T07:17:06.763365+0000","last_became_peered":"2024-05-27T07:17:06.763365+0000","last_unstale":"2024-05-27T07:22:22.296515+0000","last_undegraded":"2024-05-27T07:22:22.296515+0000","last_fullsized":"2024-05-27T07:22:22.296515+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:13.668426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301091+0000","last_change":"2024-05-27T07:17:02.344452+0000","last_active":"2024-05-27T07:22:22.301091+0000","last_peered":"2024-05-27T07:22:22.301091+0000","last_clean":"2024-05-27T07:22:22.301091+0000","last_became_active":"2024-05-27T07:17:02.343885+0000","last_became_peered":"2024-05-27T07:17:02.343885+0000","last_unstale":"2024-05-27T07:22:22.301091+0000","last_undegraded":"2024-05-27T07:22:22.301091+0000","last_fullsized":"2024-05-27T07:22:22.301091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:26.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375242+0000","last_change":"2024-05-27T07:17:21.883024+0000","last_active":"2024-05-27T07:22:22.375242+0000","last_peered":"2024-05-27T07:22:22.375242+0000","last_clean":"2024-05-27T07:22:22.375242+0000","last_became_active":"2024-05-27T07:17:21.882429+0000","last_became_peered":"2024-05-27T07:17:21.882429+0000","last_unstale":"2024-05-27T07:22:22.375242+0000","last_undegraded":"2024-05-27T07:22:22.375242+0000","last_fullsized":"2024-05-27T07:22:22.375242+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:36:40.443537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373282+0000","last_change":"2024-05-27T07:17:02.432768+0000","last_active":"2024-05-27T07:22:22.373282+0000","last_peered":"2024-05-27T07:22:22.373282+0000","last_clean":"2024-05-27T07:22:22.373282+0000","last_became_active":"2024-05-27T07:17:02.431551+0000","last_became_peered":"2024-05-27T07:17:02.431551+0000","last_unstale":"2024-05-27T07:22:22.373282+0000","last_undegraded":"2024-05-27T07:22:22.373282+0000","last_fullsized":"2024-05-27T07:22:22.373282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:39.759512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297928+0000","last_change":"2024-05-27T07:17:06.770992+0000","last_active":"2024-05-27T07:22:22.297928+0000","last_peered":"2024-05-27T07:22:22.297928+0000","last_clean":"2024-05-27T07:22:22.297928+0000","last_became_active":"2024-05-27T07:17:06.770560+0000","last_became_peered":"2024-05-27T07:17:06.770560+0000","last_unstale":"2024-05-27T07:22:22.297928+0000","last_undegraded":"2024-05-27T07:22:22.297928+0000","last_fullsized":"2024-05-27T07:22:22.297928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:07:34.443551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366856+0000","last_change":"2024-05-27T07:17:21.895362+0000","last_active":"2024-05-27T07:22:22.366856+0000","last_peered":"2024-05-27T07:22:22.366856+0000","last_clean":"2024-05-27T07:22:22.366856+0000","last_became_active":"2024-05-27T07:17:21.894563+0000","last_became_peered":"2024-05-27T07:17:21.894563+0000","last_unstale":"2024-05-27T07:22:22.366856+0000","last_undegraded":"2024-05-27T07:22:22.366856+0000","last_fullsized":"2024-05-27T07:22:22.366856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:44.394301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'49","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.689696+0000","last_change":"2024-05-27T07:17:02.338203+0000","last_active":"2024-05-27T07:22:22.689696+0000","last_peered":"2024-05-27T07:22:22.689696+0000","last_clean":"2024-05-27T07:22:22.689696+0000","last_became_active":"2024-05-27T07:17:02.336192+0000","last_became_peered":"2024-05-27T07:17:02.336192+0000","last_unstale":"2024-05-27T07:22:22.689696+0000","last_undegraded":"2024-05-27T07:22:22.689696+0000","last_fullsized":"2024-05-27T07:22:22.689696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:48.187517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368696+0000","last_change":"2024-05-27T07:17:06.759729+0000","last_active":"2024-05-27T07:22:22.368696+0000","last_peered":"2024-05-27T07:22:22.368696+0000","last_clean":"2024-05-27T07:22:22.368696+0000","last_became_active":"2024-05-27T07:17:06.756766+0000","last_became_peered":"2024-05-27T07:17:06.756766+0000","last_unstale":"2024-05-27T07:22:22.368696+0000","last_undegraded":"2024-05-27T07:22:22.368696+0000","last_fullsized":"2024-05-27T07:22:22.368696+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:01.090586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369921+0000","last_change":"2024-05-27T07:17:21.878487+0000","last_active":"2024-05-27T07:22:22.369921+0000","last_peered":"2024-05-27T07:22:22.369921+0000","last_clean":"2024-05-27T07:22:22.369921+0000","last_became_active":"2024-05-27T07:17:21.877623+0000","last_became_peered":"2024-05-27T07:17:21.877623+0000","last_unstale":"2024-05-27T07:22:22.369921+0000","last_undegraded":"2024-05-27T07:22:22.369921+0000","last_fullsized":"2024-05-27T07:22:22.369921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:28:52.618738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375223+0000","last_change":"2024-05-27T07:17:02.486495+0000","last_active":"2024-05-27T07:22:22.375223+0000","last_peered":"2024-05-27T07:22:22.375223+0000","last_clean":"2024-05-27T07:22:22.375223+0000","last_became_active":"2024-05-27T07:17:02.485493+0000","last_became_peered":"2024-05-27T07:17:02.485493+0000","last_unstale":"2024-05-27T07:22:22.375223+0000","last_undegraded":"2024-05-27T07:22:22.375223+0000","last_fullsized":"2024-05-27T07:22:22.375223+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:31.977210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595283+0000","last_change":"2024-05-27T07:17:06.760937+0000","last_active":"2024-05-27T07:22:22.595283+0000","last_peered":"2024-05-27T07:22:22.595283+0000","last_clean":"2024-05-27T07:22:22.595283+0000","last_became_active":"2024-05-27T07:17:06.760254+0000","last_became_peered":"2024-05-27T07:17:06.760254+0000","last_unstale":"2024-05-27T07:22:22.595283+0000","last_undegraded":"2024-05-27T07:22:22.595283+0000","last_fullsized":"2024-05-27T07:22:22.595283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:50.147319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373319+0000","last_change":"2024-05-27T07:17:21.901511+0000","last_active":"2024-05-27T07:22:22.373319+0000","last_peered":"2024-05-27T07:22:22.373319+0000","last_clean":"2024-05-27T07:22:22.373319+0000","last_became_active":"2024-05-27T07:17:21.899847+0000","last_became_peered":"2024-05-27T07:17:21.899847+0000","last_unstale":"2024-05-27T07:22:22.373319+0000","last_undegraded":"2024-05-27T07:22:22.373319+0000","last_fullsized":"2024-05-27T07:22:22.373319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:28.133509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370105+0000","last_change":"2024-05-27T07:17:02.432520+0000","last_active":"2024-05-27T07:22:22.370105+0000","last_peered":"2024-05-27T07:22:22.370105+0000","last_clean":"2024-05-27T07:22:22.370105+0000","last_became_active":"2024-05-27T07:17:02.431563+0000","last_became_peered":"2024-05-27T07:17:02.431563+0000","last_unstale":"2024-05-27T07:22:22.370105+0000","last_undegraded":"2024-05-27T07:22:22.370105+0000","last_fullsized":"2024-05-27T07:22:22.370105+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:17:28.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372653+0000","last_change":"2024-05-27T07:17:06.754408+0000","last_active":"2024-05-27T07:22:22.372653+0000","last_peered":"2024-05-27T07:22:22.372653+0000","last_clean":"2024-05-27T07:22:22.372653+0000","last_became_active":"2024-05-27T07:17:06.753787+0000","last_became_peered":"2024-05-27T07:17:06.753787+0000","last_unstale":"2024-05-27T07:22:22.372653+0000","last_undegraded":"2024-05-27T07:22:22.372653+0000","last_fullsized":"2024-05-27T07:22:22.372653+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:21:50.461296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369512+0000","last_change":"2024-05-27T07:17:21.895277+0000","last_active":"2024-05-27T07:22:22.369512+0000","last_peered":"2024-05-27T07:22:22.369512+0000","last_clean":"2024-05-27T07:22:22.369512+0000","last_became_active":"2024-05-27T07:17:21.894518+0000","last_became_peered":"2024-05-27T07:17:21.894518+0000","last_unstale":"2024-05-27T07:22:22.369512+0000","last_undegraded":"2024-05-27T07:22:22.369512+0000","last_fullsized":"2024-05-27T07:22:22.369512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:02.008209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367305+0000","last_change":"2024-05-27T07:17:02.343604+0000","last_active":"2024-05-27T07:22:22.367305+0000","last_peered":"2024-05-27T07:22:22.367305+0000","last_clean":"2024-05-27T07:22:22.367305+0000","last_became_active":"2024-05-27T07:17:02.341890+0000","last_became_peered":"2024-05-27T07:17:02.341890+0000","last_unstale":"2024-05-27T07:22:22.367305+0000","last_undegraded":"2024-05-27T07:22:22.367305+0000","last_fullsized":"2024-05-27T07:22:22.367305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:10:21.670193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375088+0000","last_change":"2024-05-27T07:17:06.754767+0000","last_active":"2024-05-27T07:22:22.375088+0000","last_peered":"2024-05-27T07:22:22.375088+0000","last_clean":"2024-05-27T07:22:22.375088+0000","last_became_active":"2024-05-27T07:17:06.754291+0000","last_became_peered":"2024-05-27T07:17:06.754291+0000","last_unstale":"2024-05-27T07:22:22.375088+0000","last_undegraded":"2024-05-27T07:22:22.375088+0000","last_fullsized":"2024-05-27T07:22:22.375088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:46:48.145874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377303+0000","last_change":"2024-05-27T07:17:21.892342+0000","last_active":"2024-05-27T07:22:22.377303+0000","last_peered":"2024-05-27T07:22:22.377303+0000","last_clean":"2024-05-27T07:22:22.377303+0000","last_became_active":"2024-05-27T07:17:21.891393+0000","last_became_peered":"2024-05-27T07:17:21.891393+0000","last_unstale":"2024-05-27T07:22:22.377303+0000","last_undegraded":"2024-05-27T07:22:22.377303+0000","last_fullsized":"2024-05-27T07:22:22.377303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:30.510616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370938+0000","last_change":"2024-05-27T07:17:02.340238+0000","last_active":"2024-05-27T07:22:22.370938+0000","last_peered":"2024-05-27T07:22:22.370938+0000","last_clean":"2024-05-27T07:22:22.370938+0000","last_became_active":"2024-05-27T07:17:02.339561+0000","last_became_peered":"2024-05-27T07:17:02.339561+0000","last_unstale":"2024-05-27T07:22:22.370938+0000","last_undegraded":"2024-05-27T07:22:22.370938+0000","last_fullsized":"2024-05-27T07:22:22.370938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:32.451397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368236+0000","last_change":"2024-05-27T07:17:06.769397+0000","last_active":"2024-05-27T07:22:22.368236+0000","last_peered":"2024-05-27T07:22:22.368236+0000","last_clean":"2024-05-27T07:22:22.368236+0000","last_became_active":"2024-05-27T07:17:06.768936+0000","last_became_peered":"2024-05-27T07:17:06.768936+0000","last_unstale":"2024-05-27T07:22:22.368236+0000","last_undegraded":"2024-05-27T07:22:22.368236+0000","last_fullsized":"2024-05-27T07:22:22.368236+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:52.767902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374048+0000","last_change":"2024-05-27T07:17:21.894225+0000","last_active":"2024-05-27T07:22:22.374048+0000","last_peered":"2024-05-27T07:22:22.374048+0000","last_clean":"2024-05-27T07:22:22.374048+0000","last_became_active":"2024-05-27T07:17:21.893499+0000","last_became_peered":"2024-05-27T07:17:21.893499+0000","last_unstale":"2024-05-27T07:22:22.374048+0000","last_undegraded":"2024-05-27T07:22:22.374048+0000","last_fullsized":"2024-05-27T07:22:22.374048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:54.985290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371433+0000","last_change":"2024-05-27T07:17:02.347736+0000","last_active":"2024-05-27T07:22:22.371433+0000","last_peered":"2024-05-27T07:22:22.371433+0000","last_clean":"2024-05-27T07:22:22.371433+0000","last_became_active":"2024-05-27T07:17:02.347237+0000","last_became_peered":"2024-05-27T07:17:02.347237+0000","last_unstale":"2024-05-27T07:22:22.371433+0000","last_undegraded":"2024-05-27T07:22:22.371433+0000","last_fullsized":"2024-05-27T07:22:22.371433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:36:15.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301779+0000","last_change":"2024-05-27T07:17:06.759348+0000","last_active":"2024-05-27T07:22:22.301779+0000","last_peered":"2024-05-27T07:22:22.301779+0000","last_clean":"2024-05-27T07:22:22.301779+0000","last_became_active":"2024-05-27T07:17:06.758967+0000","last_became_peered":"2024-05-27T07:17:06.758967+0000","last_unstale":"2024-05-27T07:22:22.301779+0000","last_undegraded":"2024-05-27T07:22:22.301779+0000","last_fullsized":"2024-05-27T07:22:22.301779+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:45.430719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371200+0000","last_change":"2024-05-27T07:17:21.882919+0000","last_active":"2024-05-27T07:22:22.371200+0000","last_peered":"2024-05-27T07:22:22.371200+0000","last_clean":"2024-05-27T07:22:22.371200+0000","last_became_active":"2024-05-27T07:17:21.882330+0000","last_became_peered":"2024-05-27T07:17:21.882330+0000","last_unstale":"2024-05-27T07:22:22.371200+0000","last_undegraded":"2024-05-27T07:22:22.371200+0000","last_fullsized":"2024-05-27T07:22:22.371200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:51.409079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373120+0000","last_change":"2024-05-27T07:17:02.486099+0000","last_active":"2024-05-27T07:22:22.373120+0000","last_peered":"2024-05-27T07:22:22.373120+0000","last_clean":"2024-05-27T07:22:22.373120+0000","last_became_active":"2024-05-27T07:17:02.485116+0000","last_became_peered":"2024-05-27T07:17:02.485116+0000","last_unstale":"2024-05-27T07:22:22.373120+0000","last_undegraded":"2024-05-27T07:22:22.373120+0000","last_fullsized":"2024-05-27T07:22:22.373120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:36.338287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303614+0000","last_change":"2024-05-27T07:17:06.773586+0000","last_active":"2024-05-27T07:22:22.303614+0000","last_peered":"2024-05-27T07:22:22.303614+0000","last_clean":"2024-05-27T07:22:22.303614+0000","last_became_active":"2024-05-27T07:17:06.773139+0000","last_became_peered":"2024-05-27T07:17:06.773139+0000","last_unstale":"2024-05-27T07:22:22.303614+0000","last_undegraded":"2024-05-27T07:22:22.303614+0000","last_fullsized":"2024-05-27T07:22:22.303614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:19:36.373091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.594249+0000","last_change":"2024-05-27T07:17:21.892351+0000","last_active":"2024-05-27T07:22:22.594249+0000","last_peered":"2024-05-27T07:22:22.594249+0000","last_clean":"2024-05-27T07:22:22.594249+0000","last_became_active":"2024-05-27T07:17:21.891862+0000","last_became_peered":"2024-05-27T07:17:21.891862+0000","last_unstale":"2024-05-27T07:22:22.594249+0000","last_undegraded":"2024-05-27T07:22:22.594249+0000","last_fullsized":"2024-05-27T07:22:22.594249+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:50:18.193149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.677139+0000","last_change":"2024-05-27T07:17:02.336917+0000","last_active":"2024-05-27T07:22:22.677139+0000","last_peered":"2024-05-27T07:22:22.677139+0000","last_clean":"2024-05-27T07:22:22.677139+0000","last_became_active":"2024-05-27T07:17:02.335836+0000","last_became_peered":"2024-05-27T07:17:02.335836+0000","last_unstale":"2024-05-27T07:22:22.677139+0000","last_undegraded":"2024-05-27T07:22:22.677139+0000","last_fullsized":"2024-05-27T07:22:22.677139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:54.109480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302465+0000","last_change":"2024-05-27T07:17:46.705826+0000","last_active":"2024-05-27T07:22:22.302465+0000","last_peered":"2024-05-27T07:22:22.302465+0000","last_clean":"2024-05-27T07:22:22.302465+0000","last_became_active":"2024-05-27T07:17:46.704749+0000","last_became_peered":"2024-05-27T07:17:46.704749+0000","last_unstale":"2024-05-27T07:22:22.302465+0000","last_undegraded":"2024-05-27T07:22:22.302465+0000","last_fullsized":"2024-05-27T07:22:22.302465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:26.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.299838+0000","last_change":"2024-05-27T07:17:21.885535+0000","last_active":"2024-05-27T07:22:22.299838+0000","last_peered":"2024-05-27T07:22:22.299838+0000","last_clean":"2024-05-27T07:22:22.299838+0000","last_became_active":"2024-05-27T07:17:21.884118+0000","last_became_peered":"2024-05-27T07:17:21.884118+0000","last_unstale":"2024-05-27T07:22:22.299838+0000","last_undegraded":"2024-05-27T07:22:22.299838+0000","last_fullsized":"2024-05-27T07:22:22.299838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:26.600199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297957+0000","last_change":"2024-05-27T07:17:02.340037+0000","last_active":"2024-05-27T07:22:22.297957+0000","last_peered":"2024-05-27T07:22:22.297957+0000","last_clean":"2024-05-27T07:22:22.297957+0000","last_became_active":"2024-05-27T07:17:02.339378+0000","last_became_peered":"2024-05-27T07:17:02.339378+0000","last_unstale":"2024-05-27T07:22:22.297957+0000","last_undegraded":"2024-05-27T07:22:22.297957+0000","last_fullsized":"2024-05-27T07:22:22.297957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:33.774917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.304323+0000","last_change":"2024-05-27T07:17:06.763277+0000","last_active":"2024-05-27T07:22:22.304323+0000","last_peered":"2024-05-27T07:22:22.304323+0000","last_clean":"2024-05-27T07:22:22.304323+0000","last_became_active":"2024-05-27T07:17:06.762837+0000","last_became_peered":"2024-05-27T07:17:06.762837+0000","last_unstale":"2024-05-27T07:22:22.304323+0000","last_undegraded":"2024-05-27T07:22:22.304323+0000","last_fullsized":"2024-05-27T07:22:22.304323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:52.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375646+0000","last_change":"2024-05-27T07:17:06.752556+0000","last_active":"2024-05-27T07:22:22.375646+0000","last_peered":"2024-05-27T07:22:22.375646+0000","last_clean":"2024-05-27T07:22:22.375646+0000","last_became_active":"2024-05-27T07:17:06.751085+0000","last_became_peered":"2024-05-27T07:17:06.751085+0000","last_unstale":"2024-05-27T07:22:22.375646+0000","last_undegraded":"2024-05-27T07:22:22.375646+0000","last_fullsized":"2024-05-27T07:22:22.375646+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:13:53.883845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370932+0000","last_change":"2024-05-27T07:17:06.753674+0000","last_active":"2024-05-27T07:22:22.370932+0000","last_peered":"2024-05-27T07:22:22.370932+0000","last_clean":"2024-05-27T07:22:22.370932+0000","last_became_active":"2024-05-27T07:17:06.752373+0000","last_became_peered":"2024-05-27T07:17:06.752373+0000","last_unstale":"2024-05-27T07:22:22.370932+0000","last_undegraded":"2024-05-27T07:22:22.370932+0000","last_fullsized":"2024-05-27T07:22:22.370932+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:40:17.474462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370771+0000","last_change":"2024-05-27T07:17:06.766796+0000","last_active":"2024-05-27T07:22:22.370771+0000","last_peered":"2024-05-27T07:22:22.370771+0000","last_clean":"2024-05-27T07:22:22.370771+0000","last_became_active":"2024-05-27T07:17:06.766435+0000","last_became_peered":"2024-05-27T07:17:06.766435+0000","last_unstale":"2024-05-27T07:22:22.370771+0000","last_undegraded":"2024-05-27T07:22:22.370771+0000","last_fullsized":"2024-05-27T07:22:22.370771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:38.747810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.594555+0000","last_change":"2024-05-27T07:17:06.762804+0000","last_active":"2024-05-27T07:22:22.594555+0000","last_peered":"2024-05-27T07:22:22.594555+0000","last_clean":"2024-05-27T07:22:22.594555+0000","last_became_active":"2024-05-27T07:17:06.762490+0000","last_became_peered":"2024-05-27T07:17:06.762490+0000","last_unstale":"2024-05-27T07:22:22.594555+0000","last_undegraded":"2024-05-27T07:22:22.594555+0000","last_fullsized":"2024-05-27T07:22:22.594555+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:59:33.584848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.298389+0000","last_change":"2024-05-27T07:17:06.765640+0000","last_active":"2024-05-27T07:22:22.298389+0000","last_peered":"2024-05-27T07:22:22.298389+0000","last_clean":"2024-05-27T07:22:22.298389+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:22:22.298389+0000","last_undegraded":"2024-05-27T07:22:22.298389+0000","last_fullsized":"2024-05-27T07:22:22.298389+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:19.695007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376739+0000","last_change":"2024-05-27T07:17:06.759982+0000","last_active":"2024-05-27T07:22:22.376739+0000","last_peered":"2024-05-27T07:22:22.376739+0000","last_clean":"2024-05-27T07:22:22.376739+0000","last_became_active":"2024-05-27T07:17:06.759367+0000","last_became_peered":"2024-05-27T07:17:06.759367+0000","last_unstale":"2024-05-27T07:22:22.376739+0000","last_undegraded":"2024-05-27T07:22:22.376739+0000","last_fullsized":"2024-05-27T07:22:22.376739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:35:04.686989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373962+0000","last_change":"2024-05-27T07:17:06.768264+0000","last_active":"2024-05-27T07:22:22.373962+0000","last_peered":"2024-05-27T07:22:22.373962+0000","last_clean":"2024-05-27T07:22:22.373962+0000","last_became_active":"2024-05-27T07:17:06.767952+0000","last_became_peered":"2024-05-27T07:17:06.767952+0000","last_unstale":"2024-05-27T07:22:22.373962+0000","last_undegraded":"2024-05-27T07:22:22.373962+0000","last_fullsized":"2024-05-27T07:22:22.373962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:34.867113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369176+0000","last_change":"2024-05-27T07:17:46.704705+0000","last_active":"2024-05-27T07:22:22.369176+0000","last_peered":"2024-05-27T07:22:22.369176+0000","last_clean":"2024-05-27T07:22:22.369176+0000","last_became_active":"2024-05-27T07:17:46.703503+0000","last_became_peered":"2024-05-27T07:17:46.703503+0000","last_unstale":"2024-05-27T07:22:22.369176+0000","last_undegraded":"2024-05-27T07:22:22.369176+0000","last_fullsized":"2024-05-27T07:22:22.369176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:39.975009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300202+0000","last_change":"2024-05-27T07:17:06.752080+0000","last_active":"2024-05-27T07:22:22.300202+0000","last_peered":"2024-05-27T07:22:22.300202+0000","last_clean":"2024-05-27T07:22:22.300202+0000","last_became_active":"2024-05-27T07:17:06.750486+0000","last_became_peered":"2024-05-27T07:17:06.750486+0000","last_unstale":"2024-05-27T07:22:22.300202+0000","last_undegraded":"2024-05-27T07:22:22.300202+0000","last_fullsized":"2024-05-27T07:22:22.300202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:12.461727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367380+0000","last_change":"2024-05-27T07:17:06.759200+0000","last_active":"2024-05-27T07:22:22.367380+0000","last_peered":"2024-05-27T07:22:22.367380+0000","last_clean":"2024-05-27T07:22:22.367380+0000","last_became_active":"2024-05-27T07:17:06.758819+0000","last_became_peered":"2024-05-27T07:17:06.758819+0000","last_unstale":"2024-05-27T07:22:22.367380+0000","last_undegraded":"2024-05-27T07:22:22.367380+0000","last_fullsized":"2024-05-27T07:22:22.367380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:00.274229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371730+0000","last_change":"2024-05-27T07:17:06.760412+0000","last_active":"2024-05-27T07:22:22.371730+0000","last_peered":"2024-05-27T07:22:22.371730+0000","last_clean":"2024-05-27T07:22:22.371730+0000","last_became_active":"2024-05-27T07:17:06.759949+0000","last_became_peered":"2024-05-27T07:17:06.759949+0000","last_unstale":"2024-05-27T07:22:22.371730+0000","last_undegraded":"2024-05-27T07:22:22.371730+0000","last_fullsized":"2024-05-27T07:22:22.371730+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.235488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297363+0000","last_change":"2024-05-27T07:17:06.775371+0000","last_active":"2024-05-27T07:22:22.297363+0000","last_peered":"2024-05-27T07:22:22.297363+0000","last_clean":"2024-05-27T07:22:22.297363+0000","last_became_active":"2024-05-27T07:17:06.774904+0000","last_became_peered":"2024-05-27T07:17:06.774904+0000","last_unstale":"2024-05-27T07:22:22.297363+0000","last_undegraded":"2024-05-27T07:22:22.297363+0000","last_fullsized":"2024-05-27T07:22:22.297363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:22:13.484733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376311+0000","last_change":"2024-05-27T07:17:06.757066+0000","last_active":"2024-05-27T07:22:22.376311+0000","last_peered":"2024-05-27T07:22:22.376311+0000","last_clean":"2024-05-27T07:22:22.376311+0000","last_became_active":"2024-05-27T07:17:06.756316+0000","last_became_peered":"2024-05-27T07:17:06.756316+0000","last_unstale":"2024-05-27T07:22:22.376311+0000","last_undegraded":"2024-05-27T07:22:22.376311+0000","last_fullsized":"2024-05-27T07:22:22.376311+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:15:12.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598777,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1472,"num_read_kb":847,"num_write":303,"num_write_kb":1448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1796331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710802,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":525814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907031,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33184,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711968,"statfs":{"total":95995035648,"available":95961055232,"internally_reserved":0,"allocated":1871872,"data_stored":1312141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070562,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1454080,"data_stored":922668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266792,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1445888,"data_stored":922476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463023,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1646592,"data_stored":1122653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626549,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":520532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":588352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":521920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":521170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182542,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":520726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986309,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":596155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822782,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28944,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716208,"statfs":{"total":95995035648,"available":95965396992,"internally_reserved":0,"allocated":1789952,"data_stored":1244911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:22:29.674 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-27T07:22:29.674 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:22:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.993+0000 7f26bd600640 1 -- 172.21.15.12:0/1512102537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 msgr2=0x7f26b8104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.993+0000 7f26bd600640 1 --2- 172.21.15.12:0/1512102537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b8104150 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009900 tx=0x7f26a002a410 comp rx=0 tx=0).stop 2024-05-27T07:22:29.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.993+0000 7f26bd600640 1 -- 172.21.15.12:0/1512102537 shutdown_connections 2024-05-27T07:22:29.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.993+0000 7f26bd600640 1 --2- 172.21.15.12:0/1512102537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26b8108a40 0x7f26b810ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.993+0000 7f26bd600640 1 --2- 172.21.15.12:0/1512102537 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b8104150 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.993+0000 7f26bd600640 1 --2- 172.21.15.12:0/1512102537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26b8102b00 0x7f26b8103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.993+0000 7f26bd600640 1 -- 172.21.15.12:0/1512102537 >> 172.21.15.12:0/1512102537 conn(0x7f26b80fdcf0 msgr2=0x7f26b81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:29.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.994+0000 7f26bd600640 1 -- 172.21.15.12:0/1512102537 shutdown_connections 2024-05-27T07:22:29.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.994+0000 7f26bd600640 1 -- 172.21.15.12:0/1512102537 wait complete. 2024-05-27T07:22:29.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.996+0000 7f26bd600640 1 Processor -- start 2024-05-27T07:22:29.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.996+0000 7f26bd600640 1 -- start start 2024-05-27T07:22:29.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.996+0000 7f26bd600640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26b8102b00 0x7f26b819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.996+0000 7f26bd600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26bd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26b8108a40 0x7f26b81a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26bd600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f26b810f5e0 con 0x7f26b8103d40 2024-05-27T07:22:29.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26bd600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f26b810f460 con 0x7f26b8108a40 2024-05-27T07:22:29.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26bd600640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26b810f760 con 0x7f26b8102b00 2024-05-27T07:22:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26afe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56272/0 (socket says 172.21.15.12:56272) 2024-05-27T07:22:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26afe00640 1 -- 172.21.15.12:0/275264930 learned_addr learned my addr 172.21.15.12:0/275264930 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26b6a00640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26b8102b00 0x7f26b819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.997+0000 7f26b7400640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26b8108a40 0x7f26b81a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26afe00640 1 -- 172.21.15.12:0/275264930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26b8102b00 msgr2=0x7f26b819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26afe00640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26b8102b00 0x7f26b819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26afe00640 1 -- 172.21.15.12:0/275264930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26b8108a40 msgr2=0x7f26b81a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26afe00640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26b8108a40 0x7f26b81a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26afe00640 1 -- 172.21.15.12:0/275264930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b81a1d70 con 0x7f26b8103d40 2024-05-27T07:22:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26afe00640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b819a770 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009230 tx=0x7f26a00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26af400640 1 -- 172.21.15.12:0/275264930 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a003f030 con 0x7f26b8103d40 2024-05-27T07:22:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26af400640 1 -- 172.21.15.12:0/275264930 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f26a0003c90 con 0x7f26b8103d40 2024-05-27T07:22:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.998+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b81a2000 con 0x7f26b8103d40 2024-05-27T07:22:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.999+0000 7f26af400640 1 -- 172.21.15.12:0/275264930 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a0035070 con 0x7f26b8103d40 2024-05-27T07:22:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:29.999+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b81a24f0 con 0x7f26b8103d40 2024-05-27T07:22:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.000+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26b8107850 con 0x7f26b8103d40 2024-05-27T07:22:29.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:22:29.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:22:29.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:29.994 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.994 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:29.994 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.994 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.994 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.994 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:29.994 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:29.995 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.003+0000 7f26af400640 1 -- 172.21.15.12:0/275264930 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f26a0042030 con 0x7f26b8103d40 2024-05-27T07:22:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.005+0000 7f26af400640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f269c0b29f0 0x7f269c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.005+0000 7f26af400640 1 -- 172.21.15.12:0/275264930 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f26a01030e0 con 0x7f26b8103d40 2024-05-27T07:22:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.005+0000 7f26b6a00640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f269c0b29f0 0x7f269c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.006+0000 7f26b6a00640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f269c0b29f0 0x7f269c0b4f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26a8005fd0 tx=0x7f26a800b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:30.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.012+0000 7f26af400640 1 -- 172.21.15.12:0/275264930 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f26a00c9f50 con 0x7f26b8103d40 2024-05-27T07:22:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.270+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f26b8063100 con 0x7f269c0b29f0 2024-05-27T07:22:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.287+0000 7f26af400640 1 -- 172.21.15.12:0/275264930 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339472 (secure 0 0 0) 0x7f26b8063100 con 0x7f269c0b29f0 2024-05-27T07:22:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:22:30.284 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-05-27T07:22:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f269c0b29f0 msgr2=0x7f269c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f269c0b29f0 0x7f269c0b4f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26a8005fd0 tx=0x7f26a800b410 comp rx=0 tx=0).stop 2024-05-27T07:22:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 msgr2=0x7f26b819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b819a770 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009230 tx=0x7f26a00042f0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 shutdown_connections 2024-05-27T07:22:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f269c0b29f0 0x7f269c0b4f00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26b8108a40 0x7f26b81a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26b8103d40 0x7f26b819a770 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 --2- 172.21.15.12:0/275264930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26b8102b00 0x7f26b819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 >> 172.21.15.12:0/275264930 conn(0x7f26b80fdcf0 msgr2=0x7f26b8104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 shutdown_connections 2024-05-27T07:22:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:30.295+0000 7f26bd600640 1 -- 172.21.15.12:0/275264930 wait complete. 2024-05-27T07:22:30.315 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:22:28.329802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189379,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1553,"num_read_kb":916,"num_write":428,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":832,"ondisk_log_size":832,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377364,"kb_used_data":15332,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124564460,"statfs":{"total":1151940427776,"available":1151554007040,"internally_reserved":0,"allocated":15699968,"data_stored":9319518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.974778"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377753+0000","last_change":"2024-05-27T07:17:06.763882+0000","last_active":"2024-05-27T07:22:22.377753+0000","last_peered":"2024-05-27T07:22:22.377753+0000","last_clean":"2024-05-27T07:22:22.377753+0000","last_became_active":"2024-05-27T07:17:06.759625+0000","last_became_peered":"2024-05-27T07:17:06.759625+0000","last_unstale":"2024-05-27T07:22:22.377753+0000","last_undegraded":"2024-05-27T07:22:22.377753+0000","last_fullsized":"2024-05-27T07:22:22.377753+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:24.105867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595140+0000","last_change":"2024-05-27T07:17:06.755737+0000","last_active":"2024-05-27T07:22:22.595140+0000","last_peered":"2024-05-27T07:22:22.595140+0000","last_clean":"2024-05-27T07:22:22.595140+0000","last_became_active":"2024-05-27T07:17:06.754588+0000","last_became_peered":"2024-05-27T07:17:06.754588+0000","last_unstale":"2024-05-27T07:22:22.595140+0000","last_undegraded":"2024-05-27T07:22:22.595140+0000","last_fullsized":"2024-05-27T07:22:22.595140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:02.417140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595016+0000","last_change":"2024-05-27T07:17:06.772057+0000","last_active":"2024-05-27T07:22:22.595016+0000","last_peered":"2024-05-27T07:22:22.595016+0000","last_clean":"2024-05-27T07:22:22.595016+0000","last_became_active":"2024-05-27T07:17:06.771636+0000","last_became_peered":"2024-05-27T07:17:06.771636+0000","last_unstale":"2024-05-27T07:22:22.595016+0000","last_undegraded":"2024-05-27T07:22:22.595016+0000","last_fullsized":"2024-05-27T07:22:22.595016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:45.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373701+0000","last_change":"2024-05-27T07:17:06.763599+0000","last_active":"2024-05-27T07:22:22.373701+0000","last_peered":"2024-05-27T07:22:22.373701+0000","last_clean":"2024-05-27T07:22:22.373701+0000","last_became_active":"2024-05-27T07:17:06.763125+0000","last_became_peered":"2024-05-27T07:17:06.763125+0000","last_unstale":"2024-05-27T07:22:22.373701+0000","last_undegraded":"2024-05-27T07:22:22.373701+0000","last_fullsized":"2024-05-27T07:22:22.373701+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:43.359673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372016+0000","last_change":"2024-05-27T07:17:06.755896+0000","last_active":"2024-05-27T07:22:22.372016+0000","last_peered":"2024-05-27T07:22:22.372016+0000","last_clean":"2024-05-27T07:22:22.372016+0000","last_became_active":"2024-05-27T07:17:06.755495+0000","last_became_peered":"2024-05-27T07:17:06.755495+0000","last_unstale":"2024-05-27T07:22:22.372016+0000","last_undegraded":"2024-05-27T07:22:22.372016+0000","last_fullsized":"2024-05-27T07:22:22.372016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:41:37.632012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377962+0000","last_change":"2024-05-27T07:17:46.699881+0000","last_active":"2024-05-27T07:22:22.377962+0000","last_peered":"2024-05-27T07:22:22.377962+0000","last_clean":"2024-05-27T07:22:22.377962+0000","last_became_active":"2024-05-27T07:17:46.699080+0000","last_became_peered":"2024-05-27T07:17:46.699080+0000","last_unstale":"2024-05-27T07:22:22.377962+0000","last_undegraded":"2024-05-27T07:22:22.377962+0000","last_fullsized":"2024-05-27T07:22:22.377962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:43.580762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300909+0000","last_change":"2024-05-27T07:17:06.770468+0000","last_active":"2024-05-27T07:22:22.300909+0000","last_peered":"2024-05-27T07:22:22.300909+0000","last_clean":"2024-05-27T07:22:22.300909+0000","last_became_active":"2024-05-27T07:17:06.768345+0000","last_became_peered":"2024-05-27T07:17:06.768345+0000","last_unstale":"2024-05-27T07:22:22.300909+0000","last_undegraded":"2024-05-27T07:22:22.300909+0000","last_fullsized":"2024-05-27T07:22:22.300909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:12:27.646654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366997+0000","last_change":"2024-05-27T07:17:06.758769+0000","last_active":"2024-05-27T07:22:22.366997+0000","last_peered":"2024-05-27T07:22:22.366997+0000","last_clean":"2024-05-27T07:22:22.366997+0000","last_became_active":"2024-05-27T07:17:06.758393+0000","last_became_peered":"2024-05-27T07:17:06.758393+0000","last_unstale":"2024-05-27T07:22:22.366997+0000","last_undegraded":"2024-05-27T07:22:22.366997+0000","last_fullsized":"2024-05-27T07:22:22.366997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:56.450569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377841+0000","last_change":"2024-05-27T07:17:06.770681+0000","last_active":"2024-05-27T07:22:22.377841+0000","last_peered":"2024-05-27T07:22:22.377841+0000","last_clean":"2024-05-27T07:22:22.377841+0000","last_became_active":"2024-05-27T07:17:06.766520+0000","last_became_peered":"2024-05-27T07:17:06.766520+0000","last_unstale":"2024-05-27T07:22:22.377841+0000","last_undegraded":"2024-05-27T07:22:22.377841+0000","last_fullsized":"2024-05-27T07:22:22.377841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:56.931743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376580+0000","last_change":"2024-05-27T07:17:06.761462+0000","last_active":"2024-05-27T07:22:22.376580+0000","last_peered":"2024-05-27T07:22:22.376580+0000","last_clean":"2024-05-27T07:22:22.376580+0000","last_became_active":"2024-05-27T07:17:06.759344+0000","last_became_peered":"2024-05-27T07:17:06.759344+0000","last_unstale":"2024-05-27T07:22:22.376580+0000","last_undegraded":"2024-05-27T07:22:22.376580+0000","last_fullsized":"2024-05-27T07:22:22.376580+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:12:22.688477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303430+0000","last_change":"2024-05-27T07:17:06.761947+0000","last_active":"2024-05-27T07:22:22.303430+0000","last_peered":"2024-05-27T07:22:22.303430+0000","last_clean":"2024-05-27T07:22:22.303430+0000","last_became_active":"2024-05-27T07:17:06.761464+0000","last_became_peered":"2024-05-27T07:17:06.761464+0000","last_unstale":"2024-05-27T07:22:22.303430+0000","last_undegraded":"2024-05-27T07:22:22.303430+0000","last_fullsized":"2024-05-27T07:22:22.303430+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:08:19.939311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374234+0000","last_change":"2024-05-27T07:17:06.772039+0000","last_active":"2024-05-27T07:22:22.374234+0000","last_peered":"2024-05-27T07:22:22.374234+0000","last_clean":"2024-05-27T07:22:22.374234+0000","last_became_active":"2024-05-27T07:17:06.771303+0000","last_became_peered":"2024-05-27T07:17:06.771303+0000","last_unstale":"2024-05-27T07:22:22.374234+0000","last_undegraded":"2024-05-27T07:22:22.374234+0000","last_fullsized":"2024-05-27T07:22:22.374234+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:13:30.254848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297550+0000","last_change":"2024-05-27T07:17:06.774228+0000","last_active":"2024-05-27T07:22:22.297550+0000","last_peered":"2024-05-27T07:22:22.297550+0000","last_clean":"2024-05-27T07:22:22.297550+0000","last_became_active":"2024-05-27T07:17:06.771436+0000","last_became_peered":"2024-05-27T07:17:06.771436+0000","last_unstale":"2024-05-27T07:22:22.297550+0000","last_undegraded":"2024-05-27T07:22:22.297550+0000","last_fullsized":"2024-05-27T07:22:22.297550+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:58:25.749143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.593630+0000","last_change":"2024-05-27T07:17:06.755558+0000","last_active":"2024-05-27T07:22:22.593630+0000","last_peered":"2024-05-27T07:22:22.593630+0000","last_clean":"2024-05-27T07:22:22.593630+0000","last_became_active":"2024-05-27T07:17:06.754858+0000","last_became_peered":"2024-05-27T07:17:06.754858+0000","last_unstale":"2024-05-27T07:22:22.593630+0000","last_undegraded":"2024-05-27T07:22:22.593630+0000","last_fullsized":"2024-05-27T07:22:22.593630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:09.853982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595599+0000","last_change":"2024-05-27T07:17:06.770731+0000","last_active":"2024-05-27T07:22:22.595599+0000","last_peered":"2024-05-27T07:22:22.595599+0000","last_clean":"2024-05-27T07:22:22.595599+0000","last_became_active":"2024-05-27T07:17:06.766923+0000","last_became_peered":"2024-05-27T07:17:06.766923+0000","last_unstale":"2024-05-27T07:22:22.595599+0000","last_undegraded":"2024-05-27T07:22:22.595599+0000","last_fullsized":"2024-05-27T07:22:22.595599+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:04:13.110121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369807+0000","last_change":"2024-05-27T07:17:06.758928+0000","last_active":"2024-05-27T07:22:22.369807+0000","last_peered":"2024-05-27T07:22:22.369807+0000","last_clean":"2024-05-27T07:22:22.369807+0000","last_became_active":"2024-05-27T07:17:06.758328+0000","last_became_peered":"2024-05-27T07:17:06.758328+0000","last_unstale":"2024-05-27T07:22:22.369807+0000","last_undegraded":"2024-05-27T07:22:22.369807+0000","last_fullsized":"2024-05-27T07:22:22.369807+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:36.284395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377323+0000","last_change":"2024-05-27T07:17:06.760006+0000","last_active":"2024-05-27T07:22:22.377323+0000","last_peered":"2024-05-27T07:22:22.377323+0000","last_clean":"2024-05-27T07:22:22.377323+0000","last_became_active":"2024-05-27T07:17:06.758177+0000","last_became_peered":"2024-05-27T07:17:06.758177+0000","last_unstale":"2024-05-27T07:22:22.377323+0000","last_undegraded":"2024-05-27T07:22:22.377323+0000","last_fullsized":"2024-05-27T07:22:22.377323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.516289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370659+0000","last_change":"2024-05-27T07:17:06.768484+0000","last_active":"2024-05-27T07:22:22.370659+0000","last_peered":"2024-05-27T07:22:22.370659+0000","last_clean":"2024-05-27T07:22:22.370659+0000","last_became_active":"2024-05-27T07:17:06.768105+0000","last_became_peered":"2024-05-27T07:17:06.768105+0000","last_unstale":"2024-05-27T07:22:22.370659+0000","last_undegraded":"2024-05-27T07:22:22.370659+0000","last_fullsized":"2024-05-27T07:22:22.370659+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:54:48.751678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373306+0000","last_change":"2024-05-27T07:17:06.762782+0000","last_active":"2024-05-27T07:22:22.373306+0000","last_peered":"2024-05-27T07:22:22.373306+0000","last_clean":"2024-05-27T07:22:22.373306+0000","last_became_active":"2024-05-27T07:17:06.760129+0000","last_became_peered":"2024-05-27T07:17:06.760129+0000","last_unstale":"2024-05-27T07:22:22.373306+0000","last_undegraded":"2024-05-27T07:22:22.373306+0000","last_fullsized":"2024-05-27T07:22:22.373306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:47:19.360662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370479+0000","last_change":"2024-05-27T07:17:21.899772+0000","last_active":"2024-05-27T07:22:22.370479+0000","last_peered":"2024-05-27T07:22:22.370479+0000","last_clean":"2024-05-27T07:22:22.370479+0000","last_became_active":"2024-05-27T07:17:21.899038+0000","last_became_peered":"2024-05-27T07:17:21.899038+0000","last_unstale":"2024-05-27T07:22:22.370479+0000","last_undegraded":"2024-05-27T07:22:22.370479+0000","last_fullsized":"2024-05-27T07:22:22.370479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:14:38.784433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369427+0000","last_change":"2024-05-27T07:17:46.708798+0000","last_active":"2024-05-27T07:22:22.369427+0000","last_peered":"2024-05-27T07:22:22.369427+0000","last_clean":"2024-05-27T07:22:22.369427+0000","last_became_active":"2024-05-27T07:17:46.696433+0000","last_became_peered":"2024-05-27T07:17:46.696433+0000","last_unstale":"2024-05-27T07:22:22.369427+0000","last_undegraded":"2024-05-27T07:22:22.369427+0000","last_fullsized":"2024-05-27T07:22:22.369427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:01:40.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368466+0000","last_change":"2024-05-27T07:17:06.760865+0000","last_active":"2024-05-27T07:22:22.368466+0000","last_peered":"2024-05-27T07:22:22.368466+0000","last_clean":"2024-05-27T07:22:22.368466+0000","last_became_active":"2024-05-27T07:17:06.759491+0000","last_became_peered":"2024-05-27T07:17:06.759491+0000","last_unstale":"2024-05-27T07:22:22.368466+0000","last_undegraded":"2024-05-27T07:22:22.368466+0000","last_fullsized":"2024-05-27T07:22:22.368466+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:31.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372496+0000","last_change":"2024-05-27T07:17:21.898583+0000","last_active":"2024-05-27T07:22:22.372496+0000","last_peered":"2024-05-27T07:22:22.372496+0000","last_clean":"2024-05-27T07:22:22.372496+0000","last_became_active":"2024-05-27T07:17:21.897937+0000","last_became_peered":"2024-05-27T07:17:21.897937+0000","last_unstale":"2024-05-27T07:22:22.372496+0000","last_undegraded":"2024-05-27T07:22:22.372496+0000","last_fullsized":"2024-05-27T07:22:22.372496+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:45:00.412914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368660+0000","last_change":"2024-05-27T07:17:02.432465+0000","last_active":"2024-05-27T07:22:22.368660+0000","last_peered":"2024-05-27T07:22:22.368660+0000","last_clean":"2024-05-27T07:22:22.368660+0000","last_became_active":"2024-05-27T07:17:02.431638+0000","last_became_peered":"2024-05-27T07:17:02.431638+0000","last_unstale":"2024-05-27T07:22:22.368660+0000","last_undegraded":"2024-05-27T07:22:22.368660+0000","last_fullsized":"2024-05-27T07:22:22.368660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:03:13.167952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595303+0000","last_change":"2024-05-27T07:17:06.763133+0000","last_active":"2024-05-27T07:22:22.595303+0000","last_peered":"2024-05-27T07:22:22.595303+0000","last_clean":"2024-05-27T07:22:22.595303+0000","last_became_active":"2024-05-27T07:17:06.762770+0000","last_became_peered":"2024-05-27T07:17:06.762770+0000","last_unstale":"2024-05-27T07:22:22.595303+0000","last_undegraded":"2024-05-27T07:22:22.595303+0000","last_fullsized":"2024-05-27T07:22:22.595303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:33.223722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373220+0000","last_change":"2024-05-27T07:17:21.900369+0000","last_active":"2024-05-27T07:22:22.373220+0000","last_peered":"2024-05-27T07:22:22.373220+0000","last_clean":"2024-05-27T07:22:22.373220+0000","last_became_active":"2024-05-27T07:17:21.899758+0000","last_became_peered":"2024-05-27T07:17:21.899758+0000","last_unstale":"2024-05-27T07:22:22.373220+0000","last_undegraded":"2024-05-27T07:22:22.373220+0000","last_fullsized":"2024-05-27T07:22:22.373220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:21:42.116302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372292+0000","last_change":"2024-05-27T07:17:02.347047+0000","last_active":"2024-05-27T07:22:22.372292+0000","last_peered":"2024-05-27T07:22:22.372292+0000","last_clean":"2024-05-27T07:22:22.372292+0000","last_became_active":"2024-05-27T07:17:02.346562+0000","last_became_peered":"2024-05-27T07:17:02.346562+0000","last_unstale":"2024-05-27T07:22:22.372292+0000","last_undegraded":"2024-05-27T07:22:22.372292+0000","last_fullsized":"2024-05-27T07:22:22.372292+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:25:50.133089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372097+0000","last_change":"2024-05-27T07:17:06.767930+0000","last_active":"2024-05-27T07:22:22.372097+0000","last_peered":"2024-05-27T07:22:22.372097+0000","last_clean":"2024-05-27T07:22:22.372097+0000","last_became_active":"2024-05-27T07:17:06.767439+0000","last_became_peered":"2024-05-27T07:17:06.767439+0000","last_unstale":"2024-05-27T07:22:22.372097+0000","last_undegraded":"2024-05-27T07:22:22.372097+0000","last_fullsized":"2024-05-27T07:22:22.372097+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:18:47.464602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372182+0000","last_change":"2024-05-27T07:17:21.889199+0000","last_active":"2024-05-27T07:22:22.372182+0000","last_peered":"2024-05-27T07:22:22.372182+0000","last_clean":"2024-05-27T07:22:22.372182+0000","last_became_active":"2024-05-27T07:17:21.888667+0000","last_became_peered":"2024-05-27T07:17:21.888667+0000","last_unstale":"2024-05-27T07:22:22.372182+0000","last_undegraded":"2024-05-27T07:22:22.372182+0000","last_fullsized":"2024-05-27T07:22:22.372182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.029820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371656+0000","last_change":"2024-05-27T07:17:02.339872+0000","last_active":"2024-05-27T07:22:22.371656+0000","last_peered":"2024-05-27T07:22:22.371656+0000","last_clean":"2024-05-27T07:22:22.371656+0000","last_became_active":"2024-05-27T07:17:02.338975+0000","last_became_peered":"2024-05-27T07:17:02.338975+0000","last_unstale":"2024-05-27T07:22:22.371656+0000","last_undegraded":"2024-05-27T07:22:22.371656+0000","last_fullsized":"2024-05-27T07:22:22.371656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:14.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301875+0000","last_change":"2024-05-27T07:17:06.758234+0000","last_active":"2024-05-27T07:22:22.301875+0000","last_peered":"2024-05-27T07:22:22.301875+0000","last_clean":"2024-05-27T07:22:22.301875+0000","last_became_active":"2024-05-27T07:17:06.757827+0000","last_became_peered":"2024-05-27T07:17:06.757827+0000","last_unstale":"2024-05-27T07:22:22.301875+0000","last_undegraded":"2024-05-27T07:22:22.301875+0000","last_fullsized":"2024-05-27T07:22:22.301875+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:04:21.688435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372775+0000","last_change":"2024-05-27T07:17:21.882540+0000","last_active":"2024-05-27T07:22:22.372775+0000","last_peered":"2024-05-27T07:22:22.372775+0000","last_clean":"2024-05-27T07:22:22.372775+0000","last_became_active":"2024-05-27T07:17:21.882087+0000","last_became_peered":"2024-05-27T07:17:21.882087+0000","last_unstale":"2024-05-27T07:22:22.372775+0000","last_undegraded":"2024-05-27T07:22:22.372775+0000","last_fullsized":"2024-05-27T07:22:22.372775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:53:04.363521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'9","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.712554+0000","last_change":"2024-05-27T07:17:46.704997+0000","last_active":"2024-05-27T07:22:22.712554+0000","last_peered":"2024-05-27T07:22:22.712554+0000","last_clean":"2024-05-27T07:22:22.712554+0000","last_became_active":"2024-05-27T07:17:46.704355+0000","last_became_peered":"2024-05-27T07:17:46.704355+0000","last_unstale":"2024-05-27T07:22:22.712554+0000","last_undegraded":"2024-05-27T07:22:22.712554+0000","last_fullsized":"2024-05-27T07:22:22.712554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:43:02.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":229,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372140+0000","last_change":"2024-05-27T07:17:06.765809+0000","last_active":"2024-05-27T07:22:22.372140+0000","last_peered":"2024-05-27T07:22:22.372140+0000","last_clean":"2024-05-27T07:22:22.372140+0000","last_became_active":"2024-05-27T07:17:06.763845+0000","last_became_peered":"2024-05-27T07:17:06.763845+0000","last_unstale":"2024-05-27T07:22:22.372140+0000","last_undegraded":"2024-05-27T07:22:22.372140+0000","last_fullsized":"2024-05-27T07:22:22.372140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:25.399181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372567+0000","last_change":"2024-05-27T07:17:21.886316+0000","last_active":"2024-05-27T07:22:22.372567+0000","last_peered":"2024-05-27T07:22:22.372567+0000","last_clean":"2024-05-27T07:22:22.372567+0000","last_became_active":"2024-05-27T07:17:21.885129+0000","last_became_peered":"2024-05-27T07:17:21.885129+0000","last_unstale":"2024-05-27T07:22:22.372567+0000","last_undegraded":"2024-05-27T07:22:22.372567+0000","last_fullsized":"2024-05-27T07:22:22.372567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:10.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377109+0000","last_change":"2024-05-27T07:17:02.338765+0000","last_active":"2024-05-27T07:22:22.377109+0000","last_peered":"2024-05-27T07:22:22.377109+0000","last_clean":"2024-05-27T07:22:22.377109+0000","last_became_active":"2024-05-27T07:17:02.337103+0000","last_became_peered":"2024-05-27T07:17:02.337103+0000","last_unstale":"2024-05-27T07:22:22.377109+0000","last_undegraded":"2024-05-27T07:22:22.377109+0000","last_fullsized":"2024-05-27T07:22:22.377109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:15:07.885071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301489+0000","last_change":"2024-05-27T07:17:06.766839+0000","last_active":"2024-05-27T07:22:22.301489+0000","last_peered":"2024-05-27T07:22:22.301489+0000","last_clean":"2024-05-27T07:22:22.301489+0000","last_became_active":"2024-05-27T07:17:06.766373+0000","last_became_peered":"2024-05-27T07:17:06.766373+0000","last_unstale":"2024-05-27T07:22:22.301489+0000","last_undegraded":"2024-05-27T07:22:22.301489+0000","last_fullsized":"2024-05-27T07:22:22.301489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:09:55.064045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376814+0000","last_change":"2024-05-27T07:18:47.133692+0000","last_active":"2024-05-27T07:22:22.376814+0000","last_peered":"2024-05-27T07:22:22.376814+0000","last_clean":"2024-05-27T07:22:22.376814+0000","last_became_active":"2024-05-27T07:18:47.132122+0000","last_became_peered":"2024-05-27T07:18:47.132122+0000","last_unstale":"2024-05-27T07:22:22.376814+0000","last_undegraded":"2024-05-27T07:22:22.376814+0000","last_fullsized":"2024-05-27T07:22:22.376814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:34:57.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296700+0000","last_change":"2024-05-27T07:17:02.329809+0000","last_active":"2024-05-27T07:22:22.296700+0000","last_peered":"2024-05-27T07:22:22.296700+0000","last_clean":"2024-05-27T07:22:22.296700+0000","last_became_active":"2024-05-27T07:17:02.328758+0000","last_became_peered":"2024-05-27T07:17:02.328758+0000","last_unstale":"2024-05-27T07:22:22.296700+0000","last_undegraded":"2024-05-27T07:22:22.296700+0000","last_fullsized":"2024-05-27T07:22:22.296700+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:52.479444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366880+0000","last_change":"2024-05-27T07:17:06.749836+0000","last_active":"2024-05-27T07:22:22.366880+0000","last_peered":"2024-05-27T07:22:22.366880+0000","last_clean":"2024-05-27T07:22:22.366880+0000","last_became_active":"2024-05-27T07:17:06.749512+0000","last_became_peered":"2024-05-27T07:17:06.749512+0000","last_unstale":"2024-05-27T07:22:22.366880+0000","last_undegraded":"2024-05-27T07:22:22.366880+0000","last_fullsized":"2024-05-27T07:22:22.366880+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:41:51.000415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595128+0000","last_change":"2024-05-27T07:17:21.888376+0000","last_active":"2024-05-27T07:22:22.595128+0000","last_peered":"2024-05-27T07:22:22.595128+0000","last_clean":"2024-05-27T07:22:22.595128+0000","last_became_active":"2024-05-27T07:17:21.886183+0000","last_became_peered":"2024-05-27T07:17:21.886183+0000","last_unstale":"2024-05-27T07:22:22.595128+0000","last_undegraded":"2024-05-27T07:22:22.595128+0000","last_fullsized":"2024-05-27T07:22:22.595128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:34.541338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300149+0000","last_change":"2024-05-27T07:17:02.334113+0000","last_active":"2024-05-27T07:22:22.300149+0000","last_peered":"2024-05-27T07:22:22.300149+0000","last_clean":"2024-05-27T07:22:22.300149+0000","last_became_active":"2024-05-27T07:17:02.333201+0000","last_became_peered":"2024-05-27T07:17:02.333201+0000","last_unstale":"2024-05-27T07:22:22.300149+0000","last_undegraded":"2024-05-27T07:22:22.300149+0000","last_fullsized":"2024-05-27T07:22:22.300149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:13.484654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369241+0000","last_change":"2024-05-27T07:17:06.751676+0000","last_active":"2024-05-27T07:22:22.369241+0000","last_peered":"2024-05-27T07:22:22.369241+0000","last_clean":"2024-05-27T07:22:22.369241+0000","last_became_active":"2024-05-27T07:17:06.751221+0000","last_became_peered":"2024-05-27T07:17:06.751221+0000","last_unstale":"2024-05-27T07:22:22.369241+0000","last_undegraded":"2024-05-27T07:22:22.369241+0000","last_fullsized":"2024-05-27T07:22:22.369241+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:40.012561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367255+0000","last_change":"2024-05-27T07:17:21.897640+0000","last_active":"2024-05-27T07:22:22.367255+0000","last_peered":"2024-05-27T07:22:22.367255+0000","last_clean":"2024-05-27T07:22:22.367255+0000","last_became_active":"2024-05-27T07:17:21.895355+0000","last_became_peered":"2024-05-27T07:17:21.895355+0000","last_unstale":"2024-05-27T07:22:22.367255+0000","last_undegraded":"2024-05-27T07:22:22.367255+0000","last_fullsized":"2024-05-27T07:22:22.367255+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:51:12.030069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'5","reported_seq":277,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375588+0000","last_change":"2024-05-27T07:17:02.339464+0000","last_active":"2024-05-27T07:22:22.375588+0000","last_peered":"2024-05-27T07:22:22.375588+0000","last_clean":"2024-05-27T07:22:22.375588+0000","last_became_active":"2024-05-27T07:17:02.338577+0000","last_became_peered":"2024-05-27T07:17:02.338577+0000","last_unstale":"2024-05-27T07:22:22.375588+0000","last_undegraded":"2024-05-27T07:22:22.375588+0000","last_fullsized":"2024-05-27T07:22:22.375588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:29:59.210157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376770+0000","last_change":"2024-05-27T07:17:06.766883+0000","last_active":"2024-05-27T07:22:22.376770+0000","last_peered":"2024-05-27T07:22:22.376770+0000","last_clean":"2024-05-27T07:22:22.376770+0000","last_became_active":"2024-05-27T07:17:06.766166+0000","last_became_peered":"2024-05-27T07:17:06.766166+0000","last_unstale":"2024-05-27T07:22:22.376770+0000","last_undegraded":"2024-05-27T07:22:22.376770+0000","last_fullsized":"2024-05-27T07:22:22.376770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:28.936499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369022+0000","last_change":"2024-05-27T07:17:21.893930+0000","last_active":"2024-05-27T07:22:22.369022+0000","last_peered":"2024-05-27T07:22:22.369022+0000","last_clean":"2024-05-27T07:22:22.369022+0000","last_became_active":"2024-05-27T07:17:21.893029+0000","last_became_peered":"2024-05-27T07:17:21.893029+0000","last_unstale":"2024-05-27T07:22:22.369022+0000","last_undegraded":"2024-05-27T07:22:22.369022+0000","last_fullsized":"2024-05-27T07:22:22.369022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:48:32.453078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303681+0000","last_change":"2024-05-27T07:17:02.346531+0000","last_active":"2024-05-27T07:22:22.303681+0000","last_peered":"2024-05-27T07:22:22.303681+0000","last_clean":"2024-05-27T07:22:22.303681+0000","last_became_active":"2024-05-27T07:17:02.345844+0000","last_became_peered":"2024-05-27T07:17:02.345844+0000","last_unstale":"2024-05-27T07:22:22.303681+0000","last_undegraded":"2024-05-27T07:22:22.303681+0000","last_fullsized":"2024-05-27T07:22:22.303681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:28.544616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595299+0000","last_change":"2024-05-27T07:17:06.769845+0000","last_active":"2024-05-27T07:22:22.595299+0000","last_peered":"2024-05-27T07:22:22.595299+0000","last_clean":"2024-05-27T07:22:22.595299+0000","last_became_active":"2024-05-27T07:17:06.768467+0000","last_became_peered":"2024-05-27T07:17:06.768467+0000","last_unstale":"2024-05-27T07:22:22.595299+0000","last_undegraded":"2024-05-27T07:22:22.595299+0000","last_fullsized":"2024-05-27T07:22:22.595299+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:30.311824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374196+0000","last_change":"2024-05-27T07:17:21.899885+0000","last_active":"2024-05-27T07:22:22.374196+0000","last_peered":"2024-05-27T07:22:22.374196+0000","last_clean":"2024-05-27T07:22:22.374196+0000","last_became_active":"2024-05-27T07:17:21.899340+0000","last_became_peered":"2024-05-27T07:17:21.899340+0000","last_unstale":"2024-05-27T07:22:22.374196+0000","last_undegraded":"2024-05-27T07:22:22.374196+0000","last_fullsized":"2024-05-27T07:22:22.374196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:12.665409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301032+0000","last_change":"2024-05-27T07:17:02.435770+0000","last_active":"2024-05-27T07:22:22.301032+0000","last_peered":"2024-05-27T07:22:22.301032+0000","last_clean":"2024-05-27T07:22:22.301032+0000","last_became_active":"2024-05-27T07:17:02.434535+0000","last_became_peered":"2024-05-27T07:17:02.434535+0000","last_unstale":"2024-05-27T07:22:22.301032+0000","last_undegraded":"2024-05-27T07:22:22.301032+0000","last_fullsized":"2024-05-27T07:22:22.301032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:53.745583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372886+0000","last_change":"2024-05-27T07:17:46.704100+0000","last_active":"2024-05-27T07:22:22.372886+0000","last_peered":"2024-05-27T07:22:22.372886+0000","last_clean":"2024-05-27T07:22:22.372886+0000","last_became_active":"2024-05-27T07:17:46.703475+0000","last_became_peered":"2024-05-27T07:17:46.703475+0000","last_unstale":"2024-05-27T07:22:22.372886+0000","last_undegraded":"2024-05-27T07:22:22.372886+0000","last_fullsized":"2024-05-27T07:22:22.372886+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:20:29.173923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377473+0000","last_change":"2024-05-27T07:18:47.131521+0000","last_active":"2024-05-27T07:22:22.377473+0000","last_peered":"2024-05-27T07:22:22.377473+0000","last_clean":"2024-05-27T07:22:22.377473+0000","last_became_active":"2024-05-27T07:18:47.129995+0000","last_became_peered":"2024-05-27T07:18:47.129995+0000","last_unstale":"2024-05-27T07:22:22.377473+0000","last_undegraded":"2024-05-27T07:22:22.377473+0000","last_fullsized":"2024-05-27T07:22:22.377473+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:45:27.268768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":565,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.933547+0000","last_change":"2024-05-27T07:17:46.719293+0000","last_active":"2024-05-27T07:22:22.933547+0000","last_peered":"2024-05-27T07:22:22.933547+0000","last_clean":"2024-05-27T07:22:22.933547+0000","last_became_active":"2024-05-27T07:17:46.703496+0000","last_became_peered":"2024-05-27T07:17:46.703496+0000","last_unstale":"2024-05-27T07:22:22.933547+0000","last_undegraded":"2024-05-27T07:22:22.933547+0000","last_fullsized":"2024-05-27T07:22:22.933547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:26.310524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67910,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":42,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373655+0000","last_change":"2024-05-27T07:17:06.763141+0000","last_active":"2024-05-27T07:22:22.373655+0000","last_peered":"2024-05-27T07:22:22.373655+0000","last_clean":"2024-05-27T07:22:22.373655+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:22:22.373655+0000","last_undegraded":"2024-05-27T07:22:22.373655+0000","last_fullsized":"2024-05-27T07:22:22.373655+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:47.072552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302824+0000","last_change":"2024-05-27T07:17:21.895342+0000","last_active":"2024-05-27T07:22:22.302824+0000","last_peered":"2024-05-27T07:22:22.302824+0000","last_clean":"2024-05-27T07:22:22.302824+0000","last_became_active":"2024-05-27T07:17:21.894630+0000","last_became_peered":"2024-05-27T07:17:21.894630+0000","last_unstale":"2024-05-27T07:22:22.302824+0000","last_undegraded":"2024-05-27T07:22:22.302824+0000","last_fullsized":"2024-05-27T07:22:22.302824+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:57:50.501683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371791+0000","last_change":"2024-05-27T07:17:02.341848+0000","last_active":"2024-05-27T07:22:22.371791+0000","last_peered":"2024-05-27T07:22:22.371791+0000","last_clean":"2024-05-27T07:22:22.371791+0000","last_became_active":"2024-05-27T07:17:02.339477+0000","last_became_peered":"2024-05-27T07:17:02.339477+0000","last_unstale":"2024-05-27T07:22:22.371791+0000","last_undegraded":"2024-05-27T07:22:22.371791+0000","last_fullsized":"2024-05-27T07:22:22.371791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:23.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297148+0000","last_change":"2024-05-27T07:17:06.761340+0000","last_active":"2024-05-27T07:22:22.297148+0000","last_peered":"2024-05-27T07:22:22.297148+0000","last_clean":"2024-05-27T07:22:22.297148+0000","last_became_active":"2024-05-27T07:17:06.759103+0000","last_became_peered":"2024-05-27T07:17:06.759103+0000","last_unstale":"2024-05-27T07:22:22.297148+0000","last_undegraded":"2024-05-27T07:22:22.297148+0000","last_fullsized":"2024-05-27T07:22:22.297148+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:42.263106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375699+0000","last_change":"2024-05-27T07:17:21.883252+0000","last_active":"2024-05-27T07:22:22.375699+0000","last_peered":"2024-05-27T07:22:22.375699+0000","last_clean":"2024-05-27T07:22:22.375699+0000","last_became_active":"2024-05-27T07:17:21.882468+0000","last_became_peered":"2024-05-27T07:17:21.882468+0000","last_unstale":"2024-05-27T07:22:22.375699+0000","last_undegraded":"2024-05-27T07:22:22.375699+0000","last_fullsized":"2024-05-27T07:22:22.375699+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:35:59.250357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301049+0000","last_change":"2024-05-27T07:17:02.343343+0000","last_active":"2024-05-27T07:22:22.301049+0000","last_peered":"2024-05-27T07:22:22.301049+0000","last_clean":"2024-05-27T07:22:22.301049+0000","last_became_active":"2024-05-27T07:17:02.342741+0000","last_became_peered":"2024-05-27T07:17:02.342741+0000","last_unstale":"2024-05-27T07:22:22.301049+0000","last_undegraded":"2024-05-27T07:22:22.301049+0000","last_fullsized":"2024-05-27T07:22:22.301049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:46.006865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301543+0000","last_change":"2024-05-27T07:17:06.771473+0000","last_active":"2024-05-27T07:22:22.301543+0000","last_peered":"2024-05-27T07:22:22.301543+0000","last_clean":"2024-05-27T07:22:22.301543+0000","last_became_active":"2024-05-27T07:17:06.771082+0000","last_became_peered":"2024-05-27T07:17:06.771082+0000","last_unstale":"2024-05-27T07:22:22.301543+0000","last_undegraded":"2024-05-27T07:22:22.301543+0000","last_fullsized":"2024-05-27T07:22:22.301543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:25:01.371182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369606+0000","last_change":"2024-05-27T07:17:21.897691+0000","last_active":"2024-05-27T07:22:22.369606+0000","last_peered":"2024-05-27T07:22:22.369606+0000","last_clean":"2024-05-27T07:22:22.369606+0000","last_became_active":"2024-05-27T07:17:21.895221+0000","last_became_peered":"2024-05-27T07:17:21.895221+0000","last_unstale":"2024-05-27T07:22:22.369606+0000","last_undegraded":"2024-05-27T07:22:22.369606+0000","last_fullsized":"2024-05-27T07:22:22.369606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:14.626947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":283,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371332+0000","last_change":"2024-05-27T07:17:02.343375+0000","last_active":"2024-05-27T07:22:22.371332+0000","last_peered":"2024-05-27T07:22:22.371332+0000","last_clean":"2024-05-27T07:22:22.371332+0000","last_became_active":"2024-05-27T07:17:02.341315+0000","last_became_peered":"2024-05-27T07:17:02.341315+0000","last_unstale":"2024-05-27T07:22:22.371332+0000","last_undegraded":"2024-05-27T07:22:22.371332+0000","last_fullsized":"2024-05-27T07:22:22.371332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:41.927265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301159+0000","last_change":"2024-05-27T07:17:06.765862+0000","last_active":"2024-05-27T07:22:22.301159+0000","last_peered":"2024-05-27T07:22:22.301159+0000","last_clean":"2024-05-27T07:22:22.301159+0000","last_became_active":"2024-05-27T07:17:06.765439+0000","last_became_peered":"2024-05-27T07:17:06.765439+0000","last_unstale":"2024-05-27T07:22:22.301159+0000","last_undegraded":"2024-05-27T07:22:22.301159+0000","last_fullsized":"2024-05-27T07:22:22.301159+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:47.586833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.298125+0000","last_change":"2024-05-27T07:17:21.891810+0000","last_active":"2024-05-27T07:22:22.298125+0000","last_peered":"2024-05-27T07:22:22.298125+0000","last_clean":"2024-05-27T07:22:22.298125+0000","last_became_active":"2024-05-27T07:17:21.890473+0000","last_became_peered":"2024-05-27T07:17:21.890473+0000","last_unstale":"2024-05-27T07:22:22.298125+0000","last_undegraded":"2024-05-27T07:22:22.298125+0000","last_fullsized":"2024-05-27T07:22:22.298125+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:52:12.731189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375448+0000","last_change":"2024-05-27T07:17:46.715870+0000","last_active":"2024-05-27T07:22:22.375448+0000","last_peered":"2024-05-27T07:22:22.375448+0000","last_clean":"2024-05-27T07:22:22.375448+0000","last_became_active":"2024-05-27T07:17:46.701248+0000","last_became_peered":"2024-05-27T07:17:46.701248+0000","last_unstale":"2024-05-27T07:22:22.375448+0000","last_undegraded":"2024-05-27T07:22:22.375448+0000","last_fullsized":"2024-05-27T07:22:22.375448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:54:18.693752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301040+0000","last_change":"2024-05-27T07:17:06.759937+0000","last_active":"2024-05-27T07:22:22.301040+0000","last_peered":"2024-05-27T07:22:22.301040+0000","last_clean":"2024-05-27T07:22:22.301040+0000","last_became_active":"2024-05-27T07:17:06.759526+0000","last_became_peered":"2024-05-27T07:17:06.759526+0000","last_unstale":"2024-05-27T07:22:22.301040+0000","last_undegraded":"2024-05-27T07:22:22.301040+0000","last_fullsized":"2024-05-27T07:22:22.301040+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:22.934317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300660+0000","last_change":"2024-05-27T07:17:21.898331+0000","last_active":"2024-05-27T07:22:22.300660+0000","last_peered":"2024-05-27T07:22:22.300660+0000","last_clean":"2024-05-27T07:22:22.300660+0000","last_became_active":"2024-05-27T07:17:21.897965+0000","last_became_peered":"2024-05-27T07:17:21.897965+0000","last_unstale":"2024-05-27T07:22:22.300660+0000","last_undegraded":"2024-05-27T07:22:22.300660+0000","last_fullsized":"2024-05-27T07:22:22.300660+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:20.584175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373702+0000","last_change":"2024-05-27T07:17:02.346029+0000","last_active":"2024-05-27T07:22:22.373702+0000","last_peered":"2024-05-27T07:22:22.373702+0000","last_clean":"2024-05-27T07:22:22.373702+0000","last_became_active":"2024-05-27T07:17:02.341292+0000","last_became_peered":"2024-05-27T07:17:02.341292+0000","last_unstale":"2024-05-27T07:22:22.373702+0000","last_undegraded":"2024-05-27T07:22:22.373702+0000","last_fullsized":"2024-05-27T07:22:22.373702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:31.705741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376883+0000","last_change":"2024-05-27T07:17:46.691785+0000","last_active":"2024-05-27T07:22:22.376883+0000","last_peered":"2024-05-27T07:22:22.376883+0000","last_clean":"2024-05-27T07:22:22.376883+0000","last_became_active":"2024-05-27T07:17:46.690445+0000","last_became_peered":"2024-05-27T07:17:46.690445+0000","last_unstale":"2024-05-27T07:22:22.376883+0000","last_undegraded":"2024-05-27T07:22:22.376883+0000","last_fullsized":"2024-05-27T07:22:22.376883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:59:27.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372885+0000","last_change":"2024-05-27T07:17:21.892434+0000","last_active":"2024-05-27T07:22:22.372885+0000","last_peered":"2024-05-27T07:22:22.372885+0000","last_clean":"2024-05-27T07:22:22.372885+0000","last_became_active":"2024-05-27T07:17:21.891736+0000","last_became_peered":"2024-05-27T07:17:21.891736+0000","last_unstale":"2024-05-27T07:22:22.372885+0000","last_undegraded":"2024-05-27T07:22:22.372885+0000","last_fullsized":"2024-05-27T07:22:22.372885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:48.458819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370417+0000","last_change":"2024-05-27T07:17:02.436323+0000","last_active":"2024-05-27T07:22:22.370417+0000","last_peered":"2024-05-27T07:22:22.370417+0000","last_clean":"2024-05-27T07:22:22.370417+0000","last_became_active":"2024-05-27T07:17:02.435000+0000","last_became_peered":"2024-05-27T07:17:02.435000+0000","last_unstale":"2024-05-27T07:22:22.370417+0000","last_undegraded":"2024-05-27T07:22:22.370417+0000","last_fullsized":"2024-05-27T07:22:22.370417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:08.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374831+0000","last_change":"2024-05-27T07:17:06.760418+0000","last_active":"2024-05-27T07:22:22.374831+0000","last_peered":"2024-05-27T07:22:22.374831+0000","last_clean":"2024-05-27T07:22:22.374831+0000","last_became_active":"2024-05-27T07:17:06.759875+0000","last_became_peered":"2024-05-27T07:17:06.759875+0000","last_unstale":"2024-05-27T07:22:22.374831+0000","last_undegraded":"2024-05-27T07:22:22.374831+0000","last_fullsized":"2024-05-27T07:22:22.374831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:12:54.324323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297331+0000","last_change":"2024-05-27T07:17:21.888048+0000","last_active":"2024-05-27T07:22:22.297331+0000","last_peered":"2024-05-27T07:22:22.297331+0000","last_clean":"2024-05-27T07:22:22.297331+0000","last_became_active":"2024-05-27T07:17:21.887579+0000","last_became_peered":"2024-05-27T07:17:21.887579+0000","last_unstale":"2024-05-27T07:22:22.297331+0000","last_undegraded":"2024-05-27T07:22:22.297331+0000","last_fullsized":"2024-05-27T07:22:22.297331+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:34:44.569959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301559+0000","last_change":"2024-05-27T07:17:02.337655+0000","last_active":"2024-05-27T07:22:22.301559+0000","last_peered":"2024-05-27T07:22:22.301559+0000","last_clean":"2024-05-27T07:22:22.301559+0000","last_became_active":"2024-05-27T07:17:02.337252+0000","last_became_peered":"2024-05-27T07:17:02.337252+0000","last_unstale":"2024-05-27T07:22:22.301559+0000","last_undegraded":"2024-05-27T07:22:22.301559+0000","last_fullsized":"2024-05-27T07:22:22.301559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:11:25.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376338+0000","last_change":"2024-05-27T07:17:06.762934+0000","last_active":"2024-05-27T07:22:22.376338+0000","last_peered":"2024-05-27T07:22:22.376338+0000","last_clean":"2024-05-27T07:22:22.376338+0000","last_became_active":"2024-05-27T07:17:06.762507+0000","last_became_peered":"2024-05-27T07:17:06.762507+0000","last_unstale":"2024-05-27T07:22:22.376338+0000","last_undegraded":"2024-05-27T07:22:22.376338+0000","last_fullsized":"2024-05-27T07:22:22.376338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:23:35.164892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296768+0000","last_change":"2024-05-27T07:17:21.886991+0000","last_active":"2024-05-27T07:22:22.296768+0000","last_peered":"2024-05-27T07:22:22.296768+0000","last_clean":"2024-05-27T07:22:22.296768+0000","last_became_active":"2024-05-27T07:17:21.885945+0000","last_became_peered":"2024-05-27T07:17:21.885945+0000","last_unstale":"2024-05-27T07:22:22.296768+0000","last_undegraded":"2024-05-27T07:22:22.296768+0000","last_fullsized":"2024-05-27T07:22:22.296768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:23:01.853601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300810+0000","last_change":"2024-05-27T07:17:46.864130+0000","last_active":"2024-05-27T07:22:22.300810+0000","last_peered":"2024-05-27T07:22:22.300810+0000","last_clean":"2024-05-27T07:22:22.300810+0000","last_became_active":"2024-05-27T07:17:46.862997+0000","last_became_peered":"2024-05-27T07:17:46.862997+0000","last_unstale":"2024-05-27T07:22:22.300810+0000","last_undegraded":"2024-05-27T07:22:22.300810+0000","last_fullsized":"2024-05-27T07:22:22.300810+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:10.594627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":278,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296813+0000","last_change":"2024-05-27T07:17:02.435463+0000","last_active":"2024-05-27T07:22:22.296813+0000","last_peered":"2024-05-27T07:22:22.296813+0000","last_clean":"2024-05-27T07:22:22.296813+0000","last_became_active":"2024-05-27T07:17:02.432131+0000","last_became_peered":"2024-05-27T07:17:02.432131+0000","last_unstale":"2024-05-27T07:22:22.296813+0000","last_undegraded":"2024-05-27T07:22:22.296813+0000","last_fullsized":"2024-05-27T07:22:22.296813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:41.874577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302768+0000","last_change":"2024-05-27T07:18:47.134043+0000","last_active":"2024-05-27T07:22:22.302768+0000","last_peered":"2024-05-27T07:22:22.302768+0000","last_clean":"2024-05-27T07:22:22.302768+0000","last_became_active":"2024-05-27T07:18:47.132050+0000","last_became_peered":"2024-05-27T07:18:47.132050+0000","last_unstale":"2024-05-27T07:22:22.302768+0000","last_undegraded":"2024-05-27T07:22:22.302768+0000","last_fullsized":"2024-05-27T07:22:22.302768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:12.170883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367929+0000","last_change":"2024-05-27T07:17:06.751168+0000","last_active":"2024-05-27T07:22:22.367929+0000","last_peered":"2024-05-27T07:22:22.367929+0000","last_clean":"2024-05-27T07:22:22.367929+0000","last_became_active":"2024-05-27T07:17:06.750816+0000","last_became_peered":"2024-05-27T07:17:06.750816+0000","last_unstale":"2024-05-27T07:22:22.367929+0000","last_undegraded":"2024-05-27T07:22:22.367929+0000","last_fullsized":"2024-05-27T07:22:22.367929+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:33:45.607426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373763+0000","last_change":"2024-05-27T07:17:02.337490+0000","last_active":"2024-05-27T07:22:22.373763+0000","last_peered":"2024-05-27T07:22:22.373763+0000","last_clean":"2024-05-27T07:22:22.373763+0000","last_became_active":"2024-05-27T07:17:02.335317+0000","last_became_peered":"2024-05-27T07:17:02.335317+0000","last_unstale":"2024-05-27T07:22:22.373763+0000","last_undegraded":"2024-05-27T07:22:22.373763+0000","last_fullsized":"2024-05-27T07:22:22.373763+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:33:46.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":123,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300719+0000","last_change":"2024-05-27T07:15:10.825312+0000","last_active":"2024-05-27T07:22:22.300719+0000","last_peered":"2024-05-27T07:22:22.300719+0000","last_clean":"2024-05-27T07:22:22.300719+0000","last_became_active":"2024-05-27T07:15:10.809245+0000","last_became_peered":"2024-05-27T07:15:10.809245+0000","last_unstale":"2024-05-27T07:22:22.300719+0000","last_undegraded":"2024-05-27T07:22:22.300719+0000","last_fullsized":"2024-05-27T07:22:22.300719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:10:08.008565+0000","last_clean_scrub_stamp":"2024-05-27T07:10:08.008565+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:02:32.769372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302265+0000","last_change":"2024-05-27T07:18:47.133948+0000","last_active":"2024-05-27T07:22:22.302265+0000","last_peered":"2024-05-27T07:22:22.302265+0000","last_clean":"2024-05-27T07:22:22.302265+0000","last_became_active":"2024-05-27T07:18:47.132566+0000","last_became_peered":"2024-05-27T07:18:47.132566+0000","last_unstale":"2024-05-27T07:22:22.302265+0000","last_undegraded":"2024-05-27T07:22:22.302265+0000","last_fullsized":"2024-05-27T07:22:22.302265+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:47:04.463121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.296515+0000","last_change":"2024-05-27T07:17:06.766187+0000","last_active":"2024-05-27T07:22:22.296515+0000","last_peered":"2024-05-27T07:22:22.296515+0000","last_clean":"2024-05-27T07:22:22.296515+0000","last_became_active":"2024-05-27T07:17:06.763365+0000","last_became_peered":"2024-05-27T07:17:06.763365+0000","last_unstale":"2024-05-27T07:22:22.296515+0000","last_undegraded":"2024-05-27T07:22:22.296515+0000","last_fullsized":"2024-05-27T07:22:22.296515+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:12:13.668426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301091+0000","last_change":"2024-05-27T07:17:02.344452+0000","last_active":"2024-05-27T07:22:22.301091+0000","last_peered":"2024-05-27T07:22:22.301091+0000","last_clean":"2024-05-27T07:22:22.301091+0000","last_became_active":"2024-05-27T07:17:02.343885+0000","last_became_peered":"2024-05-27T07:17:02.343885+0000","last_unstale":"2024-05-27T07:22:22.301091+0000","last_undegraded":"2024-05-27T07:22:22.301091+0000","last_fullsized":"2024-05-27T07:22:22.301091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:27:26.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375242+0000","last_change":"2024-05-27T07:17:21.883024+0000","last_active":"2024-05-27T07:22:22.375242+0000","last_peered":"2024-05-27T07:22:22.375242+0000","last_clean":"2024-05-27T07:22:22.375242+0000","last_became_active":"2024-05-27T07:17:21.882429+0000","last_became_peered":"2024-05-27T07:17:21.882429+0000","last_unstale":"2024-05-27T07:22:22.375242+0000","last_undegraded":"2024-05-27T07:22:22.375242+0000","last_fullsized":"2024-05-27T07:22:22.375242+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:36:40.443537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373282+0000","last_change":"2024-05-27T07:17:02.432768+0000","last_active":"2024-05-27T07:22:22.373282+0000","last_peered":"2024-05-27T07:22:22.373282+0000","last_clean":"2024-05-27T07:22:22.373282+0000","last_became_active":"2024-05-27T07:17:02.431551+0000","last_became_peered":"2024-05-27T07:17:02.431551+0000","last_unstale":"2024-05-27T07:22:22.373282+0000","last_undegraded":"2024-05-27T07:22:22.373282+0000","last_fullsized":"2024-05-27T07:22:22.373282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:39.759512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297928+0000","last_change":"2024-05-27T07:17:06.770992+0000","last_active":"2024-05-27T07:22:22.297928+0000","last_peered":"2024-05-27T07:22:22.297928+0000","last_clean":"2024-05-27T07:22:22.297928+0000","last_became_active":"2024-05-27T07:17:06.770560+0000","last_became_peered":"2024-05-27T07:17:06.770560+0000","last_unstale":"2024-05-27T07:22:22.297928+0000","last_undegraded":"2024-05-27T07:22:22.297928+0000","last_fullsized":"2024-05-27T07:22:22.297928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:07:34.443551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.366856+0000","last_change":"2024-05-27T07:17:21.895362+0000","last_active":"2024-05-27T07:22:22.366856+0000","last_peered":"2024-05-27T07:22:22.366856+0000","last_clean":"2024-05-27T07:22:22.366856+0000","last_became_active":"2024-05-27T07:17:21.894563+0000","last_became_peered":"2024-05-27T07:17:21.894563+0000","last_unstale":"2024-05-27T07:22:22.366856+0000","last_undegraded":"2024-05-27T07:22:22.366856+0000","last_fullsized":"2024-05-27T07:22:22.366856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:44.394301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'49","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.689696+0000","last_change":"2024-05-27T07:17:02.338203+0000","last_active":"2024-05-27T07:22:22.689696+0000","last_peered":"2024-05-27T07:22:22.689696+0000","last_clean":"2024-05-27T07:22:22.689696+0000","last_became_active":"2024-05-27T07:17:02.336192+0000","last_became_peered":"2024-05-27T07:17:02.336192+0000","last_unstale":"2024-05-27T07:22:22.689696+0000","last_undegraded":"2024-05-27T07:22:22.689696+0000","last_fullsized":"2024-05-27T07:22:22.689696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:02:48.187517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368696+0000","last_change":"2024-05-27T07:17:06.759729+0000","last_active":"2024-05-27T07:22:22.368696+0000","last_peered":"2024-05-27T07:22:22.368696+0000","last_clean":"2024-05-27T07:22:22.368696+0000","last_became_active":"2024-05-27T07:17:06.756766+0000","last_became_peered":"2024-05-27T07:17:06.756766+0000","last_unstale":"2024-05-27T07:22:22.368696+0000","last_undegraded":"2024-05-27T07:22:22.368696+0000","last_fullsized":"2024-05-27T07:22:22.368696+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:01.090586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369921+0000","last_change":"2024-05-27T07:17:21.878487+0000","last_active":"2024-05-27T07:22:22.369921+0000","last_peered":"2024-05-27T07:22:22.369921+0000","last_clean":"2024-05-27T07:22:22.369921+0000","last_became_active":"2024-05-27T07:17:21.877623+0000","last_became_peered":"2024-05-27T07:17:21.877623+0000","last_unstale":"2024-05-27T07:22:22.369921+0000","last_undegraded":"2024-05-27T07:22:22.369921+0000","last_fullsized":"2024-05-27T07:22:22.369921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:28:52.618738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375223+0000","last_change":"2024-05-27T07:17:02.486495+0000","last_active":"2024-05-27T07:22:22.375223+0000","last_peered":"2024-05-27T07:22:22.375223+0000","last_clean":"2024-05-27T07:22:22.375223+0000","last_became_active":"2024-05-27T07:17:02.485493+0000","last_became_peered":"2024-05-27T07:17:02.485493+0000","last_unstale":"2024-05-27T07:22:22.375223+0000","last_undegraded":"2024-05-27T07:22:22.375223+0000","last_fullsized":"2024-05-27T07:22:22.375223+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:31.977210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.595283+0000","last_change":"2024-05-27T07:17:06.760937+0000","last_active":"2024-05-27T07:22:22.595283+0000","last_peered":"2024-05-27T07:22:22.595283+0000","last_clean":"2024-05-27T07:22:22.595283+0000","last_became_active":"2024-05-27T07:17:06.760254+0000","last_became_peered":"2024-05-27T07:17:06.760254+0000","last_unstale":"2024-05-27T07:22:22.595283+0000","last_undegraded":"2024-05-27T07:22:22.595283+0000","last_fullsized":"2024-05-27T07:22:22.595283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:02:50.147319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373319+0000","last_change":"2024-05-27T07:17:21.901511+0000","last_active":"2024-05-27T07:22:22.373319+0000","last_peered":"2024-05-27T07:22:22.373319+0000","last_clean":"2024-05-27T07:22:22.373319+0000","last_became_active":"2024-05-27T07:17:21.899847+0000","last_became_peered":"2024-05-27T07:17:21.899847+0000","last_unstale":"2024-05-27T07:22:22.373319+0000","last_undegraded":"2024-05-27T07:22:22.373319+0000","last_fullsized":"2024-05-27T07:22:22.373319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:28.133509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370105+0000","last_change":"2024-05-27T07:17:02.432520+0000","last_active":"2024-05-27T07:22:22.370105+0000","last_peered":"2024-05-27T07:22:22.370105+0000","last_clean":"2024-05-27T07:22:22.370105+0000","last_became_active":"2024-05-27T07:17:02.431563+0000","last_became_peered":"2024-05-27T07:17:02.431563+0000","last_unstale":"2024-05-27T07:22:22.370105+0000","last_undegraded":"2024-05-27T07:22:22.370105+0000","last_fullsized":"2024-05-27T07:22:22.370105+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:17:28.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.372653+0000","last_change":"2024-05-27T07:17:06.754408+0000","last_active":"2024-05-27T07:22:22.372653+0000","last_peered":"2024-05-27T07:22:22.372653+0000","last_clean":"2024-05-27T07:22:22.372653+0000","last_became_active":"2024-05-27T07:17:06.753787+0000","last_became_peered":"2024-05-27T07:17:06.753787+0000","last_unstale":"2024-05-27T07:22:22.372653+0000","last_undegraded":"2024-05-27T07:22:22.372653+0000","last_fullsized":"2024-05-27T07:22:22.372653+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:21:50.461296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369512+0000","last_change":"2024-05-27T07:17:21.895277+0000","last_active":"2024-05-27T07:22:22.369512+0000","last_peered":"2024-05-27T07:22:22.369512+0000","last_clean":"2024-05-27T07:22:22.369512+0000","last_became_active":"2024-05-27T07:17:21.894518+0000","last_became_peered":"2024-05-27T07:17:21.894518+0000","last_unstale":"2024-05-27T07:22:22.369512+0000","last_undegraded":"2024-05-27T07:22:22.369512+0000","last_fullsized":"2024-05-27T07:22:22.369512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:02.008209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367305+0000","last_change":"2024-05-27T07:17:02.343604+0000","last_active":"2024-05-27T07:22:22.367305+0000","last_peered":"2024-05-27T07:22:22.367305+0000","last_clean":"2024-05-27T07:22:22.367305+0000","last_became_active":"2024-05-27T07:17:02.341890+0000","last_became_peered":"2024-05-27T07:17:02.341890+0000","last_unstale":"2024-05-27T07:22:22.367305+0000","last_undegraded":"2024-05-27T07:22:22.367305+0000","last_fullsized":"2024-05-27T07:22:22.367305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:10:21.670193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375088+0000","last_change":"2024-05-27T07:17:06.754767+0000","last_active":"2024-05-27T07:22:22.375088+0000","last_peered":"2024-05-27T07:22:22.375088+0000","last_clean":"2024-05-27T07:22:22.375088+0000","last_became_active":"2024-05-27T07:17:06.754291+0000","last_became_peered":"2024-05-27T07:17:06.754291+0000","last_unstale":"2024-05-27T07:22:22.375088+0000","last_undegraded":"2024-05-27T07:22:22.375088+0000","last_fullsized":"2024-05-27T07:22:22.375088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:46:48.145874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.377303+0000","last_change":"2024-05-27T07:17:21.892342+0000","last_active":"2024-05-27T07:22:22.377303+0000","last_peered":"2024-05-27T07:22:22.377303+0000","last_clean":"2024-05-27T07:22:22.377303+0000","last_became_active":"2024-05-27T07:17:21.891393+0000","last_became_peered":"2024-05-27T07:17:21.891393+0000","last_unstale":"2024-05-27T07:22:22.377303+0000","last_undegraded":"2024-05-27T07:22:22.377303+0000","last_fullsized":"2024-05-27T07:22:22.377303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:30.510616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370938+0000","last_change":"2024-05-27T07:17:02.340238+0000","last_active":"2024-05-27T07:22:22.370938+0000","last_peered":"2024-05-27T07:22:22.370938+0000","last_clean":"2024-05-27T07:22:22.370938+0000","last_became_active":"2024-05-27T07:17:02.339561+0000","last_became_peered":"2024-05-27T07:17:02.339561+0000","last_unstale":"2024-05-27T07:22:22.370938+0000","last_undegraded":"2024-05-27T07:22:22.370938+0000","last_fullsized":"2024-05-27T07:22:22.370938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:32.451397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.368236+0000","last_change":"2024-05-27T07:17:06.769397+0000","last_active":"2024-05-27T07:22:22.368236+0000","last_peered":"2024-05-27T07:22:22.368236+0000","last_clean":"2024-05-27T07:22:22.368236+0000","last_became_active":"2024-05-27T07:17:06.768936+0000","last_became_peered":"2024-05-27T07:17:06.768936+0000","last_unstale":"2024-05-27T07:22:22.368236+0000","last_undegraded":"2024-05-27T07:22:22.368236+0000","last_fullsized":"2024-05-27T07:22:22.368236+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:52.767902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.374048+0000","last_change":"2024-05-27T07:17:21.894225+0000","last_active":"2024-05-27T07:22:22.374048+0000","last_peered":"2024-05-27T07:22:22.374048+0000","last_clean":"2024-05-27T07:22:22.374048+0000","last_became_active":"2024-05-27T07:17:21.893499+0000","last_became_peered":"2024-05-27T07:17:21.893499+0000","last_unstale":"2024-05-27T07:22:22.374048+0000","last_undegraded":"2024-05-27T07:22:22.374048+0000","last_fullsized":"2024-05-27T07:22:22.374048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:52:54.985290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371433+0000","last_change":"2024-05-27T07:17:02.347736+0000","last_active":"2024-05-27T07:22:22.371433+0000","last_peered":"2024-05-27T07:22:22.371433+0000","last_clean":"2024-05-27T07:22:22.371433+0000","last_became_active":"2024-05-27T07:17:02.347237+0000","last_became_peered":"2024-05-27T07:17:02.347237+0000","last_unstale":"2024-05-27T07:22:22.371433+0000","last_undegraded":"2024-05-27T07:22:22.371433+0000","last_fullsized":"2024-05-27T07:22:22.371433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:36:15.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.301779+0000","last_change":"2024-05-27T07:17:06.759348+0000","last_active":"2024-05-27T07:22:22.301779+0000","last_peered":"2024-05-27T07:22:22.301779+0000","last_clean":"2024-05-27T07:22:22.301779+0000","last_became_active":"2024-05-27T07:17:06.758967+0000","last_became_peered":"2024-05-27T07:17:06.758967+0000","last_unstale":"2024-05-27T07:22:22.301779+0000","last_undegraded":"2024-05-27T07:22:22.301779+0000","last_fullsized":"2024-05-27T07:22:22.301779+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:45.430719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371200+0000","last_change":"2024-05-27T07:17:21.882919+0000","last_active":"2024-05-27T07:22:22.371200+0000","last_peered":"2024-05-27T07:22:22.371200+0000","last_clean":"2024-05-27T07:22:22.371200+0000","last_became_active":"2024-05-27T07:17:21.882330+0000","last_became_peered":"2024-05-27T07:17:21.882330+0000","last_unstale":"2024-05-27T07:22:22.371200+0000","last_undegraded":"2024-05-27T07:22:22.371200+0000","last_fullsized":"2024-05-27T07:22:22.371200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:51.409079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373120+0000","last_change":"2024-05-27T07:17:02.486099+0000","last_active":"2024-05-27T07:22:22.373120+0000","last_peered":"2024-05-27T07:22:22.373120+0000","last_clean":"2024-05-27T07:22:22.373120+0000","last_became_active":"2024-05-27T07:17:02.485116+0000","last_became_peered":"2024-05-27T07:17:02.485116+0000","last_unstale":"2024-05-27T07:22:22.373120+0000","last_undegraded":"2024-05-27T07:22:22.373120+0000","last_fullsized":"2024-05-27T07:22:22.373120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:36.338287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.303614+0000","last_change":"2024-05-27T07:17:06.773586+0000","last_active":"2024-05-27T07:22:22.303614+0000","last_peered":"2024-05-27T07:22:22.303614+0000","last_clean":"2024-05-27T07:22:22.303614+0000","last_became_active":"2024-05-27T07:17:06.773139+0000","last_became_peered":"2024-05-27T07:17:06.773139+0000","last_unstale":"2024-05-27T07:22:22.303614+0000","last_undegraded":"2024-05-27T07:22:22.303614+0000","last_fullsized":"2024-05-27T07:22:22.303614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:19:36.373091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.594249+0000","last_change":"2024-05-27T07:17:21.892351+0000","last_active":"2024-05-27T07:22:22.594249+0000","last_peered":"2024-05-27T07:22:22.594249+0000","last_clean":"2024-05-27T07:22:22.594249+0000","last_became_active":"2024-05-27T07:17:21.891862+0000","last_became_peered":"2024-05-27T07:17:21.891862+0000","last_unstale":"2024-05-27T07:22:22.594249+0000","last_undegraded":"2024-05-27T07:22:22.594249+0000","last_fullsized":"2024-05-27T07:22:22.594249+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:50:18.193149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.677139+0000","last_change":"2024-05-27T07:17:02.336917+0000","last_active":"2024-05-27T07:22:22.677139+0000","last_peered":"2024-05-27T07:22:22.677139+0000","last_clean":"2024-05-27T07:22:22.677139+0000","last_became_active":"2024-05-27T07:17:02.335836+0000","last_became_peered":"2024-05-27T07:17:02.335836+0000","last_unstale":"2024-05-27T07:22:22.677139+0000","last_undegraded":"2024-05-27T07:22:22.677139+0000","last_fullsized":"2024-05-27T07:22:22.677139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:54.109480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.302465+0000","last_change":"2024-05-27T07:17:46.705826+0000","last_active":"2024-05-27T07:22:22.302465+0000","last_peered":"2024-05-27T07:22:22.302465+0000","last_clean":"2024-05-27T07:22:22.302465+0000","last_became_active":"2024-05-27T07:17:46.704749+0000","last_became_peered":"2024-05-27T07:17:46.704749+0000","last_unstale":"2024-05-27T07:22:22.302465+0000","last_undegraded":"2024-05-27T07:22:22.302465+0000","last_fullsized":"2024-05-27T07:22:22.302465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:02:26.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.299838+0000","last_change":"2024-05-27T07:17:21.885535+0000","last_active":"2024-05-27T07:22:22.299838+0000","last_peered":"2024-05-27T07:22:22.299838+0000","last_clean":"2024-05-27T07:22:22.299838+0000","last_became_active":"2024-05-27T07:17:21.884118+0000","last_became_peered":"2024-05-27T07:17:21.884118+0000","last_unstale":"2024-05-27T07:22:22.299838+0000","last_undegraded":"2024-05-27T07:22:22.299838+0000","last_fullsized":"2024-05-27T07:22:22.299838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:20.832075+0000","last_clean_scrub_stamp":"2024-05-27T07:17:20.832075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:17:26.600199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297957+0000","last_change":"2024-05-27T07:17:02.340037+0000","last_active":"2024-05-27T07:22:22.297957+0000","last_peered":"2024-05-27T07:22:22.297957+0000","last_clean":"2024-05-27T07:22:22.297957+0000","last_became_active":"2024-05-27T07:17:02.339378+0000","last_became_peered":"2024-05-27T07:17:02.339378+0000","last_unstale":"2024-05-27T07:22:22.297957+0000","last_undegraded":"2024-05-27T07:22:22.297957+0000","last_fullsized":"2024-05-27T07:22:22.297957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:01.264652+0000","last_clean_scrub_stamp":"2024-05-27T07:17:01.264652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:33.774917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.304323+0000","last_change":"2024-05-27T07:17:06.763277+0000","last_active":"2024-05-27T07:22:22.304323+0000","last_peered":"2024-05-27T07:22:22.304323+0000","last_clean":"2024-05-27T07:22:22.304323+0000","last_became_active":"2024-05-27T07:17:06.762837+0000","last_became_peered":"2024-05-27T07:17:06.762837+0000","last_unstale":"2024-05-27T07:22:22.304323+0000","last_undegraded":"2024-05-27T07:22:22.304323+0000","last_fullsized":"2024-05-27T07:22:22.304323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:52.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.375646+0000","last_change":"2024-05-27T07:17:06.752556+0000","last_active":"2024-05-27T07:22:22.375646+0000","last_peered":"2024-05-27T07:22:22.375646+0000","last_clean":"2024-05-27T07:22:22.375646+0000","last_became_active":"2024-05-27T07:17:06.751085+0000","last_became_peered":"2024-05-27T07:17:06.751085+0000","last_unstale":"2024-05-27T07:22:22.375646+0000","last_undegraded":"2024-05-27T07:22:22.375646+0000","last_fullsized":"2024-05-27T07:22:22.375646+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:13:53.883845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370932+0000","last_change":"2024-05-27T07:17:06.753674+0000","last_active":"2024-05-27T07:22:22.370932+0000","last_peered":"2024-05-27T07:22:22.370932+0000","last_clean":"2024-05-27T07:22:22.370932+0000","last_became_active":"2024-05-27T07:17:06.752373+0000","last_became_peered":"2024-05-27T07:17:06.752373+0000","last_unstale":"2024-05-27T07:22:22.370932+0000","last_undegraded":"2024-05-27T07:22:22.370932+0000","last_fullsized":"2024-05-27T07:22:22.370932+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:40:17.474462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.370771+0000","last_change":"2024-05-27T07:17:06.766796+0000","last_active":"2024-05-27T07:22:22.370771+0000","last_peered":"2024-05-27T07:22:22.370771+0000","last_clean":"2024-05-27T07:22:22.370771+0000","last_became_active":"2024-05-27T07:17:06.766435+0000","last_became_peered":"2024-05-27T07:17:06.766435+0000","last_unstale":"2024-05-27T07:22:22.370771+0000","last_undegraded":"2024-05-27T07:22:22.370771+0000","last_fullsized":"2024-05-27T07:22:22.370771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:38.747810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.594555+0000","last_change":"2024-05-27T07:17:06.762804+0000","last_active":"2024-05-27T07:22:22.594555+0000","last_peered":"2024-05-27T07:22:22.594555+0000","last_clean":"2024-05-27T07:22:22.594555+0000","last_became_active":"2024-05-27T07:17:06.762490+0000","last_became_peered":"2024-05-27T07:17:06.762490+0000","last_unstale":"2024-05-27T07:22:22.594555+0000","last_undegraded":"2024-05-27T07:22:22.594555+0000","last_fullsized":"2024-05-27T07:22:22.594555+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:59:33.584848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.298389+0000","last_change":"2024-05-27T07:17:06.765640+0000","last_active":"2024-05-27T07:22:22.298389+0000","last_peered":"2024-05-27T07:22:22.298389+0000","last_clean":"2024-05-27T07:22:22.298389+0000","last_became_active":"2024-05-27T07:17:06.760459+0000","last_became_peered":"2024-05-27T07:17:06.760459+0000","last_unstale":"2024-05-27T07:22:22.298389+0000","last_undegraded":"2024-05-27T07:22:22.298389+0000","last_fullsized":"2024-05-27T07:22:22.298389+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:19.695007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376739+0000","last_change":"2024-05-27T07:17:06.759982+0000","last_active":"2024-05-27T07:22:22.376739+0000","last_peered":"2024-05-27T07:22:22.376739+0000","last_clean":"2024-05-27T07:22:22.376739+0000","last_became_active":"2024-05-27T07:17:06.759367+0000","last_became_peered":"2024-05-27T07:17:06.759367+0000","last_unstale":"2024-05-27T07:22:22.376739+0000","last_undegraded":"2024-05-27T07:22:22.376739+0000","last_fullsized":"2024-05-27T07:22:22.376739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:35:04.686989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.373962+0000","last_change":"2024-05-27T07:17:06.768264+0000","last_active":"2024-05-27T07:22:22.373962+0000","last_peered":"2024-05-27T07:22:22.373962+0000","last_clean":"2024-05-27T07:22:22.373962+0000","last_became_active":"2024-05-27T07:17:06.767952+0000","last_became_peered":"2024-05-27T07:17:06.767952+0000","last_unstale":"2024-05-27T07:22:22.373962+0000","last_undegraded":"2024-05-27T07:22:22.373962+0000","last_fullsized":"2024-05-27T07:22:22.373962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:34.867113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.369176+0000","last_change":"2024-05-27T07:17:46.704705+0000","last_active":"2024-05-27T07:22:22.369176+0000","last_peered":"2024-05-27T07:22:22.369176+0000","last_clean":"2024-05-27T07:22:22.369176+0000","last_became_active":"2024-05-27T07:17:46.703503+0000","last_became_peered":"2024-05-27T07:17:46.703503+0000","last_unstale":"2024-05-27T07:22:22.369176+0000","last_undegraded":"2024-05-27T07:22:22.369176+0000","last_fullsized":"2024-05-27T07:22:22.369176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:39.975009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.300202+0000","last_change":"2024-05-27T07:17:06.752080+0000","last_active":"2024-05-27T07:22:22.300202+0000","last_peered":"2024-05-27T07:22:22.300202+0000","last_clean":"2024-05-27T07:22:22.300202+0000","last_became_active":"2024-05-27T07:17:06.750486+0000","last_became_peered":"2024-05-27T07:17:06.750486+0000","last_unstale":"2024-05-27T07:22:22.300202+0000","last_undegraded":"2024-05-27T07:22:22.300202+0000","last_fullsized":"2024-05-27T07:22:22.300202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:14:12.461727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.367380+0000","last_change":"2024-05-27T07:17:06.759200+0000","last_active":"2024-05-27T07:22:22.367380+0000","last_peered":"2024-05-27T07:22:22.367380+0000","last_clean":"2024-05-27T07:22:22.367380+0000","last_became_active":"2024-05-27T07:17:06.758819+0000","last_became_peered":"2024-05-27T07:17:06.758819+0000","last_unstale":"2024-05-27T07:22:22.367380+0000","last_undegraded":"2024-05-27T07:22:22.367380+0000","last_fullsized":"2024-05-27T07:22:22.367380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:44:00.274229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.371730+0000","last_change":"2024-05-27T07:17:06.760412+0000","last_active":"2024-05-27T07:22:22.371730+0000","last_peered":"2024-05-27T07:22:22.371730+0000","last_clean":"2024-05-27T07:22:22.371730+0000","last_became_active":"2024-05-27T07:17:06.759949+0000","last_became_peered":"2024-05-27T07:17:06.759949+0000","last_unstale":"2024-05-27T07:22:22.371730+0000","last_undegraded":"2024-05-27T07:22:22.371730+0000","last_fullsized":"2024-05-27T07:22:22.371730+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.235488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.297363+0000","last_change":"2024-05-27T07:17:06.775371+0000","last_active":"2024-05-27T07:22:22.297363+0000","last_peered":"2024-05-27T07:22:22.297363+0000","last_clean":"2024-05-27T07:22:22.297363+0000","last_became_active":"2024-05-27T07:17:06.774904+0000","last_became_peered":"2024-05-27T07:17:06.774904+0000","last_unstale":"2024-05-27T07:22:22.297363+0000","last_undegraded":"2024-05-27T07:22:22.297363+0000","last_fullsized":"2024-05-27T07:22:22.297363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:22:13.484733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:22:22.376311+0000","last_change":"2024-05-27T07:17:06.757066+0000","last_active":"2024-05-27T07:22:22.376311+0000","last_peered":"2024-05-27T07:22:22.376311+0000","last_clean":"2024-05-27T07:22:22.376311+0000","last_became_active":"2024-05-27T07:17:06.756316+0000","last_became_peered":"2024-05-27T07:17:06.756316+0000","last_unstale":"2024-05-27T07:22:22.376311+0000","last_undegraded":"2024-05-27T07:22:22.376311+0000","last_fullsized":"2024-05-27T07:22:22.376311+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:17:05.696367+0000","last_clean_scrub_stamp":"2024-05-27T07:17:05.696367+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:15:12.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598777,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1472,"num_read_kb":847,"num_write":303,"num_write_kb":1448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1796331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710802,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":525814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907031,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33184,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711968,"statfs":{"total":95995035648,"available":95961055232,"internally_reserved":0,"allocated":1871872,"data_stored":1312141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070562,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1454080,"data_stored":922668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266792,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1445888,"data_stored":922476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863463023,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1646592,"data_stored":1122653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626549,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":520532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":588352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":521920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":521170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182542,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":520726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986309,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":596155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822782,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28944,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716208,"statfs":{"total":95995035648,"available":95965396992,"internally_reserved":0,"allocated":1789952,"data_stored":1244911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:22:30.321 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:22:30.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:22:30.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:22:30.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:30.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:30.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:30.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:30.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:30.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:22:30.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:30.796 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: Updating smithi012:/etc/ceph/ceph.conf 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:31.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:31 smithi012 ceph-mon[31473]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 146 B/s wr, 15 op/s 2024-05-27T07:22:31.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:22:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:22:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: Updating smithi012:/etc/ceph/ceph.conf 2024-05-27T07:22:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:22:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:22:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:31.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:31 smithi162 ceph-mon[35676]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 146 B/s wr, 15 op/s 2024-05-27T07:22:31.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:31.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:22:31.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:22:31.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: Updating smithi012:/etc/ceph/ceph.conf 2024-05-27T07:22:31.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: Updating smithi062:/etc/ceph/ceph.conf 2024-05-27T07:22:31.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: Updating smithi162:/etc/ceph/ceph.conf 2024-05-27T07:22:31.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:22:31.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:31 smithi062 ceph-mon[35304]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 146 B/s wr, 15 op/s 2024-05-27T07:22:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:32 smithi012 ceph-mon[31473]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:32 smithi012 ceph-mon[31473]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:32 smithi012 ceph-mon[31473]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:32 smithi012 ceph-mon[31473]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:32.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:32 smithi162 ceph-mon[35676]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:32 smithi162 ceph-mon[35676]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:32 smithi162 ceph-mon[35676]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:32 smithi162 ceph-mon[35676]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:32.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:32 smithi062 ceph-mon[35304]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:32 smithi062 ceph-mon[35304]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:32 smithi062 ceph-mon[35304]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:22:32.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:32 smithi062 ceph-mon[35304]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:33.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:33.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:33.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 114 B/s wr, 12 op/s 2024-05-27T07:22:33.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.183 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.183 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:33 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.325+0000 7f2e92e00700 1 -- 172.21.15.12:0/2239803322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 msgr2=0x7f2e8c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.325+0000 7f2e92e00700 1 --2- 172.21.15.12:0/2239803322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c109320 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2e780099c0 tx=0x7f2e78009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.326+0000 7f2e92e00700 1 -- 172.21.15.12:0/2239803322 shutdown_connections 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.326+0000 7f2e92e00700 1 --2- 172.21.15.12:0/2239803322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e8c109860 0x7f2e8c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.326+0000 7f2e92e00700 1 --2- 172.21.15.12:0/2239803322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e8c1045e0 0x7f2e8c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.326+0000 7f2e92e00700 1 --2- 172.21.15.12:0/2239803322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c109320 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.326+0000 7f2e92e00700 1 -- 172.21.15.12:0/2239803322 >> 172.21.15.12:0/2239803322 conn(0x7f2e8c100030 msgr2=0x7f2e8c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.326+0000 7f2e92e00700 1 -- 172.21.15.12:0/2239803322 shutdown_connections 2024-05-27T07:22:33.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.326+0000 7f2e92e00700 1 -- 172.21.15.12:0/2239803322 wait complete. 2024-05-27T07:22:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.327+0000 7f2e92e00700 1 Processor -- start 2024-05-27T07:22:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.327+0000 7f2e92e00700 1 -- start start 2024-05-27T07:22:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.328+0000 7f2e92e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e8c1045e0 0x7f2e8c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.328+0000 7f2e92e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.328+0000 7f2e92e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e8c109860 0x7f2e8c1a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.328+0000 7f2e92e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c114370 con 0x7f2e8c108f10 2024-05-27T07:22:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.328+0000 7f2e92e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c1141f0 con 0x7f2e8c109860 2024-05-27T07:22:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.328+0000 7f2e92e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c1144f0 con 0x7f2e8c1045e0 2024-05-27T07:22:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.328+0000 7f2e8b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c19d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36206/0 (socket says 172.21.15.12:36206) 2024-05-27T07:22:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8b400700 1 -- 172.21.15.12:0/991401311 learned_addr learned my addr 172.21.15.12:0/991401311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:33.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8be00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e8c1045e0 0x7f2e8c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e90e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e8c109860 0x7f2e8c1a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8b400700 1 -- 172.21.15.12:0/991401311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e8c1045e0 msgr2=0x7f2e8c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8b400700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e8c1045e0 0x7f2e8c19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8b400700 1 -- 172.21.15.12:0/991401311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e8c109860 msgr2=0x7f2e8c1a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8b400700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e8c109860 0x7f2e8c1a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8b400700 1 -- 172.21.15.12:0/991401311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c1a1e90 con 0x7f2e8c108f10 2024-05-27T07:22:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e8be00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e8c1045e0 0x7f2e8c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.329+0000 7f2e90e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e8c109860 0x7f2e8c1a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.330+0000 7f2e8b400700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c19d560 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2e780098e0 tx=0x7f2e7800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.330+0000 7f2e88c00700 1 -- 172.21.15.12:0/991401311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78023ad0 con 0x7f2e8c108f10 2024-05-27T07:22:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.330+0000 7f2e88c00700 1 -- 172.21.15.12:0/991401311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e78023c70 con 0x7f2e8c108f10 2024-05-27T07:22:33.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.330+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8c1a2120 con 0x7f2e8c108f10 2024-05-27T07:22:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.331+0000 7f2e88c00700 1 -- 172.21.15.12:0/991401311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78017810 con 0x7f2e8c108f10 2024-05-27T07:22:33.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.331+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e8c10b170 con 0x7f2e8c108f10 2024-05-27T07:22:33.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.333+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e8c105a90 con 0x7f2e8c108f10 2024-05-27T07:22:33.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.336+0000 7f2e88c00700 1 -- 172.21.15.12:0/991401311 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f2e78017ab0 con 0x7f2e8c108f10 2024-05-27T07:22:33.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.337+0000 7f2e88c00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2e700b2b00 0x7f2e700b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.338+0000 7f2e88c00700 1 -- 172.21.15.12:0/991401311 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2e780ec200 con 0x7f2e8c108f10 2024-05-27T07:22:33.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.338+0000 7f2e8be00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2e700b2b00 0x7f2e700b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.339+0000 7f2e8be00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2e700b2b00 0x7f2e700b5010 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c005950 tx=0x7f2e7c0058e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:33.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.343+0000 7f2e88c00700 1 -- 172.21.15.12:0/991401311 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2e78013070 con 0x7f2e8c108f10 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 114 B/s wr, 12 op/s 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.376 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:33 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: Updating smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: Updating smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: Updating smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 114 B/s wr, 12 op/s 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:33 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.647+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e8c19e320 con 0x7f2e8c108f10 2024-05-27T07:22:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.649+0000 7f2e88c00700 1 -- 172.21.15.12:0/991401311 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11867 (secure 0 0 0) 0x7f2e780b3860 con 0x7f2e8c108f10 2024-05-27T07:22:33.640 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:22:33.640 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:22:21.096387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:22:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2e700b2b00 msgr2=0x7f2e700b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2e700b2b00 0x7f2e700b5010 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c005950 tx=0x7f2e7c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 msgr2=0x7f2e8c19d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c19d560 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2e780098e0 tx=0x7f2e7800f510 comp rx=0 tx=0).stop 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 shutdown_connections 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2e700b2b00 0x7f2e700b5010 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e8c109860 0x7f2e8c1a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e8c1045e0 0x7f2e8c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 --2- 172.21.15.12:0/991401311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e8c108f10 0x7f2e8c19d560 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 >> 172.21.15.12:0/991401311 conn(0x7f2e8c100030 msgr2=0x7f2e8c10a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 shutdown_connections 2024-05-27T07:22:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:33.653+0000 7f2e92e00700 1 -- 172.21.15.12:0/991401311 wait complete. 2024-05-27T07:22:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 27 2024-05-27T07:22:34.215 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/991401311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:34.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/991401311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:34.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/991401311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:34.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:34.554 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:22:35.041 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:35.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:35 smithi012 ceph-mon[31473]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 93 B/s wr, 12 op/s 2024-05-27T07:22:35.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:35 smithi162 ceph-mon[35676]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 93 B/s wr, 12 op/s 2024-05-27T07:22:35.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:35 smithi062 ceph-mon[35304]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 93 B/s wr, 12 op/s 2024-05-27T07:22:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:36 smithi162 ceph-mon[35676]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:22:36.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:36 smithi062 ceph-mon[35304]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:22:36.823 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:36 smithi012 ceph-mon[31473]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:22:37.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.498+0000 7fd93ee00700 1 -- 172.21.15.12:0/1417874291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd938106570 msgr2=0x7fd938106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.498+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1417874291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd938106570 0x7fd938106980 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd924009990 tx=0x7fd924009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.498+0000 7fd93ee00700 1 -- 172.21.15.12:0/1417874291 shutdown_connections 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.498+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1417874291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd938106ec0 0x7fd938113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.498+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1417874291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd93810c5b0 0x7fd93810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.498+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1417874291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd938106570 0x7fd938106980 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.498+0000 7fd93ee00700 1 -- 172.21.15.12:0/1417874291 >> 172.21.15.12:0/1417874291 conn(0x7fd938102070 msgr2=0x7fd9381044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.499+0000 7fd93ee00700 1 -- 172.21.15.12:0/1417874291 shutdown_connections 2024-05-27T07:22:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.499+0000 7fd93ee00700 1 -- 172.21.15.12:0/1417874291 wait complete. 2024-05-27T07:22:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.500+0000 7fd93ee00700 1 Processor -- start 2024-05-27T07:22:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.500+0000 7fd93ee00700 1 -- start start 2024-05-27T07:22:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.501+0000 7fd93ee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd938106570 0x7fd93819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.501+0000 7fd93ee00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd938106ec0 0x7fd93819d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.501+0000 7fd93ee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93810c5b0 0x7fd9381a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.501+0000 7fd93ee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd938116590 con 0x7fd93810c5b0 2024-05-27T07:22:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.501+0000 7fd93ee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd938116410 con 0x7fd938106570 2024-05-27T07:22:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.501+0000 7fd93ee00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd938116710 con 0x7fd938106ec0 2024-05-27T07:22:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93810c5b0 0x7fd9381a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93810c5b0 0x7fd9381a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36232/0 (socket says 172.21.15.12:36232) 2024-05-27T07:22:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 -- 172.21.15.12:0/1990837772 learned_addr learned my addr 172.21.15.12:0/1990837772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd937e00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd938106570 0x7fd93819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd937400700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd938106ec0 0x7fd93819d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 -- 172.21.15.12:0/1990837772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd938106ec0 msgr2=0x7fd93819d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd938106ec0 0x7fd93819d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 -- 172.21.15.12:0/1990837772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd938106570 msgr2=0x7fd93819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd938106570 0x7fd93819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.502+0000 7fd93ce00700 1 -- 172.21.15.12:0/1990837772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9381a1da0 con 0x7fd93810c5b0 2024-05-27T07:22:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.503+0000 7fd93ce00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93810c5b0 0x7fd9381a17b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd930007ee0 tx=0x7fd93000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.503+0000 7fd934c00700 1 -- 172.21.15.12:0/1990837772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930004bb0 con 0x7fd93810c5b0 2024-05-27T07:22:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.503+0000 7fd934c00700 1 -- 172.21.15.12:0/1990837772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd930004d50 con 0x7fd93810c5b0 2024-05-27T07:22:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.503+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9381a2090 con 0x7fd93810c5b0 2024-05-27T07:22:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.503+0000 7fd934c00700 1 -- 172.21.15.12:0/1990837772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9300058b0 con 0x7fd93810c5b0 2024-05-27T07:22:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.503+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9381a2550 con 0x7fd93810c5b0 2024-05-27T07:22:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.505+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9380512b0 con 0x7fd93810c5b0 2024-05-27T07:22:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.509+0000 7fd934c00700 1 -- 172.21.15.12:0/1990837772 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7fd93001a020 con 0x7fd93810c5b0 2024-05-27T07:22:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.510+0000 7fd934c00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd91c0b2a80 0x7fd91c0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.511+0000 7fd934c00700 1 -- 172.21.15.12:0/1990837772 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fd9300e28f0 con 0x7fd93810c5b0 2024-05-27T07:22:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.511+0000 7fd937e00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd91c0b2a80 0x7fd91c0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.512+0000 7fd937e00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd91c0b2a80 0x7fd91c0b4f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9381079e0 tx=0x7fd92800a380 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:37.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.513+0000 7fd934c00700 1 -- 172.21.15.12:0/1990837772 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd930010070 con 0x7fd93810c5b0 2024-05-27T07:22:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.752+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd93802e5e0 con 0x7fd93810c5b0 2024-05-27T07:22:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.756+0000 7fd934c00700 1 -- 172.21.15.12:0/1990837772 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25182 (secure 0 0 0) 0x7fd9300a9d50 con 0x7fd93810c5b0 2024-05-27T07:22:37.746 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:22:37.746 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":90,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:22:22.282989+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/435630590":"2024-05-28T07:22:22.282926+0000","172.21.15.12:6801/2856894494":"2024-05-28T07:22:22.282926+0000","172.21.15.12:6800/2856894494":"2024-05-28T07:22:22.282926+0000","172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/725627344":"2024-05-28T07:22:22.282926+0000","172.21.15.12:0/1619685219":"2024-05-28T07:22:22.282926+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/3259206461":"2024-05-28T07:22:22.282926+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:22:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd91c0b2a80 msgr2=0x7fd91c0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd91c0b2a80 0x7fd91c0b4f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9381079e0 tx=0x7fd92800a380 comp rx=0 tx=0).stop 2024-05-27T07:22:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93810c5b0 msgr2=0x7fd9381a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93810c5b0 0x7fd9381a17b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd930007ee0 tx=0x7fd93000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 shutdown_connections 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd91c0b2a80 0x7fd91c0b4f90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd938106570 0x7fd93819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93810c5b0 0x7fd9381a17b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 --2- 172.21.15.12:0/1990837772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd938106ec0 0x7fd93819d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 >> 172.21.15.12:0/1990837772 conn(0x7fd938102070 msgr2=0x7fd9381047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.760+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 shutdown_connections 2024-05-27T07:22:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:37.761+0000 7fd93ee00700 1 -- 172.21.15.12:0/1990837772 wait complete. 2024-05-27T07:22:38.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:37 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1990837772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:38.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:37 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1990837772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:38.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:37 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1990837772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:38.682 DEBUG:tasks.fwd_scrub:Yielding 2024-05-27T07:22:38.683 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T07:22:38.693 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:22:38.694 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:22:38.694 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-27T07:22:38.694 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-27T07:22:38.694 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:22:38.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:22:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.983+0000 7f57c9400640 1 -- 172.21.15.12:0/3677694562 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410e020 msgr2=0x7f57c4110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.983+0000 7f57c9400640 1 --2- 172.21.15.12:0/3677694562 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410e020 0x7f57c4110510 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009ca0 tx=0x7f57bc02a9e0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.983+0000 7f57c9400640 1 -- 172.21.15.12:0/3677694562 shutdown_connections 2024-05-27T07:22:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.983+0000 7f57c9400640 1 --2- 172.21.15.12:0/3677694562 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410e020 0x7f57c4110510 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.983+0000 7f57c9400640 1 --2- 172.21.15.12:0/3677694562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c410b670 0x7f57c410da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.983+0000 7f57c9400640 1 --2- 172.21.15.12:0/3677694562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c4104c10 0x7f57c4105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.983+0000 7f57c9400640 1 -- 172.21.15.12:0/3677694562 >> 172.21.15.12:0/3677694562 conn(0x7f57c40fe210 msgr2=0x7f57c4100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.984+0000 7f57c9400640 1 -- 172.21.15.12:0/3677694562 shutdown_connections 2024-05-27T07:22:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.984+0000 7f57c9400640 1 -- 172.21.15.12:0/3677694562 wait complete. 2024-05-27T07:22:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 Processor -- start 2024-05-27T07:22:38.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 -- start start 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c4104c10 0x7f57c419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410b670 0x7f57c419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c410e020 0x7f57c41a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4113b80 con 0x7f57c410b670 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4113a00 con 0x7f57c4104c10 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.985+0000 7f57c9400640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4113d00 con 0x7f57c410e020 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410b670 0x7f57c419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410b670 0x7f57c419cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36252/0 (socket says 172.21.15.12:36252) 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 -- 172.21.15.12:0/4029660935 learned_addr learned my addr 172.21.15.12:0/4029660935 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57c3e00640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c4104c10 0x7f57c419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57c8a00640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c410e020 0x7f57c41a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 -- 172.21.15.12:0/4029660935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c410e020 msgr2=0x7f57c41a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c410e020 0x7f57c41a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 -- 172.21.15.12:0/4029660935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c4104c10 msgr2=0x7f57c419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c4104c10 0x7f57c419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57bb400640 1 -- 172.21.15.12:0/4029660935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c41a1620 con 0x7f57c410b670 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.986+0000 7f57c3e00640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c4104c10 0x7f57c419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.987+0000 7f57bb400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410b670 0x7f57c419cd20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00d770 tx=0x7f57ac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.987+0000 7f57c1600640 1 -- 172.21.15.12:0/4029660935 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac017070 con 0x7f57c410b670 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.987+0000 7f57c1600640 1 -- 172.21.15.12:0/4029660935 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57ac0054b0 con 0x7f57c410b670 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.987+0000 7f57c1600640 1 -- 172.21.15.12:0/4029660935 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac004c60 con 0x7f57c410b670 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.987+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c41a1910 con 0x7f57c410b670 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.987+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c41a1e10 con 0x7f57c410b670 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.989+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c406a230 con 0x7f57c410b670 2024-05-27T07:22:38.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.991+0000 7f57c1600640 1 -- 172.21.15.12:0/4029660935 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f57ac004e00 con 0x7f57c410b670 2024-05-27T07:22:38.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.993+0000 7f57c1600640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57a40b29e0 0x7f57a40b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.993+0000 7f57c1600640 1 -- 172.21.15.12:0/4029660935 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f57ac0e2ac0 con 0x7f57c410b670 2024-05-27T07:22:38.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.993+0000 7f57c3e00640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57a40b29e0 0x7f57a40b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.994+0000 7f57c3e00640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57a40b29e0 0x7f57a40b4ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f57b40058c0 tx=0x7f57b4005850 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:38.995+0000 7f57c1600640 1 -- 172.21.15.12:0/4029660935 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f57c41a1e10 con 0x7f57c410b670 2024-05-27T07:22:39.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:38 smithi012 ceph-mon[31473]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:22:39.137 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:38 smithi062 ceph-mon[35304]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:22:39.154 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:39.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.223+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f57c41a9380 con 0x7f57c410b670 2024-05-27T07:22:39.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.229+0000 7f57c1600640 1 -- 172.21.15.12:0/4029660935 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v30)=0 v30) ==== 129+0+0 (secure 0 0 0) 0x7f57ac0a9f20 con 0x7f57c410b670 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57a40b29e0 msgr2=0x7f57a40b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57a40b29e0 0x7f57a40b4ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f57b40058c0 tx=0x7f57b4005850 comp rx=0 tx=0).stop 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410b670 msgr2=0x7f57c419cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410b670 0x7f57c419cd20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00d770 tx=0x7f57ac00dc30 comp rx=0 tx=0).stop 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 shutdown_connections 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c410e020 0x7f57c41a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c410b670 0x7f57c419cd20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57a40b29e0 0x7f57a40b4ef0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 --2- 172.21.15.12:0/4029660935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c4104c10 0x7f57c419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 >> 172.21.15.12:0/4029660935 conn(0x7f57c40fe210 msgr2=0x7f57c4109c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 shutdown_connections 2024-05-27T07:22:39.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.232+0000 7f57c9400640 1 -- 172.21.15.12:0/4029660935 wait complete. 2024-05-27T07:22:39.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:38 smithi162 ceph-mon[35676]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:22:39.254 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 -- 172.21.15.12:0/2686227222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9160109c80 msgr2=0x7f9160103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 --2- 172.21.15.12:0/2686227222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9160109c80 0x7f9160103820 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9150009ca0 tx=0x7f915002a8c0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 -- 172.21.15.12:0/2686227222 shutdown_connections 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 --2- 172.21.15.12:0/2686227222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91601043d0 0x7f91601107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 --2- 172.21.15.12:0/2686227222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f916010a120 0x7f9160103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 --2- 172.21.15.12:0/2686227222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9160109c80 0x7f9160103820 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 -- 172.21.15.12:0/2686227222 >> 172.21.15.12:0/2686227222 conn(0x7f91600ff340 msgr2=0x7f9160101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 -- 172.21.15.12:0/2686227222 shutdown_connections 2024-05-27T07:22:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.523+0000 7f9165000640 1 -- 172.21.15.12:0/2686227222 wait complete. 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.524+0000 7f9165000640 1 Processor -- start 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.524+0000 7f9165000640 1 -- start start 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f9165000640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91601043d0 0x7f91600760d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f9165000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9160109c80 0x7f9160076610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f9165000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f916010a120 0x7f9160076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f9165000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9160113d70 con 0x7f916010a120 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f9165000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9160113bf0 con 0x7f9160109c80 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f9165000640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9160113ef0 con 0x7f91601043d0 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f915e800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f916010a120 0x7f9160076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f915e800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f916010a120 0x7f9160076b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36274/0 (socket says 172.21.15.12:36274) 2024-05-27T07:22:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.525+0000 7f915e800640 1 -- 172.21.15.12:0/2436958508 learned_addr learned my addr 172.21.15.12:0/2436958508 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915de00640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91601043d0 0x7f91600760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915d400640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9160109c80 0x7f9160076610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915e800640 1 -- 172.21.15.12:0/2436958508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91601043d0 msgr2=0x7f91600760d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915e800640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91601043d0 0x7f91600760d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915e800640 1 -- 172.21.15.12:0/2436958508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9160109c80 msgr2=0x7f9160076610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915e800640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9160109c80 0x7f9160076610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915e800640 1 -- 172.21.15.12:0/2436958508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91601a8b00 con 0x7f916010a120 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.526+0000 7f915e800640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f916010a120 0x7f9160076b50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f915800c970 tx=0x7f915800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.527+0000 7f9156a00640 1 -- 172.21.15.12:0/2436958508 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9158007bb0 con 0x7f916010a120 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.527+0000 7f9156a00640 1 -- 172.21.15.12:0/2436958508 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9158007d50 con 0x7f916010a120 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.527+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91601a8df0 con 0x7f916010a120 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.527+0000 7f9156a00640 1 -- 172.21.15.12:0/2436958508 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9158018660 con 0x7f916010a120 2024-05-27T07:22:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.527+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91601a9250 con 0x7f916010a120 2024-05-27T07:22:39.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.528+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9160050860 con 0x7f916010a120 2024-05-27T07:22:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.531+0000 7f9156a00640 1 -- 172.21.15.12:0/2436958508 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f915800f970 con 0x7f916010a120 2024-05-27T07:22:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.532+0000 7f9156a00640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91440b29e0 0x7f91440b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.532+0000 7f9156a00640 1 -- 172.21.15.12:0/2436958508 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f91580e2e70 con 0x7f916010a120 2024-05-27T07:22:39.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.533+0000 7f915de00640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91440b29e0 0x7f91440b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.534+0000 7f915de00640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91440b29e0 0x7f91440b4ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9150000f80 tx=0x7f9150009890 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:39.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.535+0000 7f9156a00640 1 -- 172.21.15.12:0/2436958508 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9158054df0 con 0x7f916010a120 2024-05-27T07:22:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.771+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f9160077600 con 0x7f916010a120 2024-05-27T07:22:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.781+0000 7f9156a00640 1 -- 172.21.15.12:0/2436958508 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7f9158014030 con 0x7f916010a120 2024-05-27T07:22:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91440b29e0 msgr2=0x7f91440b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91440b29e0 0x7f91440b4ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9150000f80 tx=0x7f9150009890 comp rx=0 tx=0).stop 2024-05-27T07:22:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f916010a120 msgr2=0x7f9160076b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f916010a120 0x7f9160076b50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f915800c970 tx=0x7f915800ce30 comp rx=0 tx=0).stop 2024-05-27T07:22:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 shutdown_connections 2024-05-27T07:22:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f916010a120 0x7f9160076b50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91440b29e0 0x7f91440b4ef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9160109c80 0x7f9160076610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 --2- 172.21.15.12:0/2436958508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91601043d0 0x7f91600760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.784+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 >> 172.21.15.12:0/2436958508 conn(0x7f91600ff340 msgr2=0x7f91600ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.785+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 shutdown_connections 2024-05-27T07:22:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:39.785+0000 7f9165000640 1 -- 172.21.15.12:0/2436958508 wait complete. 2024-05-27T07:22:39.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 -- 172.21.15.12:0/1275978963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 msgr2=0x7fefb410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 --2- 172.21.15.12:0/1275978963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb410fea0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009900 tx=0x7fefa002a3f0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 -- 172.21.15.12:0/1275978963 shutdown_connections 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 --2- 172.21.15.12:0/1275978963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefb41103e0 0x7fefb4112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 --2- 172.21.15.12:0/1275978963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb410fea0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 --2- 172.21.15.12:0/1275978963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefb4102840 0x7fefb4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 -- 172.21.15.12:0/1275978963 >> 172.21.15.12:0/1275978963 conn(0x7fefb40fe210 msgr2=0x7fefb4100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 -- 172.21.15.12:0/1275978963 shutdown_connections 2024-05-27T07:22:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.074+0000 7fefb9600640 1 -- 172.21.15.12:0/1275978963 wait complete. 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.075+0000 7fefb9600640 1 Processor -- start 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.075+0000 7fefb9600640 1 -- start start 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb9600640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefb4102840 0x7fefb41a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb9600640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb41a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb9600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefb41103e0 0x7fefb419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb9600640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fefb4115d30 con 0x7fefb410da80 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb9600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fefb4115bb0 con 0x7fefb41103e0 2024-05-27T07:22:40.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb9600640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefb4115eb0 con 0x7fefb4102840 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb41a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb3400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb41a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36290/0 (socket says 172.21.15.12:36290) 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb3400640 1 -- 172.21.15.12:0/103122190 learned_addr learned my addr 172.21.15.12:0/103122190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb3e00640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefb4102840 0x7fefb41a24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.076+0000 7fefb3400640 1 -- 172.21.15.12:0/103122190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefb4102840 msgr2=0x7fefb41a24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.077+0000 7fefb3400640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefb4102840 0x7fefb41a24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.077+0000 7fefb3400640 1 -- 172.21.15.12:0/103122190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefb41103e0 msgr2=0x7fefb419c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.077+0000 7fefb8c00640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefb41103e0 0x7fefb419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.077+0000 7fefb3400640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefb41103e0 0x7fefb419c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.077+0000 7fefb3400640 1 -- 172.21.15.12:0/103122190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb419ccb0 con 0x7fefb410da80 2024-05-27T07:22:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.077+0000 7fefb8c00640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefb41103e0 0x7fefb419c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.077+0000 7fefb3400640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb41a2a30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fefa0004770 tx=0x7fefa0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:40.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.078+0000 7fefb0c00640 1 -- 172.21.15.12:0/103122190 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa003f030 con 0x7fefb410da80 2024-05-27T07:22:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.078+0000 7fefb0c00640 1 -- 172.21.15.12:0/103122190 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fefa0004440 con 0x7fefb410da80 2024-05-27T07:22:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.078+0000 7fefb0c00640 1 -- 172.21.15.12:0/103122190 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0035070 con 0x7fefb410da80 2024-05-27T07:22:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.078+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb419cf40 con 0x7fefb410da80 2024-05-27T07:22:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.078+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb419d480 con 0x7fefb410da80 2024-05-27T07:22:40.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.079+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefb406ad40 con 0x7fefb410da80 2024-05-27T07:22:40.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.082+0000 7fefb0c00640 1 -- 172.21.15.12:0/103122190 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7fefa0042030 con 0x7fefb410da80 2024-05-27T07:22:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.083+0000 7fefb0c00640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef980b2a60 0x7fef980b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.084+0000 7fefb0c00640 1 -- 172.21.15.12:0/103122190 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fefa01033e0 con 0x7fefb410da80 2024-05-27T07:22:40.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.084+0000 7fefb3e00640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef980b2a60 0x7fef980b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.085+0000 7fefb3e00640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef980b2a60 0x7fef980b4f70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fefa4009b60 tx=0x7fefa4005f90 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:40.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.087+0000 7fefb0c00640 1 -- 172.21.15.12:0/103122190 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fefa00ca840 con 0x7fefb410da80 2024-05-27T07:22:40.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.298+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fefb406a230 con 0x7fefb410da80 2024-05-27T07:22:40.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.306+0000 7fefb0c00640 1 -- 172.21.15.12:0/103122190 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7fefa00ca140 con 0x7fefb410da80 2024-05-27T07:22:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef980b2a60 msgr2=0x7fef980b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef980b2a60 0x7fef980b4f70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fefa4009b60 tx=0x7fefa4005f90 comp rx=0 tx=0).stop 2024-05-27T07:22:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 msgr2=0x7fefb41a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb41a2a30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fefa0004770 tx=0x7fefa0004040 comp rx=0 tx=0).stop 2024-05-27T07:22:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 shutdown_connections 2024-05-27T07:22:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef980b2a60 0x7fef980b4f70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefb41103e0 0x7fefb419c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fefb410da80 0x7fefb41a2a30 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 --2- 172.21.15.12:0/103122190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefb4102840 0x7fefb41a24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 >> 172.21.15.12:0/103122190 conn(0x7fefb40fe210 msgr2=0x7fefb4111220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.309+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 shutdown_connections 2024-05-27T07:22:40.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.310+0000 7fefb9600640 1 -- 172.21.15.12:0/103122190 wait complete. 2024-05-27T07:22:40.324 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-27T07:22:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/4029660935' entity='client.admin' 2024-05-27T07:22:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2436958508' entity='client.admin' 2024-05-27T07:22:40.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:40.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:40.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:40.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:40 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/4029660935' entity='client.admin' 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2436958508' entity='client.admin' 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:40.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:40.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:40.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:40 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:40.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/4029660935' entity='client.admin' 2024-05-27T07:22:40.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:40.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2436958508' entity='client.admin' 2024-05-27T07:22:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:40.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:40 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 -- 172.21.15.12:0/3778021139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 msgr2=0x7f6848112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 --2- 172.21.15.12:0/3778021139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f6848112800 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6840009ca0 tx=0x7f684002a8c0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 -- 172.21.15.12:0/3778021139 shutdown_connections 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 --2- 172.21.15.12:0/3778021139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f6848112800 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 --2- 172.21.15.12:0/3778021139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684810c180 0x7f6848105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 --2- 172.21.15.12:0/3778021139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f684810bce0 0x7f6848105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 -- 172.21.15.12:0/3778021139 >> 172.21.15.12:0/3778021139 conn(0x7f6848101380 msgr2=0x7f68481037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 -- 172.21.15.12:0/3778021139 shutdown_connections 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.606+0000 7f684d000640 1 -- 172.21.15.12:0/3778021139 wait complete. 2024-05-27T07:22:40.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.607+0000 7f684d000640 1 Processor -- start 2024-05-27T07:22:40.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.607+0000 7f684d000640 1 -- start start 2024-05-27T07:22:40.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f684d000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f68481a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f684d000640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f684810bce0 0x7f68481a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f684d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684810c180 0x7f684819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f684d000640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6848115df0 con 0x7f6848106430 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f684d000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6848115c70 con 0x7f684810c180 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f684d000640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6848115f70 con 0x7f684810bce0 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f6847400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f68481a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f6847400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f68481a2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36304/0 (socket says 172.21.15.12:36304) 2024-05-27T07:22:40.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f6847400640 1 -- 172.21.15.12:0/3087411882 learned_addr learned my addr 172.21.15.12:0/3087411882 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:40.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f6846a00640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f684810bce0 0x7f68481a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.608+0000 7f6847e00640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684810c180 0x7f684819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f6847400640 1 -- 172.21.15.12:0/3087411882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f684810bce0 msgr2=0x7f68481a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f6847400640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f684810bce0 0x7f68481a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f6847400640 1 -- 172.21.15.12:0/3087411882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684810c180 msgr2=0x7f684819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:40.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f6847400640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684810c180 0x7f684819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:40.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f6847400640 1 -- 172.21.15.12:0/3087411882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684819ce40 con 0x7f6848106430 2024-05-27T07:22:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f6847400640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f68481a2540 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f683800d740 tx=0x7f683800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f683fe00640 1 -- 172.21.15.12:0/3087411882 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838017070 con 0x7f6848106430 2024-05-27T07:22:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f683fe00640 1 -- 172.21.15.12:0/3087411882 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f683800de50 con 0x7f6848106430 2024-05-27T07:22:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684819d130 con 0x7f6848106430 2024-05-27T07:22:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f683fe00640 1 -- 172.21.15.12:0/3087411882 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838004c60 con 0x7f6848106430 2024-05-27T07:22:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.609+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68481a9330 con 0x7f6848106430 2024-05-27T07:22:40.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.611+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6848050860 con 0x7f6848106430 2024-05-27T07:22:40.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.615+0000 7f683fe00640 1 -- 172.21.15.12:0/3087411882 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f6838004e00 con 0x7f6848106430 2024-05-27T07:22:40.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.616+0000 7f683fe00640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68280b2a60 0x7f68280b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:40.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.616+0000 7f683fe00640 1 -- 172.21.15.12:0/3087411882 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f68380e2940 con 0x7f6848106430 2024-05-27T07:22:40.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.617+0000 7f6846a00640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68280b2a60 0x7f68280b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:40.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.617+0000 7f6846a00640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68280b2a60 0x7f68280b4f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f684810a1c0 tx=0x7f6830005f90 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:40.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.619+0000 7f683fe00640 1 -- 172.21.15.12:0/3087411882 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f68380a9e80 con 0x7f6848106430 2024-05-27T07:22:40.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:40.863+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6848063100 con 0x7f68280b2a60 2024-05-27T07:22:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.081+0000 7f683fe00640 1 -- 172.21.15.12:0/3087411882 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6848063100 con 0x7f68280b2a60 2024-05-27T07:22:41.071 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path / 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68280b2a60 msgr2=0x7f68280b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68280b2a60 0x7f68280b4f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f684810a1c0 tx=0x7f6830005f90 comp rx=0 tx=0).stop 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 msgr2=0x7f68481a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f68481a2540 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f683800d740 tx=0x7f683800dc00 comp rx=0 tx=0).stop 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 shutdown_connections 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68280b2a60 0x7f68280b4f70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684810c180 0x7f684819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f684810bce0 0x7f68481a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 --2- 172.21.15.12:0/3087411882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6848106430 0x7f68481a2540 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 >> 172.21.15.12:0/3087411882 conn(0x7f6848101380 msgr2=0x7f68481086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.085+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 shutdown_connections 2024-05-27T07:22:41.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.086+0000 7f684d000640 1 -- 172.21.15.12:0/3087411882 wait complete. 2024-05-27T07:22:41.104 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-27T07:22:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.389+0000 7f1472800640 1 -- 172.21.15.12:0/1301158609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c10c340 msgr2=0x7f146c105f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.389+0000 7f1472800640 1 --2- 172.21.15.12:0/1301158609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c10c340 0x7f146c105f80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1454009ca0 tx=0x7f145402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.389+0000 7f1472800640 1 -- 172.21.15.12:0/1301158609 shutdown_connections 2024-05-27T07:22:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.389+0000 7f1472800640 1 --2- 172.21.15.12:0/1301158609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f146c1065f0 0x7f146c1129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.389+0000 7f1472800640 1 --2- 172.21.15.12:0/1301158609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c10c340 0x7f146c105f80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.389+0000 7f1472800640 1 --2- 172.21.15.12:0/1301158609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f146c10bea0 0x7f146c105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.389+0000 7f1472800640 1 -- 172.21.15.12:0/1301158609 >> 172.21.15.12:0/1301158609 conn(0x7f146c0771f0 msgr2=0x7f146c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.390+0000 7f1472800640 1 -- 172.21.15.12:0/1301158609 shutdown_connections 2024-05-27T07:22:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.390+0000 7f1472800640 1 -- 172.21.15.12:0/1301158609 wait complete. 2024-05-27T07:22:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.391+0000 7f1472800640 1 Processor -- start 2024-05-27T07:22:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.391+0000 7f1472800640 1 -- start start 2024-05-27T07:22:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.391+0000 7f1472800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c1065f0 0x7f146c1a2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.391+0000 7f1472800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f146c10bea0 0x7f146c1a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.391+0000 7f1472800640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f146c10c340 0x7f146c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.391+0000 7f1472800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f146c115f90 con 0x7f146c1065f0 2024-05-27T07:22:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f1472800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f146c115e10 con 0x7f146c10bea0 2024-05-27T07:22:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f1472800640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f146c116110 con 0x7f146c10c340 2024-05-27T07:22:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c1065f0 0x7f146c1a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c1065f0 0x7f146c1a2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36314/0 (socket says 172.21.15.12:36314) 2024-05-27T07:22:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 -- 172.21.15.12:0/3010367823 learned_addr learned my addr 172.21.15.12:0/3010367823 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146be00640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f146c10c340 0x7f146c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146aa00640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f146c10bea0 0x7f146c1a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 -- 172.21.15.12:0/3010367823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f146c10c340 msgr2=0x7f146c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f146c10c340 0x7f146c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 -- 172.21.15.12:0/3010367823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f146c10bea0 msgr2=0x7f146c1a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f146c10bea0 0x7f146c1a2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146b400640 1 -- 172.21.15.12:0/3010367823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146c19cf30 con 0x7f146c1065f0 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.392+0000 7f146be00640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f146c10c340 0x7f146c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.393+0000 7f146b400640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c1065f0 0x7f146c1a2620 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f145c00d770 tx=0x7f145c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.393+0000 7f1463e00640 1 -- 172.21.15.12:0/3010367823 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c017070 con 0x7f146c1065f0 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.393+0000 7f1463e00640 1 -- 172.21.15.12:0/3010367823 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f145c009c30 con 0x7f146c1065f0 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.393+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146c19d220 con 0x7f146c1065f0 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.393+0000 7f1463e00640 1 -- 172.21.15.12:0/3010367823 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c00f620 con 0x7f146c1065f0 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.393+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f146c1a9470 con 0x7f146c1065f0 2024-05-27T07:22:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.395+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f146c107180 con 0x7f146c1065f0 2024-05-27T07:22:41.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.397+0000 7f1463e00640 1 -- 172.21.15.12:0/3010367823 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f145c01a070 con 0x7f146c1065f0 2024-05-27T07:22:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.400+0000 7f1463e00640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f144c0b2a60 0x7f144c0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.400+0000 7f1463e00640 1 -- 172.21.15.12:0/3010367823 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f145c0e26a0 con 0x7f146c1065f0 2024-05-27T07:22:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.400+0000 7f146aa00640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f144c0b2a60 0x7f144c0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.401+0000 7f146aa00640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f144c0b2a60 0x7f144c0b4f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f146c111c80 tx=0x7f1454005fd0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.404+0000 7f1463e00640 1 -- 172.21.15.12:0/3010367823 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f145c009de0 con 0x7f146c1065f0 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 -- 172.21.15.12:0/2362831708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0108d20 msgr2=0x7ff2c0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/2362831708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0108d20 0x7ff2c0109130 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80099c0 tx=0x7ff2a8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 -- 172.21.15.12:0/2362831708 shutdown_connections 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/2362831708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c0109670 0x7ff2c0110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/2362831708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2c01043f0 0x7ff2c0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/2362831708 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0108d20 0x7ff2c0109130 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 -- 172.21.15.12:0/2362831708 >> 172.21.15.12:0/2362831708 conn(0x7ff2c0079d00 msgr2=0x7ff2c007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 -- 172.21.15.12:0/2362831708 shutdown_connections 2024-05-27T07:22:41.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.540+0000 7ff2bfe00700 1 -- 172.21.15.12:0/2362831708 wait complete. 2024-05-27T07:22:41.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:41 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/103122190' entity='client.admin' 2024-05-27T07:22:41.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:41 smithi012 ceph-mon[31473]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-27T07:22:41.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:41 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:41.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:41 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:41.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:41 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:41.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:41 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.541+0000 7ff2bfe00700 1 Processor -- start 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.542+0000 7ff2bfe00700 1 -- start start 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.542+0000 7ff2bfe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2c01043f0 0x7ff2c0075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.542+0000 7ff2bfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c0108d20 0x7ff2c0075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.542+0000 7ff2bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0109670 0x7ff2c0076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.542+0000 7ff2bfe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c0114130 con 0x7ff2c0109670 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.542+0000 7ff2bfe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c0113fb0 con 0x7ff2c0108d20 2024-05-27T07:22:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.542+0000 7ff2bfe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01142b0 con 0x7ff2c01043f0 2024-05-27T07:22:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.543+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0109670 0x7ff2c0076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.543+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0109670 0x7ff2c0076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36330/0 (socket says 172.21.15.12:36330) 2024-05-27T07:22:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.543+0000 7ff2bf400700 1 -- 172.21.15.12:0/1608818962 learned_addr learned my addr 172.21.15.12:0/1608818962 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.543+0000 7ff2b6000700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c0108d20 0x7ff2c0075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.543+0000 7ff2bea00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2c01043f0 0x7ff2c0075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2bf400700 1 -- 172.21.15.12:0/1608818962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2c01043f0 msgr2=0x7ff2c0075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2bf400700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2c01043f0 0x7ff2c0075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2bf400700 1 -- 172.21.15.12:0/1608818962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c0108d20 msgr2=0x7ff2c0075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2bf400700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c0108d20 0x7ff2c0075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2bf400700 1 -- 172.21.15.12:0/1608818962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c01a8f60 con 0x7ff2c0109670 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2bf400700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0109670 0x7ff2c0076530 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8007ee0 tx=0x7ff2b800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2c4a00700 1 -- 172.21.15.12:0/1608818962 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8004bb0 con 0x7ff2c0109670 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2c4a00700 1 -- 172.21.15.12:0/1608818962 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2b8004d50 con 0x7ff2c0109670 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2c4a00700 1 -- 172.21.15.12:0/1608818962 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80058b0 con 0x7ff2c0109670 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.544+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c01a9250 con 0x7ff2c0109670 2024-05-27T07:22:41.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.545+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c01a9710 con 0x7ff2c0109670 2024-05-27T07:22:41.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.549+0000 7ff2c4a00700 1 -- 172.21.15.12:0/1608818962 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7ff2b801a020 con 0x7ff2c0109670 2024-05-27T07:22:41.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 Processor -- start 2024-05-27T07:22:41.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 -- start start 2024-05-27T07:22:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abae13a70 0x557abaea77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaea7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeb3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557abadd4ec0 con 0x557abaeb1340 2024-05-27T07:22:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557abadc4a60 con 0x557abae13a70 2024-05-27T07:22:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.550+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557abadc8aa0 con 0x557abad27ec0 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.551+0000 7ff2b6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeb3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.551+0000 7ff2b6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeb3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36338/0 (socket says 172.21.15.12:36338) 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.551+0000 7ff2b6000700 1 -- 172.21.15.12:0/2377627700 learned_addr learned my addr 172.21.15.12:0/2377627700 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.551+0000 7ff2bea00700 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaea7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.551+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abae13a70 0x557abaea77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b6000700 1 -- 172.21.15.12:0/2377627700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 msgr2=0x557abaea7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaea7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b6000700 1 -- 172.21.15.12:0/2377627700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abae13a70 msgr2=0x557abaea77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abae13a70 0x557abaea77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b6000700 1 -- 172.21.15.12:0/2377627700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557abad8d4a0 con 0x557abaeb1340 2024-05-27T07:22:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeb3760 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80058e0 tx=0x7ff2a801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8ed210eac7cce9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2c4a00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff2a40b2b00 0x7ff2a40b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2c4a00700 1 -- 172.21.15.12:0/1608818962 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7ff2b80e21b0 con 0x7ff2c0109670 2024-05-27T07:22:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b4c00700 1 -- 172.21.15.12:0/2377627700 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80052d0 con 0x557abaeb1340 2024-05-27T07:22:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b4c00700 1 -- 172.21.15.12:0/2377627700 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2a800be40 con 0x557abaeb1340 2024-05-27T07:22:41.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2bea00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff2a40b2b00 0x7ff2a40b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.552+0000 7ff2b4c00700 1 -- 172.21.15.12:0/2377627700 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8003cd0 con 0x557abaeb1340 2024-05-27T07:22:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2377627700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 msgr2=0x557abaeb3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeb3760 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80058e0 tx=0x7ff2a801a040 comp rx=0 tx=0).stop 2024-05-27T07:22:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2bea00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff2a40b2b00 0x7ff2a40b5010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x557abaea39f0 tx=0x7ff2b000b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2377627700 shutdown_connections 2024-05-27T07:22:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abae13a70 0x557abaea77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeb3760 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2377627700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaea7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2377627700 >> 172.21.15.12:0/2377627700 conn(0x557abadc1410 msgr2=0x557abaea2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.554+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2377627700 shutdown_connections 2024-05-27T07:22:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.555+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2377627700 wait complete. 2024-05-27T07:22:41.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.555+0000 7ff2d4899b80 1 -- start start 2024-05-27T07:22:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.555+0000 7ff2d4899b80 1 Processor -- start 2024-05-27T07:22:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaeae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abae13a70 0x557abaeae820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaeaed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557abaccb020 con 0x557abae13a70 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557abad657e0 con 0x557abaeb1340 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557abaeb3e20 con 0x557abad27ec0 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2d4899b80 10 client.0 _ll_register_callbacks cb 0x557abad6ecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.556+0000 7ff2b6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abae13a70 0x557abaeae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2b6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abae13a70 0x557abaeae820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36348/0 (socket says 172.21.15.12:36348) 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2b6000700 1 -- 172.21.15.12:0/2860751662 learned_addr learned my addr 172.21.15.12:0/2860751662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:41.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2bea00700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaeae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2bf400700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaeaed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2b6000700 1 -- 172.21.15.12:0/2860751662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 msgr2=0x557abaeae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaeae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2b6000700 1 -- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 msgr2=0x557abaeaed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaeaed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2b6000700 1 -- 172.21.15.12:0/2860751662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557abad14950 con 0x557abae13a70 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.557+0000 7ff2bf400700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaeaed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abae13a70 0x557abaeae820 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098e0 tx=0x7ff2a800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b90e58b9d595c66 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2ad600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2ad600700 1 -- 172.21.15.12:0/2860751662 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8005360 con 0x557abae13a70 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2ad600700 1 -- 172.21.15.12:0/2860751662 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7ff2a8003950 con 0x557abae13a70 2024-05-27T07:22:41.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557abadd6e10 con 0x557abae13a70 2024-05-27T07:22:41.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2ad600700 1 -- 172.21.15.12:0/2860751662 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff2a803a050 con 0x557abae13a70 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2d4899b80 10 client.15414 fetch_fsmap learned FSMap version 27 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.558+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557abad950e0 con 0x557abae13a70 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2ad600700 1 -- 172.21.15.12:0/2860751662 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x557abadd6e10 con 0x557abae13a70 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2ad600700 1 -- 172.21.15.12:0/2860751662 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff2a8026b10 con 0x557abae13a70 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2d4899b80 10 client.15414 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2d4899b80 10 client.15414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2d4899b80 20 client.15414 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaec0c00 0x557abaec3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2d4899b80 4 client.15414 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.559+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557abadd05f0 con 0x557abaec0c00 2024-05-27T07:22:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.560+0000 7ff2bea00700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaec0c00 0x557abaec3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.561+0000 7ff2bea00700 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaec0c00 0x557abaec3020 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a5d3bd834942750 server_cookie=81419c687002be30 in_seq=0 out_seq=0 2024-05-27T07:22:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.561+0000 7ff2ad600700 10 client.15414 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.563+0000 7ff2ad600700 1 -- 172.21.15.12:0/2860751662 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557abadd05f0 con 0x557abaec0c00 2024-05-27T07:22:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.563+0000 7ff2ad600700 10 client.15414 handle_command_reply: tid=0 2024-05-27T07:22:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 client.15414 shutdown 2024-05-27T07:22:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abae13a70 msgr2=0x557abaeae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abae13a70 0x557abaeae820 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098e0 tx=0x7ff2a800b630 comp rx=0 tx=0).stop 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 shutdown_connections 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaec0c00 0x557abaec3020 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaeaed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abad27ec0 0x557abaeae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2860751662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abae13a70 0x557abaeae820 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2ad600700 0 client.15414 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 >> 172.21.15.12:0/2860751662 conn(0x557abadc1410 msgr2=0x557abaea7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 shutdown_connections 2024-05-27T07:22:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2860751662 wait complete. 2024-05-27T07:22:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.564+0000 7ff2d4899b80 20 client.15414 trim_cache size 0 max 16384 2024-05-27T07:22:41.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 Processor -- start 2024-05-27T07:22:41.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 -- start start 2024-05-27T07:22:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abad27ec0 0x557abaea27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaeb00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01b4e50 con 0x557abaeb1340 2024-05-27T07:22:41.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01b26e0 con 0x557abad27ec0 2024-05-27T07:22:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.588+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01b0530 con 0x557abae13a70 2024-05-27T07:22:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36356/0 (socket says 172.21.15.12:36356) 2024-05-27T07:22:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 -- 172.21.15.12:0/2567964477 learned_addr learned my addr 172.21.15.12:0/2567964477 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2b6000700 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaeb00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bea00700 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abad27ec0 0x557abaea27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 -- 172.21.15.12:0/2567964477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 msgr2=0x557abaeb00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaeb00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 -- 172.21.15.12:0/2567964477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abad27ec0 msgr2=0x557abaea27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abad27ec0 0x557abaea27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.589+0000 7ff2bf400700 1 -- 172.21.15.12:0/2567964477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557abaea8de0 con 0x557abaeb1340 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.590+0000 7ff2bf400700 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeab540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80a4f90 tx=0x7ff2b80a8860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c544944b1e2348 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.590+0000 7ff2afe00700 1 -- 172.21.15.12:0/2567964477 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80a9610 con 0x557abaeb1340 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.590+0000 7ff2afe00700 1 -- 172.21.15.12:0/2567964477 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2b80a8f10 con 0x557abaeb1340 2024-05-27T07:22:41.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.590+0000 7ff2afe00700 1 -- 172.21.15.12:0/2567964477 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80a9170 con 0x557abaeb1340 2024-05-27T07:22:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2567964477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 msgr2=0x557abaeab540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeab540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80a4f90 tx=0x7ff2b80a8860 comp rx=0 tx=0).stop 2024-05-27T07:22:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2567964477 shutdown_connections 2024-05-27T07:22:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abad27ec0 0x557abaea27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abaeb1340 0x557abaeab540 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/2567964477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaeb00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2567964477 >> 172.21.15.12:0/2567964477 conn(0x557abadc1410 msgr2=0x557abaeaf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2567964477 shutdown_connections 2024-05-27T07:22:41.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.591+0000 7ff2d4899b80 1 -- 172.21.15.12:0/2567964477 wait complete. 2024-05-27T07:22:41.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.592+0000 7ff2d4899b80 1 -- start start 2024-05-27T07:22:41.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.592+0000 7ff2d4899b80 1 Processor -- start 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abad27ec0 0x557abaec5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaec6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2d4899b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaec44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01b0ba0 con 0x557abad27ec0 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01b2010 con 0x557abaeb1340 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2d4899b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01b1270 con 0x557abae13a70 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2d4899b80 10 client.0 _ll_register_callbacks cb 0x557abad6ecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abad27ec0 0x557abaec5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abad27ec0 0x557abaec5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36370/0 (socket says 172.21.15.12:36370) 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2bea00700 1 -- 172.21.15.12:0/1218115735 learned_addr learned my addr 172.21.15.12:0/1218115735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2b6000700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaec6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.593+0000 7ff2bf400700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaec44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2bea00700 1 -- 172.21.15.12:0/1218115735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 msgr2=0x557abaec6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2bea00700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaec6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2bea00700 1 -- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 msgr2=0x557abaec44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2bea00700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaec44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2bea00700 1 -- 172.21.15.12:0/1218115735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557abaeaf0d0 con 0x557abad27ec0 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2bf400700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaec44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2bea00700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abad27ec0 0x557abaec5fe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x557abaeadb60 tx=0x7ff2b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd32c6dacc72ef47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2ad600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.594+0000 7ff2ad600700 1 -- 172.21.15.12:0/1218115735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0003e00 con 0x557abad27ec0 2024-05-27T07:22:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.595+0000 7ff2ad600700 1 -- 172.21.15.12:0/1218115735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7ff2b001eb90 con 0x557abad27ec0 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.595+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2b80a8f10 con 0x557abad27ec0 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.595+0000 7ff2ad600700 1 -- 172.21.15.12:0/1218115735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff2b003b050 con 0x557abad27ec0 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.595+0000 7ff2d4899b80 10 client.15420 fetch_fsmap learned FSMap version 27 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.595+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557abaf2bc40 con 0x557abad27ec0 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.595+0000 7ff2ad600700 1 -- 172.21.15.12:0/1218115735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2b80a8f10 con 0x557abad27ec0 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.595+0000 7ff2ad600700 1 -- 172.21.15.12:0/1218115735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff2b001d410 con 0x557abad27ec0 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.596+0000 7ff2d4899b80 10 client.15420 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.596+0000 7ff2d4899b80 10 client.15420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.596+0000 7ff2d4899b80 20 client.15420 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.596+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaf2c330 0x557abaf2e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.596+0000 7ff2d4899b80 4 client.15420 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:22:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.596+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557abac472d0 con 0x557abaf2c330 2024-05-27T07:22:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.596+0000 7ff2b6000700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaf2c330 0x557abaf2e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.597+0000 7ff2b6000700 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaf2c330 0x557abaf2e750 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a01ab375f5542a server_cookie=683900c6cbe18007 in_seq=0 out_seq=0 2024-05-27T07:22:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.597+0000 7ff2ad600700 10 client.15420 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2ad600700 1 -- 172.21.15.12:0/1218115735 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557abac472d0 con 0x557abaf2c330 2024-05-27T07:22:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2ad600700 10 client.15420 handle_command_reply: tid=0 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 client.15420 shutdown 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abad27ec0 msgr2=0x557abaec5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abad27ec0 0x557abaec5fe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x557abaeadb60 tx=0x7ff2b0004750 comp rx=0 tx=0).stop 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 shutdown_connections 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557abaf2c330 0x557abaf2e750 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557abaeb1340 0x557abaec44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557abad27ec0 0x557abaec5fe0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 --2- 172.21.15.12:0/1218115735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557abae13a70 0x557abaec6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2ad600700 0 client.15420 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 >> 172.21.15.12:0/1218115735 conn(0x557abadc1410 msgr2=0x557abaeaf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.601+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 shutdown_connections 2024-05-27T07:22:41.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.602+0000 7ff2d4899b80 1 -- 172.21.15.12:0/1218115735 wait complete. 2024-05-27T07:22:41.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.602+0000 7ff2d4899b80 20 client.15420 trim_cache size 0 max 16384 2024-05-27T07:22:41.592 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:22:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:22:41.592 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "28de32dd-3c49-4274-9640-dc96317593f0", 2024-05-27T07:22:41.593 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:22:41.593 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:22:41.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff2a40b2b00 msgr2=0x7ff2a40b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff2a40b2b00 0x7ff2a40b5010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x557abaea39f0 tx=0x7ff2b000b410 comp rx=0 tx=0).stop 2024-05-27T07:22:41.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0109670 msgr2=0x7ff2c0076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0109670 0x7ff2c0076530 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8007ee0 tx=0x7ff2b800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 shutdown_connections 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff2a40b2b00 0x7ff2a40b5010 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2c0109670 0x7ff2c0076530 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2c01043f0 0x7ff2c0075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 --2- 172.21.15.12:0/1608818962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c0108d20 0x7ff2c0075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 >> 172.21.15.12:0/1608818962 conn(0x7ff2c0079d00 msgr2=0x7ff2c0101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 shutdown_connections 2024-05-27T07:22:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.603+0000 7ff2bfe00700 1 -- 172.21.15.12:0/1608818962 wait complete. 2024-05-27T07:22:41.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.656+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f146c063100 con 0x7f144c0b2a60 2024-05-27T07:22:41.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:41 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/103122190' entity='client.admin' 2024-05-27T07:22:41.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:41 smithi162 ceph-mon[35676]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-27T07:22:41.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:41 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:41.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:41 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:41.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:41 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:41.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:41 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:41.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:41 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/103122190' entity='client.admin' 2024-05-27T07:22:41.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:41 smithi062 ceph-mon[35304]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-27T07:22:41.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:41 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:41.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:41 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:41.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:41 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:41.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:41 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:22:41.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.835+0000 7f1463e00640 1 -- 172.21.15.12:0/3010367823 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f146c063100 con 0x7f144c0b2a60 2024-05-27T07:22:41.825 INFO:teuthology.orchestra.run.smithi012.stdout:Retention added to path / 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f144c0b2a60 msgr2=0x7f144c0b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f144c0b2a60 0x7f144c0b4f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f146c111c80 tx=0x7f1454005fd0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c1065f0 msgr2=0x7f146c1a2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c1065f0 0x7f146c1a2620 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f145c00d770 tx=0x7f145c00dc30 comp rx=0 tx=0).stop 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 shutdown_connections 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f146c10c340 0x7f146c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f144c0b2a60 0x7f144c0b4f70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f146c10bea0 0x7f146c1a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 --2- 172.21.15.12:0/3010367823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f146c1065f0 0x7f146c1a2620 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.838+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 >> 172.21.15.12:0/3010367823 conn(0x7f146c0771f0 msgr2=0x7f146c107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.839+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 shutdown_connections 2024-05-27T07:22:41.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:41.839+0000 7f1472800640 1 -- 172.21.15.12:0/3010367823 wait complete. 2024-05-27T07:22:41.859 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T07:22:42.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 -- 172.21.15.12:0/2244241703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c10c180 msgr2=0x7f765c105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 --2- 172.21.15.12:0/2244241703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c10c180 0x7f765c105dc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f764c009900 tx=0x7f764c02a3f0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 -- 172.21.15.12:0/2244241703 shutdown_connections 2024-05-27T07:22:42.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 --2- 172.21.15.12:0/2244241703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c106430 0x7f765c112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 --2- 172.21.15.12:0/2244241703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c10c180 0x7f765c105dc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 --2- 172.21.15.12:0/2244241703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f765c10bce0 0x7f765c105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 -- 172.21.15.12:0/2244241703 >> 172.21.15.12:0/2244241703 conn(0x7f765c101380 msgr2=0x7f765c1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 -- 172.21.15.12:0/2244241703 shutdown_connections 2024-05-27T07:22:42.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.147+0000 7f7663200640 1 -- 172.21.15.12:0/2244241703 wait complete. 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.148+0000 7f7663200640 1 Processor -- start 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7663200640 1 -- start start 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7663200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c106430 0x7f765c1a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7663200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c10bce0 0x7f765c1a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7663200640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f765c10c180 0x7f765c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7663200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f765c115dd0 con 0x7f765c106430 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7663200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f765c115c50 con 0x7f765c10bce0 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7663200640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f765c115f50 con 0x7f765c10c180 2024-05-27T07:22:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7661e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c106430 0x7f765c1a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7661e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c106430 0x7f765c1a2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40012/0 (socket says 172.21.15.12:40012) 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.149+0000 7f7661e00640 1 -- 172.21.15.12:0/3723201153 learned_addr learned my addr 172.21.15.12:0/3723201153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7661400640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c10bce0 0x7f765c1a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7661e00640 1 -- 172.21.15.12:0/3723201153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f765c10c180 msgr2=0x7f765c19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7661e00640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f765c10c180 0x7f765c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7661e00640 1 -- 172.21.15.12:0/3723201153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c10bce0 msgr2=0x7f765c1a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7661e00640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c10bce0 0x7f765c1a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7661e00640 1 -- 172.21.15.12:0/3723201153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765c19cdc0 con 0x7f765c106430 2024-05-27T07:22:42.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7661e00640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c106430 0x7f765c1a2510 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f765000d740 tx=0x7f765000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.150+0000 7f7656a00640 1 -- 172.21.15.12:0/3723201153 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650017070 con 0x7f765c106430 2024-05-27T07:22:42.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.151+0000 7f7656a00640 1 -- 172.21.15.12:0/3723201153 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f765000de50 con 0x7f765c106430 2024-05-27T07:22:42.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.151+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765c19d050 con 0x7f765c106430 2024-05-27T07:22:42.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.151+0000 7f7656a00640 1 -- 172.21.15.12:0/3723201153 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650004c60 con 0x7f765c106430 2024-05-27T07:22:42.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.151+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765c1a92b0 con 0x7f765c106430 2024-05-27T07:22:42.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.152+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765c106fc0 con 0x7f765c106430 2024-05-27T07:22:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.154+0000 7f7656a00640 1 -- 172.21.15.12:0/3723201153 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f7650004e00 con 0x7f765c106430 2024-05-27T07:22:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.156+0000 7f7656a00640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f76440b29e0 0x7f76440b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.156+0000 7f7656a00640 1 -- 172.21.15.12:0/3723201153 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f76500e29c0 con 0x7f765c106430 2024-05-27T07:22:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.156+0000 7f7661400640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f76440b29e0 0x7f76440b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.157+0000 7f7661400640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f76440b29e0 0x7f76440b4ef0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f764c009230 tx=0x7f764c005fd0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.159+0000 7f7656a00640 1 -- 172.21.15.12:0/3723201153 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f76500a9e20 con 0x7f765c106430 2024-05-27T07:22:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.404+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f765c103f20 con 0x7f76440b29e0 2024-05-27T07:22:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.424+0000 7f7656a00640 1 -- 172.21.15.12:0/3723201153 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f764c004390 con 0x7f76440b29e0 2024-05-27T07:22:42.413 INFO:teuthology.orchestra.run.smithi012.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:22:40", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f76440b29e0 msgr2=0x7f76440b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f76440b29e0 0x7f76440b4ef0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f764c009230 tx=0x7f764c005fd0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c106430 msgr2=0x7f765c1a2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c106430 0x7f765c1a2510 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f765000d740 tx=0x7f765000dc00 comp rx=0 tx=0).stop 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 shutdown_connections 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f765c10c180 0x7f765c19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f76440b29e0 0x7f76440b4ef0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c10bce0 0x7f765c1a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 --2- 172.21.15.12:0/3723201153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f765c106430 0x7f765c1a2510 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 >> 172.21.15.12:0/3723201153 conn(0x7f765c101380 msgr2=0x7f765c111e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 shutdown_connections 2024-05-27T07:22:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.427+0000 7f7663200640 1 -- 172.21.15.12:0/3723201153 wait complete. 2024-05-27T07:22:42.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T07:22:42.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:22:42.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:42 smithi012 ceph-mon[31473]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:42.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:42 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:22:42.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:42 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:42.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:42 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:22:42.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.724+0000 7f28f5c00640 1 -- 172.21.15.12:0/3447483638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010f540 msgr2=0x7f28f0116ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.724+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3447483638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010f540 0x7f28f0116ce0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009ca0 tx=0x7f28e802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.725+0000 7f28f5c00640 1 -- 172.21.15.12:0/3447483638 shutdown_connections 2024-05-27T07:22:42.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.725+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3447483638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010f540 0x7f28f0116ce0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.725+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3447483638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28f010ea80 0x7f28f010eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.725+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3447483638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f010a500 0x7f28f010a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.725+0000 7f28f5c00640 1 -- 172.21.15.12:0/3447483638 >> 172.21.15.12:0/3447483638 conn(0x7f28f0077190 msgr2=0x7f28f00775d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.725+0000 7f28f5c00640 1 -- 172.21.15.12:0/3447483638 shutdown_connections 2024-05-27T07:22:42.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.725+0000 7f28f5c00640 1 -- 172.21.15.12:0/3447483638 wait complete. 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.726+0000 7f28f5c00640 1 Processor -- start 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.726+0000 7f28f5c00640 1 -- start start 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.726+0000 7f28f5c00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010a500 0x7f28f01a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28f5c00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28f010ea80 0x7f28f01a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28f5c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f010f540 0x7f28f01a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28f5c00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28f011a330 con 0x7f28f010a500 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28f5c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f28f011a1b0 con 0x7f28f010f540 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28f5c00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f28f011a4b0 con 0x7f28f010ea80 2024-05-27T07:22:42.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010a500 0x7f28f01a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010a500 0x7f28f01a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40034/0 (socket says 172.21.15.12:40034) 2024-05-27T07:22:42.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 -- 172.21.15.12:0/3787856308 learned_addr learned my addr 172.21.15.12:0/3787856308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 -- 172.21.15.12:0/3787856308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28f010ea80 msgr2=0x7f28f01a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28f010ea80 0x7f28f01a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 -- 172.21.15.12:0/3787856308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f010f540 msgr2=0x7f28f01a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28ef400640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f010f540 0x7f28f01a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f010f540 0x7f28f01a5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.727+0000 7f28eea00640 1 -- 172.21.15.12:0/3787856308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f01a5bd0 con 0x7f28f010a500 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.728+0000 7f28eea00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010a500 0x7f28f01a0d70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f28e000d950 tx=0x7f28e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.728+0000 7f28e7400640 1 -- 172.21.15.12:0/3787856308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0017070 con 0x7f28f010a500 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.728+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f01a5ec0 con 0x7f28f010a500 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.728+0000 7f28e7400640 1 -- 172.21.15.12:0/3787856308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28e0009c30 con 0x7f28f010a500 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.728+0000 7f28e7400640 1 -- 172.21.15.12:0/3787856308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e000f620 con 0x7f28f010a500 2024-05-27T07:22:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.728+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f01ad840 con 0x7f28f010a500 2024-05-27T07:22:42.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.730+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f0050860 con 0x7f28f010a500 2024-05-27T07:22:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.732+0000 7f28e7400640 1 -- 172.21.15.12:0/3787856308 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f28e0010460 con 0x7f28f010a500 2024-05-27T07:22:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:42 smithi162 ceph-mon[35676]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:42 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:22:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:42 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:42 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:22:42.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.734+0000 7f28e7400640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28d40b29e0 0x7f28d40b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.735+0000 7f28e7400640 1 -- 172.21.15.12:0/3787856308 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f28e00e28b0 con 0x7f28f010a500 2024-05-27T07:22:42.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.735+0000 7f28e7e00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28d40b29e0 0x7f28d40b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.736+0000 7f28e7e00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28d40b29e0 0x7f28d40b4ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f28d8005d80 tx=0x7f28d8005c90 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:42.739+0000 7f28e7400640 1 -- 172.21.15.12:0/3787856308 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f28e00a9d10 con 0x7f28f010a500 2024-05-27T07:22:42.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:42 smithi062 ceph-mon[35304]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:42 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:22:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:42 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:42 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:22:42.949 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:42.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.000+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f28f0063100 con 0x7f28d40b29e0 2024-05-27T07:22:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.012+0000 7f28e7400640 1 -- 172.21.15.12:0/3787856308 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f28f0063100 con 0x7f28d40b29e0 2024-05-27T07:22:43.002 INFO:teuthology.orchestra.run.smithi012.stdout:/ 1m 6m3h 2024-05-27T07:22:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28d40b29e0 msgr2=0x7f28d40b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28d40b29e0 0x7f28d40b4ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f28d8005d80 tx=0x7f28d8005c90 comp rx=0 tx=0).stop 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010a500 msgr2=0x7f28f01a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010a500 0x7f28f01a0d70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f28e000d950 tx=0x7f28e000de10 comp rx=0 tx=0).stop 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 shutdown_connections 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28d40b29e0 0x7f28d40b4ef0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f010f540 0x7f28f01a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28f010ea80 0x7f28f01a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 --2- 172.21.15.12:0/3787856308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28f010a500 0x7f28f01a0d70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 >> 172.21.15.12:0/3787856308 conn(0x7f28f0077190 msgr2=0x7f28f007a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 shutdown_connections 2024-05-27T07:22:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:43.015+0000 7f28f5c00640 1 -- 172.21.15.12:0/3787856308 wait complete. 2024-05-27T07:22:43.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-27T07:22:43.100 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-27T07:22:43.112 INFO:teuthology.run_tasks:Running task workunit... 2024-05-27T07:22:43.152 INFO:tasks.workunit:Pulling workunits from ref 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:22:43.152 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-27T07:22:43.152 INFO:tasks.workunit:timeout=3h 2024-05-27T07:22:43.152 INFO:tasks.workunit:cleanup=True 2024-05-27T07:22:43.153 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:43.212 INFO:teuthology.orchestra.run.smithi012.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:fusefs_t:s0 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-05-27 07:18:51.121325627 +0000 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-05-27 07:18:51.121325627 +0000 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-05-27 07:22:20.539959386 +0000 2024-05-27T07:22:43.213 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-05-27T07:22:43.213 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-27T07:22:43.214 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-27T07:22:43.316 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:22:43.370 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-27T07:22:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:43 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:43 smithi012 ceph-mon[31473]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:43 smithi012 ceph-mon[31473]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:22:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:43 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:43 smithi162 ceph-mon[35676]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:43 smithi162 ceph-mon[35676]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:22:43.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:43 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:43 smithi062 ceph-mon[35304]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:43 smithi062 ceph-mon[35304]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:22:44.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:44 smithi162 ceph-mon[35676]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:44.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:44 smithi162 ceph-mon[35676]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:44 smithi012 ceph-mon[31473]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:44 smithi012 ceph-mon[31473]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:44 smithi062 ceph-mon[35304]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:44.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:44 smithi062 ceph-mon[35304]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:22:45.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 -- 172.21.15.12:0/3695167525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe128106570 msgr2=0x7fe128106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 --2- 172.21.15.12:0/3695167525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe128106570 0x7fe128106980 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe114009990 tx=0x7fe114009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 -- 172.21.15.12:0/3695167525 shutdown_connections 2024-05-27T07:22:45.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 --2- 172.21.15.12:0/3695167525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe128106ec0 0x7fe128113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 --2- 172.21.15.12:0/3695167525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe12810c5b0 0x7fe12810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 --2- 172.21.15.12:0/3695167525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe128106570 0x7fe128106980 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 -- 172.21.15.12:0/3695167525 >> 172.21.15.12:0/3695167525 conn(0x7fe128102070 msgr2=0x7fe1281044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 -- 172.21.15.12:0/3695167525 shutdown_connections 2024-05-27T07:22:45.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.470+0000 7fe12e800700 1 -- 172.21.15.12:0/3695167525 wait complete. 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.472+0000 7fe12e800700 1 Processor -- start 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.472+0000 7fe12e800700 1 -- start start 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe12e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe128106570 0x7fe12819d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe12e800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe128106ec0 0x7fe12819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe12e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe12810c5b0 0x7fe1281a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe12e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe128116590 con 0x7fe12810c5b0 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe12e800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe128116410 con 0x7fe128106570 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe12e800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe128116710 con 0x7fe128106ec0 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe12810c5b0 0x7fe1281a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe12810c5b0 0x7fe1281a1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40054/0 (socket says 172.21.15.12:40054) 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe127e00700 1 -- 172.21.15.12:0/730234823 learned_addr learned my addr 172.21.15.12:0/730234823 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:45.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.473+0000 7fe127400700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe128106570 0x7fe12819d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.474+0000 7fe126a00700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe128106ec0 0x7fe12819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.474+0000 7fe127e00700 1 -- 172.21.15.12:0/730234823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe128106ec0 msgr2=0x7fe12819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.474+0000 7fe127e00700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe128106ec0 0x7fe12819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.474+0000 7fe127e00700 1 -- 172.21.15.12:0/730234823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe128106570 msgr2=0x7fe12819d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.474+0000 7fe127e00700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe128106570 0x7fe12819d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.474+0000 7fe127e00700 1 -- 172.21.15.12:0/730234823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1281a1f30 con 0x7fe12810c5b0 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.474+0000 7fe127e00700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe12810c5b0 0x7fe1281a1890 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe120007ee0 tx=0x7fe12000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.475+0000 7fe11fe00700 1 -- 172.21.15.12:0/730234823 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12000dd20 con 0x7fe12810c5b0 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.475+0000 7fe11fe00700 1 -- 172.21.15.12:0/730234823 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe120004020 con 0x7fe12810c5b0 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.475+0000 7fe11fe00700 1 -- 172.21.15.12:0/730234823 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120015660 con 0x7fe12810c5b0 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.475+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1281a2220 con 0x7fe12810c5b0 2024-05-27T07:22:45.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.475+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1281a9b40 con 0x7fe12810c5b0 2024-05-27T07:22:45.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.479+0000 7fe11fe00700 1 -- 172.21.15.12:0/730234823 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7fe1200041d0 con 0x7fe12810c5b0 2024-05-27T07:22:45.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.480+0000 7fe13d420b80 1 Processor -- start 2024-05-27T07:22:45.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.480+0000 7fe13d420b80 1 -- start start 2024-05-27T07:22:45.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac23152e40 0x55ac232d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe11fe00700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe10c0b2a80 0x7fe10c0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac232d1db0 0x55ac232d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe13d420b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ac231ffe40 con 0x55ac232d7d30 2024-05-27T07:22:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe13d420b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ac231ef9e0 con 0x55ac23152e40 2024-05-27T07:22:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe13d420b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac231f3a20 con 0x55ac232d1db0 2024-05-27T07:22:45.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe11fe00700 1 -- 172.21.15.12:0/730234823 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe1200e2590 con 0x7fe12810c5b0 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe127400700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe10c0b2a80 0x7fe10c0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe126a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe126a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40058/0 (socket says 172.21.15.12:40058) 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe126a00700 1 -- 172.21.15.12:0/3374370451 learned_addr learned my addr 172.21.15.12:0/3374370451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe127e00700 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac232d1db0 0x55ac232d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.481+0000 7fe126a00700 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac23152e40 0x55ac232d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.482+0000 7fe126a00700 1 -- 172.21.15.12:0/3374370451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac232d1db0 msgr2=0x55ac232d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.482+0000 7fe126a00700 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac232d1db0 0x55ac232d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.482+0000 7fe126a00700 1 -- 172.21.15.12:0/3374370451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac23152e40 msgr2=0x55ac232d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.482+0000 7fe126a00700 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac23152e40 0x55ac232d77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.482+0000 7fe126a00700 1 -- 172.21.15.12:0/3374370451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac231b8420 con 0x55ac232d7d30 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.482+0000 7fe127400700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe10c0b2a80 0x7fe10c0b4f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe118005950 tx=0x7fe11800b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.482+0000 7fe126a00700 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232d1870 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe128075b70 tx=0x7fe11401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daae0bc6465bba9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.483+0000 7fe11cc00700 1 -- 172.21.15.12:0/3374370451 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11400ba70 con 0x55ac232d7d30 2024-05-27T07:22:45.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.483+0000 7fe11cc00700 1 -- 172.21.15.12:0/3374370451 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe114005500 con 0x55ac232d7d30 2024-05-27T07:22:45.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.483+0000 7fe11cc00700 1 -- 172.21.15.12:0/3374370451 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114003a50 con 0x55ac232d7d30 2024-05-27T07:22:45.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 -- 172.21.15.12:0/3374370451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 msgr2=0x55ac232d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232d1870 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe128075b70 tx=0x7fe11401a040 comp rx=0 tx=0).stop 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 -- 172.21.15.12:0/3374370451 shutdown_connections 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac23152e40 0x55ac232d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac232d1db0 0x55ac232d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3374370451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232d1870 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 -- 172.21.15.12:0/3374370451 >> 172.21.15.12:0/3374370451 conn(0x55ac231b2fc0 msgr2=0x55ac232cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 -- 172.21.15.12:0/3374370451 shutdown_connections 2024-05-27T07:22:45.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.484+0000 7fe13d420b80 1 -- 172.21.15.12:0/3374370451 wait complete. 2024-05-27T07:22:45.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.485+0000 7fe13d420b80 1 -- start start 2024-05-27T07:22:45.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.485+0000 7fe13d420b80 1 Processor -- start 2024-05-27T07:22:45.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232db520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232dba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232dbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe13d420b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ac230f5fa0 con 0x55ac232d7d30 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe13d420b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ac23190760 con 0x55ac232d1db0 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe13d420b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ac232de7a0 con 0x55ac23152e40 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe13d420b80 10 client.0 _ll_register_callbacks cb 0x55ac23199c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232dbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232dbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40068/0 (socket says 172.21.15.12:40068) 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe127e00700 1 -- 172.21.15.12:0/1120118961 learned_addr learned my addr 172.21.15.12:0/1120118961 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe127400700 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232db520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.486+0000 7fe126a00700 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232dba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe127e00700 1 -- 172.21.15.12:0/1120118961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 msgr2=0x55ac232db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe127e00700 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232db520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe127e00700 1 -- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 msgr2=0x55ac232dba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe127e00700 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232dba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe127e00700 1 -- 172.21.15.12:0/1120118961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac2313f8d0 con 0x55ac232d7d30 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe127e00700 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232dbfa0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe1200a6560 tx=0x7fe120004720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7ec129ef7d040e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe111600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe111600700 1 -- 172.21.15.12:0/1120118961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200a8e00 con 0x55ac232d7d30 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe111600700 1 -- 172.21.15.12:0/1120118961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe1200a9760 con 0x55ac232d7d30 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.487+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe114005500 con 0x55ac232d7d30 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe111600700 1 -- 172.21.15.12:0/1120118961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe1200a99f0 con 0x55ac232d7d30 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 10 client.15444 fetch_fsmap learned FSMap version 27 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac231c0060 con 0x55ac232d7d30 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe111600700 1 -- 172.21.15.12:0/1120118961 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe114005500 con 0x55ac232d7d30 2024-05-27T07:22:45.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe111600700 1 -- 172.21.15.12:0/1120118961 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe120004bd0 con 0x55ac232d7d30 2024-05-27T07:22:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 10 client.15444 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:22:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 10 client.15444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 20 client.15444 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac232eb5a0 0x55ac232ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 4 client.15444 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.488+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac23201d90 con 0x55ac232eb5a0 2024-05-27T07:22:45.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.489+0000 7fe127400700 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac232eb5a0 0x55ac232ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.490+0000 7fe127400700 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac232eb5a0 0x55ac232ed9c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfbfecca63a98809 server_cookie=634805318d7b82e4 in_seq=0 out_seq=0 2024-05-27T07:22:45.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.490+0000 7fe111600700 10 client.15444 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.492+0000 7fe111600700 1 -- 172.21.15.12:0/1120118961 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ac23201d90 con 0x55ac232eb5a0 2024-05-27T07:22:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.492+0000 7fe111600700 10 client.15444 handle_command_reply: tid=0 2024-05-27T07:22:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 client.15444 shutdown 2024-05-27T07:22:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 msgr2=0x55ac232dbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232dbfa0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe1200a6560 tx=0x7fe120004720 comp rx=0 tx=0).stop 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 shutdown_connections 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232dbfa0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232db520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac232eb5a0 0x55ac232ed9c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 --2- 172.21.15.12:0/1120118961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232dba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 >> 172.21.15.12:0/1120118961 conn(0x55ac231b2fc0 msgr2=0x55ac232cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe111600700 0 client.15444 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 shutdown_connections 2024-05-27T07:22:45.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 1 -- 172.21.15.12:0/1120118961 wait complete. 2024-05-27T07:22:45.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.493+0000 7fe13d420b80 20 client.15444 trim_cache size 0 max 16384 2024-05-27T07:22:45.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.516+0000 7fe13d420b80 1 Processor -- start 2024-05-27T07:22:45.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.516+0000 7fe13d420b80 1 -- start start 2024-05-27T07:22:45.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232d5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe13d420b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281b5010 con 0x55ac232d7d30 2024-05-27T07:22:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe13d420b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281b28a0 con 0x55ac232d1db0 2024-05-27T07:22:45.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe13d420b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281b06f0 con 0x55ac23152e40 2024-05-27T07:22:45.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40084/0 (socket says 172.21.15.12:40084) 2024-05-27T07:22:45.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe127e00700 1 -- 172.21.15.12:0/4105903295 learned_addr learned my addr 172.21.15.12:0/4105903295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:45.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe127400700 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.517+0000 7fe126a00700 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232d5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe127e00700 1 -- 172.21.15.12:0/4105903295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 msgr2=0x55ac232d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe127e00700 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe127e00700 1 -- 172.21.15.12:0/4105903295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 msgr2=0x55ac232d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe127e00700 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232d5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe127e00700 1 -- 172.21.15.12:0/4105903295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac232d9ab0 con 0x55ac232d7d30 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe127400700 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe127e00700 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e7840 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe120004800 tx=0x7fe1200a8d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00d961ad7c0fd2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe113e00700 1 -- 172.21.15.12:0/4105903295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200a92f0 con 0x55ac232d7d30 2024-05-27T07:22:45.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe113e00700 1 -- 172.21.15.12:0/4105903295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1200ad6a0 con 0x55ac232d7d30 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.518+0000 7fe113e00700 1 -- 172.21.15.12:0/4105903295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120004bd0 con 0x55ac232d7d30 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 -- 172.21.15.12:0/4105903295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 msgr2=0x55ac232e7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e7840 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe120004800 tx=0x7fe1200a8d30 comp rx=0 tx=0).stop 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 -- 172.21.15.12:0/4105903295 shutdown_connections 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e7840 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 --2- 172.21.15.12:0/4105903295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232d5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 -- 172.21.15.12:0/4105903295 >> 172.21.15.12:0/4105903295 conn(0x55ac231b2fc0 msgr2=0x55ac232d8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 -- 172.21.15.12:0/4105903295 shutdown_connections 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.519+0000 7fe13d420b80 1 -- 172.21.15.12:0/4105903295 wait complete. 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.520+0000 7fe13d420b80 1 -- start start 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.520+0000 7fe13d420b80 1 Processor -- start 2024-05-27T07:22:45.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232e3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232e3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe13d420b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe13d420b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281b0d60 con 0x55ac232d7d30 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe13d420b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281b21d0 con 0x55ac232d1db0 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe13d420b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281b1430 con 0x55ac23152e40 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe13d420b80 10 client.0 _ll_register_callbacks cb 0x55ac23199c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe127e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40086/0 (socket says 172.21.15.12:40086) 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe127e00700 1 -- 172.21.15.12:0/3979947766 learned_addr learned my addr 172.21.15.12:0/3979947766 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe127400700 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232e3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.521+0000 7fe126a00700 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232e3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe127e00700 1 -- 172.21.15.12:0/3979947766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 msgr2=0x55ac232e3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe127e00700 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232e3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe127e00700 1 -- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 msgr2=0x55ac232e3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe127e00700 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232e3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe127e00700 1 -- 172.21.15.12:0/3979947766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac232e0f40 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe127e00700 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e0a00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe1200a6560 tx=0x7fe1200b2200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9320e4cedc9a734c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe111600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe111600700 1 -- 172.21.15.12:0/3979947766 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200a92f0 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe111600700 1 -- 172.21.15.12:0/3979947766 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fe1200a58e0 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1200ad6a0 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.522+0000 7fe111600700 1 -- 172.21.15.12:0/3979947766 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe1200b1cc0 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 10 client.15450 fetch_fsmap learned FSMap version 27 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac2335ee90 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe111600700 1 -- 172.21.15.12:0/3979947766 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1200ad6a0 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe111600700 1 -- 172.21.15.12:0/3979947766 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe1200ad2e0 con 0x55ac232d7d30 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 10 client.15450 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:22:45.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 10 client.15450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 20 client.15450 populate_metadata read hostname 'smithi012' 2024-05-27T07:22:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac2335f2a0 0x55ac233616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 4 client.15450 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:22:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.523+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac231fb570 con 0x55ac2335f2a0 2024-05-27T07:22:45.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.524+0000 7fe127400700 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac2335f2a0 0x55ac233616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.525+0000 7fe127400700 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac2335f2a0 0x55ac233616c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de30aa03211463ea server_cookie=cebbd3fd5fc96453 in_seq=0 out_seq=0 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.525+0000 7fe111600700 10 client.15450 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.525+0000 7fe111600700 1 -- 172.21.15.12:0/3979947766 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ac231fb570 con 0x55ac2335f2a0 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.525+0000 7fe111600700 10 client.15450 handle_command_reply: tid=0 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.525+0000 7fe13d420b80 1 client.15450 shutdown 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.525+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 msgr2=0x55ac232e0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.525+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e0a00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe1200a6560 tx=0x7fe1200b2200 comp rx=0 tx=0).stop 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 shutdown_connections 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ac232d7d30 0x55ac232e0a00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ac23152e40 0x55ac232e3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ac2335f2a0 0x55ac233616c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 --2- 172.21.15.12:0/3979947766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac232d1db0 0x55ac232e3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 >> 172.21.15.12:0/3979947766 conn(0x55ac231b2fc0 msgr2=0x55ac232da860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe111600700 0 client.15450 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:22:45.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 shutdown_connections 2024-05-27T07:22:45.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 1 -- 172.21.15.12:0/3979947766 wait complete. 2024-05-27T07:22:45.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.526+0000 7fe13d420b80 20 client.15450 trim_cache size 0 max 16384 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout: "28de32dd-3c49-4274-9640-dc96317593f0": { 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "28de32dd-3c49-4274-9640-dc96317593f0", 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:22:45.520 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:22:45.521 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:22:45.521 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:22:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe10c0b2a80 msgr2=0x7fe10c0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe10c0b2a80 0x7fe10c0b4f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe118005950 tx=0x7fe11800b410 comp rx=0 tx=0).stop 2024-05-27T07:22:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe12810c5b0 msgr2=0x7fe1281a1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe12810c5b0 0x7fe1281a1890 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe120007ee0 tx=0x7fe12000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 shutdown_connections 2024-05-27T07:22:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe10c0b2a80 0x7fe10c0b4f90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe128106570 0x7fe12819d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe12810c5b0 0x7fe1281a1890 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 --2- 172.21.15.12:0/730234823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe128106ec0 0x7fe12819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 >> 172.21.15.12:0/730234823 conn(0x7fe128102070 msgr2=0x7fe1281049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 shutdown_connections 2024-05-27T07:22:45.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:22:45.527+0000 7fe12e800700 1 -- 172.21.15.12:0/730234823 wait complete. 2024-05-27T07:22:45.586 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:45 smithi012 ceph-mon[31473]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-27T07:22:45.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:45 smithi162 ceph-mon[35676]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-27T07:22:45.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:45 smithi062 ceph-mon[35304]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-27T07:22:46.524 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:46 smithi012 ceph-mon[31473]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:46.527 INFO:tasks.cephfs.filesystem:scrub status for tag:28de32dd-3c49-4274-9640-dc96317593f0 - {'path': '/', 'tag': '28de32dd-3c49-4274-9640-dc96317593f0', 'options': 'recursive,force'} 2024-05-27T07:22:46.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:46 smithi162 ceph-mon[35676]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:46.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:46 smithi062 ceph-mon[35304]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:48.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:48 smithi162 ceph-mon[35676]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:48.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:48 smithi062 ceph-mon[35304]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:48.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:48 smithi012 ceph-mon[31473]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:50.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:50 smithi162 ceph-mon[35676]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-27T07:22:50.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:50 smithi062 ceph-mon[35304]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-27T07:22:50.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:50 smithi012 ceph-mon[31473]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-27T07:22:52.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:52 smithi162 ceph-mon[35676]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:52.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:52 smithi062 ceph-mon[35304]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:52.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:52 smithi012 ceph-mon[31473]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-27T07:22:54.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:54 smithi162 ceph-mon[35676]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 40 op/s 2024-05-27T07:22:54.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:54 smithi062 ceph-mon[35304]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 40 op/s 2024-05-27T07:22:54.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:54 smithi012 ceph-mon[31473]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 40 op/s 2024-05-27T07:22:55.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:55 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:22:55.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:55 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:22:56.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:55 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:22:56.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:56 smithi162 ceph-mon[35676]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-27T07:22:56.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:56 smithi062 ceph-mon[35304]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-27T07:22:57.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:56 smithi012 ceph-mon[31473]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-27T07:22:58.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:22:58 smithi162 ceph-mon[35676]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-27T07:22:58.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:22:58 smithi062 ceph-mon[35304]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-27T07:22:58.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:22:58 smithi012 ceph-mon[31473]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-27T07:23:00.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:00 smithi162 ceph-mon[35676]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 7 op/s 2024-05-27T07:23:00.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:00 smithi062 ceph-mon[35304]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 7 op/s 2024-05-27T07:23:00.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:00 smithi012 ceph-mon[31473]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 7 op/s 2024-05-27T07:23:01.341 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:23:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:23:00.866+0000 7f93c9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.343 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:23:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:23:00.866+0000 7f93c9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.343 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:23:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:23:00.866+0000 7f93c9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.344 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:23:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:23:00.866+0000 7f93c9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.344 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:23:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:23:00.866+0000 7f93c9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:02 smithi162 ceph-mon[35676]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 596 B/s wr, 5 op/s 2024-05-27T07:23:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:02 smithi162 ceph-mon[35676]: mgrmap e25: x(active, since 40s), standbys: z, y 2024-05-27T07:23:02.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:02 smithi062 ceph-mon[35304]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 596 B/s wr, 5 op/s 2024-05-27T07:23:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:02 smithi062 ceph-mon[35304]: mgrmap e25: x(active, since 40s), standbys: z, y 2024-05-27T07:23:02.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:02 smithi012 ceph-mon[31473]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 596 B/s wr, 5 op/s 2024-05-27T07:23:02.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:02 smithi012 ceph-mon[31473]: mgrmap e25: x(active, since 40s), standbys: z, y 2024-05-27T07:23:04.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:04 smithi162 ceph-mon[35676]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T07:23:04.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:04 smithi062 ceph-mon[35304]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T07:23:04.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:04 smithi012 ceph-mon[31473]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T07:23:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:07 smithi162 ceph-mon[35676]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:23:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:07 smithi062 ceph-mon[35304]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:23:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:07 smithi012 ceph-mon[31473]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:23:09.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:09 smithi162 ceph-mon[35676]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:23:09.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:09 smithi062 ceph-mon[35304]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:23:09.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:09 smithi012 ceph-mon[31473]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:23:11.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:11 smithi162 ceph-mon[35676]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-27T07:23:11.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:11 smithi062 ceph-mon[35304]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-27T07:23:11.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:11 smithi012 ceph-mon[31473]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-27T07:23:13.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:13 smithi162 ceph-mon[35676]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:23:13.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:13 smithi062 ceph-mon[35304]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:23:13.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:13 smithi012 ceph-mon[31473]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:23:15.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:15 smithi162 ceph-mon[35676]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:23:15.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:15 smithi062 ceph-mon[35304]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:23:15.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:15 smithi012 ceph-mon[31473]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:23:16.528 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:23:17.007 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:17.568 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:17 smithi012 ceph-mon[31473]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-27T07:23:17.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:17 smithi162 ceph-mon[35676]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-27T07:23:17.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:17 smithi062 ceph-mon[35304]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-27T07:23:19.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:19 smithi012 ceph-mon[31473]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 -- 172.21.15.12:0/416958726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc109860 msgr2=0x7fb0fc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 --2- 172.21.15.12:0/416958726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc109860 0x7fb0fc1111c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400b0d0 tx=0x7fb0f400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 -- 172.21.15.12:0/416958726 shutdown_connections 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 --2- 172.21.15.12:0/416958726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0fc1045e0 0x7fb0fc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 --2- 172.21.15.12:0/416958726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc109860 0x7fb0fc1111c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 --2- 172.21.15.12:0/416958726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0fc108f10 0x7fb0fc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 -- 172.21.15.12:0/416958726 >> 172.21.15.12:0/416958726 conn(0x7fb0fc100030 msgr2=0x7fb0fc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 -- 172.21.15.12:0/416958726 shutdown_connections 2024-05-27T07:23:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.544+0000 7fb103a00700 1 -- 172.21.15.12:0/416958726 wait complete. 2024-05-27T07:23:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.545+0000 7fb103a00700 1 Processor -- start 2024-05-27T07:23:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.546+0000 7fb103a00700 1 -- start start 2024-05-27T07:23:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.546+0000 7fb103a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0fc1045e0 0x7fb0fc19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.546+0000 7fb103a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc108f10 0x7fb0fc19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb103a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0fc109860 0x7fb0fc19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb103a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc114340 con 0x7fb0fc108f10 2024-05-27T07:23:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb103a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1141c0 con 0x7fb0fc109860 2024-05-27T07:23:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb103a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1144c0 con 0x7fb0fc1045e0 2024-05-27T07:23:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb0fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc108f10 0x7fb0fc19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb0fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc108f10 0x7fb0fc19b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51904/0 (socket says 172.21.15.12:51904) 2024-05-27T07:23:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2029817347 learned_addr learned my addr 172.21.15.12:0/2029817347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb101000700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0fc1045e0 0x7fb0fc19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb101a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0fc109860 0x7fb0fc19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.547+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2029817347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0fc1045e0 msgr2=0x7fb0fc19ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0fc1045e0 0x7fb0fc19ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2029817347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0fc109860 msgr2=0x7fb0fc19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0fc109860 0x7fb0fc19f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2029817347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc19fd20 con 0x7fb0fc108f10 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc108f10 0x7fb0fc19b3f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800d980 tx=0x7fb0e800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0f9600700 1 -- 172.21.15.12:0/2029817347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8017070 con 0x7fb0fc108f10 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0f9600700 1 -- 172.21.15.12:0/2029817347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0e8010460 con 0x7fb0fc108f10 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0fc1a0010 con 0x7fb0fc108f10 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.548+0000 7fb0f9600700 1 -- 172.21.15.12:0/2029817347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e800de60 con 0x7fb0fc108f10 2024-05-27T07:23:19.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.549+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0fc1a7930 con 0x7fb0fc108f10 2024-05-27T07:23:19.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.552+0000 7fb0f9600700 1 -- 172.21.15.12:0/2029817347 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7fb0e8010610 con 0x7fb0fc108f10 2024-05-27T07:23:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.553+0000 7fb1124e0b80 1 Processor -- start 2024-05-27T07:23:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb1124e0b80 1 -- start start 2024-05-27T07:23:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb0f9600700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb0e40b2b10 0x7fb0e40b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104b3fe40 0x565104cc8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb0f9600700 1 -- 172.21.15.12:0/2029817347 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb0e80e27c0 con 0x7fb0fc108f10 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104cc2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc31c0 0x565104cc35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x565104bece40 con 0x565104b3fe40 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565104bdc9e0 con 0x565104cc31c0 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.554+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565104be0a20 con 0x565104cc90e0 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb0e40b2b10 0x7fb0e40b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104b3fe40 0x565104cc8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104b3fe40 0x565104cc8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51906/0 (socket says 172.21.15.12:51906) 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101000700 1 -- 172.21.15.12:0/3081220191 learned_addr learned my addr 172.21.15.12:0/3081220191 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104cc2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101000700 1 -- 172.21.15.12:0/3081220191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 msgr2=0x565104cc2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101000700 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104cc2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101000700 1 -- 172.21.15.12:0/3081220191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc31c0 msgr2=0x565104cc35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.555+0000 7fb101000700 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc31c0 0x565104cc35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.556+0000 7fb101000700 1 -- 172.21.15.12:0/3081220191 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565104ba5420 con 0x565104b3fe40 2024-05-27T07:23:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.556+0000 7fb101a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb0e40b2b10 0x7fb0e40b5020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009900 tx=0x7fb0f4009c80 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.556+0000 7fb101000700 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104b3fe40 0x565104cc8ba0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000bd70 tx=0x7fb0f000f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a3a3d9882871e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.556+0000 7fb0ee000700 1 -- 172.21.15.12:0/3081220191 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0015070 con 0x565104b3fe40 2024-05-27T07:23:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.556+0000 7fb0ee000700 1 -- 172.21.15.12:0/3081220191 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0f000fd20 con 0x565104b3fe40 2024-05-27T07:23:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.556+0000 7fb0ee000700 1 -- 172.21.15.12:0/3081220191 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f00116a0 con 0x565104b3fe40 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.557+0000 7fb1124e0b80 1 -- 172.21.15.12:0/3081220191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104b3fe40 msgr2=0x565104cc8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.557+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104b3fe40 0x565104cc8ba0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000bd70 tx=0x7fb0f000f640 comp rx=0 tx=0).stop 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 -- 172.21.15.12:0/3081220191 shutdown_connections 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc31c0 0x565104cc35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104b3fe40 0x565104cc8ba0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/3081220191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104cc2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 -- 172.21.15.12:0/3081220191 >> 172.21.15.12:0/3081220191 conn(0x565104b9ffc0 msgr2=0x565104cb98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 -- 172.21.15.12:0/3081220191 shutdown_connections 2024-05-27T07:23:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 -- 172.21.15.12:0/3081220191 wait complete. 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.558+0000 7fb1124e0b80 1 -- start start 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 1 Processor -- start 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104b3fe40 0x565104ccc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104ccce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc90e0 0x565104ccd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x565104ae2fa0 con 0x565104cc31c0 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565104b7d760 con 0x565104cc90e0 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565104ccfbc0 con 0x565104b3fe40 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.559+0000 7fb1124e0b80 10 client.0 _ll_register_callbacks cb 0x565104b86c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104ccce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104ccce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51914/0 (socket says 172.21.15.12:51914) 2024-05-27T07:23:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 -- 172.21.15.12:0/1900865691 learned_addr learned my addr 172.21.15.12:0/1900865691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb101000700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104b3fe40 0x565104ccc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb101a00700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc90e0 0x565104ccd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 -- 172.21.15.12:0/1900865691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104b3fe40 msgr2=0x565104ccc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104b3fe40 0x565104ccc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 -- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc90e0 msgr2=0x565104ccd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc90e0 0x565104ccd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 -- 172.21.15.12:0/1900865691 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565104b2c8d0 con 0x565104cc31c0 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb101a00700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc90e0 0x565104ccd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104ccce30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80a9140 tx=0x7fb0e80a9980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61fc07444abb51f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.560+0000 7fb0e2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.561+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1900865691 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8020030 con 0x565104cc31c0 2024-05-27T07:23:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.561+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1900865691 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb0e80dfc60 con 0x565104cc31c0 2024-05-27T07:23:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.561+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0f000fd20 con 0x565104cc31c0 2024-05-27T07:23:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.561+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1900865691 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0e8068e60 con 0x565104cc31c0 2024-05-27T07:23:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.561+0000 7fb1124e0b80 10 client.15468 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.561+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565104bad060 con 0x565104cc31c0 2024-05-27T07:23:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.561+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1900865691 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0f000fd20 con 0x565104cc31c0 2024-05-27T07:23:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1900865691 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb0e80e2070 con 0x565104cc31c0 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb1124e0b80 10 client.15468 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb1124e0b80 10 client.15468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb1124e0b80 20 client.15468 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104cdcad0 0x565104cdef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb1124e0b80 4 client.15468 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565104beed90 con 0x565104cdcad0 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.562+0000 7fb101000700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104cdcad0 0x565104cdef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.563+0000 7fb101000700 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104cdcad0 0x565104cdef20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c026083801365dc6 server_cookie=97a9804013a2399c in_seq=0 out_seq=0 2024-05-27T07:23:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.563+0000 7fb0e2a00700 10 client.15468 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.565+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1900865691 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x565104beed90 con 0x565104cdcad0 2024-05-27T07:23:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.565+0000 7fb0e2a00700 10 client.15468 handle_command_reply: tid=0 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 client.15468 shutdown 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 msgr2=0x565104ccce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104ccce30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80a9140 tx=0x7fb0e80a9980 comp rx=0 tx=0).stop 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 shutdown_connections 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104cdcad0 0x565104cdef20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104cc90e0 0x565104ccd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104b3fe40 0x565104ccc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1900865691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104ccce30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 >> 172.21.15.12:0/1900865691 conn(0x565104b9ffc0 msgr2=0x565104cb9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb0e2a00700 0 client.15468 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 shutdown_connections 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1900865691 wait complete. 2024-05-27T07:23:19.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.566+0000 7fb1124e0b80 20 client.15468 trim_cache size 0 max 16384 2024-05-27T07:23:19.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.589+0000 7fb1124e0b80 1 Processor -- start 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.589+0000 7fb1124e0b80 1 -- start start 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104cd8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104cbdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104ccc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1b0c20 con 0x565104cc31c0 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1ae4b0 con 0x565104b3fe40 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1ac300 con 0x565104cc90e0 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb0fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104cbdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb0fbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104cbdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51928/0 (socket says 172.21.15.12:51928) 2024-05-27T07:23:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2977118490 learned_addr learned my addr 172.21.15.12:0/2977118490 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb101000700 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104cd8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.590+0000 7fb101a00700 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104ccc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2977118490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 msgr2=0x565104ccc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104ccc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2977118490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 msgr2=0x565104cd8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104cd8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0fbe00700 1 -- 172.21.15.12:0/2977118490 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565104cbae00 con 0x565104cc31c0 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104cbdf70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80a93d0 tx=0x7fb0e80a8ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e465445bd9bcb3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0ed600700 1 -- 172.21.15.12:0/2977118490 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8020070 con 0x565104cc31c0 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0ed600700 1 -- 172.21.15.12:0/2977118490 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb0e80dff30 con 0x565104cc31c0 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.591+0000 7fb0ed600700 1 -- 172.21.15.12:0/2977118490 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8009b40 con 0x565104cc31c0 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 -- 172.21.15.12:0/2977118490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 msgr2=0x565104cbdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104cbdf70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80a93d0 tx=0x7fb0e80a8ec0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 -- 172.21.15.12:0/2977118490 shutdown_connections 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104cd8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc90e0 0x565104ccc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/2977118490 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc31c0 0x565104cbdf70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 -- 172.21.15.12:0/2977118490 >> 172.21.15.12:0/2977118490 conn(0x565104b9ffc0 msgr2=0x565104cbd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.592+0000 7fb1124e0b80 1 -- 172.21.15.12:0/2977118490 shutdown_connections 2024-05-27T07:23:19.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.593+0000 7fb1124e0b80 1 -- 172.21.15.12:0/2977118490 wait complete. 2024-05-27T07:23:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.593+0000 7fb1124e0b80 1 -- start start 2024-05-27T07:23:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.593+0000 7fb1124e0b80 1 Processor -- start 2024-05-27T07:23:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104ce42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc31c0 0x565104ce4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb1124e0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc90e0 0x565104ce4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1ac970 con 0x565104cc90e0 2024-05-27T07:23:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1adde0 con 0x565104b3fe40 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb1124e0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1ad040 con 0x565104cc31c0 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb1124e0b80 10 client.0 _ll_register_callbacks cb 0x565104b86c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb101a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc90e0 0x565104ce4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb101a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc90e0 0x565104ce4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51932/0 (socket says 172.21.15.12:51932) 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb101a00700 1 -- 172.21.15.12:0/1250747826 learned_addr learned my addr 172.21.15.12:0/1250747826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb101000700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104ce42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.594+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc31c0 0x565104ce4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb101a00700 1 -- 172.21.15.12:0/1250747826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc31c0 msgr2=0x565104ce4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb101a00700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc31c0 0x565104ce4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb101a00700 1 -- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 msgr2=0x565104ce42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb101a00700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104ce42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb101a00700 1 -- 172.21.15.12:0/1250747826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565104d46600 con 0x565104cc90e0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb101000700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104ce42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb0fbe00700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc31c0 0x565104ce4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb101a00700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc90e0 0x565104ce4d50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4015000 tx=0x7fb0f400b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf2e6a6b6409c2b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.595+0000 7fb0e2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.596+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1250747826 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f401ed00 con 0x565104cc90e0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.596+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1250747826 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb0f402c460 con 0x565104cc90e0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.596+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565104be8570 con 0x565104cc90e0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.596+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1250747826 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0f402c660 con 0x565104cc90e0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.596+0000 7fb1124e0b80 10 client.15474 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.596+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565104d47820 con 0x565104cc90e0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.596+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1250747826 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x565104be8570 con 0x565104cc90e0 2024-05-27T07:23:19.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1250747826 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb0f4011070 con 0x565104cc90e0 2024-05-27T07:23:19.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb1124e0b80 10 client.15474 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:19.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb1124e0b80 10 client.15474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:19.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb1124e0b80 20 client.15474 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:19.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104d47f20 0x565104d4a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb1124e0b80 4 client.15474 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:23:19.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565104a5f250 con 0x565104d47f20 2024-05-27T07:23:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.597+0000 7fb101000700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104d47f20 0x565104d4a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.598+0000 7fb101000700 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104d47f20 0x565104d4a340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=429137ba0d7f94a5 server_cookie=e3b454861964bc70 in_seq=0 out_seq=0 2024-05-27T07:23:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.598+0000 7fb0e2a00700 10 client.15474 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb0e2a00700 1 -- 172.21.15.12:0/1250747826 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565104a5f250 con 0x565104d47f20 2024-05-27T07:23:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb0e2a00700 10 client.15474 handle_command_reply: tid=0 2024-05-27T07:23:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 client.15474 shutdown 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc90e0 msgr2=0x565104ce4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc90e0 0x565104ce4d50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4015000 tx=0x7fb0f400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 shutdown_connections 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x565104d47f20 0x565104d4a340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565104b3fe40 0x565104ce42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565104cc90e0 0x565104ce4d50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 --2- 172.21.15.12:0/1250747826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565104cc31c0 0x565104ce4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 >> 172.21.15.12:0/1250747826 conn(0x565104b9ffc0 msgr2=0x565104cc6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb0e2a00700 0 client.15474 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.599+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 shutdown_connections 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb1124e0b80 1 -- 172.21.15.12:0/1250747826 wait complete. 2024-05-27T07:23:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb1124e0b80 20 client.15474 trim_cache size 0 max 16384 2024-05-27T07:23:19.593 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:23:19.593 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:23:19.593 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:23:19.593 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb0e40b2b10 msgr2=0x7fb0e40b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb103a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb0e40b2b10 0x7fb0e40b5020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009900 tx=0x7fb0f4009c80 comp rx=0 tx=0).stop 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc108f10 msgr2=0x7fb0fc19b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb103a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc108f10 0x7fb0fc19b3f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800d980 tx=0x7fb0e800dd40 comp rx=0 tx=0).stop 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 shutdown_connections 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.600+0000 7fb103a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb0e40b2b10 0x7fb0e40b5020 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.601+0000 7fb103a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0fc109860 0x7fb0fc19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.601+0000 7fb103a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0fc1045e0 0x7fb0fc19ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.601+0000 7fb103a00700 1 --2- 172.21.15.12:0/2029817347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0fc108f10 0x7fb0fc19b3f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.601+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 >> 172.21.15.12:0/2029817347 conn(0x7fb0fc100030 msgr2=0x7fb0fc108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.601+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 shutdown_connections 2024-05-27T07:23:19.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:19.601+0000 7fb103a00700 1 -- 172.21.15.12:0/2029817347 wait complete. 2024-05-27T07:23:19.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:19 smithi162 ceph-mon[35676]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-27T07:23:19.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:19 smithi062 ceph-mon[35304]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-27T07:23:20.660 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:23:20.661 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:23:21.167 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:21.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:21 smithi012 ceph-mon[31473]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:23:21.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:21 smithi162 ceph-mon[35676]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:23:21.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:21 smithi062 ceph-mon[35304]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:23:23.519 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:23 smithi012 ceph-mon[31473]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.642+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3879440075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 msgr2=0x7fdfd810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.642+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3879440075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd810ad90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009990 tx=0x7fdfc4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.642+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3879440075 shutdown_connections 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.642+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3879440075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfd810b2d0 0x7fdfd8117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.642+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3879440075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd8110960 0x7fdfd8110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.642+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3879440075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd810ad90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.642+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3879440075 >> 172.21.15.12:0/3879440075 conn(0x7fdfd8077c70 msgr2=0x7fdfd80780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.643+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3879440075 shutdown_connections 2024-05-27T07:23:23.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.643+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3879440075 wait complete. 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.644+0000 7fdfdfa00700 1 Processor -- start 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.644+0000 7fdfdfa00700 1 -- start start 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.644+0000 7fdfdfa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd81a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.644+0000 7fdfdfa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfd810b2d0 0x7fdfd81a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdfa00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd8110960 0x7fdfd81a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdfa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd811a980 con 0x7fdfd810a980 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdfa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd811a800 con 0x7fdfd810b2d0 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdfa00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd811ab00 con 0x7fdfd8110960 2024-05-27T07:23:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd81a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd81a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38890/0 (socket says 172.21.15.12:38890) 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 -- 172.21.15.12:0/3162425302 learned_addr learned my addr 172.21.15.12:0/3162425302 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdda00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd8110960 0x7fdfd81a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfd7e00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfd810b2d0 0x7fdfd81a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 -- 172.21.15.12:0/3162425302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd8110960 msgr2=0x7fdfd81a5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd8110960 0x7fdfd81a5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 -- 172.21.15.12:0/3162425302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfd810b2d0 msgr2=0x7fdfd81a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfd810b2d0 0x7fdfd81a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.645+0000 7fdfdd000700 1 -- 172.21.15.12:0/3162425302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd81a6280 con 0x7fdfd810a980 2024-05-27T07:23:23.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.646+0000 7fdfdd000700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd81a13b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00d980 tx=0x7fdfcc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:23.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.646+0000 7fdfd5600700 1 -- 172.21.15.12:0/3162425302 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc017070 con 0x7fdfd810a980 2024-05-27T07:23:23.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.646+0000 7fdfd5600700 1 -- 172.21.15.12:0/3162425302 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdfcc010460 con 0x7fdfd810a980 2024-05-27T07:23:23.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.646+0000 7fdfd5600700 1 -- 172.21.15.12:0/3162425302 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00de60 con 0x7fdfd810a980 2024-05-27T07:23:23.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.646+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd81a6570 con 0x7fdfd810a980 2024-05-27T07:23:23.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.646+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd81ade90 con 0x7fdfd810a980 2024-05-27T07:23:23.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.648+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd810c070 con 0x7fdfd810a980 2024-05-27T07:23:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.650+0000 7fdfd5600700 1 -- 172.21.15.12:0/3162425302 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7fdfcc010610 con 0x7fdfd810a980 2024-05-27T07:23:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.652+0000 7fdfd5600700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdfc00b2b90 0x7fdfc00b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.652+0000 7fdfd5600700 1 -- 172.21.15.12:0/3162425302 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fdfcc0e26f0 con 0x7fdfd810a980 2024-05-27T07:23:23.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.652+0000 7fdfd7e00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdfc00b2b90 0x7fdfc00b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:23.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.653+0000 7fdfd7e00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdfc00b2b90 0x7fdfc00b50a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400b5c0 tx=0x7fdfc4005fb0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:23.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.655+0000 7fdfd5600700 1 -- 172.21.15.12:0/3162425302 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdfcc0a9af0 con 0x7fdfd810a980 2024-05-27T07:23:23.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:23 smithi162 ceph-mon[35676]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:23:23.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:23 smithi062 ceph-mon[35304]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:23:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.943+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd81a2710 con 0x7fdfd810a980 2024-05-27T07:23:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.945+0000 7fdfd5600700 1 -- 172.21.15.12:0/3162425302 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11867 (secure 0 0 0) 0x7fdfcc0a93f0 con 0x7fdfd810a980 2024-05-27T07:23:23.936 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:23:23.936 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:22:21.096387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:23:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdfc00b2b90 msgr2=0x7fdfc00b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdfc00b2b90 0x7fdfc00b50a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400b5c0 tx=0x7fdfc4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 msgr2=0x7fdfd81a13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd81a13b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00d980 tx=0x7fdfcc00dd40 comp rx=0 tx=0).stop 2024-05-27T07:23:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 shutdown_connections 2024-05-27T07:23:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdfc00b2b90 0x7fdfc00b50a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd8110960 0x7fdfd81a5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfd810a980 0x7fdfd81a13b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 --2- 172.21.15.12:0/3162425302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfd810b2d0 0x7fdfd81a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 >> 172.21.15.12:0/3162425302 conn(0x7fdfd8077c70 msgr2=0x7fdfd807b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:23.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 shutdown_connections 2024-05-27T07:23:23.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:23.949+0000 7fdfdfa00700 1 -- 172.21.15.12:0/3162425302 wait complete. 2024-05-27T07:23:23.941 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 27 2024-05-27T07:23:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:24 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3162425302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:24.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:24 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3162425302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:24.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:24 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3162425302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:24.832 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:23:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:25.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:25 smithi062 ceph-mon[35304]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T07:23:25.772 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:25 smithi012 ceph-mon[31473]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T07:23:25.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:25 smithi162 ceph-mon[35676]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T07:23:27.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:27 smithi062 ceph-mon[35304]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.776+0000 7f970d000700 1 -- 172.21.15.12:0/1685149243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 msgr2=0x7f97081111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.776+0000 7f970d000700 1 --2- 172.21.15.12:0/1685149243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081111c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f970000b0d0 tx=0x7f970000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.768 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:27 smithi012 ceph-mon[31473]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.777+0000 7f970d000700 1 -- 172.21.15.12:0/1685149243 shutdown_connections 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.777+0000 7f970d000700 1 --2- 172.21.15.12:0/1685149243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081111c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.777+0000 7f970d000700 1 --2- 172.21.15.12:0/1685149243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081045e0 0x7f97081049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.777+0000 7f970d000700 1 --2- 172.21.15.12:0/1685149243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9708108f10 0x7f9708109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.777+0000 7f970d000700 1 -- 172.21.15.12:0/1685149243 >> 172.21.15.12:0/1685149243 conn(0x7f9708100030 msgr2=0x7f9708102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.777+0000 7f970d000700 1 -- 172.21.15.12:0/1685149243 shutdown_connections 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.777+0000 7f970d000700 1 -- 172.21.15.12:0/1685149243 wait complete. 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.778+0000 7f970d000700 1 Processor -- start 2024-05-27T07:23:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.778+0000 7f970d000700 1 -- start start 2024-05-27T07:23:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f970d000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081045e0 0x7f970819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f970d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9708108f10 0x7f970819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f970d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f970d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9708114390 con 0x7f9708109860 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f970d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9708114210 con 0x7f9708108f10 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f970d000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9708114510 con 0x7f97081045e0 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f9706800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f9706800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38914/0 (socket says 172.21.15.12:38914) 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.779+0000 7f9706800700 1 -- 172.21.15.12:0/365285706 learned_addr learned my addr 172.21.15.12:0/365285706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f9705e00700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081045e0 0x7f970819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f96fd400700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9708108f10 0x7f970819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f9706800700 1 -- 172.21.15.12:0/365285706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081045e0 msgr2=0x7f970819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f9706800700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081045e0 0x7f970819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f9706800700 1 -- 172.21.15.12:0/365285706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9708108f10 msgr2=0x7f970819d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f9706800700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9708108f10 0x7f970819d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f9706800700 1 -- 172.21.15.12:0/365285706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97081a1e30 con 0x7f9708109860 2024-05-27T07:23:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f96fd400700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9708108f10 0x7f970819d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.780+0000 7f9706800700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081a1790 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f97000079a0 tx=0x7f9700007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.781+0000 7f96ff400700 1 -- 172.21.15.12:0/365285706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970001f070 con 0x7f9708109860 2024-05-27T07:23:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.781+0000 7f96ff400700 1 -- 172.21.15.12:0/365285706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9700007d90 con 0x7f9708109860 2024-05-27T07:23:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.781+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97081a20c0 con 0x7f9708109860 2024-05-27T07:23:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.781+0000 7f96ff400700 1 -- 172.21.15.12:0/365285706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970001b430 con 0x7f9708109860 2024-05-27T07:23:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.781+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97081a9a40 con 0x7f9708109860 2024-05-27T07:23:27.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.785+0000 7f96ff400700 1 -- 172.21.15.12:0/365285706 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7f9700022070 con 0x7f9708109860 2024-05-27T07:23:27.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 Processor -- start 2024-05-27T07:23:27.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 -- start start 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x55846205f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x558462059870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558462059db0 0x55846205a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558461f87e40 con 0x558462059db0 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558461f779e0 con 0x558461edae40 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.786+0000 7f971ba60b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558461f7ba20 con 0x55846205fd30 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558462059db0 0x55846205a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558462059db0 0x55846205a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38926/0 (socket says 172.21.15.12:38926) 2024-05-27T07:23:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f96fd400700 1 -- 172.21.15.12:0/1913401545 learned_addr learned my addr 172.21.15.12:0/1913401545 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:27.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f96ff400700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96ec0b2b90 0x7f96ec0b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f96fd400700 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x55846205f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f9705e00700 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x558462059870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f9706800700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96ec0b2b90 0x7f96ec0b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.787+0000 7f96ff400700 1 -- 172.21.15.12:0/365285706 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f970000f020 con 0x7f9708109860 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96fd400700 1 -- 172.21.15.12:0/1913401545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 msgr2=0x558462059870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96fd400700 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x558462059870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96fd400700 1 -- 172.21.15.12:0/1913401545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 msgr2=0x55846205f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96fd400700 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x55846205f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96fd400700 1 -- 172.21.15.12:0/1913401545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558461f40420 con 0x558462059db0 2024-05-27T07:23:27.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96fd400700 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558462059db0 0x55846205a1c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009fd0 tx=0x7f96f0009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcef9823383ae77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:27.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f9706800700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96ec0b2b90 0x7f96ec0b50a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97000e10c0 tx=0x7f970001c000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:27.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96f7400700 1 -- 172.21.15.12:0/1913401545 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0012de0 con 0x558462059db0 2024-05-27T07:23:27.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96f7400700 1 -- 172.21.15.12:0/1913401545 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96f000f8e0 con 0x558462059db0 2024-05-27T07:23:27.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.788+0000 7f96f7400700 1 -- 172.21.15.12:0/1913401545 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f001e7c0 con 0x558462059db0 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.789+0000 7f971ba60b80 1 -- 172.21.15.12:0/1913401545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558462059db0 msgr2=0x55846205a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.789+0000 7f971ba60b80 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558462059db0 0x55846205a1c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009fd0 tx=0x7f96f0009710 comp rx=0 tx=0).stop 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 -- 172.21.15.12:0/1913401545 shutdown_connections 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x55846205f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558462059db0 0x55846205a1c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 --2- 172.21.15.12:0/1913401545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x558462059870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 -- 172.21.15.12:0/1913401545 >> 172.21.15.12:0/1913401545 conn(0x558461f3afc0 msgr2=0x5584620548d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 -- 172.21.15.12:0/1913401545 shutdown_connections 2024-05-27T07:23:27.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 -- 172.21.15.12:0/1913401545 wait complete. 2024-05-27T07:23:27.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.790+0000 7f971ba60b80 1 -- start start 2024-05-27T07:23:27.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 1 Processor -- start 2024-05-27T07:23:27.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x5584620634d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558462059db0 0x558462063a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55846205fd30 0x558462063f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558461e7dfa0 con 0x55846205fd30 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558461f18760 con 0x558461edae40 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558462066800 con 0x558462059db0 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.791+0000 7f971ba60b80 10 client.0 _ll_register_callbacks cb 0x558461f21c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55846205fd30 0x558462063f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55846205fd30 0x558462063f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38936/0 (socket says 172.21.15.12:38936) 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 -- 172.21.15.12:0/686603912 learned_addr learned my addr 172.21.15.12:0/686603912 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f9705e00700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558462059db0 0x558462063a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x5584620634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 -- 172.21.15.12:0/686603912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558462059db0 msgr2=0x558462063a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558462059db0 0x558462063a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 -- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 msgr2=0x5584620634d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x5584620634d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f96fd400700 1 -- 172.21.15.12:0/686603912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558461ec78d0 con 0x55846205fd30 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.792+0000 7f9705e00700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558462059db0 0x558462063a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f96fd400700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55846205fd30 0x558462063f50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f96f0012ec0 tx=0x7f96f0014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6819c7274c5d29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f96ebe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f96ebe00700 1 -- 172.21.15.12:0/686603912 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0005260 con 0x55846205fd30 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f96ebe00700 1 -- 172.21.15.12:0/686603912 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f96f0005400 con 0x55846205fd30 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96f000f8e0 con 0x55846205fd30 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f96ebe00700 1 -- 172.21.15.12:0/686603912 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f96f0005600 con 0x55846205fd30 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f971ba60b80 10 client.15492 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:27.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.793+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558461f48060 con 0x55846205fd30 2024-05-27T07:23:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f96ebe00700 1 -- 172.21.15.12:0/686603912 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96f000f8e0 con 0x55846205fd30 2024-05-27T07:23:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f96ebe00700 1 -- 172.21.15.12:0/686603912 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f96f001e770 con 0x55846205fd30 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f971ba60b80 10 client.15492 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f971ba60b80 10 client.15492 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f971ba60b80 20 client.15492 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f971ba60b80 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558462073600 0x558462075a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f971ba60b80 4 client.15492 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.794+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558461f89d90 con 0x558462073600 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.795+0000 7f9705e00700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558462073600 0x558462075a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.796+0000 7f9705e00700 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558462073600 0x558462075a20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f082d15e77114e8 server_cookie=2b6a9afd1a20f3f7 in_seq=0 out_seq=0 2024-05-27T07:23:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.796+0000 7f96ebe00700 10 client.15492 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:23:27.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.798+0000 7f96ebe00700 1 -- 172.21.15.12:0/686603912 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558461f89d90 con 0x558462073600 2024-05-27T07:23:27.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.798+0000 7f96ebe00700 10 client.15492 handle_command_reply: tid=0 2024-05-27T07:23:27.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.798+0000 7f971ba60b80 1 client.15492 shutdown 2024-05-27T07:23:27.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.798+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55846205fd30 msgr2=0x558462063f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.798+0000 7f971ba60b80 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55846205fd30 0x558462063f50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f96f0012ec0 tx=0x7f96f0014040 comp rx=0 tx=0).stop 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 shutdown_connections 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558462073600 0x558462075a20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558461edae40 0x5584620634d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55846205fd30 0x558462063f50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 --2- 172.21.15.12:0/686603912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558462059db0 0x558462063a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f96ebe00700 0 client.15492 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 >> 172.21.15.12:0/686603912 conn(0x558461f3afc0 msgr2=0x558462054580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:27.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 shutdown_connections 2024-05-27T07:23:27.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 1 -- 172.21.15.12:0/686603912 wait complete. 2024-05-27T07:23:27.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.799+0000 7f971ba60b80 20 client.15492 trim_cache size 0 max 16384 2024-05-27T07:23:27.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.816+0000 7f971ba60b80 1 Processor -- start 2024-05-27T07:23:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.816+0000 7f971ba60b80 1 -- start start 2024-05-27T07:23:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.816+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x558462059870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.816+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 0x558462063ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.816+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x558462065970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f971ba60b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97081b5000 con 0x558461edae40 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f971ba60b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97081b2890 con 0x558462059db0 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f971ba60b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97081b06e0 con 0x55846205fd30 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x558462059870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x558462059870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38940/0 (socket says 172.21.15.12:38940) 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 -- 172.21.15.12:0/744494664 learned_addr learned my addr 172.21.15.12:0/744494664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 -- 172.21.15.12:0/744494664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 msgr2=0x558462065970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 --2- 172.21.15.12:0/744494664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x558462065970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 -- 172.21.15.12:0/744494664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 msgr2=0x558462063ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 --2- 172.21.15.12:0/744494664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 0x558462063ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.817+0000 7f96fd400700 1 -- 172.21.15.12:0/744494664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558462061440 con 0x558461edae40 2024-05-27T07:23:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.818+0000 7f96fd400700 1 --2- 172.21.15.12:0/744494664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x558462059870 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f96f0005830 tx=0x7f96f0009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b890d181f514e016 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.818+0000 7f96f6a00700 1 -- 172.21.15.12:0/744494664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0023020 con 0x558461edae40 2024-05-27T07:23:27.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.818+0000 7f96f6a00700 1 -- 172.21.15.12:0/744494664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96f000f8e0 con 0x558461edae40 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.818+0000 7f96f6a00700 1 -- 172.21.15.12:0/744494664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0017040 con 0x558461edae40 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 -- 172.21.15.12:0/744494664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 msgr2=0x558462059870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 --2- 172.21.15.12:0/744494664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x558462059870 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f96f0005830 tx=0x7f96f0009f30 comp rx=0 tx=0).stop 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 -- 172.21.15.12:0/744494664 shutdown_connections 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 --2- 172.21.15.12:0/744494664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x558462065970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 --2- 172.21.15.12:0/744494664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x558462059870 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 --2- 172.21.15.12:0/744494664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 0x558462063ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 -- 172.21.15.12:0/744494664 >> 172.21.15.12:0/744494664 conn(0x558461f3afc0 msgr2=0x55846205d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 -- 172.21.15.12:0/744494664 shutdown_connections 2024-05-27T07:23:27.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.819+0000 7f971ba60b80 1 -- 172.21.15.12:0/744494664 wait complete. 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 -- start start 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 Processor -- start 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x55846206c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 0x55846207a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x55846207ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97081b0d50 con 0x558461edae40 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97081b21c0 con 0x558462059db0 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.820+0000 7f971ba60b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97081b1420 con 0x55846205fd30 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f971ba60b80 10 client.0 _ll_register_callbacks cb 0x558461f21c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x55846206c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x55846206c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38948/0 (socket says 172.21.15.12:38948) 2024-05-27T07:23:27.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 -- 172.21.15.12:0/4145869810 learned_addr learned my addr 172.21.15.12:0/4145869810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f9705e00700 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 0x55846207a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 -- 172.21.15.12:0/4145869810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 msgr2=0x55846207ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x55846207ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 -- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 msgr2=0x55846207a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 0x55846207a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.821+0000 7f96fd400700 1 -- 172.21.15.12:0/4145869810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55846207b320 con 0x558461edae40 2024-05-27T07:23:27.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f96fd400700 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x55846206c0b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f96f001f500 tx=0x7f96f00096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1d5cc2b0f661f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:27.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f96ebe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:27.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f96ebe00700 1 -- 172.21.15.12:0/4145869810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0038020 con 0x558461edae40 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f96ebe00700 1 -- 172.21.15.12:0/4145869810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f96f001e5d0 con 0x558461edae40 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96f000f8e0 con 0x558461edae40 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f96ebe00700 1 -- 172.21.15.12:0/4145869810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f96f0043050 con 0x558461edae40 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f971ba60b80 10 client.15498 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.822+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5584620ddd00 con 0x558461edae40 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f96ebe00700 1 -- 172.21.15.12:0/4145869810 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96f000f8e0 con 0x558461edae40 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f96ebe00700 1 -- 172.21.15.12:0/4145869810 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f96f0032020 con 0x558461edae40 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f971ba60b80 10 client.15498 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f971ba60b80 10 client.15498 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f971ba60b80 20 client.15498 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f971ba60b80 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5584620de2e0 0x5584620e0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f971ba60b80 4 client.15498 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.823+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558461f83570 con 0x5584620de2e0 2024-05-27T07:23:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.824+0000 7f9705e00700 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5584620de2e0 0x5584620e0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.825+0000 7f9705e00700 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5584620de2e0 0x5584620e0700 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81c199221354fa78 server_cookie=7aadc65d62bef59c in_seq=0 out_seq=0 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.825+0000 7f96ebe00700 10 client.15498 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.825+0000 7f96ebe00700 1 -- 172.21.15.12:0/4145869810 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558461f83570 con 0x5584620de2e0 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.825+0000 7f96ebe00700 10 client.15498 handle_command_reply: tid=0 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.825+0000 7f971ba60b80 1 client.15498 shutdown 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.825+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 msgr2=0x55846206c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.825+0000 7f971ba60b80 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x55846206c0b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f96f001f500 tx=0x7f96f00096e0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 shutdown_connections 2024-05-27T07:23:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55846205fd30 0x55846207ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558461edae40 0x55846206c0b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5584620de2e0 0x5584620e0700 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 --2- 172.21.15.12:0/4145869810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558462059db0 0x55846207a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f96ebe00700 0 client.15498 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 >> 172.21.15.12:0/4145869810 conn(0x558461f3afc0 msgr2=0x5584620632b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 shutdown_connections 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 1 -- 172.21.15.12:0/4145869810 wait complete. 2024-05-27T07:23:27.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.826+0000 7f971ba60b80 20 client.15498 trim_cache size 0 max 16384 2024-05-27T07:23:27.819 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:23:27.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96ec0b2b90 msgr2=0x7f96ec0b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96ec0b2b90 0x7f96ec0b50a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97000e10c0 tx=0x7f970001c000 comp rx=0 tx=0).stop 2024-05-27T07:23:27.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 msgr2=0x7f97081a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081a1790 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f97000079a0 tx=0x7f9700007a80 comp rx=0 tx=0).stop 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 shutdown_connections 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96ec0b2b90 0x7f96ec0b50a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9708109860 0x7f97081a1790 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081045e0 0x7f970819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 --2- 172.21.15.12:0/365285706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9708108f10 0x7f970819d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 >> 172.21.15.12:0/365285706 conn(0x7f9708100030 msgr2=0x7f970810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 shutdown_connections 2024-05-27T07:23:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:27.827+0000 7f970d000700 1 -- 172.21.15.12:0/365285706 wait complete. 2024-05-27T07:23:27.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:27 smithi162 ceph-mon[35676]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:28.834 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:23:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:29.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:29 smithi012 ceph-mon[31473]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:29.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:29 smithi162 ceph-mon[35676]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:29.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:29 smithi062 ceph-mon[35304]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 -- 172.21.15.12:0/3750527239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 msgr2=0x7fdc50110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 --2- 172.21.15.12:0/3750527239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc50110460 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009990 tx=0x7fdc3c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 -- 172.21.15.12:0/3750527239 shutdown_connections 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 --2- 172.21.15.12:0/3750527239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc501109a0 0x7fdc50112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 --2- 172.21.15.12:0/3750527239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc50075d20 0x7fdc50076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 --2- 172.21.15.12:0/3750527239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc50110460 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 -- 172.21.15.12:0/3750527239 >> 172.21.15.12:0/3750527239 conn(0x7fdc50079d00 msgr2=0x7fdc5007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 -- 172.21.15.12:0/3750527239 shutdown_connections 2024-05-27T07:23:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.804+0000 7fdc56e00700 1 -- 172.21.15.12:0/3750527239 wait complete. 2024-05-27T07:23:31.796 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:31 smithi012 ceph-mon[31473]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:23:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.806+0000 7fdc56e00700 1 Processor -- start 2024-05-27T07:23:31.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.806+0000 7fdc56e00700 1 -- start start 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc56e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc50075d20 0x7fdc5019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc56e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc5019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc56e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc501109a0 0x7fdc501a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc56e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501163c0 con 0x7fdc50076670 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc56e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc50116240 con 0x7fdc50075d20 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc56e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc50116540 con 0x7fdc501109a0 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc55000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc5019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc55000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc5019d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38962/0 (socket says 172.21.15.12:38962) 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc55000700 1 -- 172.21.15.12:0/2726853633 learned_addr learned my addr 172.21.15.12:0/2726853633 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc55a00700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc50075d20 0x7fdc5019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.807+0000 7fdc56400700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc501109a0 0x7fdc501a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.808+0000 7fdc55000700 1 -- 172.21.15.12:0/2726853633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc501109a0 msgr2=0x7fdc501a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.808+0000 7fdc55000700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc501109a0 0x7fdc501a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.808+0000 7fdc55000700 1 -- 172.21.15.12:0/2726853633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc50075d20 msgr2=0x7fdc5019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.808+0000 7fdc55000700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc50075d20 0x7fdc5019ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.808+0000 7fdc55000700 1 -- 172.21.15.12:0/2726853633 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc501a1d60 con 0x7fdc50076670 2024-05-27T07:23:31.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.808+0000 7fdc55000700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc5019d430 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005910 tx=0x7fdc3c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.809+0000 7fdc4e000700 1 -- 172.21.15.12:0/2726853633 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c027070 con 0x7fdc50076670 2024-05-27T07:23:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.809+0000 7fdc4e000700 1 -- 172.21.15.12:0/2726853633 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc3c00fc00 con 0x7fdc50076670 2024-05-27T07:23:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.809+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc501a1ff0 con 0x7fdc50076670 2024-05-27T07:23:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.809+0000 7fdc4e000700 1 -- 172.21.15.12:0/2726853633 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c022d80 con 0x7fdc50076670 2024-05-27T07:23:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.809+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc501a9970 con 0x7fdc50076670 2024-05-27T07:23:31.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.813+0000 7fdc4e000700 1 -- 172.21.15.12:0/2726853633 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7fdc3c02a070 con 0x7fdc50076670 2024-05-27T07:23:31.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.814+0000 7fdc66fbbb80 1 Processor -- start 2024-05-27T07:23:31.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.814+0000 7fdc66fbbb80 1 -- start start 2024-05-27T07:23:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba502750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba502c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d0ba42fe40 con 0x55d0ba46e9f0 2024-05-27T07:23:31.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d0ba41f9e0 con 0x55d0ba382e40 2024-05-27T07:23:31.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0ba423a20 con 0x55d0ba50c2c0 2024-05-27T07:23:31.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc56400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba502750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc56400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba502750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38964/0 (socket says 172.21.15.12:38964) 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.815+0000 7fdc56400700 1 -- 172.21.15.12:0/2658985480 learned_addr learned my addr 172.21.15.12:0/2658985480 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc55a00700 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba502c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc55000700 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc4e000700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc380b2b90 0x7fdc380b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc56400700 1 -- 172.21.15.12:0/2658985480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 msgr2=0x55d0ba50e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc56400700 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc56400700 1 -- 172.21.15.12:0/2658985480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 msgr2=0x55d0ba502c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc56400700 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba502c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc56400700 1 -- 172.21.15.12:0/2658985480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ba3e8420 con 0x55d0ba46e9f0 2024-05-27T07:23:31.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.816+0000 7fdc4e000700 1 -- 172.21.15.12:0/2726853633 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fdc3c0e3b70 con 0x7fdc50076670 2024-05-27T07:23:31.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.817+0000 7fdc56400700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc380b2b90 0x7fdc380b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.817+0000 7fdc56400700 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba502750 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdc480084c0 tx=0x7fdc480087d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315c3de4f856c920 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.817+0000 7fdc42a00700 1 -- 172.21.15.12:0/2658985480 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc48015950 con 0x55d0ba46e9f0 2024-05-27T07:23:31.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.817+0000 7fdc42a00700 1 -- 172.21.15.12:0/2658985480 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc480103c0 con 0x55d0ba46e9f0 2024-05-27T07:23:31.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.817+0000 7fdc42a00700 1 -- 172.21.15.12:0/2658985480 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4801f600 con 0x55d0ba46e9f0 2024-05-27T07:23:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.818+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2658985480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 msgr2=0x55d0ba502750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.818+0000 7fdc56400700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc380b2b90 0x7fdc380b50a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc4801a000 tx=0x7fdc4801fc10 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.818+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba502750 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdc480084c0 tx=0x7fdc480087d0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2658985480 shutdown_connections 2024-05-27T07:23:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba502750 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2658985480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba502c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2658985480 >> 172.21.15.12:0/2658985480 conn(0x55d0ba41c390 msgr2=0x55d0ba4fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2658985480 shutdown_connections 2024-05-27T07:23:31.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2658985480 wait complete. 2024-05-27T07:23:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.819+0000 7fdc66fbbb80 1 -- start start 2024-05-27T07:23:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 1 Processor -- start 2024-05-27T07:23:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba509200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba509740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba509c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d0ba325fa0 con 0x55d0ba46e9f0 2024-05-27T07:23:31.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d0ba3c0760 con 0x55d0ba382e40 2024-05-27T07:23:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0ba50ee00 con 0x55d0ba50c2c0 2024-05-27T07:23:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.820+0000 7fdc66fbbb80 10 client.0 _ll_register_callbacks cb 0x55d0ba3c9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba509740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba509740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38976/0 (socket says 172.21.15.12:38976) 2024-05-27T07:23:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 -- 172.21.15.12:0/1656272239 learned_addr learned my addr 172.21.15.12:0/1656272239 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55a00700 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba509200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc56400700 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba509c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 -- 172.21.15.12:0/1656272239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 msgr2=0x55d0ba509c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba509c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 -- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 msgr2=0x55d0ba509200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba509200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 -- 172.21.15.12:0/1656272239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0ba36f8d0 con 0x55d0ba46e9f0 2024-05-27T07:23:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc55000700 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba509740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000c00 tx=0x7fdc3c0aadc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af94f3b5f9c755e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.821+0000 7fdc37400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.822+0000 7fdc37400700 1 -- 172.21.15.12:0/1656272239 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c0aa390 con 0x55d0ba46e9f0 2024-05-27T07:23:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.822+0000 7fdc37400700 1 -- 172.21.15.12:0/1656272239 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fdc3c00f770 con 0x55d0ba46e9f0 2024-05-27T07:23:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.822+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc480103c0 con 0x55d0ba46e9f0 2024-05-27T07:23:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.822+0000 7fdc37400700 1 -- 172.21.15.12:0/1656272239 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc3c0aaf80 con 0x55d0ba46e9f0 2024-05-27T07:23:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.822+0000 7fdc66fbbb80 10 client.15510 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.822+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0ba3f0060 con 0x55d0ba46e9f0 2024-05-27T07:23:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc37400700 1 -- 172.21.15.12:0/1656272239 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc480103c0 con 0x55d0ba46e9f0 2024-05-27T07:23:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc37400700 1 -- 172.21.15.12:0/1656272239 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdc3c0a9ae0 con 0x55d0ba46e9f0 2024-05-27T07:23:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc66fbbb80 10 client.15510 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc66fbbb80 10 client.15510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:23:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc66fbbb80 20 client.15510 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:31.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba51bb20 0x55d0ba51df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc66fbbb80 4 client.15510 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:31.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.823+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0ba431d90 con 0x55d0ba51bb20 2024-05-27T07:23:31.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.824+0000 7fdc55a00700 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba51bb20 0x55d0ba51df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.825+0000 7fdc55a00700 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba51bb20 0x55d0ba51df40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e95a54a0ad6e5964 server_cookie=2b0104e9f3cfd90d in_seq=0 out_seq=0 2024-05-27T07:23:31.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.825+0000 7fdc37400700 10 client.15510 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:23:31.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.827+0000 7fdc37400700 1 -- 172.21.15.12:0/1656272239 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d0ba431d90 con 0x55d0ba51bb20 2024-05-27T07:23:31.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.827+0000 7fdc37400700 10 client.15510 handle_command_reply: tid=0 2024-05-27T07:23:31.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.827+0000 7fdc66fbbb80 1 client.15510 shutdown 2024-05-27T07:23:31.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.827+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 msgr2=0x55d0ba509740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.827+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba509740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000c00 tx=0x7fdc3c0aadc0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.827+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 shutdown_connections 2024-05-27T07:23:31.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.827+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba51bb20 0x55d0ba51df40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba382e40 0x55d0ba509200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba50c2c0 0x55d0ba509c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/1656272239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba46e9f0 0x55d0ba509740 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc37400700 0 client.15510 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:23:31.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 >> 172.21.15.12:0/1656272239 conn(0x55d0ba41c390 msgr2=0x55d0ba502750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 shutdown_connections 2024-05-27T07:23:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/1656272239 wait complete. 2024-05-27T07:23:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.828+0000 7fdc66fbbb80 20 client.15510 trim_cache size 0 max 16384 2024-05-27T07:23:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 Processor -- start 2024-05-27T07:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 -- start start 2024-05-27T07:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba50b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba517ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba5236b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b4dc0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b2650 con 0x55d0ba46e9f0 2024-05-27T07:23:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.846+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b04a0 con 0x55d0ba382e40 2024-05-27T07:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba5236b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba5236b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53674/0 (socket says 172.21.15.12:53674) 2024-05-27T07:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 -- 172.21.15.12:0/2081802625 learned_addr learned my addr 172.21.15.12:0/2081802625 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc55a00700 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba50b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc55000700 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba517ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 -- 172.21.15.12:0/2081802625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 msgr2=0x55d0ba50b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba50b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 -- 172.21.15.12:0/2081802625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 msgr2=0x55d0ba517ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba517ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc56400700 1 -- 172.21.15.12:0/2081802625 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ba5183e0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.847+0000 7fdc55a00700 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba50b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.848+0000 7fdc56400700 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba5236b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdc48015450 tx=0x7fdc48025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f909eb4c1063a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.848+0000 7fdc42000700 1 -- 172.21.15.12:0/2081802625 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4802f030 con 0x55d0ba50c2c0 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.848+0000 7fdc42000700 1 -- 172.21.15.12:0/2081802625 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc480079b0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.848+0000 7fdc42000700 1 -- 172.21.15.12:0/2081802625 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc480156f0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2081802625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 msgr2=0x55d0ba5236b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba5236b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdc48015450 tx=0x7fdc48025000 comp rx=0 tx=0).stop 2024-05-27T07:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2081802625 shutdown_connections 2024-05-27T07:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba5236b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba50b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/2081802625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba517ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2081802625 >> 172.21.15.12:0/2081802625 conn(0x55d0ba41c390 msgr2=0x55d0ba504820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2081802625 shutdown_connections 2024-05-27T07:23:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.849+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/2081802625 wait complete. 2024-05-27T07:23:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.850+0000 7fdc66fbbb80 1 -- start start 2024-05-27T07:23:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.850+0000 7fdc66fbbb80 1 Processor -- start 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.850+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba514510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.850+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba50f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.850+0000 7fdc66fbbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.850+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b0b10 con 0x55d0ba50c2c0 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.850+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b1f80 con 0x55d0ba46e9f0 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc66fbbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b11e0 con 0x55d0ba382e40 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc66fbbb80 10 client.0 _ll_register_callbacks cb 0x55d0ba3c9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53678/0 (socket says 172.21.15.12:53678) 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 -- 172.21.15.12:0/4172077477 learned_addr learned my addr 172.21.15.12:0/4172077477 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc55a00700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba514510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc55000700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba50f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 -- 172.21.15.12:0/4172077477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 msgr2=0x55d0ba514510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba514510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 -- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 msgr2=0x55d0ba50f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba50f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc56400700 1 -- 172.21.15.12:0/4172077477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0ba5102f0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.851+0000 7fdc55a00700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba514510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.852+0000 7fdc56400700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50fdb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdc48013a00 tx=0x7fdc48013ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31f4394c6a18cff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.852+0000 7fdc37400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.852+0000 7fdc37400700 1 -- 172.21.15.12:0/4172077477 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc48037030 con 0x55d0ba50c2c0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.852+0000 7fdc37400700 1 -- 172.21.15.12:0/4172077477 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fdc4801f3a0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.852+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc480079b0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc37400700 1 -- 172.21.15.12:0/4172077477 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc48041050 con 0x55d0ba50c2c0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc66fbbb80 10 client.15516 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0ba586200 con 0x55d0ba50c2c0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc37400700 1 -- 172.21.15.12:0/4172077477 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc480079b0 con 0x55d0ba50c2c0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc37400700 1 -- 172.21.15.12:0/4172077477 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdc48013070 con 0x55d0ba50c2c0 2024-05-27T07:23:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc66fbbb80 10 client.15516 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc66fbbb80 10 client.15516 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:23:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc66fbbb80 20 client.15516 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.853+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba586650 0x55d0ba588a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.854+0000 7fdc66fbbb80 4 client.15516 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.854+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0ba42b570 con 0x55d0ba586650 2024-05-27T07:23:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.854+0000 7fdc55a00700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba586650 0x55d0ba588a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.855+0000 7fdc55a00700 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba586650 0x55d0ba588a70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f16de26e84a5c939 server_cookie=27a68869adf216f7 in_seq=0 out_seq=0 2024-05-27T07:23:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.855+0000 7fdc37400700 10 client.15516 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:23:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.856+0000 7fdc37400700 1 -- 172.21.15.12:0/4172077477 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0ba42b570 con 0x55d0ba586650 2024-05-27T07:23:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.856+0000 7fdc37400700 10 client.15516 handle_command_reply: tid=0 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.856+0000 7fdc66fbbb80 1 client.15516 shutdown 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.856+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 msgr2=0x55d0ba50fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.856+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50fdb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdc48013a00 tx=0x7fdc48013ae0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.856+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 shutdown_connections 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55d0ba586650 0x55d0ba588a70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0ba50c2c0 0x55d0ba50fdb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0ba382e40 0x55d0ba514510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 1 --2- 172.21.15.12:0/4172077477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0ba46e9f0 0x55d0ba50f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc37400700 0 client.15516 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 >> 172.21.15.12:0/4172077477 conn(0x55d0ba41c390 msgr2=0x55d0ba501120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 shutdown_connections 2024-05-27T07:23:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 1 -- 172.21.15.12:0/4172077477 wait complete. 2024-05-27T07:23:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.857+0000 7fdc66fbbb80 20 client.15516 trim_cache size 0 max 16384 2024-05-27T07:23:31.850 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc380b2b90 msgr2=0x7fdc380b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc380b2b90 0x7fdc380b50a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc4801a000 tx=0x7fdc4801fc10 comp rx=0 tx=0).stop 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 msgr2=0x7fdc5019d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc5019d430 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005910 tx=0x7fdc3c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 shutdown_connections 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc380b2b90 0x7fdc380b50a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc50075d20 0x7fdc5019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc501109a0 0x7fdc501a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 --2- 172.21.15.12:0/2726853633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc50076670 0x7fdc5019d430 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 >> 172.21.15.12:0/2726853633 conn(0x7fdc50079d00 msgr2=0x7fdc50102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 shutdown_connections 2024-05-27T07:23:31.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:31.858+0000 7fdc56e00700 1 -- 172.21.15.12:0/2726853633 wait complete. 2024-05-27T07:23:31.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:31 smithi162 ceph-mon[35676]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:23:31.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:31 smithi062 ceph-mon[35304]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-27T07:23:32.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:23:33.280 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:33.703 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:33 smithi012 ceph-mon[31473]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:33 smithi162 ceph-mon[35676]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:33.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:33 smithi062 ceph-mon[35304]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:35.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:35 smithi162 ceph-mon[35676]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 -- 172.21.15.12:0/1135638061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 msgr2=0x7f55bc10e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 --2- 172.21.15.12:0/1135638061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc10e3b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009990 tx=0x7f55a8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 -- 172.21.15.12:0/1135638061 shutdown_connections 2024-05-27T07:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 --2- 172.21.15.12:0/1135638061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc10e8f0 0x7f55bc110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 --2- 172.21.15.12:0/1135638061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55bc105510 0x7f55bc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 --2- 172.21.15.12:0/1135638061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc10e3b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 -- 172.21.15.12:0/1135638061 >> 172.21.15.12:0/1135638061 conn(0x7f55bc0fedb0 msgr2=0x7f55bc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:35.823 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:35 smithi012 ceph-mon[31473]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.832+0000 7f55c0e00700 1 -- 172.21.15.12:0/1135638061 shutdown_connections 2024-05-27T07:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.833+0000 7f55c0e00700 1 -- 172.21.15.12:0/1135638061 wait complete. 2024-05-27T07:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 Processor -- start 2024-05-27T07:23:35.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 -- start start 2024-05-27T07:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55bc105510 0x7f55bc103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc10e8f0 0x7f55bc103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc114330 con 0x7f55bc105e60 2024-05-27T07:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1141b0 con 0x7f55bc10e8f0 2024-05-27T07:23:35.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.834+0000 7f55c0e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1144b0 con 0x7f55bc105510 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc103740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53700/0 (socket says 172.21.15.12:53700) 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 -- 172.21.15.12:0/907944440 learned_addr learned my addr 172.21.15.12:0/907944440 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55bc105510 0x7f55bc103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55ba800700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc10e8f0 0x7f55bc103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 -- 172.21.15.12:0/907944440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55bc105510 msgr2=0x7f55bc103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55bc105510 0x7f55bc103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 -- 172.21.15.12:0/907944440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc10e8f0 msgr2=0x7f55bc103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc10e8f0 0x7f55bc103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.835+0000 7f55b9400700 1 -- 172.21.15.12:0/907944440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc1a90b0 con 0x7f55bc105e60 2024-05-27T07:23:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.836+0000 7f55b9e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55bc105510 0x7f55bc103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.836+0000 7f55b9400700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc103740 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f55a8005910 tx=0x7f55a80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.836+0000 7f55b2a00700 1 -- 172.21.15.12:0/907944440 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8027070 con 0x7f55bc105e60 2024-05-27T07:23:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.836+0000 7f55b2a00700 1 -- 172.21.15.12:0/907944440 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55a800fc00 con 0x7f55bc105e60 2024-05-27T07:23:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.836+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc1a9340 con 0x7f55bc105e60 2024-05-27T07:23:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.837+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc1a9740 con 0x7f55bc105e60 2024-05-27T07:23:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.836+0000 7f55b2a00700 1 -- 172.21.15.12:0/907944440 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8022d80 con 0x7f55bc105e60 2024-05-27T07:23:35.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.842+0000 7f55b2a00700 1 -- 172.21.15.12:0/907944440 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7f55a802a070 con 0x7f55bc105e60 2024-05-27T07:23:35.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.843+0000 7f55cf8f0b80 1 Processor -- start 2024-05-27T07:23:35.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.843+0000 7f55cf8f0b80 1 -- start start 2024-05-27T07:23:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.843+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d08ffe40 0x5573d0a847f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.843+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a84d30 0x5573d0a7e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a7f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5573d09ace40 con 0x5573d0a7edb0 2024-05-27T07:23:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5573d099c9e0 con 0x5573d08ffe40 2024-05-27T07:23:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5573d09a0a20 con 0x5573d0a84d30 2024-05-27T07:23:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a7f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a7f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53706/0 (socket says 172.21.15.12:53706) 2024-05-27T07:23:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55b9400700 1 -- 172.21.15.12:0/3778092119 learned_addr learned my addr 172.21.15.12:0/3778092119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55b9e00700 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a84d30 0x5573d0a7e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.844+0000 7f55ba800700 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d08ffe40 0x5573d0a847f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b9400700 1 -- 172.21.15.12:0/3778092119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a84d30 msgr2=0x5573d0a7e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b9400700 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a84d30 0x5573d0a7e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b9400700 1 -- 172.21.15.12:0/3778092119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d08ffe40 msgr2=0x5573d0a847f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b9400700 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d08ffe40 0x5573d0a847f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b9400700 1 -- 172.21.15.12:0/3778092119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573d0965420 con 0x5573d0a7edb0 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b2a00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55a00b2b10 0x7f55a00b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b9400700 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a7f1c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f55a80e0f20 tx=0x7f55a80ec640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed63506a7116e80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55b2a00700 1 -- 172.21.15.12:0/907944440 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f55a80e3bf0 con 0x7f55bc105e60 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.845+0000 7f55ba800700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55a00b2b10 0x7f55a00b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.846+0000 7f55a7400700 1 -- 172.21.15.12:0/3778092119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a80a6820 con 0x5573d0a7edb0 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.846+0000 7f55a7400700 1 -- 172.21.15.12:0/3778092119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55a80aa470 con 0x5573d0a7edb0 2024-05-27T07:23:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.846+0000 7f55a7400700 1 -- 172.21.15.12:0/3778092119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a80aa0d0 con 0x5573d0a7edb0 2024-05-27T07:23:35.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3778092119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 msgr2=0x5573d0a7f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55ba800700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55a00b2b10 0x7f55a00b5020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x5573d0a7fb30 tx=0x7f55b4009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:35.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a7f1c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f55a80e0f20 tx=0x7f55a80ec640 comp rx=0 tx=0).stop 2024-05-27T07:23:35.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3778092119 shutdown_connections 2024-05-27T07:23:35.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d08ffe40 0x5573d0a847f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a7f1c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3778092119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a84d30 0x5573d0a7e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3778092119 >> 172.21.15.12:0/3778092119 conn(0x5573d095ffc0 msgr2=0x5573d0a798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3778092119 shutdown_connections 2024-05-27T07:23:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.847+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3778092119 wait complete. 2024-05-27T07:23:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.848+0000 7f55cf8f0b80 1 -- start start 2024-05-27T07:23:35.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.848+0000 7f55cf8f0b80 1 Processor -- start 2024-05-27T07:23:35.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.848+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a885a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a88ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a89020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5573d08a2fa0 con 0x5573d0a7edb0 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5573d093d760 con 0x5573d0a84d30 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5573d0a8b5c0 con 0x5573d08ffe40 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55cf8f0b80 10 client.0 _ll_register_callbacks cb 0x5573d0946c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55b9e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a885a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55b9e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a885a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35586/0 (socket says 172.21.15.12:35586) 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55b9e00700 1 -- 172.21.15.12:0/3420286153 learned_addr learned my addr 172.21.15.12:0/3420286153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.849+0000 7f55ba800700 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a89020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f55b9400700 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a88ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f55b9e00700 1 -- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 msgr2=0x5573d0a89020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f55b9e00700 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a89020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f55b9e00700 1 -- 172.21.15.12:0/3420286153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 msgr2=0x5573d0a88ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f55b9e00700 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a88ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f55b9e00700 1 -- 172.21.15.12:0/3420286153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573d08ec8d0 con 0x5573d08ffe40 2024-05-27T07:23:35.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f55b9e00700 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a885a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f55ac003ca0 tx=0x7f55ac0113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3de4cae577f6d9a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:35.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.850+0000 7f559be00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.851+0000 7f559be00700 1 -- 172.21.15.12:0/3420286153 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac011d10 con 0x5573d08ffe40 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.851+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55a80aa470 con 0x5573d08ffe40 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.851+0000 7f559be00700 1 -- 172.21.15.12:0/3420286153 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f55ac012450 con 0x5573d08ffe40 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.851+0000 7f559be00700 1 -- 172.21.15.12:0/3420286153 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f55ac0126b0 con 0x5573d08ffe40 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.851+0000 7f55cf8f0b80 10 client.25231 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.851+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5573d096d060 con 0x5573d08ffe40 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f559be00700 1 -- 172.21.15.12:0/3420286153 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55a80aa470 con 0x5573d08ffe40 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f559be00700 1 -- 172.21.15.12:0/3420286153 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f55ac024750 con 0x5573d08ffe40 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f55cf8f0b80 10 client.25231 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:35.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f55cf8f0b80 10 client.25231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f55cf8f0b80 20 client.25231 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0a98490 0x5573d0a9a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f55cf8f0b80 4 client.25231 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.852+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573d09aed90 con 0x5573d0a98490 2024-05-27T07:23:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.853+0000 7f55b9400700 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0a98490 0x5573d0a9a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.854+0000 7f55b9400700 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0a98490 0x5573d0a9a910 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f4267b74446ac53 server_cookie=800665b947e96157 in_seq=0 out_seq=0 2024-05-27T07:23:35.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.854+0000 7f559be00700 10 client.25231 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:35.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.856+0000 7f559be00700 1 -- 172.21.15.12:0/3420286153 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5573d09aed90 con 0x5573d0a98490 2024-05-27T07:23:35.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.856+0000 7f559be00700 10 client.25231 handle_command_reply: tid=0 2024-05-27T07:23:35.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.856+0000 7f55cf8f0b80 1 client.25231 shutdown 2024-05-27T07:23:35.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.856+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 msgr2=0x5573d0a885a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.856+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a885a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f55ac003ca0 tx=0x7f55ac0113e0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 shutdown_connections 2024-05-27T07:23:35.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0a98490 0x5573d0a9a910 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a89020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a885a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/3420286153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a88ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f559be00700 0 client.25231 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:35.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 >> 172.21.15.12:0/3420286153 conn(0x5573d095ffc0 msgr2=0x5573d0a79580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:35.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 shutdown_connections 2024-05-27T07:23:35.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/3420286153 wait complete. 2024-05-27T07:23:35.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.857+0000 7f55cf8f0b80 20 client.25231 trim_cache size 0 max 16384 2024-05-27T07:23:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 Processor -- start 2024-05-27T07:23:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 -- start start 2024-05-27T07:23:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a7e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a94560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a88a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1b4e90 con 0x5573d0a7edb0 2024-05-27T07:23:35.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1b2720 con 0x5573d0a84d30 2024-05-27T07:23:35.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.874+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1b0570 con 0x5573d08ffe40 2024-05-27T07:23:35.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a94560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a94560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53730/0 (socket says 172.21.15.12:53730) 2024-05-27T07:23:35.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 -- 172.21.15.12:0/1271166737 learned_addr learned my addr 172.21.15.12:0/1271166737 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:35.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9e00700 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a7e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 -- 172.21.15.12:0/1271166737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 msgr2=0x5573d0a7e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a7e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55ba800700 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a88a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 -- 172.21.15.12:0/1271166737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 msgr2=0x5573d0a88a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a88a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55b9400700 1 -- 172.21.15.12:0/1271166737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573d0a95f10 con 0x5573d0a7edb0 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.875+0000 7f55ba800700 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a88a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.876+0000 7f55b9400700 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a94560 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f55a80e1590 tx=0x7f55a8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54ec8f4b99e3cfd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.876+0000 7f55a6a00700 1 -- 172.21.15.12:0/1271166737 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a80aa2b0 con 0x5573d0a7edb0 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.876+0000 7f55a6a00700 1 -- 172.21.15.12:0/1271166737 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55a80aa470 con 0x5573d0a7edb0 2024-05-27T07:23:35.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.876+0000 7f55a6a00700 1 -- 172.21.15.12:0/1271166737 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a80aa730 con 0x5573d0a7edb0 2024-05-27T07:23:35.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/1271166737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 msgr2=0x5573d0a94560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a94560 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f55a80e1590 tx=0x7f55a8009310 comp rx=0 tx=0).stop 2024-05-27T07:23:35.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/1271166737 shutdown_connections 2024-05-27T07:23:35.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a88a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d08ffe40 0x5573d0a7e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/1271166737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d0a7edb0 0x5573d0a94560 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/1271166737 >> 172.21.15.12:0/1271166737 conn(0x5573d095ffc0 msgr2=0x5573d0a88380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:35.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/1271166737 shutdown_connections 2024-05-27T07:23:35.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.877+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/1271166737 wait complete. 2024-05-27T07:23:35.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.878+0000 7f55cf8f0b80 1 -- start start 2024-05-27T07:23:35.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.878+0000 7f55cf8f0b80 1 Processor -- start 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.878+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d08ffe40 0x5573d0a90f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a7edb0 0x5573d0a9f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55cf8f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a9ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1b0be0 con 0x5573d08ffe40 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1b2050 con 0x5573d0a84d30 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55cf8f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1b12b0 con 0x5573d0a7edb0 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55cf8f0b80 10 client.0 _ll_register_callbacks cb 0x5573d0946c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d08ffe40 0x5573d0a90f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d08ffe40 0x5573d0a90f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53732/0 (socket says 172.21.15.12:53732) 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 -- 172.21.15.12:0/2831330154 learned_addr learned my addr 172.21.15.12:0/2831330154 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9400700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a7edb0 0x5573d0a9f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 -- 172.21.15.12:0/2831330154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a7edb0 msgr2=0x5573d0a9f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55ba800700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a9ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a7edb0 0x5573d0a9f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 -- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 msgr2=0x5573d0a9ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a9ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9e00700 1 -- 172.21.15.12:0/2831330154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573d0aa06c0 con 0x5573d08ffe40 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.879+0000 7f55b9400700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a7edb0 0x5573d0a9f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.880+0000 7f55b9e00700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d08ffe40 0x5573d0a90f70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f55ac003800 tx=0x7f55ac011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47b29f2b2154628 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.880+0000 7f559be00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.880+0000 7f559be00700 1 -- 172.21.15.12:0/2831330154 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac02a070 con 0x5573d08ffe40 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.880+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55a80aa470 con 0x5573d08ffe40 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.880+0000 7f559be00700 1 -- 172.21.15.12:0/2831330154 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f55ac024620 con 0x5573d08ffe40 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.880+0000 7f559be00700 1 -- 172.21.15.12:0/2831330154 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f55ac011ea0 con 0x5573d08ffe40 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.880+0000 7f55cf8f0b80 10 client.15534 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:35.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5573d0b02ea0 con 0x5573d08ffe40 2024-05-27T07:23:35.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f559be00700 1 -- 172.21.15.12:0/2831330154 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55a80aa470 con 0x5573d08ffe40 2024-05-27T07:23:35.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f559be00700 1 -- 172.21.15.12:0/2831330154 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f55ac0051a0 con 0x5573d08ffe40 2024-05-27T07:23:35.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f55cf8f0b80 10 client.15534 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f55cf8f0b80 10 client.15534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f55cf8f0b80 20 client.15534 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0b03630 0x5573d0b05a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f55cf8f0b80 4 client.15534 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.881+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573d09a8570 con 0x5573d0b03630 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.882+0000 7f55b9400700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0b03630 0x5573d0b05a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.883+0000 7f55b9400700 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0b03630 0x5573d0b05a50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1cde93dc3e3f017 server_cookie=9c42e518e694700e in_seq=0 out_seq=0 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.883+0000 7f559be00700 10 client.15534 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.883+0000 7f559be00700 1 -- 172.21.15.12:0/2831330154 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573d09a8570 con 0x5573d0b03630 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.883+0000 7f559be00700 10 client.15534 handle_command_reply: tid=0 2024-05-27T07:23:35.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.883+0000 7f55cf8f0b80 1 client.15534 shutdown 2024-05-27T07:23:35.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.883+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d08ffe40 msgr2=0x5573d0a90f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.883+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d08ffe40 0x5573d0a90f70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f55ac003800 tx=0x7f55ac011920 comp rx=0 tx=0).stop 2024-05-27T07:23:35.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 shutdown_connections 2024-05-27T07:23:35.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5573d0b03630 0x5573d0b05a50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573d0a84d30 0x5573d0a9ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5573d08ffe40 0x5573d0a90f70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 --2- 172.21.15.12:0/2831330154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573d0a7edb0 0x5573d0a9f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f559be00700 0 client.15534 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 >> 172.21.15.12:0/2831330154 conn(0x5573d095ffc0 msgr2=0x5573d0a88db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 shutdown_connections 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 1 -- 172.21.15.12:0/2831330154 wait complete. 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.884+0000 7f55cf8f0b80 20 client.15534 trim_cache size 0 max 16384 2024-05-27T07:23:35.876 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55a00b2b10 msgr2=0x7f55a00b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55a00b2b10 0x7f55a00b5020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x5573d0a7fb30 tx=0x7f55b4009040 comp rx=0 tx=0).stop 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 msgr2=0x7f55bc103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc103740 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f55a8005910 tx=0x7f55a80047b0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 shutdown_connections 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55a00b2b10 0x7f55a00b5020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc10e8f0 0x7f55bc103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55bc105510 0x7f55bc103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 --2- 172.21.15.12:0/907944440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55bc105e60 0x7f55bc103740 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:35.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 >> 172.21.15.12:0/907944440 conn(0x7f55bc0fedb0 msgr2=0x7f55bc0ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:35.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 shutdown_connections 2024-05-27T07:23:35.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:35.885+0000 7f55c0e00700 1 -- 172.21.15.12:0/907944440 wait complete. 2024-05-27T07:23:35.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:35 smithi062 ceph-mon[35304]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:37.813 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:23:37.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:23:37.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:37 smithi012 ceph-mon[31473]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:37.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:37 smithi162 ceph-mon[35676]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:37.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:37 smithi062 ceph-mon[35304]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:38.299 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:39.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:39 smithi012 ceph-mon[31473]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:39.788 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:39 smithi062 ceph-mon[35304]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:39.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:39 smithi162 ceph-mon[35676]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 -- 172.21.15.12:0/4140622979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5410b890 msgr2=0x7f3a541131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 --2- 172.21.15.12:0/4140622979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5410b890 0x7f3a541131f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00b0d0 tx=0x7f3a4c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 -- 172.21.15.12:0/4140622979 shutdown_connections 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 --2- 172.21.15.12:0/4140622979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a5410b890 0x7f3a541131f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 --2- 172.21.15.12:0/4140622979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a54106610 0x7f3a54106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 --2- 172.21.15.12:0/4140622979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5410af40 0x7f3a5410b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 -- 172.21.15.12:0/4140622979 >> 172.21.15.12:0/4140622979 conn(0x7f3a54102020 msgr2=0x7f3a54104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 -- 172.21.15.12:0/4140622979 shutdown_connections 2024-05-27T07:23:40.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.803+0000 7f3a53200700 1 -- 172.21.15.12:0/4140622979 wait complete. 2024-05-27T07:23:40.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.804+0000 7f3a53200700 1 Processor -- start 2024-05-27T07:23:40.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.805+0000 7f3a53200700 1 -- start start 2024-05-27T07:23:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.805+0000 7f3a53200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a54106610 0x7f3a5419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.805+0000 7f3a53200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5410af40 0x7f3a5419d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.805+0000 7f3a53200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a5410b890 0x7f3a541a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.805+0000 7f3a53200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541163c0 con 0x7f3a54106610 2024-05-27T07:23:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.805+0000 7f3a53200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54116240 con 0x7f3a5410af40 2024-05-27T07:23:40.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.805+0000 7f3a53200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54116540 con 0x7f3a5410b890 2024-05-27T07:23:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a54106610 0x7f3a5419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a54106610 0x7f3a5419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53754/0 (socket says 172.21.15.12:53754) 2024-05-27T07:23:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 -- 172.21.15.12:0/1720936431 learned_addr learned my addr 172.21.15.12:0/1720936431 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51400700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5410af40 0x7f3a5419d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a52800700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a5410b890 0x7f3a541a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 -- 172.21.15.12:0/1720936431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a5410b890 msgr2=0x7f3a541a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a5410b890 0x7f3a541a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 -- 172.21.15.12:0/1720936431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5410af40 msgr2=0x7f3a5419d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5410af40 0x7f3a5419d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.806+0000 7f3a51e00700 1 -- 172.21.15.12:0/1720936431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a541a1be0 con 0x7f3a54106610 2024-05-27T07:23:40.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.807+0000 7f3a51e00700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a54106610 0x7f3a5419cd10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400d950 tx=0x7f3a4400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:40.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.807+0000 7f3a4aa00700 1 -- 172.21.15.12:0/1720936431 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44016070 con 0x7f3a54106610 2024-05-27T07:23:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.807+0000 7f3a4aa00700 1 -- 172.21.15.12:0/1720936431 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a44010460 con 0x7f3a54106610 2024-05-27T07:23:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.807+0000 7f3a4aa00700 1 -- 172.21.15.12:0/1720936431 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4400f620 con 0x7f3a54106610 2024-05-27T07:23:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.807+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a541a1ed0 con 0x7f3a54106610 2024-05-27T07:23:40.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.807+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a541a97f0 con 0x7f3a54106610 2024-05-27T07:23:40.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.811+0000 7f3a4aa00700 1 -- 172.21.15.12:0/1720936431 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7f3a44019070 con 0x7f3a54106610 2024-05-27T07:23:40.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.812+0000 7f3a67526b80 1 Processor -- start 2024-05-27T07:23:40.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a67526b80 1 -- start start 2024-05-27T07:23:40.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecc3eec0 0x5599ecdbcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecdca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a67526b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5599eccebec0 con 0x5599ecc3eec0 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a67526b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5599eccdba60 con 0x5599ecdbd8a0 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a67526b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5599eccdfaa0 con 0x5599ecdbd490 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a4aa00700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3a380b2b10 0x7f3a380b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.813+0000 7f3a4aa00700 1 -- 172.21.15.12:0/1720936431 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f3a440e2920 con 0x7f3a54106610 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.814+0000 7f3a52800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecc3eec0 0x5599ecdbcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.814+0000 7f3a51400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecdca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.814+0000 7f3a51e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.814+0000 7f3a51e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35620/0 (socket says 172.21.15.12:35620) 2024-05-27T07:23:40.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.814+0000 7f3a51e00700 1 -- 172.21.15.12:0/2639763651 learned_addr learned my addr 172.21.15.12:0/2639763651 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:40.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.814+0000 7f3a52800700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3a380b2b10 0x7f3a380b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a51e00700 1 -- 172.21.15.12:0/2639763651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 msgr2=0x5599ecdca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a51e00700 1 --2- 172.21.15.12:0/2639763651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecdca0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a51e00700 1 -- 172.21.15.12:0/2639763651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecc3eec0 msgr2=0x5599ecdbcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a51e00700 1 --2- 172.21.15.12:0/2639763651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecc3eec0 0x5599ecdbcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a51e00700 1 -- 172.21.15.12:0/2639763651 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ecca44a0 con 0x5599ecdbd490 2024-05-27T07:23:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a51e00700 1 --2- 172.21.15.12:0/2639763651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3a44005950 tx=0x7f3a440a9130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a121fa1ef6ebfe0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:40.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a52800700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3a380b2b10 0x7f3a380b5020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x5599ecdb9710 tx=0x7f3a4c00b040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.815+0000 7f3a3f400700 1 -- 172.21.15.12:0/2639763651 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44020070 con 0x5599ecdbd490 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.816+0000 7f3a3f400700 1 -- 172.21.15.12:0/2639763651 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a440ad950 con 0x5599ecdbd490 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.816+0000 7f3a3f400700 1 -- 172.21.15.12:0/2639763651 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a440a9a60 con 0x5599ecdbd490 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.816+0000 7f3a67526b80 1 -- 172.21.15.12:0/2639763651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 msgr2=0x5599ecdc7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.816+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2639763651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3a44005950 tx=0x7f3a440a9130 comp rx=0 tx=0).stop 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 -- 172.21.15.12:0/2639763651 shutdown_connections 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2639763651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecdca0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2639763651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecc3eec0 0x5599ecdbcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2639763651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 -- 172.21.15.12:0/2639763651 >> 172.21.15.12:0/2639763651 conn(0x5599ecc9f040 msgr2=0x5599ecdb8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 -- 172.21.15.12:0/2639763651 shutdown_connections 2024-05-27T07:23:40.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 -- 172.21.15.12:0/2639763651 wait complete. 2024-05-27T07:23:40.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.817+0000 7f3a67526b80 1 -- start start 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 1 Processor -- start 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdc8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5599ecbe2020 con 0x5599ecdbd8a0 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5599ecc7c7e0 con 0x5599ecdbd490 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5599ecdcaa10 con 0x5599ecc3eec0 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.818+0000 7f3a67526b80 10 client.0 _ll_register_callbacks cb 0x5599ecc85cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdc8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdc8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53766/0 (socket says 172.21.15.12:53766) 2024-05-27T07:23:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 -- 172.21.15.12:0/2879249265 learned_addr learned my addr 172.21.15.12:0/2879249265 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a51400700 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a51e00700 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 -- 172.21.15.12:0/2879249265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 msgr2=0x5599ecdc75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 -- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd490 msgr2=0x5599ecdc7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.819+0000 7f3a52800700 1 -- 172.21.15.12:0/2879249265 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599ecc2b950 con 0x5599ecdbd8a0 2024-05-27T07:23:40.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a52800700 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdc8070 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00b010 tx=0x7f3a4c0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f25738f4af7903c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a33e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a33e00700 1 -- 172.21.15.12:0/2879249265 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c020c40 con 0x5599ecdbd8a0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a440ad950 con 0x5599ecdbd8a0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a33e00700 1 -- 172.21.15.12:0/2879249265 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f3a4c020de0 con 0x5599ecdbd8a0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a33e00700 1 -- 172.21.15.12:0/2879249265 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3a4c03b050 con 0x5599ecdbd8a0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a67526b80 10 client.15546 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.820+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599eccac0e0 con 0x5599ecdbd8a0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a33e00700 1 -- 172.21.15.12:0/2879249265 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a440ad950 con 0x5599ecdbd8a0 2024-05-27T07:23:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a33e00700 1 -- 172.21.15.12:0/2879249265 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3a4c011070 con 0x5599ecdbd8a0 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a67526b80 10 client.15546 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a67526b80 10 client.15546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a67526b80 20 client.15546 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ecdd7940 0x5599ecdd9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a67526b80 4 client.15546 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.821+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599eccede10 con 0x5599ecdd7940 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.822+0000 7f3a51e00700 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ecdd7940 0x5599ecdd9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.823+0000 7f3a51e00700 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ecdd7940 0x5599ecdd9d80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a40a7acec7ebe0f server_cookie=941541a0aebc7d55 in_seq=0 out_seq=0 2024-05-27T07:23:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.823+0000 7f3a33e00700 10 client.15546 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.825+0000 7f3a33e00700 1 -- 172.21.15.12:0/2879249265 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5599eccede10 con 0x5599ecdd7940 2024-05-27T07:23:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.825+0000 7f3a33e00700 10 client.15546 handle_command_reply: tid=0 2024-05-27T07:23:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.825+0000 7f3a67526b80 1 client.15546 shutdown 2024-05-27T07:23:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 msgr2=0x5599ecdc8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdc8070 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00b010 tx=0x7f3a4c0099e0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 shutdown_connections 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdc8070 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ecdd7940 0x5599ecdd9d80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 --2- 172.21.15.12:0/2879249265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd490 0x5599ecdc7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 >> 172.21.15.12:0/2879249265 conn(0x5599ecc9f040 msgr2=0x5599ecdb8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a33e00700 0 client.15546 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 shutdown_connections 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 1 -- 172.21.15.12:0/2879249265 wait complete. 2024-05-27T07:23:40.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.826+0000 7f3a67526b80 20 client.15546 trim_cache size 0 max 16384 2024-05-27T07:23:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 Processor -- start 2024-05-27T07:23:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 -- start start 2024-05-27T07:23:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd490 0x5599ecdd3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecddf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541b4ea0 con 0x5599ecdbd490 2024-05-27T07:23:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541b2730 con 0x5599ecdbd8a0 2024-05-27T07:23:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.855+0000 7f3a67526b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541b0580 con 0x5599ecc3eec0 2024-05-27T07:23:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd490 0x5599ecdd3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35644/0 (socket says 172.21.15.12:35644) 2024-05-27T07:23:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51e00700 1 -- 172.21.15.12:0/581754074 learned_addr learned my addr 172.21.15.12:0/581754074 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a52800700 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecddf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51400700 1 -- 172.21.15.12:0/581754074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 msgr2=0x5599ecdc2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51400700 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51400700 1 -- 172.21.15.12:0/581754074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 msgr2=0x5599ecddf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51400700 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecddf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.856+0000 7f3a51400700 1 -- 172.21.15.12:0/581754074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ecdc5b60 con 0x5599ecdbd490 2024-05-27T07:23:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.857+0000 7f3a51400700 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd490 0x5599ecdd3ab0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x5599ecdb9310 tx=0x7f3a40003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569cdb264d7d6c12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.857+0000 7f3a3ea00700 1 -- 172.21.15.12:0/581754074 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4000d640 con 0x5599ecdbd490 2024-05-27T07:23:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.857+0000 7f3a3ea00700 1 -- 172.21.15.12:0/581754074 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a4000dc70 con 0x5599ecdbd490 2024-05-27T07:23:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.857+0000 7f3a3ea00700 1 -- 172.21.15.12:0/581754074 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40011b70 con 0x5599ecdbd490 2024-05-27T07:23:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.858+0000 7f3a67526b80 1 -- 172.21.15.12:0/581754074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd490 msgr2=0x5599ecdd3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.858+0000 7f3a67526b80 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd490 0x5599ecdd3ab0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x5599ecdb9310 tx=0x7f3a40003960 comp rx=0 tx=0).stop 2024-05-27T07:23:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.859+0000 7f3a67526b80 1 -- 172.21.15.12:0/581754074 shutdown_connections 2024-05-27T07:23:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.859+0000 7f3a67526b80 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecdbd8a0 0x5599ecddf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.859+0000 7f3a67526b80 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecc3eec0 0x5599ecdc2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.859+0000 7f3a67526b80 1 --2- 172.21.15.12:0/581754074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd490 0x5599ecdd3ab0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.859+0000 7f3a67526b80 1 -- 172.21.15.12:0/581754074 >> 172.21.15.12:0/581754074 conn(0x5599ecc9f040 msgr2=0x5599ecdc0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.859+0000 7f3a67526b80 1 -- 172.21.15.12:0/581754074 shutdown_connections 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.859+0000 7f3a67526b80 1 -- 172.21.15.12:0/581754074 wait complete. 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.860+0000 7f3a67526b80 1 -- start start 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.860+0000 7f3a67526b80 1 Processor -- start 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.860+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecc3eec0 0x5599ecdd0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.860+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdcc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a67526b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a67526b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541b0bf0 con 0x5599ecdbd8a0 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a67526b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541b2060 con 0x5599ecc3eec0 2024-05-27T07:23:40.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a67526b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541b12c0 con 0x5599ecdbd490 2024-05-27T07:23:40.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a67526b80 10 client.0 _ll_register_callbacks cb 0x5599ecc85cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:40.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a51e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecc3eec0 0x5599ecdd0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a51400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdcc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a52800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a51400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdcc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35654/0 (socket says 172.21.15.12:35654) 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.861+0000 7f3a51400700 1 -- 172.21.15.12:0/1648060299 learned_addr learned my addr 172.21.15.12:0/1648060299 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a51400700 1 -- 172.21.15.12:0/1648060299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecc3eec0 msgr2=0x5599ecdd0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a51400700 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecc3eec0 0x5599ecdd0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a51400700 1 -- 172.21.15.12:0/1648060299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 msgr2=0x5599ecdccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a51400700 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a51400700 1 -- 172.21.15.12:0/1648060299 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599ecdcd0a0 con 0x5599ecdbd490 2024-05-27T07:23:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a52800700 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:23:40.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a51400700 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdcc570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3a40011720 tx=0x7f3a40011750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=545cbd4ed7af82f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:40.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.862+0000 7f3a33e00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:23:40.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.863+0000 7f3a33e00700 1 -- 172.21.15.12:0/1648060299 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4000d910 con 0x5599ecdbd490 2024-05-27T07:23:40.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.863+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a4000dc70 con 0x5599ecdbd490 2024-05-27T07:23:40.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.863+0000 7f3a33e00700 1 -- 172.21.15.12:0/1648060299 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f3a40018040 con 0x5599ecdbd490 2024-05-27T07:23:40.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.863+0000 7f3a33e00700 1 -- 172.21.15.12:0/1648060299 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3a4000dab0 con 0x5599ecdbd490 2024-05-27T07:23:40.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.863+0000 7f3a67526b80 10 client.25252 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:40.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.863+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599ece4b9e0 con 0x5599ecdbd490 2024-05-27T07:23:40.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.864+0000 7f3a33e00700 1 -- 172.21.15.12:0/1648060299 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a4000dc70 con 0x5599ecdbd490 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.864+0000 7f3a33e00700 1 -- 172.21.15.12:0/1648060299 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3a40004750 con 0x5599ecdbd490 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.864+0000 7f3a67526b80 10 client.25252 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.864+0000 7f3a67526b80 10 client.25252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.864+0000 7f3a67526b80 20 client.25252 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.865+0000 7f3a67526b80 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ece4c0f0 0x5599ece4e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.865+0000 7f3a67526b80 4 client.25252 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.865+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5599ecce75f0 con 0x5599ece4c0f0 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.865+0000 7f3a51e00700 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ece4c0f0 0x5599ece4e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:40.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.866+0000 7f3a51e00700 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ece4c0f0 0x5599ece4e510 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73de80f45a50a6a server_cookie=ea3f439415e20732 in_seq=0 out_seq=0 2024-05-27T07:23:40.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.866+0000 7f3a33e00700 10 client.25252 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a33e00700 1 -- 172.21.15.12:0/1648060299 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5599ecce75f0 con 0x5599ece4c0f0 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a33e00700 10 client.25252 handle_command_reply: tid=0 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 client.25252 shutdown 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 msgr2=0x5599ecdcc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdcc570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3a40011720 tx=0x7f3a40011750 comp rx=0 tx=0).stop 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 shutdown_connections 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5599ece4c0f0 0x5599ece4e510 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599ecc3eec0 0x5599ecdd0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5599ecdbd8a0 0x5599ecdccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 --2- 172.21.15.12:0/1648060299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599ecdbd490 0x5599ecdcc570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 >> 172.21.15.12:0/1648060299 conn(0x5599ecc9f040 msgr2=0x5599ecdc94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a33e00700 0 client.25252 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:40.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.868+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 shutdown_connections 2024-05-27T07:23:40.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.869+0000 7f3a67526b80 1 -- 172.21.15.12:0/1648060299 wait complete. 2024-05-27T07:23:40.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.869+0000 7f3a67526b80 20 client.25252 trim_cache size 0 max 16384 2024-05-27T07:23:40.860 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:23:40.860 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:23:40.861 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "ef6fc17f-c5cd-4057-8f4e-a36b65d3d980", 2024-05-27T07:23:40.861 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:23:40.861 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:23:40.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3a380b2b10 msgr2=0x7f3a380b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3a380b2b10 0x7f3a380b5020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x5599ecdb9710 tx=0x7f3a4c00b040 comp rx=0 tx=0).stop 2024-05-27T07:23:40.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a54106610 msgr2=0x7f3a5419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:40.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a54106610 0x7f3a5419cd10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400d950 tx=0x7f3a4400dd10 comp rx=0 tx=0).stop 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 shutdown_connections 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3a380b2b10 0x7f3a380b5020 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a5410b890 0x7f3a541a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a54106610 0x7f3a5419cd10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 --2- 172.21.15.12:0/1720936431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5410af40 0x7f3a5419d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.870+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 >> 172.21.15.12:0/1720936431 conn(0x7f3a54102020 msgr2=0x7f3a5410a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.871+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 shutdown_connections 2024-05-27T07:23:40.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:40.871+0000 7f3a53200700 1 -- 172.21.15.12:0/1720936431 wait complete. 2024-05-27T07:23:41.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:41 smithi012 ceph-mon[31473]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T07:23:41.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:41 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:41.833 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:23:41.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:41 smithi162 ceph-mon[35676]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T07:23:41.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:41 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:41.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:41 smithi062 ceph-mon[35304]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-27T07:23:41.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:41 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:42.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:42 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:23:42.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:42 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:42.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:42 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:23:42.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:42 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:42.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:42 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:23:42.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:42 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:23:42.834 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:42 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:23:42.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:42 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:23:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:42 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:42 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:23:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:42 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:42 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:23:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:42 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:23:42.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:42 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:23:42.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:42 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:23:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:42 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:42 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:23:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:42 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:42 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:23:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:42 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:23:42.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:42 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:23:43.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:43 smithi012 ceph-mon[31473]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:43.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:43 smithi162 ceph-mon[35676]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:43.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:43 smithi062 ceph-mon[35304]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:23:44.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.859+0000 7f2d36e00700 1 -- 172.21.15.12:0/1525569968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105bd0 msgr2=0x7f2d3010e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.859+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1525569968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105bd0 0x7f2d3010e120 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009990 tx=0x7f2d1c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.860+0000 7f2d36e00700 1 -- 172.21.15.12:0/1525569968 shutdown_connections 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.860+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1525569968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3010e6f0 0x7f2d30110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.860+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1525569968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d30105280 0x7f2d30105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.860+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1525569968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105bd0 0x7f2d3010e120 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.860+0000 7f2d36e00700 1 -- 172.21.15.12:0/1525569968 >> 172.21.15.12:0/1525569968 conn(0x7f2d30078ab0 msgr2=0x7f2d30078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.860+0000 7f2d36e00700 1 -- 172.21.15.12:0/1525569968 shutdown_connections 2024-05-27T07:23:44.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.860+0000 7f2d36e00700 1 -- 172.21.15.12:0/1525569968 wait complete. 2024-05-27T07:23:44.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.861+0000 7f2d36e00700 1 Processor -- start 2024-05-27T07:23:44.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.861+0000 7f2d36e00700 1 -- start start 2024-05-27T07:23:44.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.861+0000 7f2d36e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105280 0x7f2d3019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.861+0000 7f2d36e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d30105bd0 0x7f2d3019d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d36e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d3010e6f0 0x7f2d301a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d36e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d30114150 con 0x7f2d30105280 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d36e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2d30113fd0 con 0x7f2d30105bd0 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d36e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301142d0 con 0x7f2d3010e6f0 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105280 0x7f2d3019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105280 0x7f2d3019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42740/0 (socket says 172.21.15.12:42740) 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 -- 172.21.15.12:0/1959512853 learned_addr learned my addr 172.21.15.12:0/1959512853 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d36400700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d3010e6f0 0x7f2d301a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35000700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d30105bd0 0x7f2d3019d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 -- 172.21.15.12:0/1959512853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d3010e6f0 msgr2=0x7f2d301a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d3010e6f0 0x7f2d301a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 -- 172.21.15.12:0/1959512853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d30105bd0 msgr2=0x7f2d3019d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d30105bd0 0x7f2d3019d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.862+0000 7f2d35a00700 1 -- 172.21.15.12:0/1959512853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d301a1c70 con 0x7f2d30105280 2024-05-27T07:23:44.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.863+0000 7f2d36400700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d3010e6f0 0x7f2d301a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:44.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.863+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105280 0x7f2d3019cda0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400d950 tx=0x7f2d2400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:44.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.863+0000 7f2d2e000700 1 -- 172.21.15.12:0/1959512853 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24016070 con 0x7f2d30105280 2024-05-27T07:23:44.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.863+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d301a1f60 con 0x7f2d30105280 2024-05-27T07:23:44.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.863+0000 7f2d2e000700 1 -- 172.21.15.12:0/1959512853 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d24010460 con 0x7f2d30105280 2024-05-27T07:23:44.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.863+0000 7f2d2e000700 1 -- 172.21.15.12:0/1959512853 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400f620 con 0x7f2d30105280 2024-05-27T07:23:44.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.864+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d301a9880 con 0x7f2d30105280 2024-05-27T07:23:44.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 Processor -- start 2024-05-27T07:23:44.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 -- start start 2024-05-27T07:23:44.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b5b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c779dbe40 0x563c77b5bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b676e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563c77a88e40 con 0x563c779dbe40 2024-05-27T07:23:44.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563c77a789e0 con 0x563c77ac79f0 2024-05-27T07:23:44.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.869+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563c77a7ca20 con 0x563c77b652c0 2024-05-27T07:23:44.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.870+0000 7f2d2e000700 1 -- 172.21.15.12:0/1959512853 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7f2d24019070 con 0x7f2d30105280 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.870+0000 7f2d35a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c779dbe40 0x563c77b5bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.870+0000 7f2d35a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c779dbe40 0x563c77b5bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42746/0 (socket says 172.21.15.12:42746) 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.870+0000 7f2d35a00700 1 -- 172.21.15.12:0/1547248282 learned_addr learned my addr 172.21.15.12:0/1547248282 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.870+0000 7f2d35000700 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.870+0000 7f2d36400700 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b5b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.871+0000 7f2d35a00700 1 -- 172.21.15.12:0/1547248282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 msgr2=0x563c77b676e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.871+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b676e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.871+0000 7f2d35a00700 1 -- 172.21.15.12:0/1547248282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 msgr2=0x563c77b5b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.871+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b5b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.871+0000 7f2d35a00700 1 -- 172.21.15.12:0/1547248282 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c77a41420 con 0x563c779dbe40 2024-05-27T07:23:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.871+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c779dbe40 0x563c77b5bc90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d240dfd90 tx=0x7f2d240e6d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dba580532e904b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.872+0000 7f2d22a00700 1 -- 172.21.15.12:0/1547248282 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240f2070 con 0x563c779dbe40 2024-05-27T07:23:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.872+0000 7f2d22a00700 1 -- 172.21.15.12:0/1547248282 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d240e7040 con 0x563c779dbe40 2024-05-27T07:23:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.872+0000 7f2d22a00700 1 -- 172.21.15.12:0/1547248282 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240eecc0 con 0x563c779dbe40 2024-05-27T07:23:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.872+0000 7f2d2e000700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2d180b2b90 0x7f2d180b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.872+0000 7f2d2e000700 1 -- 172.21.15.12:0/1959512853 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2d240e2c30 con 0x7f2d30105280 2024-05-27T07:23:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.872+0000 7f2d36400700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2d180b2b90 0x7f2d180b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1547248282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c779dbe40 msgr2=0x563c77b5bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c779dbe40 0x563c77b5bc90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d240dfd90 tx=0x7f2d240e6d30 comp rx=0 tx=0).stop 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1547248282 shutdown_connections 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b5b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1547248282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c779dbe40 0x563c77b5bc90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1547248282 >> 172.21.15.12:0/1547248282 conn(0x563c77a75390 msgr2=0x563c77b56710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1547248282 shutdown_connections 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d36400700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2d180b2b90 0x7f2d180b50a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x563c77b57a90 tx=0x7f2d28009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.873+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1547248282 wait complete. 2024-05-27T07:23:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.874+0000 7f2d46e78b80 1 -- start start 2024-05-27T07:23:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.874+0000 7f2d46e78b80 1 Processor -- start 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b62710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b62c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563c7797efa0 con 0x563c77b652c0 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563c77a19760 con 0x563c77ac79f0 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563c77b67e00 con 0x563c779dbe40 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d46e78b80 10 client.0 _ll_register_callbacks cb 0x563c77a22c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.875+0000 7f2d36400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b62c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d36400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b62c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42760/0 (socket says 172.21.15.12:42760) 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d36400700 1 -- 172.21.15.12:0/793507383 learned_addr learned my addr 172.21.15.12:0/793507383 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d35a00700 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d35000700 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b62710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d36400700 1 -- 172.21.15.12:0/793507383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 msgr2=0x563c77b621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d36400700 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d36400700 1 -- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 msgr2=0x563c77b62710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d36400700 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b62710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.876+0000 7f2d36400700 1 -- 172.21.15.12:0/793507383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c779c88d0 con 0x563c77b652c0 2024-05-27T07:23:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d36400700 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b62c50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009fd0 tx=0x7f2d28004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a148723cb4a436 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:44.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d17400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d17400700 1 -- 172.21.15.12:0/793507383 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2801e050 con 0x563c77b652c0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d240e7040 con 0x563c77b652c0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d17400700 1 -- 172.21.15.12:0/793507383 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2d28004bb0 con 0x563c77b652c0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d17400700 1 -- 172.21.15.12:0/793507383 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2d28030050 con 0x563c77b652c0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d46e78b80 10 client.15564 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.877+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c77a49060 con 0x563c77b652c0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d17400700 1 -- 172.21.15.12:0/793507383 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d240e7040 con 0x563c77b652c0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d17400700 1 -- 172.21.15.12:0/793507383 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2d280236e0 con 0x563c77b652c0 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d46e78b80 10 client.15564 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d46e78b80 10 client.15564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d46e78b80 20 client.15564 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:44.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77b74d70 0x563c77b771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d46e78b80 4 client.15564 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:44.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.878+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c77a8ad90 con 0x563c77b74d70 2024-05-27T07:23:44.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.879+0000 7f2d35a00700 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77b74d70 0x563c77b771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.880+0000 7f2d17400700 10 client.15564 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:44.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.880+0000 7f2d35a00700 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77b74d70 0x563c77b771f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c355f91b7561743 server_cookie=64c58493b3f17cbf in_seq=0 out_seq=0 2024-05-27T07:23:44.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.882+0000 7f2d17400700 1 -- 172.21.15.12:0/793507383 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563c77a8ad90 con 0x563c77b74d70 2024-05-27T07:23:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.882+0000 7f2d17400700 10 client.15564 handle_command_reply: tid=0 2024-05-27T07:23:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.882+0000 7f2d46e78b80 1 client.15564 shutdown 2024-05-27T07:23:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.882+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 msgr2=0x563c77b62c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.882+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b62c50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009fd0 tx=0x7f2d28004ad0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 shutdown_connections 2024-05-27T07:23:44.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b62c50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77b74d70 0x563c77b771f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/793507383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b62710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 >> 172.21.15.12:0/793507383 conn(0x563c77a75390 msgr2=0x563c77b56260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d17400700 0 client.15564 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 shutdown_connections 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 1 -- 172.21.15.12:0/793507383 wait complete. 2024-05-27T07:23:44.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.883+0000 7f2d46e78b80 20 client.15564 trim_cache size 0 max 16384 2024-05-27T07:23:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.920+0000 7f2d46e78b80 1 Processor -- start 2024-05-27T07:23:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d46e78b80 1 -- start start 2024-05-27T07:23:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b5be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b70f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301b4f80 con 0x563c77b652c0 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301b2810 con 0x563c77ac79f0 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301b0660 con 0x563c779dbe40 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.921+0000 7f2d36400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b70f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b70f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42770/0 (socket says 172.21.15.12:42770) 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 -- 172.21.15.12:0/2614299614 learned_addr learned my addr 172.21.15.12:0/2614299614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d35000700 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d35a00700 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b5be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 -- 172.21.15.12:0/2614299614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 msgr2=0x563c77b5be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b5be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 -- 172.21.15.12:0/2614299614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 msgr2=0x563c77b613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 -- 172.21.15.12:0/2614299614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c77b5ddd0 con 0x563c77b652c0 2024-05-27T07:23:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d35a00700 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b5be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.922+0000 7f2d36400700 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b70f00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2d28000f80 tx=0x7f2d28005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7035eb950725136a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.923+0000 7f2d22000700 1 -- 172.21.15.12:0/2614299614 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28039070 con 0x563c77b652c0 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.923+0000 7f2d22000700 1 -- 172.21.15.12:0/2614299614 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d2801f040 con 0x563c77b652c0 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.923+0000 7f2d22000700 1 -- 172.21.15.12:0/2614299614 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2801e040 con 0x563c77b652c0 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 -- 172.21.15.12:0/2614299614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 msgr2=0x563c77b70f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b70f00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2d28000f80 tx=0x7f2d28005e90 comp rx=0 tx=0).stop 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 -- 172.21.15.12:0/2614299614 shutdown_connections 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77b652c0 0x563c77b70f00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c779dbe40 0x563c77b5be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/2614299614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c77ac79f0 0x563c77b613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 -- 172.21.15.12:0/2614299614 >> 172.21.15.12:0/2614299614 conn(0x563c77a75390 msgr2=0x563c77b5a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 -- 172.21.15.12:0/2614299614 shutdown_connections 2024-05-27T07:23:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.924+0000 7f2d46e78b80 1 -- 172.21.15.12:0/2614299614 wait complete. 2024-05-27T07:23:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.925+0000 7f2d46e78b80 1 -- start start 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.925+0000 7f2d46e78b80 1 Processor -- start 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c779dbe40 0x563c77b6d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77ac79f0 0x563c77b68c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d46e78b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b69220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301b0cd0 con 0x563c77ac79f0 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301b2140 con 0x563c779dbe40 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d46e78b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301b13a0 con 0x563c77b652c0 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d46e78b80 10 client.0 _ll_register_callbacks cb 0x563c77a22c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d35000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77ac79f0 0x563c77b68c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d35000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77ac79f0 0x563c77b68c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42780/0 (socket says 172.21.15.12:42780) 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d35000700 1 -- 172.21.15.12:0/1850237010 learned_addr learned my addr 172.21.15.12:0/1850237010 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:23:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c779dbe40 0x563c77b6d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.926+0000 7f2d36400700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b69220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d35000700 1 -- 172.21.15.12:0/1850237010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 msgr2=0x563c77b69220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d35000700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b69220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d35000700 1 -- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c779dbe40 msgr2=0x563c77b6d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d35000700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c779dbe40 0x563c77b6d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d35000700 1 -- 172.21.15.12:0/1850237010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c77b699c0 con 0x563c77ac79f0 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d36400700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d35000700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77ac79f0 0x563c77b68c80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c000c00 tx=0x7f2d1c005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7674d655877576f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.927+0000 7f2d17400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d17400700 1 -- 172.21.15.12:0/1850237010 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c003bd0 con 0x563c77ac79f0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d2801f040 con 0x563c77ac79f0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d17400700 1 -- 172.21.15.12:0/1850237010 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f2d1c003d70 con 0x563c77ac79f0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d17400700 1 -- 172.21.15.12:0/1850237010 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2d1c032050 con 0x563c77ac79f0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d46e78b80 10 client.15570 fetch_fsmap learned FSMap version 27 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c77bdf790 con 0x563c77ac79f0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d17400700 1 -- 172.21.15.12:0/1850237010 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d2801f040 con 0x563c77ac79f0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.928+0000 7f2d17400700 1 -- 172.21.15.12:0/1850237010 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2d1c0248a0 con 0x563c77ac79f0 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.929+0000 7f2d46e78b80 10 client.15570 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.929+0000 7f2d46e78b80 10 client.15570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.929+0000 7f2d46e78b80 20 client.15570 populate_metadata read hostname 'smithi012' 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.929+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77bdff20 0x563c77be2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.929+0000 7f2d46e78b80 4 client.15570 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:23:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.929+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c77a84570 con 0x563c77bdff20 2024-05-27T07:23:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.930+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77bdff20 0x563c77be2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.931+0000 7f2d17400700 10 client.15570 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.931+0000 7f2d35a00700 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77bdff20 0x563c77be2340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbb6bf2306a59afa server_cookie=b3e0e9099645fef2 in_seq=0 out_seq=0 2024-05-27T07:23:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d17400700 1 -- 172.21.15.12:0/1850237010 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563c77a84570 con 0x563c77bdff20 2024-05-27T07:23:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d17400700 10 client.15570 handle_command_reply: tid=0 2024-05-27T07:23:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 client.15570 shutdown 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77ac79f0 msgr2=0x563c77b68c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77ac79f0 0x563c77b68c80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c000c00 tx=0x7f2d1c005790 comp rx=0 tx=0).stop 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 shutdown_connections 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563c77bdff20 0x563c77be2340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c779dbe40 0x563c77b6d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c77b652c0 0x563c77b69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 --2- 172.21.15.12:0/1850237010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563c77ac79f0 0x563c77b68c80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 >> 172.21.15.12:0/1850237010 conn(0x563c77a75390 msgr2=0x563c77b61fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d17400700 0 client.15570 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 shutdown_connections 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 1 -- 172.21.15.12:0/1850237010 wait complete. 2024-05-27T07:23:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.932+0000 7f2d46e78b80 20 client.15570 trim_cache size 0 max 16384 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "ef6fc17f-c5cd-4057-8f4e-a36b65d3d980": { 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "ef6fc17f-c5cd-4057-8f4e-a36b65d3d980", 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:23:44.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2d180b2b90 msgr2=0x7f2d180b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2d180b2b90 0x7f2d180b50a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x563c77b57a90 tx=0x7f2d28009040 comp rx=0 tx=0).stop 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105280 msgr2=0x7f2d3019cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105280 0x7f2d3019cda0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400d950 tx=0x7f2d2400dd10 comp rx=0 tx=0).stop 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 shutdown_connections 2024-05-27T07:23:44.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2d180b2b90 0x7f2d180b50a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d3010e6f0 0x7f2d301a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d30105280 0x7f2d3019cda0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 --2- 172.21.15.12:0/1959512853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d30105bd0 0x7f2d3019d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.933+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 >> 172.21.15.12:0/1959512853 conn(0x7f2d30078ab0 msgr2=0x7f2d300ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.934+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 shutdown_connections 2024-05-27T07:23:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:23:44.934+0000 7f2d36e00700 1 -- 172.21.15.12:0/1959512853 wait complete. 2024-05-27T07:23:45.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:45 smithi012 ceph-mon[31473]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-27T07:23:45.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:45 smithi162 ceph-mon[35676]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-27T07:23:45.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:45 smithi062 ceph-mon[35304]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-27T07:23:46.088 INFO:tasks.cephfs.filesystem:scrub status for tag:ef6fc17f-c5cd-4057-8f4e-a36b65d3d980 - {'path': '/', 'tag': 'ef6fc17f-c5cd-4057-8f4e-a36b65d3d980', 'options': 'recursive,force'} 2024-05-27T07:23:46.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:46 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:23:46.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:46 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:23:46.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:46 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:23:47.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:47 smithi012 ceph-mon[31473]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:47.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:47 smithi162 ceph-mon[35676]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:47.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:47 smithi062 ceph-mon[35304]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:49.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:49 smithi012 ceph-mon[31473]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-27T07:23:49.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:49 smithi162 ceph-mon[35676]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-27T07:23:49.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:49 smithi062 ceph-mon[35304]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-27T07:23:51.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:51 smithi012 ceph-mon[31473]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:51.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:51 smithi162 ceph-mon[35676]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:51.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:51 smithi062 ceph-mon[35304]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:53.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:53 smithi012 ceph-mon[31473]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:53.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:53 smithi162 ceph-mon[35676]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:53.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:53 smithi062 ceph-mon[35304]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-27T07:23:55.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:55 smithi012 ceph-mon[31473]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-27T07:23:55.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:55 smithi012 ceph-mon[31473]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:23:55.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:55 smithi162 ceph-mon[35676]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-27T07:23:55.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:55 smithi162 ceph-mon[35676]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:23:55.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:55 smithi062 ceph-mon[35304]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-27T07:23:55.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:55 smithi062 ceph-mon[35304]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:23:57.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:57 smithi162 ceph-mon[35676]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:23:57.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:57 smithi062 ceph-mon[35304]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:23:58.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:57 smithi012 ceph-mon[31473]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:23:59.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:23:59 smithi162 ceph-mon[35676]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-27T07:23:59.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:23:59 smithi062 ceph-mon[35304]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-27T07:24:00.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:23:59 smithi012 ceph-mon[31473]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-27T07:24:01.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:01 smithi162 ceph-mon[35676]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:24:01.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:01 smithi062 ceph-mon[35304]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:24:02.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:01 smithi012 ceph-mon[31473]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:24:03.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:03 smithi162 ceph-mon[35676]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:24:03.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:03 smithi062 ceph-mon[35304]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:24:04.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:03 smithi012 ceph-mon[31473]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-27T07:24:05.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:05 smithi162 ceph-mon[35676]: pgmap v54: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T07:24:05.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:05 smithi062 ceph-mon[35304]: pgmap v54: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T07:24:06.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:05 smithi012 ceph-mon[31473]: pgmap v54: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-27T07:24:08.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:08 smithi162 ceph-mon[35676]: pgmap v55: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-27T07:24:08.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:08 smithi062 ceph-mon[35304]: pgmap v55: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-27T07:24:08.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:08 smithi012 ceph-mon[31473]: pgmap v55: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-27T07:24:10.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:10 smithi162 ceph-mon[35676]: pgmap v56: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:24:10.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:10 smithi062 ceph-mon[35304]: pgmap v56: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:24:10.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:10 smithi012 ceph-mon[31473]: pgmap v56: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:24:12.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:12 smithi162 ceph-mon[35676]: pgmap v57: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:24:12.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:12 smithi062 ceph-mon[35304]: pgmap v57: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:24:12.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:12 smithi012 ceph-mon[31473]: pgmap v57: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:24:14.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:14 smithi162 ceph-mon[35676]: pgmap v58: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:24:14.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:14 smithi062 ceph-mon[35304]: pgmap v58: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:24:14.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:14 smithi012 ceph-mon[31473]: pgmap v58: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-27T07:24:16.090 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:24:16.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:16 smithi162 ceph-mon[35676]: pgmap v59: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:24:16.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:16 smithi062 ceph-mon[35304]: pgmap v59: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:24:16.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:16 smithi012 ceph-mon[31473]: pgmap v59: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-27T07:24:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:18.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:18 smithi012 ceph-mon[31473]: pgmap v60: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:24:18.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:18 smithi162 ceph-mon[35676]: pgmap v60: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:24:18.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:18 smithi062 ceph-mon[35304]: pgmap v60: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-27T07:24:19.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 -- 172.21.15.12:0/2891401754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa82810e6f0 msgr2=0x7fa828110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 --2- 172.21.15.12:0/2891401754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa82810e6f0 0x7fa828110bd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa82000b0a0 tx=0x7fa82000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 -- 172.21.15.12:0/2891401754 shutdown_connections 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 --2- 172.21.15.12:0/2891401754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa82810e6f0 0x7fa828110bd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 --2- 172.21.15.12:0/2891401754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa828105280 0x7fa828105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 --2- 172.21.15.12:0/2891401754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa828105bd0 0x7fa82810e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 -- 172.21.15.12:0/2891401754 >> 172.21.15.12:0/2891401754 conn(0x7fa828078af0 msgr2=0x7fa828078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 -- 172.21.15.12:0/2891401754 shutdown_connections 2024-05-27T07:24:19.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.443+0000 7fa82d200700 1 -- 172.21.15.12:0/2891401754 wait complete. 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 Processor -- start 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 -- start start 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa828105280 0x7fa82819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa828105bd0 0x7fa82819d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa82810e6f0 0x7fa8281a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa828114100 con 0x7fa828105bd0 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa828113f80 con 0x7fa82810e6f0 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.445+0000 7fa82d200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa828114280 con 0x7fa828105280 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa828105bd0 0x7fa82819d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa828105bd0 0x7fa82819d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35716/0 (socket says 172.21.15.12:35716) 2024-05-27T07:24:19.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 -- 172.21.15.12:0/3419597931 learned_addr learned my addr 172.21.15.12:0/3419597931 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa827400700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa828105280 0x7fa82819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa827e00700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa82810e6f0 0x7fa8281a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 -- 172.21.15.12:0/3419597931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa828105280 msgr2=0x7fa82819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa828105280 0x7fa82819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 -- 172.21.15.12:0/3419597931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa82810e6f0 msgr2=0x7fa8281a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa82810e6f0 0x7fa8281a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.446+0000 7fa826a00700 1 -- 172.21.15.12:0/3419597931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8281a1be0 con 0x7fa828105bd0 2024-05-27T07:24:19.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.447+0000 7fa826a00700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa828105bd0 0x7fa82819d2b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa81400d950 tx=0x7fa81400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:19.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.447+0000 7fa81fe00700 1 -- 172.21.15.12:0/3419597931 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814016070 con 0x7fa828105bd0 2024-05-27T07:24:19.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.447+0000 7fa81fe00700 1 -- 172.21.15.12:0/3419597931 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa814010460 con 0x7fa828105bd0 2024-05-27T07:24:19.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.447+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8281a1ed0 con 0x7fa828105bd0 2024-05-27T07:24:19.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.447+0000 7fa81fe00700 1 -- 172.21.15.12:0/3419597931 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81400f620 con 0x7fa828105bd0 2024-05-27T07:24:19.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.447+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8281a97f0 con 0x7fa828105bd0 2024-05-27T07:24:19.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 Processor -- start 2024-05-27T07:24:19.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa81fe00700 1 -- 172.21.15.12:0/3419597931 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa814019070 con 0x7fa828105bd0 2024-05-27T07:24:19.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 -- start start 2024-05-27T07:24:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d391ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d39cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392820 0x55df8d39f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55df8d2c0e40 con 0x55df8d392820 2024-05-27T07:24:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55df8d2b09e0 con 0x55df8d392410 2024-05-27T07:24:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.454+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55df8d2b4a20 con 0x55df8d213e40 2024-05-27T07:24:19.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392820 0x55df8d39f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392820 0x55df8d39f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35722/0 (socket says 172.21.15.12:35722) 2024-05-27T07:24:19.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 -- 172.21.15.12:0/2528986824 learned_addr learned my addr 172.21.15.12:0/2528986824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:19.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa827e00700 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d391ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa827400700 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d39cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 -- 172.21.15.12:0/2528986824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 msgr2=0x55df8d391ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d391ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 -- 172.21.15.12:0/2528986824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 msgr2=0x55df8d39cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d39cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa826a00700 1 -- 172.21.15.12:0/2528986824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df8d279420 con 0x55df8d392820 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.455+0000 7fa827400700 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d39cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.456+0000 7fa81fe00700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa80c0b2a70 0x7fa80c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.456+0000 7fa826a00700 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392820 0x55df8d39f020 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa8140df7b0 tx=0x7fa8140e8950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8ff03b5d8d4812 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.456+0000 7fa827e00700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa80c0b2a70 0x7fa80c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.456+0000 7fa81fe00700 1 -- 172.21.15.12:0/3419597931 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa8140e2530 con 0x7fa828105bd0 2024-05-27T07:24:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.456+0000 7fa81cc00700 1 -- 172.21.15.12:0/2528986824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140a52c0 con 0x55df8d392820 2024-05-27T07:24:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.456+0000 7fa81cc00700 1 -- 172.21.15.12:0/2528986824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8140a8f10 con 0x55df8d392820 2024-05-27T07:24:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.456+0000 7fa81cc00700 1 -- 172.21.15.12:0/2528986824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140a8b70 con 0x55df8d392820 2024-05-27T07:24:19.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/2528986824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392820 msgr2=0x55df8d39f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa827e00700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa80c0b2a70 0x7fa80c0b4f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55df8d38e690 tx=0x7fa8200061f0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392820 0x55df8d39f020 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa8140df7b0 tx=0x7fa8140e8950 comp rx=0 tx=0).stop 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/2528986824 shutdown_connections 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392820 0x55df8d39f020 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d391ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/2528986824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d39cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/2528986824 >> 172.21.15.12:0/2528986824 conn(0x55df8d273fc0 msgr2=0x55df8d38dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/2528986824 shutdown_connections 2024-05-27T07:24:19.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.458+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/2528986824 wait complete. 2024-05-27T07:24:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.459+0000 7fa83d2f5b80 1 -- start start 2024-05-27T07:24:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.459+0000 7fa83d2f5b80 1 Processor -- start 2024-05-27T07:24:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d39c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392410 0x55df8d39ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55df8d1b6fa0 con 0x55df8d392410 2024-05-27T07:24:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55df8d251760 con 0x55df8d392820 2024-05-27T07:24:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55df8d39f990 con 0x55df8d213e40 2024-05-27T07:24:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa83d2f5b80 10 client.0 _ll_register_callbacks cb 0x55df8d25ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa826a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392410 0x55df8d39ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa826a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392410 0x55df8d39ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35738/0 (socket says 172.21.15.12:35738) 2024-05-27T07:24:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa826a00700 1 -- 172.21.15.12:0/3096417719 learned_addr learned my addr 172.21.15.12:0/3096417719 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:19.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa827400700 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d39c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.460+0000 7fa827e00700 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa826a00700 1 -- 172.21.15.12:0/3096417719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 msgr2=0x55df8d39c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa826a00700 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d39c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa826a00700 1 -- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 msgr2=0x55df8d39cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa826a00700 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa826a00700 1 -- 172.21.15.12:0/3096417719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df8d2008d0 con 0x55df8d392410 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa826a00700 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392410 0x55df8d39ca90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a5480 tx=0x7fa8140e27b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc82bbefe40baa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa811600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa811600700 1 -- 172.21.15.12:0/3096417719 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814020070 con 0x55df8d392410 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa811600700 1 -- 172.21.15.12:0/3096417719 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa8140e2150 con 0x55df8d392410 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.461+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8140a8f10 con 0x55df8d392410 2024-05-27T07:24:19.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa811600700 1 -- 172.21.15.12:0/3096417719 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa8140a8d50 con 0x55df8d392410 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 10 client.15582 fetch_fsmap learned FSMap version 27 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55df8d281060 con 0x55df8d392410 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa811600700 1 -- 172.21.15.12:0/3096417719 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8140a8f10 con 0x55df8d392410 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa811600700 1 -- 172.21.15.12:0/3096417719 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa8140dfaf0 con 0x55df8d392410 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 10 client.15582 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 20 client.15582 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d3aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 4 client.15582 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.462+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df8d2c2d90 con 0x55df8d3ac810 2024-05-27T07:24:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.463+0000 7fa827400700 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d3aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.464+0000 7fa827400700 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d3aec30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c100a3f19e174ec5 server_cookie=6cbebb0d08dbb24 in_seq=0 out_seq=0 2024-05-27T07:24:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.464+0000 7fa811600700 10 client.15582 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.466+0000 7fa811600700 1 -- 172.21.15.12:0/3096417719 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55df8d2c2d90 con 0x55df8d3ac810 2024-05-27T07:24:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.466+0000 7fa811600700 10 client.15582 handle_command_reply: tid=0 2024-05-27T07:24:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.466+0000 7fa83d2f5b80 1 client.15582 shutdown 2024-05-27T07:24:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392410 msgr2=0x55df8d39ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392410 0x55df8d39ca90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a5480 tx=0x7fa8140e27b0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 shutdown_connections 2024-05-27T07:24:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d3aec30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa811600700 0 client.15582 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d213e40 0x55df8d39c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/3096417719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d392410 0x55df8d39ca90 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 >> 172.21.15.12:0/3096417719 conn(0x55df8d273fc0 msgr2=0x55df8d38cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 shutdown_connections 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/3096417719 wait complete. 2024-05-27T07:24:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.467+0000 7fa83d2f5b80 20 client.15582 trim_cache size 0 max 16384 2024-05-27T07:24:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 Processor -- start 2024-05-27T07:24:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 -- start start 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d38de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392410 0x55df8d3a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa8281b4ee0 con 0x55df8d213e40 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa8281b2770 con 0x55df8d392820 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.490+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa8281b05c0 con 0x55df8d392410 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d38de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d38de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35746/0 (socket says 172.21.15.12:35746) 2024-05-27T07:24:19.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 -- 172.21.15.12:0/4088264089 learned_addr learned my addr 172.21.15.12:0/4088264089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa826a00700 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392410 0x55df8d3a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827e00700 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 -- 172.21.15.12:0/4088264089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392410 msgr2=0x55df8d3a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392410 0x55df8d3a8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 -- 172.21.15.12:0/4088264089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 msgr2=0x55df8d39e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827400700 1 -- 172.21.15.12:0/4088264089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df8d39f870 con 0x55df8d213e40 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.491+0000 7fa827e00700 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:19.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.492+0000 7fa827400700 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d38de30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x55df8d393430 tx=0x7fa818005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639112e00499c320 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:19.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.492+0000 7fa813e00700 1 -- 172.21.15.12:0/4088264089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81801f020 con 0x55df8d213e40 2024-05-27T07:24:19.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.492+0000 7fa813e00700 1 -- 172.21.15.12:0/4088264089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa818003cc0 con 0x55df8d213e40 2024-05-27T07:24:19.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.492+0000 7fa813e00700 1 -- 172.21.15.12:0/4088264089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818012650 con 0x55df8d213e40 2024-05-27T07:24:19.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4088264089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 msgr2=0x55df8d38de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d38de30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x55df8d393430 tx=0x7fa818005d00 comp rx=0 tx=0).stop 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4088264089 shutdown_connections 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392820 0x55df8d39e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d38de30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4088264089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392410 0x55df8d3a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4088264089 >> 172.21.15.12:0/4088264089 conn(0x55df8d273fc0 msgr2=0x55df8d39cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4088264089 shutdown_connections 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.493+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4088264089 wait complete. 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.494+0000 7fa83d2f5b80 1 -- start start 2024-05-27T07:24:19.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.494+0000 7fa83d2f5b80 1 Processor -- start 2024-05-27T07:24:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d3b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d3af610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa83d2f5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392820 0x55df8d3afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa8281b0c30 con 0x55df8d213e40 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa8281b20a0 con 0x55df8d392410 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa83d2f5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa8281b1300 con 0x55df8d392820 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa83d2f5b80 10 client.0 _ll_register_callbacks cb 0x55df8d25ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa827400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d3b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa827400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d3b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35760/0 (socket says 172.21.15.12:35760) 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa827400700 1 -- 172.21.15.12:0/4027756891 learned_addr learned my addr 172.21.15.12:0/4027756891 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa827e00700 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392820 0x55df8d3afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.495+0000 7fa826a00700 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d3af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa827400700 1 -- 172.21.15.12:0/4027756891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392820 msgr2=0x55df8d3afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa827400700 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392820 0x55df8d3afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa827400700 1 -- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 msgr2=0x55df8d3af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa827400700 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d3af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa827400700 1 -- 172.21.15.12:0/4027756891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df8d3b01c0 con 0x55df8d213e40 2024-05-27T07:24:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa827400700 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d3b2240 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa818003ef0 tx=0x7fa81800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2087b31057db974a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:19.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa811600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.496+0000 7fa811600700 1 -- 172.21.15.12:0/4027756891 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818032060 con 0x55df8d213e40 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa818003cc0 con 0x55df8d213e40 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa811600700 1 -- 172.21.15.12:0/4027756891 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fa81801b890 con 0x55df8d213e40 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa811600700 1 -- 172.21.15.12:0/4027756891 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa818039050 con 0x55df8d213e40 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa83d2f5b80 10 client.15588 fetch_fsmap learned FSMap version 27 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55df8d416de0 con 0x55df8d213e40 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa811600700 1 -- 172.21.15.12:0/4027756891 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa818003cc0 con 0x55df8d213e40 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa811600700 1 -- 172.21.15.12:0/4027756891 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa81802c030 con 0x55df8d213e40 2024-05-27T07:24:19.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa83d2f5b80 10 client.15588 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:24:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa83d2f5b80 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.497+0000 7fa83d2f5b80 20 client.15588 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.498+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d419450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.498+0000 7fa83d2f5b80 4 client.15588 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:24:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.498+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55df8d2bc570 con 0x55df8d3ac810 2024-05-27T07:24:19.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.498+0000 7fa826a00700 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d419450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.499+0000 7fa826a00700 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d419450 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81525f5088980ef7 server_cookie=e5df667f440a5ba3 in_seq=0 out_seq=0 2024-05-27T07:24:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.499+0000 7fa811600700 10 client.15588 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa811600700 1 -- 172.21.15.12:0/4027756891 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55df8d2bc570 con 0x55df8d3ac810 2024-05-27T07:24:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa811600700 10 client.15588 handle_command_reply: tid=0 2024-05-27T07:24:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 client.15588 shutdown 2024-05-27T07:24:19.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 msgr2=0x55df8d3b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d3b2240 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa818003ef0 tx=0x7fa81800bfd0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 shutdown_connections 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df8d392820 0x55df8d3afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55df8d213e40 0x55df8d3b2240 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55df8d3ac810 0x55df8d419450 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 --2- 172.21.15.12:0/4027756891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df8d392410 0x55df8d3af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 >> 172.21.15.12:0/4027756891 conn(0x55df8d273fc0 msgr2=0x55df8d3a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa811600700 0 client.15588 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 shutdown_connections 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 1 -- 172.21.15.12:0/4027756891 wait complete. 2024-05-27T07:24:19.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.500+0000 7fa83d2f5b80 20 client.15588 trim_cache size 0 max 16384 2024-05-27T07:24:19.493 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:24:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:24:19.493 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:24:19.493 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa80c0b2a70 msgr2=0x7fa80c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa80c0b2a70 0x7fa80c0b4f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55df8d38e690 tx=0x7fa8200061f0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa828105bd0 msgr2=0x7fa82819d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa828105bd0 0x7fa82819d2b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa81400d950 tx=0x7fa81400dd10 comp rx=0 tx=0).stop 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 shutdown_connections 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa80c0b2a70 0x7fa80c0b4f80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa82810e6f0 0x7fa8281a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa828105280 0x7fa82819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 --2- 172.21.15.12:0/3419597931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa828105bd0 0x7fa82819d2b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:19.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 >> 172.21.15.12:0/3419597931 conn(0x7fa828078af0 msgr2=0x7fa8280ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:19.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.501+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 shutdown_connections 2024-05-27T07:24:19.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:19.502+0000 7fa82d200700 1 -- 172.21.15.12:0/3419597931 wait complete. 2024-05-27T07:24:20.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:20 smithi012 ceph-mon[31473]: pgmap v61: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T07:24:20.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:20 smithi162 ceph-mon[35676]: pgmap v61: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T07:24:20.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:20 smithi062 ceph-mon[35304]: pgmap v61: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-27T07:24:20.548 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:24:20.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:24:20.996 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:21.365 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 57% (7099/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 93% (11454/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-27T07:24:22.282 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'. 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2024-05-27T07:24:22.283 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:24:22.284 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2024-05-27T07:24:22.284 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:24:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T07:24:22.284 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:24:22.284 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at 38f9d900c83 Merge PR #55758 into wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:24:22.297 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-27T07:24:22.438 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-27T07:24:22.448 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:24:22.448 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-27T07:24:22.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:22 smithi162 ceph-mon[35676]: pgmap v62: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:22.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:22 smithi062 ceph-mon[35304]: pgmap v62: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:22.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:22 smithi012 ceph-mon[31473]: pgmap v62: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:23.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.656+0000 7fb52f400700 1 -- 172.21.15.12:0/2469153483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528107de0 msgr2=0x7fb5281081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.656+0000 7fb52f400700 1 --2- 172.21.15.12:0/2469153483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528107de0 0x7fb5281081f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb5140099c0 tx=0x7fb514009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.656+0000 7fb52f400700 1 -- 172.21.15.12:0/2469153483 shutdown_connections 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.657+0000 7fb52f400700 1 --2- 172.21.15.12:0/2469153483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb528108730 0x7fb528110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.657+0000 7fb52f400700 1 --2- 172.21.15.12:0/2469153483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5281034b0 0x7fb5281038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.657+0000 7fb52f400700 1 --2- 172.21.15.12:0/2469153483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528107de0 0x7fb5281081f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.657+0000 7fb52f400700 1 -- 172.21.15.12:0/2469153483 >> 172.21.15.12:0/2469153483 conn(0x7fb5280fedb0 msgr2=0x7fb5281011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.657+0000 7fb52f400700 1 -- 172.21.15.12:0/2469153483 shutdown_connections 2024-05-27T07:24:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.657+0000 7fb52f400700 1 -- 172.21.15.12:0/2469153483 wait complete. 2024-05-27T07:24:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.658+0000 7fb52f400700 1 Processor -- start 2024-05-27T07:24:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.659+0000 7fb52f400700 1 -- start start 2024-05-27T07:24:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.659+0000 7fb52f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5281034b0 0x7fb5280ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.659+0000 7fb52f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb528107de0 0x7fb5280ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.659+0000 7fb52f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528108730 0x7fb5280ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.659+0000 7fb52f400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb5281132b0 con 0x7fb528108730 2024-05-27T07:24:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.659+0000 7fb52f400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb528113130 con 0x7fb5281034b0 2024-05-27T07:24:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.659+0000 7fb52f400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb528113430 con 0x7fb528107de0 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528108730 0x7fb5280ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528108730 0x7fb5280ffd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40090/0 (socket says 172.21.15.12:40090) 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 -- 172.21.15.12:0/1679259720 learned_addr learned my addr 172.21.15.12:0/1679259720 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52ca00700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5281034b0 0x7fb5280ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb527e00700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb528107de0 0x7fb5280ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 -- 172.21.15.12:0/1679259720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb528107de0 msgr2=0x7fb5280ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb528107de0 0x7fb5280ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 -- 172.21.15.12:0/1679259720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5281034b0 msgr2=0x7fb5280ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5281034b0 0x7fb5280ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.660+0000 7fb52d400700 1 -- 172.21.15.12:0/1679259720 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5281a9210 con 0x7fb528108730 2024-05-27T07:24:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.661+0000 7fb52d400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528108730 0x7fb5280ffd10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb52000b970 tx=0x7fb52000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.661+0000 7fb525600700 1 -- 172.21.15.12:0/1679259720 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520019070 con 0x7fb528108730 2024-05-27T07:24:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.661+0000 7fb525600700 1 -- 172.21.15.12:0/1679259720 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb52000cdc0 con 0x7fb528108730 2024-05-27T07:24:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.661+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5281a9500 con 0x7fb528108730 2024-05-27T07:24:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.661+0000 7fb525600700 1 -- 172.21.15.12:0/1679259720 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520014e30 con 0x7fb528108730 2024-05-27T07:24:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.661+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5281a99c0 con 0x7fb528108730 2024-05-27T07:24:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.663+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5280512b0 con 0x7fb528108730 2024-05-27T07:24:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.667+0000 7fb525600700 1 -- 172.21.15.12:0/1679259720 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb52001c070 con 0x7fb528108730 2024-05-27T07:24:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.668+0000 7fb525600700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb5100b29f0 0x7fb5100b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.669+0000 7fb525600700 1 -- 172.21.15.12:0/1679259720 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fb5200e2860 con 0x7fb528108730 2024-05-27T07:24:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.669+0000 7fb52ca00700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb5100b29f0 0x7fb5100b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.669+0000 7fb52ca00700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb5100b29f0 0x7fb5100b4f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005950 tx=0x7fb51c00b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:23.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.671+0000 7fb525600700 1 -- 172.21.15.12:0/1679259720 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb520023030 con 0x7fb528108730 2024-05-27T07:24:23.819 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-27T07:24:23.934 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-27T07:24:23.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.962+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5281007c0 con 0x7fb528108730 2024-05-27T07:24:23.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.964+0000 7fb525600700 1 -- 172.21.15.12:0/1679259720 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11867 (secure 0 0 0) 0x7fb5200a9f40 con 0x7fb528108730 2024-05-27T07:24:23.955 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:24:23.955 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:22:21.096387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:24:23.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb5100b29f0 msgr2=0x7fb5100b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb5100b29f0 0x7fb5100b4f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005950 tx=0x7fb51c00b410 comp rx=0 tx=0).stop 2024-05-27T07:24:23.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528108730 msgr2=0x7fb5280ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528108730 0x7fb5280ffd10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb52000b970 tx=0x7fb52000bc80 comp rx=0 tx=0).stop 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 shutdown_connections 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb5100b29f0 0x7fb5100b4f00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5281034b0 0x7fb5280ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb528108730 0x7fb5280ffd10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 --2- 172.21.15.12:0/1679259720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb528107de0 0x7fb5280ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 >> 172.21.15.12:0/1679259720 conn(0x7fb5280fedb0 msgr2=0x7fb528109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 shutdown_connections 2024-05-27T07:24:23.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:23.967+0000 7fb52f400700 1 -- 172.21.15.12:0/1679259720 wait complete. 2024-05-27T07:24:23.959 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 27 2024-05-27T07:24:24.025 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:24:24.030 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:24:24.030 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-27T07:24:24.105 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:24:24.115 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:24:24.115 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-27T07:24:24.157 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-05-27T07:24:24.158 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-05-27T07:24:24.158 DEBUG:teuthology.orchestra.run.smithi012:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-05-27T07:24:24.295 INFO:tasks.workunit.client.0.smithi012.stderr:+ set -e 2024-05-27T07:24:24.296 INFO:tasks.workunit.client.0.smithi012.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-05-27T07:24:24.313 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into 'xfstests-dev'... 2024-05-27T07:24:24.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:24 smithi162 ceph-mon[35676]: pgmap v63: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:24.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:24 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1679259720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:24.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:24 smithi062 ceph-mon[35304]: pgmap v63: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:24.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:24 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1679259720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:24 smithi012 ceph-mon[31473]: pgmap v63: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:24.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:24 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1679259720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:25.162 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:24:25.647 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:26.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:26 smithi162 ceph-mon[35676]: pgmap v64: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:24:26.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:26 smithi062 ceph-mon[35304]: pgmap v64: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:24:26.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:26 smithi012 ceph-mon[31473]: pgmap v64: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-27T07:24:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 -- 172.21.15.12:0/3273429281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388076670 msgr2=0x7f7388110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 --2- 172.21.15.12:0/3273429281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388076670 0x7f7388110460 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7370009990 tx=0x7f7370009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 -- 172.21.15.12:0/3273429281 shutdown_connections 2024-05-27T07:24:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 --2- 172.21.15.12:0/3273429281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7388075d20 0x7f7388076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 --2- 172.21.15.12:0/3273429281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73881109a0 0x7f7388112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 --2- 172.21.15.12:0/3273429281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388076670 0x7f7388110460 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 -- 172.21.15.12:0/3273429281 >> 172.21.15.12:0/3273429281 conn(0x7f7388079d00 msgr2=0x7f738807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 -- 172.21.15.12:0/3273429281 shutdown_connections 2024-05-27T07:24:28.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.155+0000 7f738dc00700 1 -- 172.21.15.12:0/3273429281 wait complete. 2024-05-27T07:24:28.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 Processor -- start 2024-05-27T07:24:28.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 -- start start 2024-05-27T07:24:28.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388075d20 0x7f738819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7388076670 0x7f738819d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73881109a0 0x7f73881a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f73881163a0 con 0x7f7388075d20 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7388116220 con 0x7f7388076670 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.157+0000 7f738dc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7388116520 con 0x7f73881109a0 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388075d20 0x7f738819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388075d20 0x7f738819cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40106/0 (socket says 172.21.15.12:40106) 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 -- 172.21.15.12:0/1090531676 learned_addr learned my addr 172.21.15.12:0/1090531676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f737f400700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7388076670 0x7f738819d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f738d200700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73881109a0 0x7f73881a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 -- 172.21.15.12:0/1090531676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73881109a0 msgr2=0x7f73881a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73881109a0 0x7f73881a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 -- 172.21.15.12:0/1090531676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7388076670 msgr2=0x7f738819d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7388076670 0x7f738819d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.158+0000 7f7387e00700 1 -- 172.21.15.12:0/1090531676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73881a1cc0 con 0x7f7388075d20 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.159+0000 7f7387e00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388075d20 0x7f738819cdf0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f737800d950 tx=0x7f737800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.159+0000 7f7385600700 1 -- 172.21.15.12:0/1090531676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7378016070 con 0x7f7388075d20 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.159+0000 7f7385600700 1 -- 172.21.15.12:0/1090531676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7378010460 con 0x7f7388075d20 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.159+0000 7f7385600700 1 -- 172.21.15.12:0/1090531676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737800f620 con 0x7f7388075d20 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.159+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73881a1fb0 con 0x7f7388075d20 2024-05-27T07:24:28.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.159+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73881a98d0 con 0x7f7388075d20 2024-05-27T07:24:28.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.164+0000 7f7385600700 1 -- 172.21.15.12:0/1090531676 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7378019070 con 0x7f7388075d20 2024-05-27T07:24:28.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.165+0000 7f739de33b80 1 Processor -- start 2024-05-27T07:24:28.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.165+0000 7f739de33b80 1 -- start start 2024-05-27T07:24:28.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f739de33b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351f3c9f0 0x559351fd0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f739de33b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351e50e40 0x559351fd0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f739de33b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351fda2c0 0x559351fdc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f739de33b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559351efde40 con 0x559351f3c9f0 2024-05-27T07:24:28.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f739de33b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559351eed9e0 con 0x559351fda2c0 2024-05-27T07:24:28.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f739de33b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559351ef1a20 con 0x559351e50e40 2024-05-27T07:24:28.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f738d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351f3c9f0 0x559351fd0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f738d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351f3c9f0 0x559351fd0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40122/0 (socket says 172.21.15.12:40122) 2024-05-27T07:24:28.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.166+0000 7f738d200700 1 -- 172.21.15.12:0/2129546210 learned_addr learned my addr 172.21.15.12:0/2129546210 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:28.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f7385600700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f736c0b2a70 0x7f736c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f7387e00700 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351e50e40 0x559351fd0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f737f400700 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351fda2c0 0x559351fdc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f738d200700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f736c0b2a70 0x7f736c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f7385600700 1 -- 172.21.15.12:0/1090531676 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f73780e2560 con 0x7f7388075d20 2024-05-27T07:24:28.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f738d200700 1 -- 172.21.15.12:0/2129546210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351e50e40 msgr2=0x559351fd0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f738d200700 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351e50e40 0x559351fd0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f738d200700 1 -- 172.21.15.12:0/2129546210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351fda2c0 msgr2=0x559351fdc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f738d200700 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351fda2c0 0x559351fdc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.167+0000 7f738d200700 1 -- 172.21.15.12:0/2129546210 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559351eb6420 con 0x559351f3c9f0 2024-05-27T07:24:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.168+0000 7f738d200700 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351f3c9f0 0x559351fd0750 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7380007950 tx=0x7f7380007c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab4b931b3027b80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.169+0000 7f738d200700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f736c0b2a70 0x7f736c0b4f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7380007fb0 tx=0x7f738001c000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.169+0000 7f737d600700 1 -- 172.21.15.12:0/2129546210 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738001d040 con 0x559351f3c9f0 2024-05-27T07:24:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.169+0000 7f737d600700 1 -- 172.21.15.12:0/2129546210 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7380009d40 con 0x559351f3c9f0 2024-05-27T07:24:28.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.169+0000 7f737d600700 1 -- 172.21.15.12:0/2129546210 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380012040 con 0x559351f3c9f0 2024-05-27T07:24:28.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 -- 172.21.15.12:0/2129546210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351f3c9f0 msgr2=0x559351fd0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351f3c9f0 0x559351fd0750 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7380007950 tx=0x7f7380007c60 comp rx=0 tx=0).stop 2024-05-27T07:24:28.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 -- 172.21.15.12:0/2129546210 shutdown_connections 2024-05-27T07:24:28.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351fda2c0 0x559351fdc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351f3c9f0 0x559351fd0750 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 --2- 172.21.15.12:0/2129546210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351e50e40 0x559351fd0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 -- 172.21.15.12:0/2129546210 >> 172.21.15.12:0/2129546210 conn(0x559351eea390 msgr2=0x559351fcb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 -- 172.21.15.12:0/2129546210 shutdown_connections 2024-05-27T07:24:28.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.170+0000 7f739de33b80 1 -- 172.21.15.12:0/2129546210 wait complete. 2024-05-27T07:24:28.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.171+0000 7f739de33b80 1 -- start start 2024-05-27T07:24:28.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.171+0000 7f739de33b80 1 Processor -- start 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f739de33b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fd7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f739de33b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 0x559351fd77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f739de33b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fd7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f739de33b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559351df3fa0 con 0x559351fda2c0 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f739de33b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559351e8e760 con 0x559351e50e40 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f739de33b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559351fdcda0 con 0x559351f3c9f0 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f739de33b80 10 client.0 _ll_register_callbacks cb 0x559351e97c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f738d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fd7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f738d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fd7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40124/0 (socket says 172.21.15.12:40124) 2024-05-27T07:24:28.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.172+0000 7f738d200700 1 -- 172.21.15.12:0/1342287337 learned_addr learned my addr 172.21.15.12:0/1342287337 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.173+0000 7f7387e00700 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fd7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.173+0000 7f738d200700 1 -- 172.21.15.12:0/1342287337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 msgr2=0x559351fd77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.173+0000 7f738d200700 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 0x559351fd77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.173+0000 7f738d200700 1 -- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 msgr2=0x559351fd7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.173+0000 7f738d200700 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fd7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.173+0000 7f738d200700 1 -- 172.21.15.12:0/1342287337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559351e3d8d0 con 0x559351fda2c0 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f738d200700 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fd7ce0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7380009600 tx=0x7f7380006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5eda70e13323208 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f7376000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:28.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f7376000700 1 -- 172.21.15.12:0/1342287337 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738002f030 con 0x559351fda2c0 2024-05-27T07:24:28.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7380009d40 con 0x559351fda2c0 2024-05-27T07:24:28.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f7376000700 1 -- 172.21.15.12:0/1342287337 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f7380022470 con 0x559351fda2c0 2024-05-27T07:24:28.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f7376000700 1 -- 172.21.15.12:0/1342287337 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7380039050 con 0x559351fda2c0 2024-05-27T07:24:28.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f739de33b80 10 client.15606 fetch_fsmap learned FSMap version 27 2024-05-27T07:24:28.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.174+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559351ebe060 con 0x559351fda2c0 2024-05-27T07:24:28.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f7376000700 1 -- 172.21.15.12:0/1342287337 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7380009d40 con 0x559351fda2c0 2024-05-27T07:24:28.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f7376000700 1 -- 172.21.15.12:0/1342287337 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7380018040 con 0x559351fda2c0 2024-05-27T07:24:28.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f739de33b80 10 client.15606 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:24:28.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f739de33b80 10 client.15606 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:24:28.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f739de33b80 20 client.15606 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:28.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f739de33b80 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x559351fe9c00 0x559351fec020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f739de33b80 4 client.15606 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:28.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.175+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559351effd90 con 0x559351fe9c00 2024-05-27T07:24:28.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.176+0000 7f7387e00700 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x559351fe9c00 0x559351fec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.177+0000 7f7376000700 10 client.15606 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:24:28.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.177+0000 7f7387e00700 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x559351fe9c00 0x559351fec020 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1b9b2e3663ae3c1 server_cookie=29e80642d8468501 in_seq=0 out_seq=0 2024-05-27T07:24:28.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.179+0000 7f7376000700 1 -- 172.21.15.12:0/1342287337 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559351effd90 con 0x559351fe9c00 2024-05-27T07:24:28.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.179+0000 7f7376000700 10 client.15606 handle_command_reply: tid=0 2024-05-27T07:24:28.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.179+0000 7f739de33b80 1 client.15606 shutdown 2024-05-27T07:24:28.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 msgr2=0x559351fd7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fd7ce0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7380009600 tx=0x7f7380006000 comp rx=0 tx=0).stop 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 shutdown_connections 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x559351fe9c00 0x559351fec020 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fd7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fd7ce0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 --2- 172.21.15.12:0/1342287337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 0x559351fd77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f7376000700 0 client.15606 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 >> 172.21.15.12:0/1342287337 conn(0x559351eea390 msgr2=0x559351fcb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 shutdown_connections 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 1 -- 172.21.15.12:0/1342287337 wait complete. 2024-05-27T07:24:28.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.180+0000 7f739de33b80 20 client.15606 trim_cache size 0 max 16384 2024-05-27T07:24:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 Processor -- start 2024-05-27T07:24:28.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 -- start start 2024-05-27T07:24:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fcb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 0x559351fd6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fe6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f73881b4f60 con 0x559351fda2c0 2024-05-27T07:24:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f73881b27f0 con 0x559351e50e40 2024-05-27T07:24:28.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.201+0000 7f739de33b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73881b0640 con 0x559351f3c9f0 2024-05-27T07:24:28.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.202+0000 7f738d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fe6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.202+0000 7f738d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fe6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40128/0 (socket says 172.21.15.12:40128) 2024-05-27T07:24:28.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.202+0000 7f738d200700 1 -- 172.21.15.12:0/2043868103 learned_addr learned my addr 172.21.15.12:0/2043868103 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:28.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.202+0000 7f737f400700 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 0x559351fd6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.202+0000 7f7387e00700 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fcb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f738d200700 1 -- 172.21.15.12:0/2043868103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 msgr2=0x559351fd6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f738d200700 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 0x559351fd6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f738d200700 1 -- 172.21.15.12:0/2043868103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 msgr2=0x559351fcb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f738d200700 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fcb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f738d200700 1 -- 172.21.15.12:0/2043868103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559351fd1390 con 0x559351fda2c0 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f7387e00700 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fcb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f738d200700 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fe6070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7380019f30 tx=0x7f73800061a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6eec29cf16a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f737cc00700 1 -- 172.21.15.12:0/2043868103 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738002f070 con 0x559351fda2c0 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f737cc00700 1 -- 172.21.15.12:0/2043868103 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7380022470 con 0x559351fda2c0 2024-05-27T07:24:28.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.203+0000 7f737cc00700 1 -- 172.21.15.12:0/2043868103 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738001d070 con 0x559351fda2c0 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 -- 172.21.15.12:0/2043868103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 msgr2=0x559351fe6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fe6070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7380019f30 tx=0x7f73800061a0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 -- 172.21.15.12:0/2043868103 shutdown_connections 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351e50e40 0x559351fcb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351fda2c0 0x559351fe6070 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 --2- 172.21.15.12:0/2043868103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351f3c9f0 0x559351fd6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 -- 172.21.15.12:0/2043868103 >> 172.21.15.12:0/2043868103 conn(0x559351eea390 msgr2=0x559351fd28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 -- 172.21.15.12:0/2043868103 shutdown_connections 2024-05-27T07:24:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.205+0000 7f739de33b80 1 -- 172.21.15.12:0/2043868103 wait complete. 2024-05-27T07:24:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.206+0000 7f739de33b80 1 -- start start 2024-05-27T07:24:28.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.206+0000 7f739de33b80 1 Processor -- start 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.206+0000 7f739de33b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351e50e40 0x559351fec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.206+0000 7f739de33b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351f3c9f0 0x559351fecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.206+0000 7f739de33b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351fda2c0 0x559352052460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f739de33b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f73881b0cb0 con 0x559351e50e40 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f739de33b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f73881b2120 con 0x559351f3c9f0 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f739de33b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73881b1380 con 0x559351fda2c0 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f739de33b80 10 client.0 _ll_register_callbacks cb 0x559351e97c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:28.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351e50e40 0x559351fec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351e50e40 0x559351fec570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40140/0 (socket says 172.21.15.12:40140) 2024-05-27T07:24:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 -- 172.21.15.12:0/2485468652 learned_addr learned my addr 172.21.15.12:0/2485468652 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f737f400700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351f3c9f0 0x559351fecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f738d200700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351fda2c0 0x559352052460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 -- 172.21.15.12:0/2485468652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351fda2c0 msgr2=0x559352052460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351fda2c0 0x559352052460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 -- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351f3c9f0 msgr2=0x559351fecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351f3c9f0 0x559351fecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.207+0000 7f7387e00700 1 -- 172.21.15.12:0/2485468652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559351fd8050 con 0x559351e50e40 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f738d200700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351fda2c0 0x559352052460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:28.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f7387e00700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351e50e40 0x559351fec570 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f737800e010 tx=0x7f73780a5450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e445ed7844bec8f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f7376000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f7376000700 1 -- 172.21.15.12:0/2485468652 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7378020040 con 0x559351e50e40 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f7376000700 1 -- 172.21.15.12:0/2485468652 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f73780a98f0 con 0x559351e50e40 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7380022470 con 0x559351e50e40 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f7376000700 1 -- 172.21.15.12:0/2485468652 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f73780a9b90 con 0x559351e50e40 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f739de33b80 10 client.15612 fetch_fsmap learned FSMap version 27 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.208+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559352053a60 con 0x559351e50e40 2024-05-27T07:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f7376000700 1 -- 172.21.15.12:0/2485468652 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7380022470 con 0x559351e50e40 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f7376000700 1 -- 172.21.15.12:0/2485468652 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7380022cb0 con 0x559351e50e40 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f739de33b80 10 client.15612 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f739de33b80 10 client.15612 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f739de33b80 20 client.15612 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f739de33b80 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5593520540c0 0x5593520564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f739de33b80 4 client.15612 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.209+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559351ef9570 con 0x5593520540c0 2024-05-27T07:24:28.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.210+0000 7f737f400700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5593520540c0 0x5593520564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.211+0000 7f737f400700 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5593520540c0 0x5593520564e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af0a6d6a6f03e106 server_cookie=1b8c7900d886db33 in_seq=0 out_seq=0 2024-05-27T07:24:28.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.211+0000 7f7376000700 10 client.15612 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:24:28.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f7376000700 1 -- 172.21.15.12:0/2485468652 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559351ef9570 con 0x5593520540c0 2024-05-27T07:24:28.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f7376000700 10 client.15612 handle_command_reply: tid=0 2024-05-27T07:24:28.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 client.15612 shutdown 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351e50e40 msgr2=0x559351fec570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351e50e40 0x559351fec570 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f737800e010 tx=0x7f73780a5450 comp rx=0 tx=0).stop 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 shutdown_connections 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559351fda2c0 0x559352052460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559351e50e40 0x559351fec570 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5593520540c0 0x5593520564e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 --2- 172.21.15.12:0/2485468652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559351f3c9f0 0x559351fecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 >> 172.21.15.12:0/2485468652 conn(0x559351eea390 msgr2=0x559351fd28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f7376000700 0 client.15612 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:24:28.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 shutdown_connections 2024-05-27T07:24:28.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.212+0000 7f739de33b80 1 -- 172.21.15.12:0/2485468652 wait complete. 2024-05-27T07:24:28.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.213+0000 7f739de33b80 20 client.15612 trim_cache size 0 max 16384 2024-05-27T07:24:28.203 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.213+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f736c0b2a70 msgr2=0x7f736c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.213+0000 7f738dc00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f736c0b2a70 0x7f736c0b4f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7380007fb0 tx=0x7f738001c000 comp rx=0 tx=0).stop 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.213+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388075d20 msgr2=0x7f738819cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388075d20 0x7f738819cdf0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f737800d950 tx=0x7f737800dd10 comp rx=0 tx=0).stop 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 shutdown_connections 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f736c0b2a70 0x7f736c0b4f80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73881109a0 0x7f73881a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7388075d20 0x7f738819cdf0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 --2- 172.21.15.12:0/1090531676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7388076670 0x7f738819d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 >> 172.21.15.12:0/1090531676 conn(0x7f7388079d00 msgr2=0x7f7388100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 shutdown_connections 2024-05-27T07:24:28.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:28.214+0000 7f738dc00700 1 -- 172.21.15.12:0/1090531676 wait complete. 2024-05-27T07:24:28.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:28 smithi162 ceph-mon[35676]: pgmap v65: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:28.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:28 smithi062 ceph-mon[35304]: pgmap v65: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:28.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:28 smithi012 ceph-mon[31473]: pgmap v65: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-27T07:24:29.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:24:29.675 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:30.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:30 smithi062 ceph-mon[35304]: pgmap v66: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T07:24:30.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:30 smithi012 ceph-mon[31473]: pgmap v66: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T07:24:30.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:30 smithi162 ceph-mon[35676]: pgmap v66: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-05-27T07:24:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.168+0000 7f2908200700 1 -- 172.21.15.12:0/3702554734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 msgr2=0x7f2900110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.168+0000 7f2908200700 1 --2- 172.21.15.12:0/3702554734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f2900110760 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009990 tx=0x7f28ec009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.168+0000 7f2908200700 1 -- 172.21.15.12:0/3702554734 shutdown_connections 2024-05-27T07:24:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.168+0000 7f2908200700 1 --2- 172.21.15.12:0/3702554734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29001030f0 0x7f2900103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.168+0000 7f2908200700 1 --2- 172.21.15.12:0/3702554734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2900110ca0 0x7f29001130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.168+0000 7f2908200700 1 --2- 172.21.15.12:0/3702554734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f2900110760 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.168+0000 7f2908200700 1 -- 172.21.15.12:0/3702554734 >> 172.21.15.12:0/3702554734 conn(0x7f29000fedb0 msgr2=0x7f29001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.169+0000 7f2908200700 1 -- 172.21.15.12:0/3702554734 shutdown_connections 2024-05-27T07:24:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.169+0000 7f2908200700 1 -- 172.21.15.12:0/3702554734 wait complete. 2024-05-27T07:24:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.170+0000 7f2908200700 1 Processor -- start 2024-05-27T07:24:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.171+0000 7f2908200700 1 -- start start 2024-05-27T07:24:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.171+0000 7f2908200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29001030f0 0x7f290019d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.171+0000 7f2908200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f290019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.171+0000 7f2908200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2900110ca0 0x7f29001a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.171+0000 7f2908200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2900116550 con 0x7f2900103a40 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.171+0000 7f2908200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f29001163d0 con 0x7f29001030f0 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.171+0000 7f2908200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29001166d0 con 0x7f2900110ca0 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f290019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f290019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42562/0 (socket says 172.21.15.12:42562) 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 -- 172.21.15.12:0/1420671430 learned_addr learned my addr 172.21.15.12:0/1420671430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2905800700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29001030f0 0x7f290019d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 -- 172.21.15.12:0/1420671430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2900110ca0 msgr2=0x7f29001a1870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:24:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2900110ca0 0x7f29001a1870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 -- 172.21.15.12:0/1420671430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29001030f0 msgr2=0x7f290019d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29001030f0 0x7f290019d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.172+0000 7f2904e00700 1 -- 172.21.15.12:0/1420671430 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29001a1f10 con 0x7f2900103a40 2024-05-27T07:24:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.173+0000 7f2904e00700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f290019d5e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f28ec000c00 tx=0x7f28ec005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.173+0000 7f28fe000700 1 -- 172.21.15.12:0/1420671430 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec027070 con 0x7f2900103a40 2024-05-27T07:24:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.173+0000 7f28fe000700 1 -- 172.21.15.12:0/1420671430 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28ec00fc40 con 0x7f2900103a40 2024-05-27T07:24:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.173+0000 7f28fe000700 1 -- 172.21.15.12:0/1420671430 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec022d00 con 0x7f2900103a40 2024-05-27T07:24:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.173+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29001a21a0 con 0x7f2900103a40 2024-05-27T07:24:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.174+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29001a2630 con 0x7f2900103a40 2024-05-27T07:24:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.180+0000 7f28fe000700 1 -- 172.21.15.12:0/1420671430 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f28ec02a070 con 0x7f2900103a40 2024-05-27T07:24:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.181+0000 7f2916ce3b80 1 Processor -- start 2024-05-27T07:24:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.181+0000 7f2916ce3b80 1 -- start start 2024-05-27T07:24:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.182+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61042330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.182+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61042870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.182+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d61057920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.182+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560d60f70e40 con 0x560d60ec3e40 2024-05-27T07:24:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.182+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560d60f609e0 con 0x560d61042e70 2024-05-27T07:24:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.182+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560d60f64a20 con 0x560d60faf9f0 2024-05-27T07:24:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.182+0000 7f28fe000700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28e80c3a00 0x7f28e80c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.183+0000 7f2906200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28e80c3a00 0x7f28e80c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.183+0000 7f2905800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61042870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.183+0000 7f2905800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61042870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42566/0 (socket says 172.21.15.12:42566) 2024-05-27T07:24:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.183+0000 7f2905800700 1 -- 172.21.15.12:0/2751500932 learned_addr learned my addr 172.21.15.12:0/2751500932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.183+0000 7f2906200700 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61042330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.183+0000 7f2904e00700 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d61057920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.183+0000 7f28fe000700 1 -- 172.21.15.12:0/1420671430 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f28ec0e39b0 con 0x7f2900103a40 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.184+0000 7f2904e00700 1 -- 172.21.15.12:0/2751500932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 msgr2=0x560d61042330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.184+0000 7f2904e00700 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61042330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.184+0000 7f2904e00700 1 -- 172.21.15.12:0/2751500932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 msgr2=0x560d61042870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.184+0000 7f2904e00700 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61042870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.184+0000 7f2904e00700 1 -- 172.21.15.12:0/2751500932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d60f29420 con 0x560d61042e70 2024-05-27T07:24:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.184+0000 7f2906200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28e80c3a00 0x7f28e80c5f10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009510 tx=0x7f28f80093c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.185+0000 7f2904e00700 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d61057920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0e0cf0 tx=0x7f28ec09f190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=441a32d8179886e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.185+0000 7f28f2a00700 1 -- 172.21.15.12:0/2751500932 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec0e1180 con 0x560d61042e70 2024-05-27T07:24:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.185+0000 7f28f2a00700 1 -- 172.21.15.12:0/2751500932 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28ec0afe50 con 0x560d61042e70 2024-05-27T07:24:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.185+0000 7f28f2a00700 1 -- 172.21.15.12:0/2751500932 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec0aecc0 con 0x560d61042e70 2024-05-27T07:24:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.186+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2751500932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 msgr2=0x560d61057920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.186+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d61057920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0e0cf0 tx=0x7f28ec09f190 comp rx=0 tx=0).stop 2024-05-27T07:24:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.187+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2751500932 shutdown_connections 2024-05-27T07:24:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.187+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d61057920 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.187+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61042330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.187+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2751500932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61042870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.187+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2751500932 >> 172.21.15.12:0/2751500932 conn(0x560d60f5d390 msgr2=0x560d6103e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.187+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2751500932 shutdown_connections 2024-05-27T07:24:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.187+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2751500932 wait complete. 2024-05-27T07:24:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.188+0000 7f2916ce3b80 1 -- start start 2024-05-27T07:24:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.188+0000 7f2916ce3b80 1 Processor -- start 2024-05-27T07:24:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60ec3e40 0x560d61051440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60faf9f0 0x560d610519e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d61057170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560d60e66fa0 con 0x560d60faf9f0 2024-05-27T07:24:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560d60f01760 con 0x560d60ec3e40 2024-05-27T07:24:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560d60f609e0 con 0x560d61042e70 2024-05-27T07:24:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2916ce3b80 10 client.0 _ll_register_callbacks cb 0x560d60f0ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2904e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60faf9f0 0x560d610519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2904e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60faf9f0 0x560d610519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42580/0 (socket says 172.21.15.12:42580) 2024-05-27T07:24:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.189+0000 7f2904e00700 1 -- 172.21.15.12:0/3672545536 learned_addr learned my addr 172.21.15.12:0/3672545536 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.190+0000 7f2905800700 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60ec3e40 0x560d61051440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.190+0000 7f2906200700 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d61057170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.190+0000 7f2904e00700 1 -- 172.21.15.12:0/3672545536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 msgr2=0x560d61057170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.190+0000 7f2904e00700 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d61057170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.190+0000 7f2904e00700 1 -- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60ec3e40 msgr2=0x560d61051440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.190+0000 7f2904e00700 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60ec3e40 0x560d61051440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.190+0000 7f2904e00700 1 -- 172.21.15.12:0/3672545536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d60eb08d0 con 0x560d60faf9f0 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f2904e00700 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60faf9f0 0x560d610519e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0e0cc0 tx=0x7f28ec022470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50594a18eae51a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f28e7400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f28e7400700 1 -- 172.21.15.12:0/3672545536 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec0e1180 con 0x560d60faf9f0 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f28e7400700 1 -- 172.21.15.12:0/3672545536 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f28ec0b3250 con 0x560d60faf9f0 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28ec0afe50 con 0x560d60faf9f0 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f28e7400700 1 -- 172.21.15.12:0/3672545536 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f28ec0a6090 con 0x560d60faf9f0 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f2916ce3b80 10 client.15624 fetch_fsmap learned FSMap version 27 2024-05-27T07:24:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.191+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d60f31060 con 0x560d60faf9f0 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f28e7400700 1 -- 172.21.15.12:0/3672545536 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28ec0afe50 con 0x560d60faf9f0 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f28e7400700 1 -- 172.21.15.12:0/3672545536 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f28ec0050c0 con 0x560d60faf9f0 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f2916ce3b80 10 client.15624 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f2916ce3b80 10 client.15624 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f2916ce3b80 20 client.15624 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d6105c980 0x560d6105eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f2916ce3b80 4 client.15624 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.192+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d60f72d90 con 0x560d6105c980 2024-05-27T07:24:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.193+0000 7f2905800700 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d6105c980 0x560d6105eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.194+0000 7f2905800700 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d6105c980 0x560d6105eda0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2bd79259fc5f9df server_cookie=98441d16dec9a5 in_seq=0 out_seq=0 2024-05-27T07:24:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.194+0000 7f28e7400700 10 client.15624 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:24:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.196+0000 7f28e7400700 1 -- 172.21.15.12:0/3672545536 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560d60f72d90 con 0x560d6105c980 2024-05-27T07:24:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.196+0000 7f28e7400700 10 client.15624 handle_command_reply: tid=0 2024-05-27T07:24:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.196+0000 7f2916ce3b80 1 client.15624 shutdown 2024-05-27T07:24:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60faf9f0 msgr2=0x560d610519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60faf9f0 0x560d610519e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0e0cc0 tx=0x7f28ec022470 comp rx=0 tx=0).stop 2024-05-27T07:24:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 shutdown_connections 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d6105c980 0x560d6105eda0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60ec3e40 0x560d61051440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d61057170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3672545536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60faf9f0 0x560d610519e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 >> 172.21.15.12:0/3672545536 conn(0x560d60f5d390 msgr2=0x560d61042310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.197+0000 7f28e7400700 0 client.15624 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.198+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 shutdown_connections 2024-05-27T07:24:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.198+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3672545536 wait complete. 2024-05-27T07:24:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.198+0000 7f2916ce3b80 20 client.15624 trim_cache size 0 max 16384 2024-05-27T07:24:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 Processor -- start 2024-05-27T07:24:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 -- start start 2024-05-27T07:24:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61057130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61047e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d6104a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f29001b0ba0 con 0x560d60ec3e40 2024-05-27T07:24:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f29001ae430 con 0x560d61042e70 2024-05-27T07:24:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.220+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29001ac280 con 0x560d60faf9f0 2024-05-27T07:24:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.221+0000 7f2906200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d6104a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.221+0000 7f2905800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61057130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.221+0000 7f2906200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d6104a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:56968/0 (socket says 172.21.15.12:56968) 2024-05-27T07:24:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.221+0000 7f2906200700 1 -- 172.21.15.12:0/3665344101 learned_addr learned my addr 172.21.15.12:0/3665344101 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.221+0000 7f2904e00700 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61047e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.221+0000 7f2905800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61057130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42584/0 (socket says 172.21.15.12:42584) 2024-05-27T07:24:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.222+0000 7f2906200700 1 -- 172.21.15.12:0/3665344101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 msgr2=0x560d61047e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.222+0000 7f2906200700 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61047e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.222+0000 7f2906200700 1 -- 172.21.15.12:0/3665344101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 msgr2=0x560d61057130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.222+0000 7f2906200700 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61057130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.222+0000 7f2906200700 1 -- 172.21.15.12:0/3665344101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d6103a240 con 0x560d61042e70 2024-05-27T07:24:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.223+0000 7f2906200700 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d6104a1c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28f80091c0 tx=0x7f28f8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63626113121e76ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.223+0000 7f28f2000700 1 -- 172.21.15.12:0/3665344101 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8012040 con 0x560d61042e70 2024-05-27T07:24:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.223+0000 7f28f2000700 1 -- 172.21.15.12:0/3665344101 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28f801e040 con 0x560d61042e70 2024-05-27T07:24:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.223+0000 7f28f2000700 1 -- 172.21.15.12:0/3665344101 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8023ba0 con 0x560d61042e70 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3665344101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 msgr2=0x560d6104a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d6104a1c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28f80091c0 tx=0x7f28f8017000 comp rx=0 tx=0).stop 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3665344101 shutdown_connections 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d61042e70 0x560d6104a1c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d61057130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/3665344101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d60faf9f0 0x560d61047e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3665344101 >> 172.21.15.12:0/3665344101 conn(0x560d60f5d390 msgr2=0x560d6104b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3665344101 shutdown_connections 2024-05-27T07:24:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.225+0000 7f2916ce3b80 1 -- 172.21.15.12:0/3665344101 wait complete. 2024-05-27T07:24:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.226+0000 7f2916ce3b80 1 -- start start 2024-05-27T07:24:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.226+0000 7f2916ce3b80 1 Processor -- start 2024-05-27T07:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.227+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d6105c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.227+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60faf9f0 0x560d6105cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.227+0000 7f2916ce3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d6105d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.227+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f29001ac8f0 con 0x560d60ec3e40 2024-05-27T07:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.227+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f29001add60 con 0x560d60faf9f0 2024-05-27T07:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.227+0000 7f2916ce3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29001acfc0 con 0x560d61042e70 2024-05-27T07:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.227+0000 7f2916ce3b80 10 client.0 _ll_register_callbacks cb 0x560d60f0ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.228+0000 7f2904e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60faf9f0 0x560d6105cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.228+0000 7f2904e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60faf9f0 0x560d6105cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:56978/0 (socket says 172.21.15.12:56978) 2024-05-27T07:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.228+0000 7f2904e00700 1 -- 172.21.15.12:0/2796204657 learned_addr learned my addr 172.21.15.12:0/2796204657 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.228+0000 7f2905800700 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d6105c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.228+0000 7f2906200700 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d6105d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.229+0000 7f2905800700 1 -- 172.21.15.12:0/2796204657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 msgr2=0x560d6105d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.229+0000 7f2905800700 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d6105d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.229+0000 7f2905800700 1 -- 172.21.15.12:0/2796204657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60faf9f0 msgr2=0x560d6105cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.229+0000 7f2905800700 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60faf9f0 0x560d6105cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.229+0000 7f2905800700 1 -- 172.21.15.12:0/2796204657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d610c51d0 con 0x560d60ec3e40 2024-05-27T07:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.229+0000 7f2905800700 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d6105c980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x560d60f31140 tx=0x7f28f4011880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da4b7749b7faff2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.229+0000 7f28e7400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.230+0000 7f28e7400700 1 -- 172.21.15.12:0/2796204657 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f401c720 con 0x560d60ec3e40 2024-05-27T07:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.230+0000 7f28e7400700 1 -- 172.21.15.12:0/2796204657 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f28f401c8c0 con 0x560d60ec3e40 2024-05-27T07:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.230+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28f801e040 con 0x560d60ec3e40 2024-05-27T07:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.230+0000 7f28e7400700 1 -- 172.21.15.12:0/2796204657 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f28f402d050 con 0x560d60ec3e40 2024-05-27T07:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.230+0000 7f2916ce3b80 10 client.15630 fetch_fsmap learned FSMap version 27 2024-05-27T07:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.230+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d610c6470 con 0x560d60ec3e40 2024-05-27T07:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f28e7400700 1 -- 172.21.15.12:0/2796204657 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28f801e040 con 0x560d60ec3e40 2024-05-27T07:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f28e7400700 1 -- 172.21.15.12:0/2796204657 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f28f4024c70 con 0x560d60ec3e40 2024-05-27T07:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f2916ce3b80 10 client.15630 fetch_fsmap finished waiting for FSMap version 27 2024-05-27T07:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f2916ce3b80 10 client.15630 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f2916ce3b80 20 client.15630 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d610c6b00 0x560d610c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f2916ce3b80 4 client.15630 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.231+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d60f6c570 con 0x560d610c6b00 2024-05-27T07:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.232+0000 7f2904e00700 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d610c6b00 0x560d610c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.233+0000 7f2904e00700 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d610c6b00 0x560d610c8f20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d8f20f3093c1a48a server_cookie=505f1c93e71f57d2 in_seq=0 out_seq=0 2024-05-27T07:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.233+0000 7f28e7400700 10 client.15630 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f28e7400700 1 -- 172.21.15.12:0/2796204657 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d60f6c570 con 0x560d610c6b00 2024-05-27T07:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f28e7400700 10 client.15630 handle_command_reply: tid=0 2024-05-27T07:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 client.15630 shutdown 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 msgr2=0x560d6105c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d6105c980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x560d60f31140 tx=0x7f28f4011880 comp rx=0 tx=0).stop 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 shutdown_connections 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560d610c6b00 0x560d610c8f20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d61042e70 0x560d6105d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560d60ec3e40 0x560d6105c980 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 --2- 172.21.15.12:0/2796204657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d60faf9f0 0x560d6105cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f28e7400700 0 client.15630 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:24:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.234+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 >> 172.21.15.12:0/2796204657 conn(0x560d60f5d390 msgr2=0x560d6103de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.235+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 shutdown_connections 2024-05-27T07:24:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.235+0000 7f2916ce3b80 1 -- 172.21.15.12:0/2796204657 wait complete. 2024-05-27T07:24:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.235+0000 7f2916ce3b80 20 client.15630 trim_cache size 0 max 16384 2024-05-27T07:24:32.225 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28e80c3a00 msgr2=0x7f28e80c5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28e80c3a00 0x7f28e80c5f10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009510 tx=0x7f28f80093c0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 msgr2=0x7f290019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f290019d5e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f28ec000c00 tx=0x7f28ec005ef0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 shutdown_connections 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f28e80c3a00 0x7f28e80c5f10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29001030f0 0x7f290019d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2900110ca0 0x7f29001a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 --2- 172.21.15.12:0/1420671430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2900103a40 0x7f290019d5e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 >> 172.21.15.12:0/1420671430 conn(0x7f29000fedb0 msgr2=0x7f2900101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 shutdown_connections 2024-05-27T07:24:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:32.236+0000 7f2908200700 1 -- 172.21.15.12:0/1420671430 wait complete. 2024-05-27T07:24:32.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:32 smithi062 ceph-mon[35304]: pgmap v67: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-05-27T07:24:32.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:32 smithi012 ceph-mon[31473]: pgmap v67: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-05-27T07:24:32.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:32 smithi162 ceph-mon[35676]: pgmap v67: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-05-27T07:24:33.258 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:24:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:34.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:34 smithi012 ceph-mon[31473]: pgmap v68: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-05-27T07:24:34.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:34 smithi162 ceph-mon[35676]: pgmap v68: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-05-27T07:24:34.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:34 smithi062 ceph-mon[35304]: pgmap v68: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-05-27T07:24:36.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.345+0000 7f0e39e00700 1 -- 172.21.15.12:0/731658318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e34106590 msgr2=0x7f0e341069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.345+0000 7f0e39e00700 1 --2- 172.21.15.12:0/731658318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e34106590 0x7f0e341069a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009990 tx=0x7f0e1c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.335 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:36 smithi012 ceph-mon[31473]: pgmap v69: 129 pgs: 129 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2024-05-27T07:24:36.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.345+0000 7f0e39e00700 1 -- 172.21.15.12:0/731658318 shutdown_connections 2024-05-27T07:24:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.345+0000 7f0e39e00700 1 --2- 172.21.15.12:0/731658318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e3410c5d0 0x7f0e3410c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.345+0000 7f0e39e00700 1 --2- 172.21.15.12:0/731658318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e34106ee0 0x7f0e34113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.345+0000 7f0e39e00700 1 --2- 172.21.15.12:0/731658318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e34106590 0x7f0e341069a0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.345+0000 7f0e39e00700 1 -- 172.21.15.12:0/731658318 >> 172.21.15.12:0/731658318 conn(0x7f0e34102070 msgr2=0x7f0e341044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.346+0000 7f0e39e00700 1 -- 172.21.15.12:0/731658318 shutdown_connections 2024-05-27T07:24:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.346+0000 7f0e39e00700 1 -- 172.21.15.12:0/731658318 wait complete. 2024-05-27T07:24:36.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.347+0000 7f0e39e00700 1 Processor -- start 2024-05-27T07:24:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.347+0000 7f0e39e00700 1 -- start start 2024-05-27T07:24:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e39e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e34106590 0x7f0e3419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e39e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e34106ee0 0x7f0e3419d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e39e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3410c5d0 0x7f0e341a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e39e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0e34116560 con 0x7f0e3410c5d0 2024-05-27T07:24:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e39e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341163e0 con 0x7f0e34106ee0 2024-05-27T07:24:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e39e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341166e0 con 0x7f0e34106590 2024-05-27T07:24:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3410c5d0 0x7f0e341a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3410c5d0 0x7f0e341a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42606/0 (socket says 172.21.15.12:42606) 2024-05-27T07:24:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e33400700 1 -- 172.21.15.12:0/1760569003 learned_addr learned my addr 172.21.15.12:0/1760569003 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:36.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.348+0000 7f0e32a00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e34106590 0x7f0e3419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e33400700 1 -- 172.21.15.12:0/1760569003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e34106590 msgr2=0x7f0e3419cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e33400700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e34106590 0x7f0e3419cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e33400700 1 -- 172.21.15.12:0/1760569003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e34106ee0 msgr2=0x7f0e3419d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e2a000700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e34106ee0 0x7f0e3419d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e33400700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e34106ee0 0x7f0e3419d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e33400700 1 -- 172.21.15.12:0/1760569003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e341a1e20 con 0x7f0e3410c5d0 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e32a00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e34106590 0x7f0e3419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:36.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.349+0000 7f0e33400700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3410c5d0 0x7f0e341a1780 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c007ee0 tx=0x7f0e2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.350+0000 7f0e38a00700 1 -- 172.21.15.12:0/1760569003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00dbd0 con 0x7f0e3410c5d0 2024-05-27T07:24:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.350+0000 7f0e38a00700 1 -- 172.21.15.12:0/1760569003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e2c00dd70 con 0x7f0e3410c5d0 2024-05-27T07:24:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.350+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e341a2110 con 0x7f0e3410c5d0 2024-05-27T07:24:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.350+0000 7f0e38a00700 1 -- 172.21.15.12:0/1760569003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0058d0 con 0x7f0e3410c5d0 2024-05-27T07:24:36.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.350+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e341a9a30 con 0x7f0e3410c5d0 2024-05-27T07:24:36.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.354+0000 7f0e38a00700 1 -- 172.21.15.12:0/1760569003 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0e2c01a020 con 0x7f0e3410c5d0 2024-05-27T07:24:36.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.355+0000 7f0e488e6b80 1 Processor -- start 2024-05-27T07:24:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e488e6b80 1 -- start start 2024-05-27T07:24:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e38a00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0e180b29f0 0x7f0e180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398ec7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398ecd30 0x55fc398e6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e38a00700 1 -- 172.21.15.12:0/1760569003 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0e2c0e22d0 con 0x7f0e3410c5d0 2024-05-27T07:24:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fc39814e40 con 0x55fc398ecd30 2024-05-27T07:24:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fc398049e0 con 0x55fc398e6db0 2024-05-27T07:24:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.356+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fc39808a20 con 0x55fc39767e40 2024-05-27T07:24:36.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.357+0000 7f0e32a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398ec7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.357+0000 7f0e2a000700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0e180b29f0 0x7f0e180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.357+0000 7f0e32a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398ec7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:37332/0 (socket says 172.21.15.12:37332) 2024-05-27T07:24:36.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.357+0000 7f0e32a00700 1 -- 172.21.15.12:0/2668051528 learned_addr learned my addr 172.21.15.12:0/2668051528 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:36.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.357+0000 7f0e33400700 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.357+0000 7f0e2a000700 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398ecd30 0x55fc398e6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.358+0000 7f0e33400700 1 -- 172.21.15.12:0/2668051528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 msgr2=0x55fc398ec7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.358+0000 7f0e33400700 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398ec7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.358+0000 7f0e33400700 1 -- 172.21.15.12:0/2668051528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398ecd30 msgr2=0x55fc398e6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.358+0000 7f0e33400700 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398ecd30 0x55fc398e6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.358+0000 7f0e33400700 1 -- 172.21.15.12:0/2668051528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc397cd420 con 0x55fc398e6db0 2024-05-27T07:24:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.358+0000 7f0e2a000700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0e180b29f0 0x7f0e180b4f00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00b5c0 tx=0x7f0e1c009ee0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.359+0000 7f0e33400700 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398e71c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0df610 tx=0x7f0e2c0a8910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c804b0f4b7eba1d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.359+0000 7f0e28c00700 1 -- 172.21.15.12:0/2668051528 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0a91a0 con 0x55fc398e6db0 2024-05-27T07:24:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.359+0000 7f0e28c00700 1 -- 172.21.15.12:0/2668051528 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e2c0ad550 con 0x55fc398e6db0 2024-05-27T07:24:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.359+0000 7f0e28c00700 1 -- 172.21.15.12:0/2668051528 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0a9400 con 0x55fc398e6db0 2024-05-27T07:24:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.360+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2668051528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 msgr2=0x55fc398e71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.360+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398e71c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0df610 tx=0x7f0e2c0a8910 comp rx=0 tx=0).stop 2024-05-27T07:24:36.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.360+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2668051528 shutdown_connections 2024-05-27T07:24:36.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.361+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398e71c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.361+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398ec7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.361+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2668051528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398ecd30 0x55fc398e6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.361+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2668051528 >> 172.21.15.12:0/2668051528 conn(0x55fc397c7fc0 msgr2=0x55fc398e18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.361+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2668051528 shutdown_connections 2024-05-27T07:24:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.361+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2668051528 wait complete. 2024-05-27T07:24:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.361+0000 7f0e488e6b80 1 -- start start 2024-05-27T07:24:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.362+0000 7f0e488e6b80 1 Processor -- start 2024-05-27T07:24:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.362+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc398f0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.362+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398e6db0 0x55fc398f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.362+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.362+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fc3970afa0 con 0x55fc39767e40 2024-05-27T07:24:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.362+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fc397a5760 con 0x55fc398ecd30 2024-05-27T07:24:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.362+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fc398f37a0 con 0x55fc398e6db0 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.363+0000 7f0e32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc398f0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.363+0000 7f0e488e6b80 10 client.0 _ll_register_callbacks cb 0x55fc397aec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.363+0000 7f0e2a000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398e6db0 0x55fc398f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.363+0000 7f0e33400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.363+0000 7f0e2a000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398e6db0 0x55fc398f0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:37346/0 (socket says 172.21.15.12:37346) 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.363+0000 7f0e2a000700 1 -- 172.21.15.12:0/2978850876 learned_addr learned my addr 172.21.15.12:0/2978850876 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.364+0000 7f0e2a000700 1 -- 172.21.15.12:0/2978850876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 msgr2=0x55fc398f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.364+0000 7f0e2a000700 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.364+0000 7f0e2a000700 1 -- 172.21.15.12:0/2978850876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 msgr2=0x55fc398f0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.364+0000 7f0e2a000700 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc398f0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.364+0000 7f0e2a000700 1 -- 172.21.15.12:0/2978850876 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc397548d0 con 0x55fc398e6db0 2024-05-27T07:24:36.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.364+0000 7f0e32a00700 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc398f0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:36.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.365+0000 7f0e2a000700 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398e6db0 0x55fc398f0b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c005b40 tx=0x7f0e1c023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2ac95c3d6890910 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.365+0000 7f0e21600700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:24:36.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.365+0000 7f0e21600700 1 -- 172.21.15.12:0/2978850876 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c00fd30 con 0x55fc398e6db0 2024-05-27T07:24:36.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.365+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e2c0ad550 con 0x55fc398e6db0 2024-05-27T07:24:36.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.366+0000 7f0e21600700 1 -- 172.21.15.12:0/2978850876 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0e1c004020 con 0x55fc398e6db0 2024-05-27T07:24:36.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.366+0000 7f0e21600700 1 -- 172.21.15.12:0/2978850876 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0e1c004220 con 0x55fc398e6db0 2024-05-27T07:24:36.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.366+0000 7f0e488e6b80 10 client.25330 fetch_fsmap learned FSMap version 28 2024-05-27T07:24:36.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.366+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fc397d5060 con 0x55fc398e6db0 2024-05-27T07:24:36.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.367+0000 7f0e21600700 1 -- 172.21.15.12:0/2978850876 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e2c0ad550 con 0x55fc398e6db0 2024-05-27T07:24:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.368+0000 7f0e21600700 1 -- 172.21.15.12:0/2978850876 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0e1c026070 con 0x55fc398e6db0 2024-05-27T07:24:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.368+0000 7f0e488e6b80 10 client.25330 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:24:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.368+0000 7f0e488e6b80 10 client.25330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.368+0000 7f0e488e6b80 20 client.25330 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.368+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39900830 0x55fc39902c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.368+0000 7f0e488e6b80 4 client.25330 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:36.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.368+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc39816d90 con 0x55fc39900830 2024-05-27T07:24:36.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.369+0000 7f0e32a00700 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39900830 0x55fc39902c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.370+0000 7f0e21600700 10 client.25330 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:36.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.370+0000 7f0e32a00700 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39900830 0x55fc39902c70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd8323c12337ce24 server_cookie=99a7d4fecc8477de in_seq=0 out_seq=0 2024-05-27T07:24:36.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e21600700 1 -- 172.21.15.12:0/2978850876 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fc39816d90 con 0x55fc39900830 2024-05-27T07:24:36.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e21600700 10 client.25330 handle_command_reply: tid=0 2024-05-27T07:24:36.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 client.25330 shutdown 2024-05-27T07:24:36.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398e6db0 msgr2=0x55fc398f0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398e6db0 0x55fc398f0b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c005b40 tx=0x7f0e1c023000 comp rx=0 tx=0).stop 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 shutdown_connections 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39900830 0x55fc39902c70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc398f0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/2978850876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398e6db0 0x55fc398f0b90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e21600700 0 client.25330 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.377+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 >> 172.21.15.12:0/2978850876 conn(0x55fc397c7fc0 msgr2=0x55fc398e1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.378+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 shutdown_connections 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.378+0000 7f0e488e6b80 1 -- 172.21.15.12:0/2978850876 wait complete. 2024-05-27T07:24:36.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.378+0000 7f0e488e6b80 20 client.25330 trim_cache size 0 max 16384 2024-05-27T07:24:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.398+0000 7f0e488e6b80 1 Processor -- start 2024-05-27T07:24:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.398+0000 7f0e488e6b80 1 -- start start 2024-05-27T07:24:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398e6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398e6db0 0x55fc398f1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341b4fc0 con 0x55fc398e6db0 2024-05-27T07:24:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341b2850 con 0x55fc398ecd30 2024-05-27T07:24:36.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341b06a0 con 0x55fc39767e40 2024-05-27T07:24:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e2a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398e6db0 0x55fc398f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e2a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398e6db0 0x55fc398f1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42638/0 (socket says 172.21.15.12:42638) 2024-05-27T07:24:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e2a000700 1 -- 172.21.15.12:0/3525523310 learned_addr learned my addr 172.21.15.12:0/3525523310 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e33400700 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.399+0000 7f0e32a00700 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398e6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.400+0000 7f0e2a000700 1 -- 172.21.15.12:0/3525523310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 msgr2=0x55fc398e6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.400+0000 7f0e2a000700 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398e6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.400+0000 7f0e2a000700 1 -- 172.21.15.12:0/3525523310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 msgr2=0x55fc398f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.400+0000 7f0e2a000700 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.400+0000 7f0e2a000700 1 -- 172.21.15.12:0/3525523310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc398efcc0 con 0x55fc398e6db0 2024-05-27T07:24:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.400+0000 7f0e2a000700 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398e6db0 0x55fc398f1040 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c004740 tx=0x7f0e1c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7652985a78009fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.401+0000 7f0e23e00700 1 -- 172.21.15.12:0/3525523310 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c00fd30 con 0x55fc398e6db0 2024-05-27T07:24:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.401+0000 7f0e23e00700 1 -- 172.21.15.12:0/3525523310 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e1c031e40 con 0x55fc398e6db0 2024-05-27T07:24:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.401+0000 7f0e23e00700 1 -- 172.21.15.12:0/3525523310 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c003a20 con 0x55fc398e6db0 2024-05-27T07:24:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3525523310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398e6db0 msgr2=0x55fc398f1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398e6db0 0x55fc398f1040 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c004740 tx=0x7f0e1c009310 comp rx=0 tx=0).stop 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3525523310 shutdown_connections 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398ecd30 0x55fc398f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc39767e40 0x55fc398e6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3525523310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc398e6db0 0x55fc398f1040 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3525523310 >> 172.21.15.12:0/3525523310 conn(0x55fc397c7fc0 msgr2=0x55fc398ef5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3525523310 shutdown_connections 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.402+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3525523310 wait complete. 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.403+0000 7f0e488e6b80 1 -- start start 2024-05-27T07:24:36.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.403+0000 7f0e488e6b80 1 Processor -- start 2024-05-27T07:24:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc39908b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e488e6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398ecd30 0x55fc398f8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341b0d10 con 0x55fc39767e40 2024-05-27T07:24:36.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341b2180 con 0x55fc398e6db0 2024-05-27T07:24:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e488e6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e341b13e0 con 0x55fc398ecd30 2024-05-27T07:24:36.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e488e6b80 10 client.0 _ll_register_callbacks cb 0x55fc397aec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc39908b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc39908b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42640/0 (socket says 172.21.15.12:42640) 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.404+0000 7f0e32a00700 1 -- 172.21.15.12:0/3815285744 learned_addr learned my addr 172.21.15.12:0/3815285744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e2a000700 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e33400700 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398ecd30 0x55fc398f8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e32a00700 1 -- 172.21.15.12:0/3815285744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398ecd30 msgr2=0x55fc398f8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e32a00700 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398ecd30 0x55fc398f8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e32a00700 1 -- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 msgr2=0x55fc398f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e32a00700 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e32a00700 1 -- 172.21.15.12:0/3815285744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc39907420 con 0x55fc39767e40 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e32a00700 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc39908b80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005c20 tx=0x7f0e24003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381288a9314afa50 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.405+0000 7f0e21600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.406+0000 7f0e21600700 1 -- 172.21.15.12:0/3815285744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2401d5b0 con 0x55fc39767e40 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.406+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fc39810570 con 0x55fc39767e40 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.406+0000 7f0e21600700 1 -- 172.21.15.12:0/3815285744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f0e2401d750 con 0x55fc39767e40 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.406+0000 7f0e21600700 1 -- 172.21.15.12:0/3815285744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0e2402d050 con 0x55fc39767e40 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.406+0000 7f0e488e6b80 10 client.15648 fetch_fsmap learned FSMap version 28 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.406+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fc39972d60 con 0x55fc39767e40 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e21600700 1 -- 172.21.15.12:0/3815285744 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55fc39810570 con 0x55fc39767e40 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e21600700 1 -- 172.21.15.12:0/3815285744 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0e24011be0 con 0x55fc39767e40 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e488e6b80 10 client.15648 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e488e6b80 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e488e6b80 20 client.15648 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:36.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39973480 0x55fc399758a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e488e6b80 4 client.15648 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.407+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fc39687250 con 0x55fc39973480 2024-05-27T07:24:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.408+0000 7f0e2a000700 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39973480 0x55fc399758a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.409+0000 7f0e2a000700 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39973480 0x55fc399758a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc6f2b5b9b560f32 server_cookie=1d64c783113c639 in_seq=0 out_seq=0 2024-05-27T07:24:36.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.409+0000 7f0e21600700 10 client.15648 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.420+0000 7f0e21600700 1 -- 172.21.15.12:0/3815285744 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fc39687250 con 0x55fc39973480 2024-05-27T07:24:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.420+0000 7f0e21600700 10 client.15648 handle_command_reply: tid=0 2024-05-27T07:24:36.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.420+0000 7f0e488e6b80 1 client.15648 shutdown 2024-05-27T07:24:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 msgr2=0x55fc39908b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc39908b80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005c20 tx=0x7f0e24003680 comp rx=0 tx=0).stop 2024-05-27T07:24:36.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 shutdown_connections 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fc398ecd30 0x55fc398f8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc39767e40 0x55fc39908b80 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fc39973480 0x55fc399758a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 --2- 172.21.15.12:0/3815285744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc398e6db0 0x55fc398f89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e21600700 0 client.15648 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 >> 172.21.15.12:0/3815285744 conn(0x55fc397c7fc0 msgr2=0x55fc398ef5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 shutdown_connections 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.421+0000 7f0e488e6b80 1 -- 172.21.15.12:0/3815285744 wait complete. 2024-05-27T07:24:36.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.422+0000 7f0e488e6b80 20 client.15648 trim_cache size 0 max 16384 2024-05-27T07:24:36.412 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:24:36.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.422+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0e180b29f0 msgr2=0x7f0e180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.422+0000 7f0e39e00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0e180b29f0 0x7f0e180b4f00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00b5c0 tx=0x7f0e1c009ee0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.422+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3410c5d0 msgr2=0x7f0e341a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.422+0000 7f0e39e00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3410c5d0 0x7f0e341a1780 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c007ee0 tx=0x7f0e2c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 shutdown_connections 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0e180b29f0 0x7f0e180b4f00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e3410c5d0 0x7f0e341a1780 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e34106590 0x7f0e3419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 --2- 172.21.15.12:0/1760569003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e34106ee0 0x7f0e3419d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 >> 172.21.15.12:0/1760569003 conn(0x7f0e34102070 msgr2=0x7f0e34103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 shutdown_connections 2024-05-27T07:24:36.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:36.423+0000 7f0e39e00700 1 -- 172.21.15.12:0/1760569003 wait complete. 2024-05-27T07:24:36.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:36 smithi162 ceph-mon[35676]: pgmap v69: 129 pgs: 129 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2024-05-27T07:24:36.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:36 smithi062 ceph-mon[35304]: pgmap v69: 129 pgs: 129 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2024-05-27T07:24:37.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:37 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:24:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:37 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:24:37.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:37 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:24:38.436 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:24:38.437 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:24:38.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:38 smithi012 ceph-mon[31473]: pgmap v70: 129 pgs: 129 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-05-27T07:24:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:38 smithi162 ceph-mon[35676]: pgmap v70: 129 pgs: 129 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-05-27T07:24:38.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:38 smithi062 ceph-mon[35304]: pgmap v70: 129 pgs: 129 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-05-27T07:24:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:40.538 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:40 smithi062 ceph-mon[35304]: pgmap v71: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-05-27T07:24:40.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:40 smithi012 ceph-mon[31473]: pgmap v71: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-05-27T07:24:40.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:40 smithi162 ceph-mon[35676]: pgmap v71: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-05-27T07:24:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 -- 172.21.15.12:0/1054128898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e0810c5a0 msgr2=0x7f6e0810c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/1054128898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e0810c5a0 0x7f6e0810c9b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009990 tx=0x7f6df8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 -- 172.21.15.12:0/1054128898 shutdown_connections 2024-05-27T07:24:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/1054128898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e08106eb0 0x7f6e08113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/1054128898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e0810c5a0 0x7f6e0810c9b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/1054128898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e08106560 0x7f6e08106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 -- 172.21.15.12:0/1054128898 >> 172.21.15.12:0/1054128898 conn(0x7f6e08102040 msgr2=0x7f6e08104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.618+0000 7f6e0dc00700 1 -- 172.21.15.12:0/1054128898 shutdown_connections 2024-05-27T07:24:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.619+0000 7f6e0dc00700 1 -- 172.21.15.12:0/1054128898 wait complete. 2024-05-27T07:24:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.620+0000 7f6e0dc00700 1 Processor -- start 2024-05-27T07:24:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.620+0000 7f6e0dc00700 1 -- start start 2024-05-27T07:24:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e0dc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e08106560 0x7f6e0819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e0dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e08106eb0 0x7f6e0819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e0dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e0810c5a0 0x7f6e081a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e0dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081165b0 con 0x7f6e08106eb0 2024-05-27T07:24:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e0dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08116430 con 0x7f6e0810c5a0 2024-05-27T07:24:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e0dc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08116730 con 0x7f6e08106560 2024-05-27T07:24:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e06000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e08106eb0 0x7f6e0819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e06000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e08106eb0 0x7f6e0819d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42660/0 (socket says 172.21.15.12:42660) 2024-05-27T07:24:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e06000700 1 -- 172.21.15.12:0/2436156242 learned_addr learned my addr 172.21.15.12:0/2436156242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.621+0000 7f6e06a00700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e08106560 0x7f6e0819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.622+0000 7f6e07400700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e0810c5a0 0x7f6e081a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.622+0000 7f6e06000700 1 -- 172.21.15.12:0/2436156242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e08106560 msgr2=0x7f6e0819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.622+0000 7f6e06000700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e08106560 0x7f6e0819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.622+0000 7f6e06000700 1 -- 172.21.15.12:0/2436156242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e0810c5a0 msgr2=0x7f6e081a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.622+0000 7f6e06000700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e0810c5a0 0x7f6e081a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.622+0000 7f6e06000700 1 -- 172.21.15.12:0/2436156242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e081a1e30 con 0x7f6e08106eb0 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.622+0000 7f6e06000700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e08106eb0 0x7f6e0819d500 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6df400d950 tx=0x7f6df400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.623+0000 7f6dff400700 1 -- 172.21.15.12:0/2436156242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4017070 con 0x7f6e08106eb0 2024-05-27T07:24:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.623+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e081a2120 con 0x7f6e08106eb0 2024-05-27T07:24:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.623+0000 7f6dff400700 1 -- 172.21.15.12:0/2436156242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6df400de30 con 0x7f6e08106eb0 2024-05-27T07:24:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.623+0000 7f6dff400700 1 -- 172.21.15.12:0/2436156242 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df400f3c0 con 0x7f6e08106eb0 2024-05-27T07:24:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.623+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e081a9a40 con 0x7f6e08106eb0 2024-05-27T07:24:41.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.628+0000 7f6dff400700 1 -- 172.21.15.12:0/2436156242 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6df401a070 con 0x7f6e08106eb0 2024-05-27T07:24:41.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.629+0000 7f6e1c77db80 1 Processor -- start 2024-05-27T07:24:41.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.629+0000 7f6e1c77db80 1 -- start start 2024-05-27T07:24:41.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563750f4dec0 0x5637510d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563750ffaec0 con 0x5637510d2db0 2024-05-27T07:24:41.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563750feaa60 con 0x5637510cce30 2024-05-27T07:24:41.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563750feeaa0 con 0x563750f4dec0 2024-05-27T07:24:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42662/0 (socket says 172.21.15.12:42662) 2024-05-27T07:24:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e06a00700 1 -- 172.21.15.12:0/2170030726 learned_addr learned my addr 172.21.15.12:0/2170030726 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e07400700 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563750f4dec0 0x5637510d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.630+0000 7f6e06000700 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.631+0000 7f6dff400700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dec0b29f0 0x7f6dec0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.631+0000 7f6e06a00700 1 -- 172.21.15.12:0/2170030726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563750f4dec0 msgr2=0x5637510d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.631+0000 7f6e06a00700 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563750f4dec0 0x5637510d2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.631+0000 7f6e06a00700 1 -- 172.21.15.12:0/2170030726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 msgr2=0x5637510cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.631+0000 7f6e06a00700 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.631+0000 7f6e06a00700 1 -- 172.21.15.12:0/2170030726 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563750fb34a0 con 0x5637510d2db0 2024-05-27T07:24:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.631+0000 7f6dff400700 1 -- 172.21.15.12:0/2436156242 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f6df40e23c0 con 0x7f6e08106eb0 2024-05-27T07:24:41.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.632+0000 7f6e06a00700 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510cc8f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6df8015000 tx=0x7f6df80096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fdcf173c584219 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.632+0000 7f6e07400700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dec0b29f0 0x7f6dec0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.632+0000 7f6df3e00700 1 -- 172.21.15.12:0/2170030726 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8026070 con 0x5637510d2db0 2024-05-27T07:24:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.632+0000 7f6df3e00700 1 -- 172.21.15.12:0/2170030726 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6df8003c50 con 0x5637510d2db0 2024-05-27T07:24:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.632+0000 7f6df3e00700 1 -- 172.21.15.12:0/2170030726 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df800b7d0 con 0x5637510d2db0 2024-05-27T07:24:41.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e1c77db80 1 -- 172.21.15.12:0/2170030726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 msgr2=0x5637510cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510cc8f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6df8015000 tx=0x7f6df80096a0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e07400700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dec0b29f0 0x7f6dec0b4f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x5637510cdbb0 tx=0x7f6e0000b040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e1c77db80 1 -- 172.21.15.12:0/2170030726 shutdown_connections 2024-05-27T07:24:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563750f4dec0 0x5637510d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/2170030726 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510cc8f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.633+0000 7f6e1c77db80 1 -- 172.21.15.12:0/2170030726 >> 172.21.15.12:0/2170030726 conn(0x563750fae040 msgr2=0x5637510c7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.634+0000 7f6e1c77db80 1 -- 172.21.15.12:0/2170030726 shutdown_connections 2024-05-27T07:24:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.634+0000 7f6e1c77db80 1 -- 172.21.15.12:0/2170030726 wait complete. 2024-05-27T07:24:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.635+0000 7f6e1c77db80 1 -- start start 2024-05-27T07:24:41.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.635+0000 7f6e1c77db80 1 Processor -- start 2024-05-27T07:24:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510d64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510cce30 0x5637510d6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563750ef1020 con 0x5637510cce30 2024-05-27T07:24:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563750f8b7e0 con 0x563750f4dec0 2024-05-27T07:24:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5637510d9820 con 0x5637510d2db0 2024-05-27T07:24:41.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e1c77db80 10 client.0 _ll_register_callbacks cb 0x563750f94cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e06000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510cce30 0x5637510d6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e06000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510cce30 0x5637510d6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42666/0 (socket says 172.21.15.12:42666) 2024-05-27T07:24:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e06000700 1 -- 172.21.15.12:0/770345204 learned_addr learned my addr 172.21.15.12:0/770345204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:41.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e07400700 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.636+0000 7f6e06a00700 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510d64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.637+0000 7f6e06000700 1 -- 172.21.15.12:0/770345204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 msgr2=0x5637510d6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.637+0000 7f6e06000700 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510d6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.637+0000 7f6e06000700 1 -- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 msgr2=0x5637510d64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.637+0000 7f6e06000700 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510d64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.637+0000 7f6e06000700 1 -- 172.21.15.12:0/770345204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563750f3a950 con 0x5637510cce30 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.637+0000 7f6e06000700 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510cce30 0x5637510d6a20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6df40a5310 tx=0x7f6df40dfc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ec681f119e9d15 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.637+0000 7f6df0c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.638+0000 7f6df0c00700 1 -- 172.21.15.12:0/770345204 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4021030 con 0x5637510cce30 2024-05-27T07:24:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.638+0000 7f6df0c00700 1 -- 172.21.15.12:0/770345204 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f6df40e2150 con 0x5637510cce30 2024-05-27T07:24:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.638+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6df8003c50 con 0x5637510cce30 2024-05-27T07:24:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.638+0000 7f6df0c00700 1 -- 172.21.15.12:0/770345204 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6df40106f0 con 0x5637510cce30 2024-05-27T07:24:41.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.638+0000 7f6e1c77db80 10 client.15660 fetch_fsmap learned FSMap version 28 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.638+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563750fbb0e0 con 0x5637510cce30 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6df0c00700 1 -- 172.21.15.12:0/770345204 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6df8003c50 con 0x5637510cce30 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6df0c00700 1 -- 172.21.15.12:0/770345204 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6df4009b40 con 0x5637510cce30 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6e1c77db80 10 client.15660 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6e1c77db80 10 client.15660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6e1c77db80 20 client.15660 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5637510e6650 0x5637510e8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6e1c77db80 4 client.15660 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:41.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.639+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563750ffce10 con 0x5637510e6650 2024-05-27T07:24:41.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.640+0000 7f6e06a00700 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5637510e6650 0x5637510e8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.641+0000 7f6e06a00700 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5637510e6650 0x5637510e8a90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1a021e4c9a4be96 server_cookie=bcabfca80c03cf48 in_seq=0 out_seq=0 2024-05-27T07:24:41.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.641+0000 7f6df0c00700 10 client.15660 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:41.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.643+0000 7f6df0c00700 1 -- 172.21.15.12:0/770345204 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563750ffce10 con 0x5637510e6650 2024-05-27T07:24:41.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.643+0000 7f6df0c00700 10 client.15660 handle_command_reply: tid=0 2024-05-27T07:24:41.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.643+0000 7f6e1c77db80 1 client.15660 shutdown 2024-05-27T07:24:41.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.643+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510cce30 msgr2=0x5637510d6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.643+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510cce30 0x5637510d6a20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6df40a5310 tx=0x7f6df40dfc60 comp rx=0 tx=0).stop 2024-05-27T07:24:41.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 shutdown_connections 2024-05-27T07:24:41.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5637510e6650 0x5637510e8a90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510d64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510d6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/770345204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510cce30 0x5637510d6a20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 >> 172.21.15.12:0/770345204 conn(0x563750fae040 msgr2=0x5637510c7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6df0c00700 0 client.15660 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 shutdown_connections 2024-05-27T07:24:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 1 -- 172.21.15.12:0/770345204 wait complete. 2024-05-27T07:24:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.644+0000 7f6e1c77db80 20 client.15660 trim_cache size 0 max 16384 2024-05-27T07:24:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.674+0000 7f6e1c77db80 1 Processor -- start 2024-05-27T07:24:41.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.674+0000 7f6e1c77db80 1 -- start start 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510cce30 0x5637510d8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b4f30 con 0x5637510d2db0 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b27c0 con 0x563750f4dec0 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b0610 con 0x5637510cce30 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510d0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42670/0 (socket says 172.21.15.12:42670) 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.675+0000 7f6e07400700 1 -- 172.21.15.12:0/3775619979 learned_addr learned my addr 172.21.15.12:0/3775619979 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e06a00700 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e06000700 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510cce30 0x5637510d8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e07400700 1 -- 172.21.15.12:0/3775619979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510cce30 msgr2=0x5637510d8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e07400700 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510cce30 0x5637510d8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e07400700 1 -- 172.21.15.12:0/3775619979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 msgr2=0x5637510cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e07400700 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e07400700 1 -- 172.21.15.12:0/3775619979 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637510d44c0 con 0x5637510d2db0 2024-05-27T07:24:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.676+0000 7f6e07400700 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510d0bf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6e00016600 tx=0x7f6e00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fa1c78cae03b5a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.677+0000 7f6df3400700 1 -- 172.21.15.12:0/3775619979 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0001c030 con 0x5637510d2db0 2024-05-27T07:24:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.677+0000 7f6df3400700 1 -- 172.21.15.12:0/3775619979 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e00027070 con 0x5637510d2db0 2024-05-27T07:24:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.677+0000 7f6df3400700 1 -- 172.21.15.12:0/3775619979 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00012040 con 0x5637510d2db0 2024-05-27T07:24:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3775619979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 msgr2=0x5637510d0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510d0bf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6e00016600 tx=0x7f6e00014000 comp rx=0 tx=0).stop 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3775619979 shutdown_connections 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563750f4dec0 0x5637510cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5637510d2db0 0x5637510d0bf0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3775619979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510cce30 0x5637510d8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3775619979 >> 172.21.15.12:0/3775619979 conn(0x563750fae040 msgr2=0x5637510d6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3775619979 shutdown_connections 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.678+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3775619979 wait complete. 2024-05-27T07:24:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.679+0000 7f6e1c77db80 1 -- start start 2024-05-27T07:24:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.679+0000 7f6e1c77db80 1 Processor -- start 2024-05-27T07:24:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563750f4dec0 0x5637510def50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510dbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e1c77db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510dbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b0c80 con 0x563750f4dec0 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b20f0 con 0x5637510cce30 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e1c77db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b1350 con 0x5637510d2db0 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e1c77db80 10 client.0 _ll_register_callbacks cb 0x563750f94cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563750f4dec0 0x5637510def50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563750f4dec0 0x5637510def50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42682/0 (socket says 172.21.15.12:42682) 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e06a00700 1 -- 172.21.15.12:0/3545572226 learned_addr learned my addr 172.21.15.12:0/3545572226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e06000700 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510dbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.680+0000 7f6e07400700 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510dbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6e06a00700 1 -- 172.21.15.12:0/3545572226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 msgr2=0x5637510dbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6e06a00700 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510dbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6e06a00700 1 -- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 msgr2=0x5637510dbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6e06a00700 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510dbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6e06a00700 1 -- 172.21.15.12:0/3545572226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637510dc530 con 0x563750f4dec0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6e06a00700 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563750f4dec0 0x5637510def50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009710 tx=0x7f6df800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfc51bd34c4fe3b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6df0c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6df0c00700 1 -- 172.21.15.12:0/3545572226 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8026040 con 0x563750f4dec0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6df0c00700 1 -- 172.21.15.12:0/3545572226 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f6df8005300 con 0x563750f4dec0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.681+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e00027070 con 0x563750f4dec0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.682+0000 7f6df0c00700 1 -- 172.21.15.12:0/3545572226 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6df803e050 con 0x563750f4dec0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.682+0000 7f6e1c77db80 10 client.15666 fetch_fsmap learned FSMap version 28 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.682+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563751151e00 con 0x563750f4dec0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.682+0000 7f6df0c00700 1 -- 172.21.15.12:0/3545572226 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e00027070 con 0x563750f4dec0 2024-05-27T07:24:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.682+0000 7f6df0c00700 1 -- 172.21.15.12:0/3545572226 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6df8034030 con 0x563750f4dec0 2024-05-27T07:24:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.684+0000 7f6e1c77db80 10 client.15666 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:24:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.684+0000 7f6e1c77db80 10 client.15666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.684+0000 7f6e1c77db80 20 client.15666 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.684+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563751151fd0 0x5637511543f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.684+0000 7f6e1c77db80 4 client.15666 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:24:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.684+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563750ff65f0 con 0x563751151fd0 2024-05-27T07:24:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.685+0000 7f6e06000700 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563751151fd0 0x5637511543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.686+0000 7f6df0c00700 10 client.15666 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.686+0000 7f6e06000700 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563751151fd0 0x5637511543f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=637e3413d722b66f server_cookie=a91077f34c26bfd9 in_seq=0 out_seq=0 2024-05-27T07:24:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6df0c00700 1 -- 172.21.15.12:0/3545572226 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563750ff65f0 con 0x563751151fd0 2024-05-27T07:24:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6df0c00700 10 client.15666 handle_command_reply: tid=0 2024-05-27T07:24:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 client.15666 shutdown 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563750f4dec0 msgr2=0x5637510def50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563750f4dec0 0x5637510def50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009710 tx=0x7f6df800b650 comp rx=0 tx=0).stop 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 shutdown_connections 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5637510d2db0 0x5637510dbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563750f4dec0 0x5637510def50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x563751151fd0 0x5637511543f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 --2- 172.21.15.12:0/3545572226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5637510cce30 0x5637510dbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 >> 172.21.15.12:0/3545572226 conn(0x563750fae040 msgr2=0x5637510d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.691+0000 7f6df0c00700 0 client.15666 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:41.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.692+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 shutdown_connections 2024-05-27T07:24:41.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.692+0000 7f6e1c77db80 1 -- 172.21.15.12:0/3545572226 wait complete. 2024-05-27T07:24:41.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.692+0000 7f6e1c77db80 20 client.15666 trim_cache size 0 max 16384 2024-05-27T07:24:41.682 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:24:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:24:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "402863a5-f9d9-44db-b0c5-dba37415ff16", 2024-05-27T07:24:41.683 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:24:41.683 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:24:41.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dec0b29f0 msgr2=0x7f6dec0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dec0b29f0 0x7f6dec0b4f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x5637510cdbb0 tx=0x7f6e0000b040 comp rx=0 tx=0).stop 2024-05-27T07:24:41.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e08106eb0 msgr2=0x7f6e0819d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e08106eb0 0x7f6e0819d500 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6df400d950 tx=0x7f6df400dd10 comp rx=0 tx=0).stop 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 shutdown_connections 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dec0b29f0 0x7f6dec0b4f00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e0810c5a0 0x7f6e081a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e08106560 0x7f6e0819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 --2- 172.21.15.12:0/2436156242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e08106eb0 0x7f6e0819d500 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.693+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 >> 172.21.15.12:0/2436156242 conn(0x7f6e08102040 msgr2=0x7f6e08103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.694+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 shutdown_connections 2024-05-27T07:24:41.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:41.694+0000 7f6e0dc00700 1 -- 172.21.15.12:0/2436156242 wait complete. 2024-05-27T07:24:42.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:42 smithi012 ceph-mon[31473]: pgmap v72: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-05-27T07:24:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:42 smithi162 ceph-mon[35676]: pgmap v72: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-05-27T07:24:42.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:42 smithi062 ceph-mon[35304]: pgmap v72: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-05-27T07:24:42.769 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:24:43.326 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:24:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:24:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:24:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:24:43.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:43 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:24:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:24:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:24:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:24:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:24:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:43 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:24:43.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:24:43.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:24:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:24:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:24:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:24:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:43 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:24:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:44 smithi012 ceph-mon[31473]: pgmap v73: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-05-27T07:24:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:44 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:44 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:44 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:24:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:44 smithi162 ceph-mon[35676]: pgmap v73: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-05-27T07:24:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:44 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:44 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:44.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:44 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:24:44.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:44 smithi062 ceph-mon[35304]: pgmap v73: 129 pgs: 129 active+clean; 18 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 1.4 MiB/s wr, 47 op/s 2024-05-27T07:24:44.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:44 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:44.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:44 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:44.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:44 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:24:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 -- 172.21.15.12:0/3976735320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c107de0 msgr2=0x7fef3c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 --2- 172.21.15.12:0/3976735320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c107de0 0x7fef3c1081f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fef280099c0 tx=0x7fef28009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 -- 172.21.15.12:0/3976735320 shutdown_connections 2024-05-27T07:24:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 --2- 172.21.15.12:0/3976735320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 0x7fef3c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 --2- 172.21.15.12:0/3976735320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef3c108730 0x7fef3c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 --2- 172.21.15.12:0/3976735320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c107de0 0x7fef3c1081f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 -- 172.21.15.12:0/3976735320 >> 172.21.15.12:0/3976735320 conn(0x7fef3c0fedb0 msgr2=0x7fef3c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 -- 172.21.15.12:0/3976735320 shutdown_connections 2024-05-27T07:24:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.149+0000 7fef41600700 1 -- 172.21.15.12:0/3976735320 wait complete. 2024-05-27T07:24:46.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.151+0000 7fef41600700 1 Processor -- start 2024-05-27T07:24:46.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.151+0000 7fef41600700 1 -- start start 2024-05-27T07:24:46.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.151+0000 7fef41600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 0x7fef3c1a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef41600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef3c107de0 0x7fef3c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef41600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c108730 0x7fef3c198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef41600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c113240 con 0x7fef3c108730 2024-05-27T07:24:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef41600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c1130c0 con 0x7fef3c1034b0 2024-05-27T07:24:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef41600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c1133c0 con 0x7fef3c107de0 2024-05-27T07:24:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 0x7fef3c1a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 0x7fef3c1a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:32890/0 (socket says 172.21.15.12:32890) 2024-05-27T07:24:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.152+0000 7fef3aa00700 1 -- 172.21.15.12:0/1520688985 learned_addr learned my addr 172.21.15.12:0/1520688985 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.153+0000 7fef3b400700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c108730 0x7fef3c198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.153+0000 7fef3a000700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef3c107de0 0x7fef3c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.153+0000 7fef3aa00700 1 -- 172.21.15.12:0/1520688985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef3c107de0 msgr2=0x7fef3c198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.153+0000 7fef3aa00700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef3c107de0 0x7fef3c198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.153+0000 7fef3aa00700 1 -- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c108730 msgr2=0x7fef3c198ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.154+0000 7fef3aa00700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c108730 0x7fef3c198ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.154+0000 7fef3aa00700 1 -- 172.21.15.12:0/1520688985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3c199820 con 0x7fef3c1034b0 2024-05-27T07:24:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.154+0000 7fef3b400700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c108730 0x7fef3c198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.154+0000 7fef3aa00700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 0x7fef3c1a0940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00d980 tx=0x7fef2c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.154+0000 7fef33400700 1 -- 172.21.15.12:0/1520688985 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c016070 con 0x7fef3c1034b0 2024-05-27T07:24:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.154+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3c1b6070 con 0x7fef3c1034b0 2024-05-27T07:24:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.155+0000 7fef33400700 1 -- 172.21.15.12:0/1520688985 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef2c004830 con 0x7fef3c1034b0 2024-05-27T07:24:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.155+0000 7fef33400700 1 -- 172.21.15.12:0/1520688985 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0054c0 con 0x7fef3c1034b0 2024-05-27T07:24:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.155+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3c1b6580 con 0x7fef3c1034b0 2024-05-27T07:24:46.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.160+0000 7fef33400700 1 -- 172.21.15.12:0/1520688985 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fef2c00f690 con 0x7fef3c1034b0 2024-05-27T07:24:46.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.161+0000 7fef50224b80 1 Processor -- start 2024-05-27T07:24:46.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef50224b80 1 -- start start 2024-05-27T07:24:46.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef50224b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef50224b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20f0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef50224b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef50224b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55bdd201ee40 con 0x55bdd20f6d30 2024-05-27T07:24:46.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef50224b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bdd200e9e0 con 0x55bdd20f0db0 2024-05-27T07:24:46.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef50224b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bdd2012a20 con 0x55bdd1f71e40 2024-05-27T07:24:46.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.162+0000 7fef33400700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef200b29f0 0x7fef200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.163+0000 7fef3b400700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef200b29f0 0x7fef200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.163+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.163+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20f0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52724/0 (socket says 172.21.15.12:52724) 2024-05-27T07:24:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.163+0000 7fef3b400700 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.163+0000 7fef3aa00700 1 -- 172.21.15.12:0/1871495826 learned_addr learned my addr 172.21.15.12:0/1871495826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.163+0000 7fef3a000700 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.164+0000 7fef33400700 1 -- 172.21.15.12:0/1520688985 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fef2c0e26d0 con 0x7fef3c1034b0 2024-05-27T07:24:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.165+0000 7fef3b400700 1 -- 172.21.15.12:0/1871495826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 msgr2=0x55bdd20f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.165+0000 7fef3b400700 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20f11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.165+0000 7fef3b400700 1 -- 172.21.15.12:0/1871495826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 msgr2=0x55bdd20f0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.165+0000 7fef3b400700 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20f0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.165+0000 7fef3b400700 1 -- 172.21.15.12:0/1871495826 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd1fd7420 con 0x55bdd1f71e40 2024-05-27T07:24:46.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.165+0000 7fef3aa00700 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:46.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.165+0000 7fef3a000700 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:46.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.166+0000 7fef3b400700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef200b29f0 0x7fef200b4f00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fef34009550 tx=0x7fef34007500 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:46.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.166+0000 7fef3b400700 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20f67f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef340073e0 tx=0x7fef34007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57850bece19119e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:46.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.167+0000 7fef27e00700 1 -- 172.21.15.12:0/1871495826 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3401b760 con 0x55bdd1f71e40 2024-05-27T07:24:46.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.167+0000 7fef27e00700 1 -- 172.21.15.12:0/1871495826 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef3401f070 con 0x55bdd1f71e40 2024-05-27T07:24:46.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.167+0000 7fef27e00700 1 -- 172.21.15.12:0/1871495826 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34023890 con 0x55bdd1f71e40 2024-05-27T07:24:46.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.169+0000 7fef50224b80 1 -- 172.21.15.12:0/1871495826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 msgr2=0x55bdd20f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.169+0000 7fef50224b80 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20f67f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef340073e0 tx=0x7fef34007f90 comp rx=0 tx=0).stop 2024-05-27T07:24:46.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.169+0000 7fef50224b80 1 -- 172.21.15.12:0/1871495826 shutdown_connections 2024-05-27T07:24:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.169+0000 7fef50224b80 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.169+0000 7fef50224b80 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20f67f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.169+0000 7fef50224b80 1 --2- 172.21.15.12:0/1871495826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.169+0000 7fef50224b80 1 -- 172.21.15.12:0/1871495826 >> 172.21.15.12:0/1871495826 conn(0x55bdd1fd1fc0 msgr2=0x55bdd20eb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.170+0000 7fef50224b80 1 -- 172.21.15.12:0/1871495826 shutdown_connections 2024-05-27T07:24:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.170+0000 7fef50224b80 1 -- 172.21.15.12:0/1871495826 wait complete. 2024-05-27T07:24:46.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.170+0000 7fef50224b80 1 -- start start 2024-05-27T07:24:46.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 1 Processor -- start 2024-05-27T07:24:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20fa490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20fa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20faf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55bdd1f14fa0 con 0x55bdd20f6d30 2024-05-27T07:24:46.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bdd1faf760 con 0x55bdd20f0db0 2024-05-27T07:24:46.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bdd20fd7a0 con 0x55bdd1f71e40 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.171+0000 7fef50224b80 10 client.0 _ll_register_callbacks cb 0x55bdd1fb8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.172+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20fa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.172+0000 7fef3b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20faf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.172+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20fa490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:42226/0 (socket says 172.21.15.12:42226) 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.172+0000 7fef3aa00700 1 -- 172.21.15.12:0/552569043 learned_addr learned my addr 172.21.15.12:0/552569043 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.173+0000 7fef3a000700 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20fa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.173+0000 7fef3aa00700 1 -- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 msgr2=0x55bdd20fa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.173+0000 7fef3aa00700 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20fa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.173+0000 7fef3aa00700 1 -- 172.21.15.12:0/552569043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 msgr2=0x55bdd20faf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.173+0000 7fef3aa00700 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20faf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.173+0000 7fef3aa00700 1 -- 172.21.15.12:0/552569043 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd1f5e8d0 con 0x55bdd1f71e40 2024-05-27T07:24:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.174+0000 7fef3b400700 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20faf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.174+0000 7fef3aa00700 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20fa490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef2c004800 tx=0x7fef2c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77c6fa6f71ad35d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.174+0000 7fef24c00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:24:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.174+0000 7fef24c00700 1 -- 172.21.15.12:0/552569043 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0dfea0 con 0x55bdd1f71e40 2024-05-27T07:24:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.175+0000 7fef24c00700 1 -- 172.21.15.12:0/552569043 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fef2c0af0e0 con 0x55bdd1f71e40 2024-05-27T07:24:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.175+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef3401f070 con 0x55bdd1f71e40 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.175+0000 7fef24c00700 1 -- 172.21.15.12:0/552569043 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef2c0af2e0 con 0x55bdd1f71e40 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.175+0000 7fef50224b80 10 client.25366 fetch_fsmap learned FSMap version 28 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.175+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bdd1fdf060 con 0x55bdd1f71e40 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.176+0000 7fef24c00700 1 -- 172.21.15.12:0/552569043 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef3401f070 con 0x55bdd1f71e40 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef24c00700 1 -- 172.21.15.12:0/552569043 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fef2c0e22a0 con 0x55bdd1f71e40 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef50224b80 10 client.25366 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef50224b80 10 client.25366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef50224b80 20 client.25366 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef50224b80 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd210a620 0x55bdd210ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef50224b80 4 client.25366 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdd2020d90 con 0x55bdd210a620 2024-05-27T07:24:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.177+0000 7fef3a000700 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd210a620 0x55bdd210ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.179+0000 7fef3a000700 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd210a620 0x55bdd210ca80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c159395a8c4d702 server_cookie=c0df982b2b8353c2 in_seq=0 out_seq=0 2024-05-27T07:24:46.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.179+0000 7fef24c00700 10 client.25366 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:46.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef24c00700 1 -- 172.21.15.12:0/552569043 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bdd2020d90 con 0x55bdd210a620 2024-05-27T07:24:46.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef24c00700 10 client.25366 handle_command_reply: tid=0 2024-05-27T07:24:46.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 client.25366 shutdown 2024-05-27T07:24:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 msgr2=0x55bdd20fa490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20fa490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef2c004800 tx=0x7fef2c004ab0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 shutdown_connections 2024-05-27T07:24:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd20faf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd1f71e40 0x55bdd20fa490 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd210a620 0x55bdd210ca80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 --2- 172.21.15.12:0/552569043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd20f0db0 0x55bdd20fa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 >> 172.21.15.12:0/552569043 conn(0x55bdd1fd1fc0 msgr2=0x55bdd20eb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.182+0000 7fef24c00700 0 client.25366 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.183+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 shutdown_connections 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.183+0000 7fef50224b80 1 -- 172.21.15.12:0/552569043 wait complete. 2024-05-27T07:24:46.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.183+0000 7fef50224b80 20 client.25366 trim_cache size 0 max 16384 2024-05-27T07:24:46.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 Processor -- start 2024-05-27T07:24:46.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 -- start start 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd20f0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f0db0 0x55bdd20f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f6d30 0x55bdd20fc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c011690 con 0x55bdd20f0db0 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c00ef20 con 0x55bdd1f71e40 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.212+0000 7fef50224b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c00cd70 con 0x55bdd20f6d30 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f0db0 0x55bdd20f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f0db0 0x55bdd20f4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52746/0 (socket says 172.21.15.12:52746) 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 -- 172.21.15.12:0/3983066682 learned_addr learned my addr 172.21.15.12:0/3983066682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3aa00700 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd20f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 -- 172.21.15.12:0/3983066682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f6d30 msgr2=0x55bdd20fc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f6d30 0x55bdd20fc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 -- 172.21.15.12:0/3983066682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 msgr2=0x55bdd20f0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd20f0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 -- 172.21.15.12:0/3983066682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd20f6380 con 0x55bdd20f0db0 2024-05-27T07:24:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.213+0000 7fef3a000700 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f0db0 0x55bdd20f4c40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fef28009990 tx=0x7fef28005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32521b000f8d84ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.214+0000 7fef27400700 1 -- 172.21.15.12:0/3983066682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2802a070 con 0x55bdd20f0db0 2024-05-27T07:24:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.214+0000 7fef27400700 1 -- 172.21.15.12:0/3983066682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef2800be20 con 0x55bdd20f0db0 2024-05-27T07:24:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.214+0000 7fef27400700 1 -- 172.21.15.12:0/3983066682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28003dd0 con 0x55bdd20f0db0 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 -- 172.21.15.12:0/3983066682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f0db0 msgr2=0x55bdd20f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f0db0 0x55bdd20f4c40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fef28009990 tx=0x7fef28005040 comp rx=0 tx=0).stop 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 -- 172.21.15.12:0/3983066682 shutdown_connections 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd20f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f6d30 0x55bdd20fc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 --2- 172.21.15.12:0/3983066682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f0db0 0x55bdd20f4c40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 -- 172.21.15.12:0/3983066682 >> 172.21.15.12:0/3983066682 conn(0x55bdd1fd1fc0 msgr2=0x55bdd20fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 -- 172.21.15.12:0/3983066682 shutdown_connections 2024-05-27T07:24:46.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.216+0000 7fef50224b80 1 -- 172.21.15.12:0/3983066682 wait complete. 2024-05-27T07:24:46.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.217+0000 7fef50224b80 1 -- start start 2024-05-27T07:24:46.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.217+0000 7fef50224b80 1 Processor -- start 2024-05-27T07:24:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef50224b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd2102c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef50224b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f0db0 0x55bdd21031d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef50224b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd2111450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef50224b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c00d3e0 con 0x55bdd20f6d30 2024-05-27T07:24:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef50224b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c00e850 con 0x55bdd1f71e40 2024-05-27T07:24:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef50224b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c00dab0 con 0x55bdd20f0db0 2024-05-27T07:24:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef50224b80 10 client.0 _ll_register_callbacks cb 0x55bdd1fb8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd2102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef3aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd2102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:32930/0 (socket says 172.21.15.12:32930) 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef3aa00700 1 -- 172.21.15.12:0/3290093941 learned_addr learned my addr 172.21.15.12:0/3290093941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef3a000700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f0db0 0x55bdd21031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.218+0000 7fef3b400700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd2111450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef3aa00700 1 -- 172.21.15.12:0/3290093941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f0db0 msgr2=0x55bdd21031d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef3aa00700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f0db0 0x55bdd21031d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef3aa00700 1 -- 172.21.15.12:0/3290093941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 msgr2=0x55bdd2111450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef3aa00700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd2111450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef3aa00700 1 -- 172.21.15.12:0/3290093941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd2111ac0 con 0x55bdd1f71e40 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef3a000700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f0db0 0x55bdd21031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef3aa00700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd2102c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0aed90 tx=0x7fef2c0a9d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359eade532db6f24 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:46.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.219+0000 7fef24c00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:24:46.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.220+0000 7fef24c00700 1 -- 172.21.15.12:0/3290093941 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0a5460 con 0x55bdd1f71e40 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.220+0000 7fef24c00700 1 -- 172.21.15.12:0/3290093941 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7fef2c0b2180 con 0x55bdd1f71e40 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.220+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef2800be20 con 0x55bdd1f71e40 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.220+0000 7fef24c00700 1 -- 172.21.15.12:0/3290093941 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef2c0a90b0 con 0x55bdd1f71e40 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.220+0000 7fef50224b80 10 client.35330 fetch_fsmap learned FSMap version 28 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.220+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bdd21749c0 con 0x55bdd1f71e40 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef24c00700 1 -- 172.21.15.12:0/3290093941 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef2800be20 con 0x55bdd1f71e40 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef24c00700 1 -- 172.21.15.12:0/3290093941 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fef2c0a8800 con 0x55bdd1f71e40 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef50224b80 10 client.35330 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef50224b80 10 client.35330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:46.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef50224b80 20 client.35330 populate_metadata read hostname 'smithi012' 2024-05-27T07:24:46.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef50224b80 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd2175090 0x55bdd21774b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:46.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef50224b80 4 client.35330 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:24:46.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.221+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bdd201a570 con 0x55bdd2175090 2024-05-27T07:24:46.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.222+0000 7fef3a000700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd2175090 0x55bdd21774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:46.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.222+0000 7fef3a000700 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd2175090 0x55bdd21774b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a57a596af12250dc server_cookie=13d4b6f3951bd2e1 in_seq=0 out_seq=0 2024-05-27T07:24:46.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.222+0000 7fef24c00700 10 client.35330 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:46.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.227+0000 7fef24c00700 1 -- 172.21.15.12:0/3290093941 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bdd201a570 con 0x55bdd2175090 2024-05-27T07:24:46.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.227+0000 7fef24c00700 10 client.35330 handle_command_reply: tid=0 2024-05-27T07:24:46.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.227+0000 7fef50224b80 1 client.35330 shutdown 2024-05-27T07:24:46.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 msgr2=0x55bdd2102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd2102c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0aed90 tx=0x7fef2c0a9d60 comp rx=0 tx=0).stop 2024-05-27T07:24:46.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 shutdown_connections 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bdd2175090 0x55bdd21774b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bdd1f71e40 0x55bdd2102c90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bdd20f6d30 0x55bdd2111450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 --2- 172.21.15.12:0/3290093941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdd20f0db0 0x55bdd21031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 >> 172.21.15.12:0/3290093941 conn(0x55bdd1fd1fc0 msgr2=0x55bdd20faca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef24c00700 0 client.35330 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 shutdown_connections 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 1 -- 172.21.15.12:0/3290093941 wait complete. 2024-05-27T07:24:46.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.228+0000 7fef50224b80 20 client.35330 trim_cache size 0 max 16384 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: "402863a5-f9d9-44db-b0c5-dba37415ff16": { 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "402863a5-f9d9-44db-b0c5-dba37415ff16", 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:24:46.219 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:24:46.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef200b29f0 msgr2=0x7fef200b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef200b29f0 0x7fef200b4f00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fef34009550 tx=0x7fef34007500 comp rx=0 tx=0).stop 2024-05-27T07:24:46.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 msgr2=0x7fef3c1a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:46.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 0x7fef3c1a0940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00d980 tx=0x7fef2c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:24:46.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 shutdown_connections 2024-05-27T07:24:46.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef200b29f0 0x7fef200b4f00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef3c1034b0 0x7fef3c1a0940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef3c108730 0x7fef3c198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 --2- 172.21.15.12:0/1520688985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef3c107de0 0x7fef3c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:46.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.230+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 >> 172.21.15.12:0/1520688985 conn(0x7fef3c0fedb0 msgr2=0x7fef3c1066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:46.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.231+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 shutdown_connections 2024-05-27T07:24:46.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:24:46.231+0000 7fef41600700 1 -- 172.21.15.12:0/1520688985 wait complete. 2024-05-27T07:24:46.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:46 smithi012 ceph-mon[31473]: pgmap v74: 129 pgs: 129 active+clean; 20 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-05-27T07:24:46.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:46 smithi162 ceph-mon[35676]: pgmap v74: 129 pgs: 129 active+clean; 20 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-05-27T07:24:46.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:46 smithi062 ceph-mon[35304]: pgmap v74: 129 pgs: 129 active+clean; 20 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-05-27T07:24:47.411 INFO:tasks.cephfs.filesystem:scrub status for tag:402863a5-f9d9-44db-b0c5-dba37415ff16 - {'path': '/', 'tag': '402863a5-f9d9-44db-b0c5-dba37415ff16', 'options': 'recursive,force'} 2024-05-27T07:24:48.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:48 smithi012 ceph-mon[31473]: pgmap v75: 129 pgs: 129 active+clean; 20 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 539 KiB/s wr, 84 op/s 2024-05-27T07:24:48.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:48 smithi162 ceph-mon[35676]: pgmap v75: 129 pgs: 129 active+clean; 20 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 539 KiB/s wr, 84 op/s 2024-05-27T07:24:48.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:48 smithi062 ceph-mon[35304]: pgmap v75: 129 pgs: 129 active+clean; 20 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 539 KiB/s wr, 84 op/s 2024-05-27T07:24:50.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:50 smithi162 ceph-mon[35676]: pgmap v76: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 820 KiB/s wr, 131 op/s 2024-05-27T07:24:50.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:50 smithi062 ceph-mon[35304]: pgmap v76: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 820 KiB/s wr, 131 op/s 2024-05-27T07:24:50.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:50 smithi012 ceph-mon[31473]: pgmap v76: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 820 KiB/s wr, 131 op/s 2024-05-27T07:24:52.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:52 smithi162 ceph-mon[35676]: pgmap v77: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 477 KiB/s wr, 89 op/s 2024-05-27T07:24:52.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:52 smithi062 ceph-mon[35304]: pgmap v77: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 477 KiB/s wr, 89 op/s 2024-05-27T07:24:52.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:52 smithi012 ceph-mon[31473]: pgmap v77: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 477 KiB/s wr, 89 op/s 2024-05-27T07:24:54.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:54 smithi162 ceph-mon[35676]: pgmap v78: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 477 KiB/s wr, 89 op/s 2024-05-27T07:24:54.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:54 smithi062 ceph-mon[35304]: pgmap v78: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 477 KiB/s wr, 89 op/s 2024-05-27T07:24:54.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:54 smithi012 ceph-mon[31473]: pgmap v78: 129 pgs: 129 active+clean; 23 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 477 KiB/s wr, 89 op/s 2024-05-27T07:24:56.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:56 smithi162 ceph-mon[35676]: pgmap v79: 129 pgs: 129 active+clean; 25 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 672 KiB/s wr, 127 op/s 2024-05-27T07:24:56.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:56 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:24:56.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:56 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:24:56.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:56 smithi062 ceph-mon[35304]: pgmap v79: 129 pgs: 129 active+clean; 25 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 672 KiB/s wr, 127 op/s 2024-05-27T07:24:56.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:56 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:24:56.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:56 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:24:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:56 smithi012 ceph-mon[31473]: pgmap v79: 129 pgs: 129 active+clean; 25 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 672 KiB/s wr, 127 op/s 2024-05-27T07:24:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:56 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:24:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:56 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:24:57.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:57 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:24:57.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:57 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:24:57.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:57 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:24:58.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:58 smithi012 ceph-mon[31473]: pgmap v80: 129 pgs: 129 active+clean; 25 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 476 KiB/s wr, 85 op/s 2024-05-27T07:24:58.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:58 smithi162 ceph-mon[35676]: pgmap v80: 129 pgs: 129 active+clean; 25 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 476 KiB/s wr, 85 op/s 2024-05-27T07:24:58.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:58 smithi062 ceph-mon[35304]: pgmap v80: 129 pgs: 129 active+clean; 25 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 476 KiB/s wr, 85 op/s 2024-05-27T07:24:59.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:24:59 smithi162 ceph-mon[35676]: pgmap v81: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 751 KiB/s wr, 124 op/s 2024-05-27T07:24:59.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:24:59 smithi062 ceph-mon[35304]: pgmap v81: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 751 KiB/s wr, 124 op/s 2024-05-27T07:24:59.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:24:59 smithi012 ceph-mon[31473]: pgmap v81: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 751 KiB/s wr, 124 op/s 2024-05-27T07:25:01.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:01 smithi162 ceph-mon[35676]: pgmap v82: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 470 KiB/s wr, 76 op/s 2024-05-27T07:25:01.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:01 smithi062 ceph-mon[35304]: pgmap v82: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 470 KiB/s wr, 76 op/s 2024-05-27T07:25:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:01 smithi012 ceph-mon[31473]: pgmap v82: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 470 KiB/s wr, 76 op/s 2024-05-27T07:25:03.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:03 smithi162 ceph-mon[35676]: pgmap v83: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 470 KiB/s wr, 76 op/s 2024-05-27T07:25:03.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:03 smithi062 ceph-mon[35304]: pgmap v83: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 470 KiB/s wr, 76 op/s 2024-05-27T07:25:03.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:03 smithi012 ceph-mon[31473]: pgmap v83: 129 pgs: 129 active+clean; 28 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 470 KiB/s wr, 76 op/s 2024-05-27T07:25:05.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:05 smithi062 ceph-mon[35304]: pgmap v84: 129 pgs: 129 active+clean; 31 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 823 KiB/s wr, 131 op/s 2024-05-27T07:25:05.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:05 smithi012 ceph-mon[31473]: pgmap v84: 129 pgs: 129 active+clean; 31 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 823 KiB/s wr, 131 op/s 2024-05-27T07:25:05.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:05 smithi162 ceph-mon[35676]: pgmap v84: 129 pgs: 129 active+clean; 31 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 823 KiB/s wr, 131 op/s 2024-05-27T07:25:07.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:07 smithi062 ceph-mon[35304]: pgmap v85: 129 pgs: 129 active+clean; 31 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 628 KiB/s wr, 92 op/s 2024-05-27T07:25:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:07 smithi162 ceph-mon[35676]: pgmap v85: 129 pgs: 129 active+clean; 31 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 628 KiB/s wr, 92 op/s 2024-05-27T07:25:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:07 smithi012 ceph-mon[31473]: pgmap v85: 129 pgs: 129 active+clean; 31 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 628 KiB/s wr, 92 op/s 2024-05-27T07:25:09.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:09 smithi012 ceph-mon[31473]: pgmap v86: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 788 KiB/s wr, 133 op/s 2024-05-27T07:25:09.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:09 smithi162 ceph-mon[35676]: pgmap v86: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 788 KiB/s wr, 133 op/s 2024-05-27T07:25:09.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:09 smithi062 ceph-mon[35304]: pgmap v86: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 788 KiB/s wr, 133 op/s 2024-05-27T07:25:11.787 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:11 smithi062 ceph-mon[35304]: pgmap v87: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 512 KiB/s wr, 95 op/s 2024-05-27T07:25:11.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:11 smithi012 ceph-mon[31473]: pgmap v87: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 512 KiB/s wr, 95 op/s 2024-05-27T07:25:11.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:11 smithi162 ceph-mon[35676]: pgmap v87: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 512 KiB/s wr, 95 op/s 2024-05-27T07:25:13.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:13 smithi012 ceph-mon[31473]: pgmap v88: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 512 KiB/s wr, 95 op/s 2024-05-27T07:25:13.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:13 smithi162 ceph-mon[35676]: pgmap v88: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 512 KiB/s wr, 95 op/s 2024-05-27T07:25:13.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:13 smithi062 ceph-mon[35304]: pgmap v88: 129 pgs: 129 active+clean; 32 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 512 KiB/s wr, 95 op/s 2024-05-27T07:25:15.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:15 smithi012 ceph-mon[31473]: pgmap v89: 129 pgs: 129 active+clean; 37 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 914 KiB/s wr, 140 op/s 2024-05-27T07:25:15.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:15 smithi162 ceph-mon[35676]: pgmap v89: 129 pgs: 129 active+clean; 37 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 914 KiB/s wr, 140 op/s 2024-05-27T07:25:15.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:15 smithi062 ceph-mon[35304]: pgmap v89: 129 pgs: 129 active+clean; 37 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 914 KiB/s wr, 140 op/s 2024-05-27T07:25:17.412 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:25:17.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:17 smithi012 ceph-mon[31473]: pgmap v90: 129 pgs: 129 active+clean; 37 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 561 KiB/s wr, 85 op/s 2024-05-27T07:25:17.949 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:17.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:17 smithi162 ceph-mon[35676]: pgmap v90: 129 pgs: 129 active+clean; 37 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 561 KiB/s wr, 85 op/s 2024-05-27T07:25:17.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:17 smithi062 ceph-mon[35304]: pgmap v90: 129 pgs: 129 active+clean; 37 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 561 KiB/s wr, 85 op/s 2024-05-27T07:25:19.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:19 smithi012 ceph-mon[31473]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 779 KiB/s wr, 127 op/s 2024-05-27T07:25:19.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:19 smithi162 ceph-mon[35676]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 779 KiB/s wr, 127 op/s 2024-05-27T07:25:19.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:19 smithi062 ceph-mon[35304]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 779 KiB/s wr, 127 op/s 2024-05-27T07:25:20.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9dffe00700 1 -- 172.21.15.12:0/40470005 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df4003b80 con 0x7f9e001034b0 2024-05-27T07:25:20.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 -- 172.21.15.12:0/40470005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e001034b0 msgr2=0x7f9e001038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 --2- 172.21.15.12:0/40470005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e001034b0 0x7f9e001038c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9df40099c0 tx=0x7f9df4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 -- 172.21.15.12:0/40470005 shutdown_connections 2024-05-27T07:25:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 --2- 172.21.15.12:0/40470005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00108730 0x7f9e00110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 --2- 172.21.15.12:0/40470005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e001034b0 0x7f9e001038c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 --2- 172.21.15.12:0/40470005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e00107de0 0x7f9e001081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 -- 172.21.15.12:0/40470005 >> 172.21.15.12:0/40470005 conn(0x7f9e000fedb0 msgr2=0x7f9e001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.571+0000 7f9e08200700 1 -- 172.21.15.12:0/40470005 shutdown_connections 2024-05-27T07:25:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.572+0000 7f9e08200700 1 -- 172.21.15.12:0/40470005 wait complete. 2024-05-27T07:25:20.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.573+0000 7f9e08200700 1 Processor -- start 2024-05-27T07:25:20.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.573+0000 7f9e08200700 1 -- start start 2024-05-27T07:25:20.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.574+0000 7f9e08200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e001034b0 0x7f9e00199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.574+0000 7f9e08200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00107de0 0x7f9e00199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.574+0000 7f9e08200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e00108730 0x7f9e0019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.574+0000 7f9e08200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e00113320 con 0x7f9e001034b0 2024-05-27T07:25:20.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.574+0000 7f9e08200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001131a0 con 0x7f9e00107de0 2024-05-27T07:25:20.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.574+0000 7f9e08200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001134a0 con 0x7f9e00108730 2024-05-27T07:25:20.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.575+0000 7f9e04e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00107de0 0x7f9e00199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.575+0000 7f9e04e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00107de0 0x7f9e00199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:50470/0 (socket says 172.21.15.12:50470) 2024-05-27T07:25:20.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.575+0000 7f9e04e00700 1 -- 172.21.15.12:0/3556338079 learned_addr learned my addr 172.21.15.12:0/3556338079 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:20.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.576+0000 7f9e06200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e00108730 0x7f9e0019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.576+0000 7f9e04e00700 1 -- 172.21.15.12:0/3556338079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e00108730 msgr2=0x7f9e0019e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.576+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e00108730 0x7f9e0019e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.576+0000 7f9e04e00700 1 -- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e001034b0 msgr2=0x7f9e00199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.576+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e001034b0 0x7f9e00199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.576+0000 7f9e04e00700 1 -- 172.21.15.12:0/3556338079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0019a760 con 0x7f9e00107de0 2024-05-27T07:25:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.577+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00107de0 0x7f9e00199eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00b500 tx=0x7f9dec00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.577+0000 7f9dfe000700 1 -- 172.21.15.12:0/3556338079 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec015020 con 0x7f9e00107de0 2024-05-27T07:25:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.577+0000 7f9dfe000700 1 -- 172.21.15.12:0/3556338079 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9dec011cc0 con 0x7f9e00107de0 2024-05-27T07:25:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.577+0000 7f9dfe000700 1 -- 172.21.15.12:0/3556338079 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec010c80 con 0x7f9e00107de0 2024-05-27T07:25:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.578+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0019f020 con 0x7f9e00107de0 2024-05-27T07:25:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.578+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0019f510 con 0x7f9e00107de0 2024-05-27T07:25:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.584+0000 7f9e16e00b80 1 Processor -- start 2024-05-27T07:25:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.585+0000 7f9e16e00b80 1 -- start start 2024-05-27T07:25:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.585+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.585+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.585+0000 7f9dfe000700 1 -- 172.21.15.12:0/3556338079 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9dec018020 con 0x7f9e00107de0 2024-05-27T07:25:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.585+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cb9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.586+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555a16bdae40 con 0x555a16b2de40 2024-05-27T07:25:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.586+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555a16bca9e0 con 0x555a16cace70 2024-05-27T07:25:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.586+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555a16bcea20 con 0x555a16c199f0 2024-05-27T07:25:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.586+0000 7f9e06200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.586+0000 7f9e05800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.587+0000 7f9e05800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60188/0 (socket says 172.21.15.12:60188) 2024-05-27T07:25:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.587+0000 7f9e05800700 1 -- 172.21.15.12:0/3487897704 learned_addr learned my addr 172.21.15.12:0/3487897704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.587+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cb9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.587+0000 7f9e05800700 1 -- 172.21.15.12:0/3487897704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 msgr2=0x555a16cac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.587+0000 7f9e05800700 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.588+0000 7f9e05800700 1 -- 172.21.15.12:0/3487897704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 msgr2=0x555a16cb9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.588+0000 7f9e05800700 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cb9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.588+0000 7f9e05800700 1 -- 172.21.15.12:0/3487897704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a16b93420 con 0x555a16b2de40 2024-05-27T07:25:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.588+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cb9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.589+0000 7f9e05800700 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cac870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9df4006010 tx=0x7f9df4003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46c2bf0a1c41efd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:20.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.590+0000 7f9df2a00700 1 -- 172.21.15.12:0/3487897704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df40039c0 con 0x555a16b2de40 2024-05-27T07:25:20.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.590+0000 7f9df2a00700 1 -- 172.21.15.12:0/3487897704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9df4005040 con 0x555a16b2de40 2024-05-27T07:25:20.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.590+0000 7f9df2a00700 1 -- 172.21.15.12:0/3487897704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df400bb10 con 0x555a16b2de40 2024-05-27T07:25:20.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.590+0000 7f9dfe000700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9de80b2bc0 0x7f9de80b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.591+0000 7f9dfe000700 1 -- 172.21.15.12:0/3556338079 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f9dec0e2010 con 0x7f9e00107de0 2024-05-27T07:25:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.591+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3487897704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 msgr2=0x555a16cac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.591+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cac870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9df4006010 tx=0x7f9df4003680 comp rx=0 tx=0).stop 2024-05-27T07:25:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.592+0000 7f9e06200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9de80b2bc0 0x7f9de80b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.592+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3487897704 shutdown_connections 2024-05-27T07:25:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.592+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cb9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.592+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.592+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3487897704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cac870 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.592+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3487897704 >> 172.21.15.12:0/3487897704 conn(0x555a16bc7390 msgr2=0x555a16ca84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.593+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3487897704 shutdown_connections 2024-05-27T07:25:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.594+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3487897704 wait complete. 2024-05-27T07:25:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.594+0000 7f9e06200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9de80b2bc0 0x7f9de80b50d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9df8009fd0 tx=0x7f9df8009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.594+0000 7f9e16e00b80 1 -- start start 2024-05-27T07:25:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.595+0000 7f9e16e00b80 1 Processor -- start 2024-05-27T07:25:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.596+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16b2de40 0x555a16cb66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.596+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16c199f0 0x555a16cb6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.596+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16cace70 0x555a16cb7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.596+0000 7f9e04e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16c199f0 0x555a16cb6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.596+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555a16ad0fa0 con 0x555a16cace70 2024-05-27T07:25:20.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.596+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555a16b6b760 con 0x555a16c199f0 2024-05-27T07:25:20.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.596+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555a16cb9b20 con 0x555a16b2de40 2024-05-27T07:25:20.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.597+0000 7f9e16e00b80 10 client.0 _ll_register_callbacks cb 0x555a16b74c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:20.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.597+0000 7f9e04e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16c199f0 0x555a16cb6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:50488/0 (socket says 172.21.15.12:50488) 2024-05-27T07:25:20.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.597+0000 7f9e04e00700 1 -- 172.21.15.12:0/2825083330 learned_addr learned my addr 172.21.15.12:0/2825083330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:20.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.597+0000 7f9e05800700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16b2de40 0x555a16cb66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.597+0000 7f9e06200700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16cace70 0x555a16cb7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.598+0000 7f9e04e00700 1 -- 172.21.15.12:0/2825083330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16b2de40 msgr2=0x555a16cb66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.598+0000 7f9e04e00700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16b2de40 0x555a16cb66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.598+0000 7f9e04e00700 1 -- 172.21.15.12:0/2825083330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16cace70 msgr2=0x555a16cb7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.598+0000 7f9e04e00700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16cace70 0x555a16cb7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.598+0000 7f9e04e00700 1 -- 172.21.15.12:0/2825083330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a16b1a8d0 con 0x555a16c199f0 2024-05-27T07:25:20.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.598+0000 7f9e06200700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16cace70 0x555a16cb7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:20.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.599+0000 7f9e04e00700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16c199f0 0x555a16cb6c10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0a6260 tx=0x7f9dec0e5e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c54737e9f836427d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:20.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.599+0000 7f9de7400700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:25:20.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.599+0000 7f9de7400700 1 -- 172.21.15.12:0/2825083330 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec0a8940 con 0x555a16c199f0 2024-05-27T07:25:20.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.600+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9df4005040 con 0x555a16c199f0 2024-05-27T07:25:20.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.600+0000 7f9de7400700 1 -- 172.21.15.12:0/2825083330 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f9dec018310 con 0x555a16c199f0 2024-05-27T07:25:20.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.600+0000 7f9de7400700 1 -- 172.21.15.12:0/2825083330 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9dec0a8760 con 0x555a16c199f0 2024-05-27T07:25:20.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.600+0000 7f9e16e00b80 10 client.35342 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:20.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.600+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555a16b9b060 con 0x555a16c199f0 2024-05-27T07:25:20.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9de7400700 1 -- 172.21.15.12:0/2825083330 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9df4005040 con 0x555a16c199f0 2024-05-27T07:25:20.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9de7400700 1 -- 172.21.15.12:0/2825083330 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9dec0a9250 con 0x555a16c199f0 2024-05-27T07:25:20.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9e16e00b80 10 client.35342 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:20.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9e16e00b80 10 client.35342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:20.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9e16e00b80 20 client.35342 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:20.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16cc6b70 0x555a16cc8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9e16e00b80 4 client.35342 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:20.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.601+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a16bdcd90 con 0x555a16cc6b70 2024-05-27T07:25:20.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.602+0000 7f9e05800700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16cc6b70 0x555a16cc8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.603+0000 7f9e05800700 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16cc6b70 0x555a16cc8fc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be23b1f23a6357d3 server_cookie=51a302ed839fbb7d in_seq=0 out_seq=0 2024-05-27T07:25:20.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.603+0000 7f9de7400700 10 client.35342 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:20.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9de7400700 1 -- 172.21.15.12:0/2825083330 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555a16bdcd90 con 0x555a16cc6b70 2024-05-27T07:25:20.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9de7400700 10 client.35342 handle_command_reply: tid=0 2024-05-27T07:25:20.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 client.35342 shutdown 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16c199f0 msgr2=0x555a16cb6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16c199f0 0x555a16cb6c10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0a6260 tx=0x7f9dec0e5e60 comp rx=0 tx=0).stop 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 shutdown_connections 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16cace70 0x555a16cb7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16b2de40 0x555a16cb66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16cc6b70 0x555a16cc8fc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/2825083330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16c199f0 0x555a16cb6c10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 >> 172.21.15.12:0/2825083330 conn(0x555a16bc7390 msgr2=0x555a16cac310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:20.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.606+0000 7f9de7400700 0 client.35342 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:20.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.607+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 shutdown_connections 2024-05-27T07:25:20.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.607+0000 7f9e16e00b80 1 -- 172.21.15.12:0/2825083330 wait complete. 2024-05-27T07:25:20.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.607+0000 7f9e16e00b80 20 client.35342 trim_cache size 0 max 16384 2024-05-27T07:25:20.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 Processor -- start 2024-05-27T07:25:20.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 -- start start 2024-05-27T07:25:20.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cb1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cb8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001b93e0 con 0x555a16b2de40 2024-05-27T07:25:20.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001b6c70 con 0x555a16cace70 2024-05-27T07:25:20.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.637+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001b4ac0 con 0x555a16c199f0 2024-05-27T07:25:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cb1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cb1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60212/0 (socket says 172.21.15.12:60212) 2024-05-27T07:25:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 -- 172.21.15.12:0/1032529112 learned_addr learned my addr 172.21.15.12:0/1032529112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e06200700 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e04e00700 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cb8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 -- 172.21.15.12:0/1032529112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 msgr2=0x555a16cb8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cb8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 -- 172.21.15.12:0/1032529112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 msgr2=0x555a16cce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e05800700 1 -- 172.21.15.12:0/1032529112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a16ca4240 con 0x555a16b2de40 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.638+0000 7f9e06200700 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.639+0000 7f9e05800700 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cb1f30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x555a16ca9a20 tx=0x7f9df40092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd7ef2a95cee524 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.639+0000 7f9df2000700 1 -- 172.21.15.12:0/1032529112 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df40053b0 con 0x555a16b2de40 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.639+0000 7f9df2000700 1 -- 172.21.15.12:0/1032529112 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9df4005670 con 0x555a16b2de40 2024-05-27T07:25:20.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.639+0000 7f9df2000700 1 -- 172.21.15.12:0/1032529112 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df400bb80 con 0x555a16b2de40 2024-05-27T07:25:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.640+0000 7f9e16e00b80 1 -- 172.21.15.12:0/1032529112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 msgr2=0x555a16cb1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.640+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cb1f30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x555a16ca9a20 tx=0x7f9df40092a0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.641+0000 7f9e16e00b80 1 -- 172.21.15.12:0/1032529112 shutdown_connections 2024-05-27T07:25:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.641+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.641+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16cb1f30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.641+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/1032529112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16cb8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.641+0000 7f9e16e00b80 1 -- 172.21.15.12:0/1032529112 >> 172.21.15.12:0/1032529112 conn(0x555a16bc7390 msgr2=0x555a16cb7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:20.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.641+0000 7f9e16e00b80 1 -- 172.21.15.12:0/1032529112 shutdown_connections 2024-05-27T07:25:20.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.641+0000 7f9e16e00b80 1 -- 172.21.15.12:0/1032529112 wait complete. 2024-05-27T07:25:20.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.642+0000 7f9e16e00b80 1 -- start start 2024-05-27T07:25:20.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.642+0000 7f9e16e00b80 1 Processor -- start 2024-05-27T07:25:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16d2e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16d2eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e16e00b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cbedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001b5130 con 0x555a16b2de40 2024-05-27T07:25:20.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001b65a0 con 0x555a16cace70 2024-05-27T07:25:20.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e16e00b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001b5800 con 0x555a16c199f0 2024-05-27T07:25:20.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e16e00b80 10 client.0 _ll_register_callbacks cb 0x555a16b74c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:20.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.643+0000 7f9e06200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cbedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e06200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cbedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:50506/0 (socket says 172.21.15.12:50506) 2024-05-27T07:25:20.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e06200700 1 -- 172.21.15.12:0/3945573663 learned_addr learned my addr 172.21.15.12:0/3945573663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e05800700 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16d2e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16d2eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e05800700 1 -- 172.21.15.12:0/3945573663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 msgr2=0x555a16d2eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e05800700 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16d2eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e05800700 1 -- 172.21.15.12:0/3945573663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 msgr2=0x555a16cbedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e05800700 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cbedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.644+0000 7f9e05800700 1 -- 172.21.15.12:0/3945573663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a16cbf480 con 0x555a16b2de40 2024-05-27T07:25:20.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.645+0000 7f9e05800700 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16d2e990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x555a16ca9a20 tx=0x7f9df40050c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b8d0cc9c46ae1f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:20.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.645+0000 7f9de7400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:20.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.646+0000 7f9de7400700 1 -- 172.21.15.12:0/3945573663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df4026040 con 0x555a16b2de40 2024-05-27T07:25:20.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.646+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9df4005670 con 0x555a16b2de40 2024-05-27T07:25:20.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.646+0000 7f9de7400700 1 -- 172.21.15.12:0/3945573663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f9df4003b00 con 0x555a16b2de40 2024-05-27T07:25:20.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.646+0000 7f9de7400700 1 -- 172.21.15.12:0/3945573663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9df4003d20 con 0x555a16b2de40 2024-05-27T07:25:20.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.646+0000 7f9e16e00b80 10 client.15699 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:20.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.646+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555a16cb1690 con 0x555a16b2de40 2024-05-27T07:25:20.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9de7400700 1 -- 172.21.15.12:0/3945573663 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9df4005670 con 0x555a16b2de40 2024-05-27T07:25:20.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9de7400700 1 -- 172.21.15.12:0/3945573663 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9df401f070 con 0x555a16b2de40 2024-05-27T07:25:20.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9e16e00b80 10 client.15699 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:20.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9e16e00b80 10 client.15699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:20.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9e16e00b80 20 client.15699 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:20.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16d426a0 0x555a16d44ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:20.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9e16e00b80 4 client.15699 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:25:20.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.647+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555a16bd6570 con 0x555a16d426a0 2024-05-27T07:25:20.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.648+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16d426a0 0x555a16d44ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:20.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.650+0000 7f9e04e00700 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16d426a0 0x555a16d44ac0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3dc7dbcf2176907 server_cookie=200a4e820927dbbe in_seq=0 out_seq=0 2024-05-27T07:25:20.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.650+0000 7f9de7400700 10 client.15699 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:20.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9de7400700 1 -- 172.21.15.12:0/3945573663 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555a16bd6570 con 0x555a16d426a0 2024-05-27T07:25:20.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9de7400700 10 client.15699 handle_command_reply: tid=0 2024-05-27T07:25:20.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 client.15699 shutdown 2024-05-27T07:25:20.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 msgr2=0x555a16d2e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16d2e990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x555a16ca9a20 tx=0x7f9df40050c0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 shutdown_connections 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555a16d426a0 0x555a16d44ac0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555a16cace70 0x555a16cbedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555a16b2de40 0x555a16d2e990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 --2- 172.21.15.12:0/3945573663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a16c199f0 0x555a16d2eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 >> 172.21.15.12:0/3945573663 conn(0x555a16bc7390 msgr2=0x555a16cc1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.656+0000 7f9de7400700 0 client.15699 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:20.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.657+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 shutdown_connections 2024-05-27T07:25:20.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.657+0000 7f9e16e00b80 1 -- 172.21.15.12:0/3945573663 wait complete. 2024-05-27T07:25:20.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.657+0000 7f9e16e00b80 20 client.15699 trim_cache size 0 max 16384 2024-05-27T07:25:20.648 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:25:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:25:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:25:20.648 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9de80b2bc0 msgr2=0x7f9de80b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9de80b2bc0 0x7f9de80b50d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9df8009fd0 tx=0x7f9df8009040 comp rx=0 tx=0).stop 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00107de0 msgr2=0x7f9e00199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00107de0 0x7f9e00199eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00b500 tx=0x7f9dec00b8c0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 shutdown_connections 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9de80b2bc0 0x7f9de80b50d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e00108730 0x7f9e0019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e001034b0 0x7f9e00199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 --2- 172.21.15.12:0/3556338079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e00107de0 0x7f9e00199eb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 >> 172.21.15.12:0/3556338079 conn(0x7f9e000fedb0 msgr2=0x7f9e00101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 shutdown_connections 2024-05-27T07:25:20.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:20.658+0000 7f9e08200700 1 -- 172.21.15.12:0/3556338079 wait complete. 2024-05-27T07:25:21.560 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:25:21.560 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:25:21.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:21 smithi012 ceph-mon[31473]: pgmap v92: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 619 KiB/s wr, 86 op/s 2024-05-27T07:25:21.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:21 smithi162 ceph-mon[35676]: pgmap v92: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 619 KiB/s wr, 86 op/s 2024-05-27T07:25:21.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:21 smithi062 ceph-mon[35304]: pgmap v92: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 619 KiB/s wr, 86 op/s 2024-05-27T07:25:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:24.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:24 smithi012 ceph-mon[31473]: pgmap v93: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 619 KiB/s wr, 86 op/s 2024-05-27T07:25:24.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:24 smithi162 ceph-mon[35676]: pgmap v93: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 619 KiB/s wr, 86 op/s 2024-05-27T07:25:24.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:24 smithi062 ceph-mon[35304]: pgmap v93: 129 pgs: 129 active+clean; 39 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 619 KiB/s wr, 86 op/s 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.551+0000 7f9792400700 1 -- 172.21.15.12:0/1083359729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 msgr2=0x7f978c117820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.551+0000 7f9792400700 1 --2- 172.21.15.12:0/1083359729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c117820 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f978400b0a0 tx=0x7f978400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.552+0000 7f9792400700 1 -- 172.21.15.12:0/1083359729 shutdown_connections 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.552+0000 7f9792400700 1 --2- 172.21.15.12:0/1083359729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c117820 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.552+0000 7f9792400700 1 --2- 172.21.15.12:0/1083359729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f978c110980 0x7f978c110d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.552+0000 7f9792400700 1 --2- 172.21.15.12:0/1083359729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978c10a950 0x7f978c10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.552+0000 7f9792400700 1 -- 172.21.15.12:0/1083359729 >> 172.21.15.12:0/1083359729 conn(0x7f978c076d20 msgr2=0x7f978c077130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.552+0000 7f9792400700 1 -- 172.21.15.12:0/1083359729 shutdown_connections 2024-05-27T07:25:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.552+0000 7f9792400700 1 -- 172.21.15.12:0/1083359729 wait complete. 2024-05-27T07:25:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.553+0000 7f9792400700 1 Processor -- start 2024-05-27T07:25:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.553+0000 7f9792400700 1 -- start start 2024-05-27T07:25:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f9792400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f978c10a950 0x7f978c1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f9792400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c1a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f9792400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978c110980 0x7f978c1a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f9792400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f978c11a970 con 0x7f978c10b2a0 2024-05-27T07:25:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f9792400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f978c11a7f0 con 0x7f978c110980 2024-05-27T07:25:25.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f9792400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f978c11aaf0 con 0x7f978c10a950 2024-05-27T07:25:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f978aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c1a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f978aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c1a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59266/0 (socket says 172.21.15.12:59266) 2024-05-27T07:25:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f978aa00700 1 -- 172.21.15.12:0/1182269100 learned_addr learned my addr 172.21.15.12:0/1182269100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.554+0000 7f978b400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f978c10a950 0x7f978c1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.555+0000 7f978be00700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978c110980 0x7f978c1a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.555+0000 7f978aa00700 1 -- 172.21.15.12:0/1182269100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f978c10a950 msgr2=0x7f978c1a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.555+0000 7f978aa00700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f978c10a950 0x7f978c1a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.555+0000 7f978aa00700 1 -- 172.21.15.12:0/1182269100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978c110980 msgr2=0x7f978c1a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.555+0000 7f978aa00700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978c110980 0x7f978c1a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.555+0000 7f978aa00700 1 -- 172.21.15.12:0/1182269100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c1a61f0 con 0x7f978c10b2a0 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.555+0000 7f978aa00700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c1a18c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f977800d980 tx=0x7f977800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.556+0000 7f9783e00700 1 -- 172.21.15.12:0/1182269100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778017070 con 0x7f978c10b2a0 2024-05-27T07:25:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.556+0000 7f9783e00700 1 -- 172.21.15.12:0/1182269100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f977800fd20 con 0x7f978c10b2a0 2024-05-27T07:25:25.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.556+0000 7f9783e00700 1 -- 172.21.15.12:0/1182269100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977800e3b0 con 0x7f978c10b2a0 2024-05-27T07:25:25.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.556+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978c1a6480 con 0x7f978c10b2a0 2024-05-27T07:25:25.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.556+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978c1a6970 con 0x7f978c10b2a0 2024-05-27T07:25:25.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.558+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f978c0512b0 con 0x7f978c10b2a0 2024-05-27T07:25:25.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.561+0000 7f9783e00700 1 -- 172.21.15.12:0/1182269100 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f977800f890 con 0x7f978c10b2a0 2024-05-27T07:25:25.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.563+0000 7f9783e00700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f97700b2a70 0x7f97700b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.563+0000 7f9783e00700 1 -- 172.21.15.12:0/1182269100 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f97780e2440 con 0x7f978c10b2a0 2024-05-27T07:25:25.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.564+0000 7f978b400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f97700b2a70 0x7f97700b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.565+0000 7f978b400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f97700b2a70 0x7f97700b4f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f978c10ba90 tx=0x7f977c00b3c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:25.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.569+0000 7f9783e00700 1 -- 172.21.15.12:0/1182269100 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f97780a9a10 con 0x7f978c10b2a0 2024-05-27T07:25:25.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.898+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f978c02e5e0 con 0x7f978c10b2a0 2024-05-27T07:25:25.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.900+0000 7f9783e00700 1 -- 172.21.15.12:0/1182269100 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11870 (secure 0 0 0) 0x7f97780a9310 con 0x7f978c10b2a0 2024-05-27T07:25:25.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:25:25.891 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2577053624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2577053624},{"type":"v1","addr":"172.21.15.162:6835","nonce":2577053624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:24:35.565239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1296782695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1296782695},{"type":"v1","addr":"172.21.15.162:6837","nonce":1296782695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/884400330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":884400330},{"type":"v1","addr":"172.21.15.162:6839","nonce":884400330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34391":{"gid":34391,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1529598052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1529598052},{"type":"v1","addr":"172.21.15.62:6833","nonce":1529598052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34475,34415,34439]},"id":1}]} 2024-05-27T07:25:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f97700b2a70 msgr2=0x7f97700b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f97700b2a70 0x7f97700b4f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f978c10ba90 tx=0x7f977c00b3c0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 msgr2=0x7f978c1a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c1a18c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f977800d980 tx=0x7f977800dc90 comp rx=0 tx=0).stop 2024-05-27T07:25:25.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 shutdown_connections 2024-05-27T07:25:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f97700b2a70 0x7f97700b4f80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978c110980 0x7f978c1a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f978c10a950 0x7f978c1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 --2- 172.21.15.12:0/1182269100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f978c10b2a0 0x7f978c1a18c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 >> 172.21.15.12:0/1182269100 conn(0x7f978c076d20 msgr2=0x7f978c1088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 shutdown_connections 2024-05-27T07:25:25.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:25.904+0000 7f9792400700 1 -- 172.21.15.12:0/1182269100 wait complete. 2024-05-27T07:25:25.896 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 28 2024-05-27T07:25:26.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:26 smithi162 ceph-mon[35676]: pgmap v94: 129 pgs: 129 active+clean; 41 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 805 KiB/s wr, 128 op/s 2024-05-27T07:25:26.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:26 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1182269100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:26.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:26 smithi062 ceph-mon[35304]: pgmap v94: 129 pgs: 129 active+clean; 41 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 805 KiB/s wr, 128 op/s 2024-05-27T07:25:26.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:26 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1182269100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:26.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:26 smithi012 ceph-mon[31473]: pgmap v94: 129 pgs: 129 active+clean; 41 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 805 KiB/s wr, 128 op/s 2024-05-27T07:25:26.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:26 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1182269100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:26.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:25:27.354 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:28.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:28 smithi062 ceph-mon[35304]: pgmap v95: 129 pgs: 129 active+clean; 41 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 403 KiB/s wr, 83 op/s 2024-05-27T07:25:28.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:28 smithi012 ceph-mon[31473]: pgmap v95: 129 pgs: 129 active+clean; 41 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 403 KiB/s wr, 83 op/s 2024-05-27T07:25:28.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:28 smithi162 ceph-mon[35676]: pgmap v95: 129 pgs: 129 active+clean; 41 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 403 KiB/s wr, 83 op/s 2024-05-27T07:25:29.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 -- 172.21.15.12:0/1359937610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e101109a0 msgr2=0x7f5e10112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 --2- 172.21.15.12:0/1359937610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e101109a0 0x7f5e10112e00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009990 tx=0x7f5e08009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 -- 172.21.15.12:0/1359937610 shutdown_connections 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 --2- 172.21.15.12:0/1359937610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e101109a0 0x7f5e10112e00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 --2- 172.21.15.12:0/1359937610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e10075d20 0x7f5e10076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 --2- 172.21.15.12:0/1359937610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e10076670 0x7f5e10110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 -- 172.21.15.12:0/1359937610 >> 172.21.15.12:0/1359937610 conn(0x7f5e10079d00 msgr2=0x7f5e1007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 -- 172.21.15.12:0/1359937610 shutdown_connections 2024-05-27T07:25:29.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.761+0000 7f5e15000700 1 -- 172.21.15.12:0/1359937610 wait complete. 2024-05-27T07:25:29.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.763+0000 7f5e15000700 1 Processor -- start 2024-05-27T07:25:29.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.763+0000 7f5e15000700 1 -- start start 2024-05-27T07:25:29.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e15000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e10075d20 0x7f5e1019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e15000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e10076670 0x7f5e1019d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e15000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e101109a0 0x7f5e101a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e15000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5e10116330 con 0x7f5e10076670 2024-05-27T07:25:29.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e15000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5e101161b0 con 0x7f5e10075d20 2024-05-27T07:25:29.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e15000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e101164b0 con 0x7f5e101109a0 2024-05-27T07:25:29.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e10076670 0x7f5e1019d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e10076670 0x7f5e1019d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59278/0 (socket says 172.21.15.12:59278) 2024-05-27T07:25:29.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e06a00700 1 -- 172.21.15.12:0/887211873 learned_addr learned my addr 172.21.15.12:0/887211873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:29.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e0f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e10075d20 0x7f5e1019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.764+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e101109a0 0x7f5e101a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e06a00700 1 -- 172.21.15.12:0/887211873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e101109a0 msgr2=0x7f5e101a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e06a00700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e101109a0 0x7f5e101a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e06a00700 1 -- 172.21.15.12:0/887211873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e10075d20 msgr2=0x7f5e1019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e06a00700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e10075d20 0x7f5e1019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e06a00700 1 -- 172.21.15.12:0/887211873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e101a1bf0 con 0x7f5e10076670 2024-05-27T07:25:29.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e06a00700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e10076670 0x7f5e1019d2c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5df800d950 tx=0x7f5df800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:29.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e0cc00700 1 -- 172.21.15.12:0/887211873 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df8016070 con 0x7f5e10076670 2024-05-27T07:25:29.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e0cc00700 1 -- 172.21.15.12:0/887211873 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5df800fd20 con 0x7f5e10076670 2024-05-27T07:25:29.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.765+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e101a1ee0 con 0x7f5e10076670 2024-05-27T07:25:29.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.766+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e101a9800 con 0x7f5e10076670 2024-05-27T07:25:29.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.766+0000 7f5e0cc00700 1 -- 172.21.15.12:0/887211873 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df800e3b0 con 0x7f5e10076670 2024-05-27T07:25:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.771+0000 7f5e0cc00700 1 -- 172.21.15.12:0/887211873 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5df8019070 con 0x7f5e10076670 2024-05-27T07:25:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.772+0000 7f5e25068b80 1 Processor -- start 2024-05-27T07:25:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.772+0000 7f5e25068b80 1 -- start start 2024-05-27T07:25:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be24ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be25a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e25068b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d8be17be40 con 0x55d8be0cee40 2024-05-27T07:25:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e25068b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d8be16b9e0 con 0x55d8be24d820 2024-05-27T07:25:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e25068b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8be16fa20 con 0x55d8be24d410 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e0fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be24ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e0fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be24ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59286/0 (socket says 172.21.15.12:59286) 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e0fe00700 1 -- 172.21.15.12:0/2678903613 learned_addr learned my addr 172.21.15.12:0/2678903613 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e0cc00700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5df40b29f0 0x7f5df40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.773+0000 7f5e0f400700 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e0cc00700 1 -- 172.21.15.12:0/887211873 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f5df80e26d0 con 0x7f5e10076670 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e06a00700 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be25a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5df40b29f0 0x7f5df40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e0fe00700 1 -- 172.21.15.12:0/2678903613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 msgr2=0x55d8be257ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e0fe00700 1 -- 172.21.15.12:0/2678903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 msgr2=0x55d8be25a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be25a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.774+0000 7f5e0fe00700 1 -- 172.21.15.12:0/2678903613 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8be134420 con 0x55d8be0cee40 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.775+0000 7f5e06a00700 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be25a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.775+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be24ced0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e10077bc0 tx=0x7f5e08005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2981c3f866cb7d35 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.775+0000 7f5e04c00700 1 -- 172.21.15.12:0/2678903613 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800b9d0 con 0x55d8be0cee40 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.775+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5df40b29f0 0x7f5df40b4f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e08026000 tx=0x7f5e08028000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.775+0000 7f5e04c00700 1 -- 172.21.15.12:0/2678903613 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e0802b070 con 0x55d8be0cee40 2024-05-27T07:25:29.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.775+0000 7f5e04c00700 1 -- 172.21.15.12:0/2678903613 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08027900 con 0x55d8be0cee40 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.776+0000 7f5e25068b80 1 -- 172.21.15.12:0/2678903613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 msgr2=0x55d8be24ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.776+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be24ced0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e10077bc0 tx=0x7f5e08005250 comp rx=0 tx=0).stop 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 -- 172.21.15.12:0/2678903613 shutdown_connections 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be25a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be24ced0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2678903613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 -- 172.21.15.12:0/2678903613 >> 172.21.15.12:0/2678903613 conn(0x55d8be12efc0 msgr2=0x55d8be248d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 -- 172.21.15.12:0/2678903613 shutdown_connections 2024-05-27T07:25:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 -- 172.21.15.12:0/2678903613 wait complete. 2024-05-27T07:25:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.777+0000 7f5e25068b80 1 -- start start 2024-05-27T07:25:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 1 Processor -- start 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be257620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be2580a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d8be071fa0 con 0x55d8be0cee40 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d8be10c760 con 0x55d8be24d820 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8be25a930 con 0x55d8be24d410 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.778+0000 7f5e25068b80 10 client.0 _ll_register_callbacks cb 0x55d8be115c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be257620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be257620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59298/0 (socket says 172.21.15.12:59298) 2024-05-27T07:25:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 -- 172.21.15.12:0/2403233842 learned_addr learned my addr 172.21.15.12:0/2403233842 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e06a00700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be2580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 -- 172.21.15.12:0/2403233842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 msgr2=0x55d8be257b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 -- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 msgr2=0x55d8be2580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be2580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0f400700 1 -- 172.21.15.12:0/2403233842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8be0bb8d0 con 0x55d8be0cee40 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e06a00700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.779+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be2580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5e0f400700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be257620 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5e000040e0 tx=0x7f5e000041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad79924d5b40722 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5dfd600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5dfd600700 1 -- 172.21.15.12:0/2403233842 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0001a070 con 0x55d8be0cee40 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5dfd600700 1 -- 172.21.15.12:0/2403233842 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f5e00004db0 con 0x55d8be0cee40 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e0802b070 con 0x55d8be0cee40 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5dfd600700 1 -- 172.21.15.12:0/2403233842 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5e00025050 con 0x55d8be0cee40 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5e25068b80 10 client.15717 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.780+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d8be13c060 con 0x55d8be0cee40 2024-05-27T07:25:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5dfd600700 1 -- 172.21.15.12:0/2403233842 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e0802b070 con 0x55d8be0cee40 2024-05-27T07:25:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5dfd600700 1 -- 172.21.15.12:0/2403233842 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5e00016710 con 0x55d8be0cee40 2024-05-27T07:25:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5e25068b80 10 client.15717 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5e25068b80 10 client.15717 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:25:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5e25068b80 20 client.15717 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be267960 0x55d8be269e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5e25068b80 4 client.15717 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.781+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8be17dd90 con 0x55d8be267960 2024-05-27T07:25:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.782+0000 7f5e06a00700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be267960 0x55d8be269e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.783+0000 7f5e06a00700 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be267960 0x55d8be269e30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=278e4cdb4076f42d server_cookie=2042f70e17c83870 in_seq=0 out_seq=0 2024-05-27T07:25:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.783+0000 7f5dfd600700 10 client.15717 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:25:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5dfd600700 1 -- 172.21.15.12:0/2403233842 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d8be17dd90 con 0x55d8be267960 2024-05-27T07:25:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5dfd600700 10 client.15717 handle_command_reply: tid=0 2024-05-27T07:25:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 client.15717 shutdown 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 msgr2=0x55d8be257620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be257620 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5e000040e0 tx=0x7f5e000041c0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 shutdown_connections 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be267960 0x55d8be269e30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5dfd600700 0 client.15717 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be2580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be0cee40 0x55d8be257620 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.785+0000 7f5e25068b80 1 --2- 172.21.15.12:0/2403233842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be257b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.786+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 >> 172.21.15.12:0/2403233842 conn(0x55d8be12efc0 msgr2=0x55d8be247ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.786+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 shutdown_connections 2024-05-27T07:25:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.786+0000 7f5e25068b80 1 -- 172.21.15.12:0/2403233842 wait complete. 2024-05-27T07:25:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.786+0000 7f5e25068b80 20 client.15717 trim_cache size 0 max 16384 2024-05-27T07:25:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.804+0000 7f5e25068b80 1 Processor -- start 2024-05-27T07:25:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e25068b80 1 -- start start 2024-05-27T07:25:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be0cee40 0x55d8be252dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d410 0x55d8be26f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be26fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e25068b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec011660 con 0x55d8be24d410 2024-05-27T07:25:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e25068b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec00eef0 con 0x55d8be24d820 2024-05-27T07:25:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e25068b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec00cd40 con 0x55d8be0cee40 2024-05-27T07:25:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d410 0x55d8be26f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.805+0000 7f5e06a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d410 0x55d8be26f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59312/0 (socket says 172.21.15.12:59312) 2024-05-27T07:25:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e06a00700 1 -- 172.21.15.12:0/1966972769 learned_addr learned my addr 172.21.15.12:0/1966972769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e0f400700 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be0cee40 0x55d8be252dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be26fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e06a00700 1 -- 172.21.15.12:0/1966972769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be0cee40 msgr2=0x55d8be252dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e06a00700 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be0cee40 0x55d8be252dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e06a00700 1 -- 172.21.15.12:0/1966972769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 msgr2=0x55d8be26fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e06a00700 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be26fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e06a00700 1 -- 172.21.15.12:0/1966972769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8be255b90 con 0x55d8be24d410 2024-05-27T07:25:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.806+0000 7f5e06a00700 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d410 0x55d8be26f5f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5df80a5620 tx=0x7f5df8091040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e98940ecc14a43 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.807+0000 7f5dffe00700 1 -- 172.21.15.12:0/1966972769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df8020040 con 0x55d8be24d410 2024-05-27T07:25:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.807+0000 7f5dffe00700 1 -- 172.21.15.12:0/1966972769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5df80a95a0 con 0x55d8be24d410 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.807+0000 7f5dffe00700 1 -- 172.21.15.12:0/1966972769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df80a9800 con 0x55d8be24d410 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 -- 172.21.15.12:0/1966972769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d410 msgr2=0x55d8be26f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d410 0x55d8be26f5f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5df80a5620 tx=0x7f5df8091040 comp rx=0 tx=0).stop 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 -- 172.21.15.12:0/1966972769 shutdown_connections 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be24d820 0x55d8be26fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be0cee40 0x55d8be252dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1966972769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d410 0x55d8be26f5f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 -- 172.21.15.12:0/1966972769 >> 172.21.15.12:0/1966972769 conn(0x55d8be12efc0 msgr2=0x55d8be257e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 -- 172.21.15.12:0/1966972769 shutdown_connections 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.808+0000 7f5e25068b80 1 -- 172.21.15.12:0/1966972769 wait complete. 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.809+0000 7f5e25068b80 1 -- start start 2024-05-27T07:25:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.809+0000 7f5e25068b80 1 Processor -- start 2024-05-27T07:25:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be0cee40 0x55d8be25ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be25b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e25068b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d820 0x55d8be2d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e25068b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec00d3b0 con 0x55d8be24d820 2024-05-27T07:25:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e25068b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec00e820 con 0x55d8be0cee40 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e25068b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec00da80 con 0x55d8be24d410 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e25068b80 10 client.0 _ll_register_callbacks cb 0x55d8be115c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e0fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d820 0x55d8be2d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e0fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d820 0x55d8be2d04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59316/0 (socket says 172.21.15.12:59316) 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e0fe00700 1 -- 172.21.15.12:0/1148950649 learned_addr learned my addr 172.21.15.12:0/1148950649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e06a00700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be25b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.810+0000 7f5e0f400700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be0cee40 0x55d8be25ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e0fe00700 1 -- 172.21.15.12:0/1148950649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 msgr2=0x55d8be25b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be25b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e0fe00700 1 -- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be0cee40 msgr2=0x55d8be25ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be0cee40 0x55d8be25ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e0fe00700 1 -- 172.21.15.12:0/1148950649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8be25bb20 con 0x55d8be24d820 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e06a00700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be25b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e0f400700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be0cee40 0x55d8be25ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5e0fe00700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d820 0x55d8be2d04b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5e10077bc0 tx=0x7f5e0800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa012c175dea97df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.811+0000 7f5dfd600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.812+0000 7f5dfd600700 1 -- 172.21.15.12:0/1148950649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0803f060 con 0x55d8be24d820 2024-05-27T07:25:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.812+0000 7f5dfd600700 1 -- 172.21.15.12:0/1148950649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f5e08027880 con 0x55d8be24d820 2024-05-27T07:25:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.812+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5df80a95a0 con 0x55d8be24d820 2024-05-27T07:25:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.812+0000 7f5dfd600700 1 -- 172.21.15.12:0/1148950649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5e08046050 con 0x55d8be24d820 2024-05-27T07:25:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.812+0000 7f5e25068b80 10 client.15723 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.812+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d8be2d1a80 con 0x55d8be24d820 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.812+0000 7f5dfd600700 1 -- 172.21.15.12:0/1148950649 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5df80a95a0 con 0x55d8be24d820 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.813+0000 7f5dfd600700 1 -- 172.21.15.12:0/1148950649 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5e0800f660 con 0x55d8be24d820 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.813+0000 7f5e25068b80 10 client.15723 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.813+0000 7f5e25068b80 10 client.15723 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.813+0000 7f5e25068b80 20 client.15723 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.813+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be2d1dc0 0x55d8be2d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.813+0000 7f5e25068b80 4 client.15723 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.813+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8be177570 con 0x55d8be2d1dc0 2024-05-27T07:25:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.814+0000 7f5e0f400700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be2d1dc0 0x55d8be2d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:29.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.816+0000 7f5e0f400700 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be2d1dc0 0x55d8be2d41e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=669488dca8fa16b4 server_cookie=2adb044ef6845b18 in_seq=0 out_seq=0 2024-05-27T07:25:29.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.816+0000 7f5dfd600700 10 client.15723 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:25:29.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.816+0000 7f5dfd600700 1 -- 172.21.15.12:0/1148950649 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8be177570 con 0x55d8be2d1dc0 2024-05-27T07:25:29.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.816+0000 7f5dfd600700 10 client.15723 handle_command_reply: tid=0 2024-05-27T07:25:29.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.816+0000 7f5e25068b80 1 client.15723 shutdown 2024-05-27T07:25:29.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d820 msgr2=0x55d8be2d04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d820 0x55d8be2d04b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5e10077bc0 tx=0x7f5e0800be60 comp rx=0 tx=0).stop 2024-05-27T07:25:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 shutdown_connections 2024-05-27T07:25:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55d8be2d1dc0 0x55d8be2d41e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d8be0cee40 0x55d8be25ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8be24d820 0x55d8be2d04b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 --2- 172.21.15.12:0/1148950649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8be24d410 0x55d8be25b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5dfd600700 0 client.15723 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:25:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 >> 172.21.15.12:0/1148950649 conn(0x55d8be12efc0 msgr2=0x55d8be263450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 shutdown_connections 2024-05-27T07:25:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 1 -- 172.21.15.12:0/1148950649 wait complete. 2024-05-27T07:25:29.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.817+0000 7f5e25068b80 20 client.15723 trim_cache size 0 max 16384 2024-05-27T07:25:29.811 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:25:29.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5df40b29f0 msgr2=0x7f5df40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5df40b29f0 0x7f5df40b4f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e08026000 tx=0x7f5e08028000 comp rx=0 tx=0).stop 2024-05-27T07:25:29.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e10076670 msgr2=0x7f5e1019d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:29.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e10076670 0x7f5e1019d2c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5df800d950 tx=0x7f5df800dc60 comp rx=0 tx=0).stop 2024-05-27T07:25:29.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 shutdown_connections 2024-05-27T07:25:29.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5df40b29f0 0x7f5df40b4f00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e10075d20 0x7f5e1019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e101109a0 0x7f5e101a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 --2- 172.21.15.12:0/887211873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e10076670 0x7f5e1019d2c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:29.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 >> 172.21.15.12:0/887211873 conn(0x7f5e10079d00 msgr2=0x7f5e10100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:29.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 shutdown_connections 2024-05-27T07:25:29.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:29.818+0000 7f5e15000700 1 -- 172.21.15.12:0/887211873 wait complete. 2024-05-27T07:25:30.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:30 smithi062 ceph-mon[35304]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 707 KiB/s wr, 128 op/s 2024-05-27T07:25:30.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:30 smithi012 ceph-mon[31473]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 707 KiB/s wr, 128 op/s 2024-05-27T07:25:30.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:30 smithi162 ceph-mon[35676]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 707 KiB/s wr, 128 op/s 2024-05-27T07:25:30.810 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:25:31.298 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:32.497 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:32 smithi012 ceph-mon[31473]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 489 KiB/s wr, 86 op/s 2024-05-27T07:25:32.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:32 smithi162 ceph-mon[35676]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 489 KiB/s wr, 86 op/s 2024-05-27T07:25:32.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:32 smithi062 ceph-mon[35304]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 489 KiB/s wr, 86 op/s 2024-05-27T07:25:33.336 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 2% (97/4296) Updating files: 3% (129/4296) Updating files: 4% (172/4296) Updating files: 4% (202/4296) Updating files: 5% (215/4296) Updating files: 6% (258/4296) Updating files: 7% (301/4296) Updating files: 7% (314/4296) Updating files: 8% (344/4296) Updating files: 8% (385/4296) Updating files: 9% (387/4296) Updating files: 10% (430/4296) Updating files: 11% (473/4296) Updating files: 11% (490/4296) Updating files: 12% (516/4296) Updating files: 12% (525/4296) Updating files: 13% (559/4296) Updating files: 14% (602/4296) Updating files: 14% (634/4296) Updating files: 15% (645/4296) Updating files: 16% (688/4296) Updating files: 17% (731/4296) Updating files: 17% (755/4296) Updating files: 18% (774/4296) Updating files: 19% (817/4296) Updating files: 20% (860/4296) Updating files: 20% (884/4296) Updating files: 21% (903/4296) Updating files: 22% (946/4296) Updating files: 22% (975/4296) Updating files: 23% (989/4296) Updating files: 24% (1032/4296) Updating files: 25% (1074/4296) Updating files: 25% (1084/4296) Updating files: 26% (1117/4296) Updating files: 27% (1160/4296) Updating files: 27% (1198/4296) Updating files: 28% (1203/4296) Updating files: 29% (1246/4296) Updating files: 29% (1274/4296) Updating files: 30% (1289/4296) Updating files: 31% (1332/4296) Updating files: 31% (1336/4296) Updating files: 32% (1375/4296) Updating files: 33% (1418/4296) Updating files: 34% (1461/4296) Updating files: 34% (1483/4296) Updating files: 35% (1504/4296) Updating files: 36% (1547/4296) Updating files: 36% (1557/4296) Updating files: 37% (1590/4296) Updating files: 37% (1629/4296) Updating files: 38% (1633/4296) Updating files: 39% (1676/4296) Updating files: 39% (1704/4296) Updating files: 40% (1719/4296) Updating files: 41% (1762/4296) Updating files: 41% (1781/4296) Updating files: 42% (1805/4296) Updating files: 43% (1848/4296) Updating files: 43% (1851/4296) Updating files: 44% (1891/4296) Updating files: 44% (1925/4296) Updating files: 45% (1934/4296) Updating files: 46% (1977/4296) Updating files: 46% (2002/4296) Updating files: 47% (2020/4296) Updating files: 48% (2063/4296) Updating files: 48% (2082/4296) Updating files: 49% (2106/4296) Updating files: 50% (2148/4296) Updating files: 50% (2158/4296) Updating files: 50% (2186/4296) Updating files: 51% (2191/4296) Updating files: 52% (2234/4296) Updating files: 52% (2245/4296) Updating files: 53% (2277/4296) Updating files: 54% (2320/4296) Updating files: 55% (2363/4296) Updating files: 55% (2388/4296) Updating files: 56% (2406/4296) Updating files: 57% (2449/4296) Updating files: 57% (2465/4296) Updating files: 58% (2492/4296) Updating files: 59% (2535/4296) Updating files: 59% (2546/4296) Updating files: 60% (2578/4296) Updating files: 61% (2621/4296) Updating files: 61% (2624/4296) Updating files: 62% (2664/4296) Updating files: 62% (2701/4296) Updating files: 63% (2707/4296) Updating files: 64% (2750/4296) Updating files: 64% (2775/4296) Updating files: 65% (2793/4296) Updating files: 66% (2836/4296) Updating files: 66% (2838/4296) Updating files: 67% (2879/4296) Updating files: 67% (2909/4296) Updating files: 68% (2922/4296) Updating files: 69% (2965/4296) Updating files: 69% (2989/4296) Updating files: 70% (3008/4296) Updating files: 71% (3051/4296) Updating files: 71% (3052/4296) Updating files: 72% (3094/4296) Updating files: 72% (3136/4296) Updating files: 73% (3137/4296) Updating files: 74% (3180/4296) Updating files: 74% (3217/4296) Updating files: 75% (3222/4296) Updating files: 76% (3265/4296) Updating files: 76% (3295/4296) Updating files: 77% (3308/4296) Updating files: 78% (3351/4296) Updating files: 78% (3373/4296) Updating files: 79% (3394/4296) Updating files: 80% (3437/4296) Updating files: 81% (3480/4296) Updating files: 81% (3483/4296) Updating files: 82% (3523/4296) Updating files: 83% (3566/4296) Updating files: 83% (3568/4296) Updating files: 84% (3609/4296) Updating files: 85% (3652/4296) Updating files: 85% (3657/4296) Updating files: 86% (3695/4296) Updating files: 87% (3738/4296) Updating files: 87% (3772/4296) Updating files: 88% (3781/4296) Updating files: 89% (3824/4296) Updating files: 89% (3866/4296) Updating files: 90% (3867/4296) Updating files: 91% (3910/4296) Updating files: 91% (3934/4296) Updating files: 92% (3953/4296) Updating files: 93% (3996/4296) Updating files: 93% (4000/4296) Updating files: 94% (4039/4296) Updating files: 94% (4075/4296) Updating files: 95% (4082/4296) Updating files: 95% (4101/4296) Updating files: 96% (4125/4296) Updating files: 96% (4127/4296) Updating files: 96% (4153/4296) Updating files: 97% (4168/4296) Updating files: 97% (4178/4296) Updating files: 97% (4202/4296) Updating files: 98% (4211/4296) Updating files: 98% (4230/4296) Updating files: 99% (4254/4296) Updating files: 99% (4257/4296) Updating files: 99% (4288/4296) Updating files: 100% (4296/4296) Updating files: 100% (4296/4296), done. 2024-05-27T07:25:33.483 INFO:tasks.workunit.client.0.smithi012.stderr:+ cd xfstests-dev 2024-05-27T07:25:33.489 INFO:tasks.workunit.client.0.smithi012.stderr:+ git checkout 0e5c12dfd008efc2848c98108c9237487e91ef35 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 -- 172.21.15.12:0/3620414557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f6810e980 msgr2=0x7f3f68110e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3620414557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f6810e980 0x7f3f68110e00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000b0a0 tx=0x7f3f6000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 -- 172.21.15.12:0/3620414557 shutdown_connections 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3620414557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f681055a0 0x7f3f681059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3620414557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f6810e980 0x7f3f68110e00 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3620414557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f68105ef0 0x7f3f6810e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 -- 172.21.15.12:0/3620414557 >> 172.21.15.12:0/3620414557 conn(0x7f3f68078df0 msgr2=0x7f3f68079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 -- 172.21.15.12:0/3620414557 shutdown_connections 2024-05-27T07:25:33.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.741+0000 7f3f6d000700 1 -- 172.21.15.12:0/3620414557 wait complete. 2024-05-27T07:25:33.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.743+0000 7f3f6d000700 1 Processor -- start 2024-05-27T07:25:33.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.743+0000 7f3f6d000700 1 -- start start 2024-05-27T07:25:33.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.743+0000 7f3f6d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f681055a0 0x7f3f68103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.743+0000 7f3f6d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f68105ef0 0x7f3f681037d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f6d000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f6810e980 0x7f3f68103d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f6d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f68114380 con 0x7f3f681055a0 2024-05-27T07:25:33.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f6d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f68114200 con 0x7f3f68105ef0 2024-05-27T07:25:33.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f6d000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f68114500 con 0x7f3f6810e980 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f65e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f681055a0 0x7f3f68103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f65e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f681055a0 0x7f3f68103290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49496/0 (socket says 172.21.15.12:49496) 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f65e00700 1 -- 172.21.15.12:0/3290667011 learned_addr learned my addr 172.21.15.12:0/3290667011 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f65400700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f68105ef0 0x7f3f681037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.744+0000 7f3f66800700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f6810e980 0x7f3f68103d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f65e00700 1 -- 172.21.15.12:0/3290667011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f6810e980 msgr2=0x7f3f68103d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f65e00700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f6810e980 0x7f3f68103d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f65e00700 1 -- 172.21.15.12:0/3290667011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f68105ef0 msgr2=0x7f3f681037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f65e00700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f68105ef0 0x7f3f681037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f65e00700 1 -- 172.21.15.12:0/3290667011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f681a91d0 con 0x7f3f681055a0 2024-05-27T07:25:33.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f66800700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f6810e980 0x7f3f68103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:33.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f65e00700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f681055a0 0x7f3f68103290 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800d950 tx=0x7f3f5800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f5ea00700 1 -- 172.21.15.12:0/3290667011 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58016070 con 0x7f3f681055a0 2024-05-27T07:25:33.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f5ea00700 1 -- 172.21.15.12:0/3290667011 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f58010460 con 0x7f3f681055a0 2024-05-27T07:25:33.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f681a9460 con 0x7f3f681055a0 2024-05-27T07:25:33.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.745+0000 7f3f5ea00700 1 -- 172.21.15.12:0/3290667011 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5800f620 con 0x7f3f681055a0 2024-05-27T07:25:33.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.746+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f681a99a0 con 0x7f3f681055a0 2024-05-27T07:25:33.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.750+0000 7f3f5ea00700 1 -- 172.21.15.12:0/3290667011 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3f58019070 con 0x7f3f681055a0 2024-05-27T07:25:33.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.750+0000 7f3f7bc01b80 1 Processor -- start 2024-05-27T07:25:33.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.751+0000 7f3f7bc01b80 1 -- start start 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.751+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.751+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.751+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1d41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.751+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55675e101e40 con 0x55675e1d9d30 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.751+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55675e0f19e0 con 0x55675e1d3db0 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.751+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55675e0f5a20 con 0x55675e054e40 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1d3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49500/0 (socket says 172.21.15.12:49500) 2024-05-27T07:25:33.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 -- 172.21.15.12:0/845514199 learned_addr learned my addr 172.21.15.12:0/845514199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f66800700 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65400700 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1d41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f5ea00700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3f4c0b29f0 0x7f3f4c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 -- 172.21.15.12:0/845514199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 msgr2=0x55675e1d97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 -- 172.21.15.12:0/845514199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 msgr2=0x55675e1d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1d41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.752+0000 7f3f65e00700 1 -- 172.21.15.12:0/845514199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55675e0ba420 con 0x55675e1d9d30 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.753+0000 7f3f66800700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3f4c0b29f0 0x7f3f4c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.753+0000 7f3f5ea00700 1 -- 172.21.15.12:0/3290667011 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f3f580e2670 con 0x7f3f681055a0 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.753+0000 7f3f65e00700 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1d3870 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3f580df8f0 tx=0x7f3f58022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cafa3cb2b96da67 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.753+0000 7f3f53400700 1 -- 172.21.15.12:0/845514199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f580a9310 con 0x55675e1d9d30 2024-05-27T07:25:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.753+0000 7f3f53400700 1 -- 172.21.15.12:0/845514199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f5805f1b0 con 0x55675e1d9d30 2024-05-27T07:25:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.753+0000 7f3f53400700 1 -- 172.21.15.12:0/845514199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5806f1d0 con 0x55675e1d9d30 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/845514199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 msgr2=0x55675e1d3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1d3870 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3f580df8f0 tx=0x7f3f58022000 comp rx=0 tx=0).stop 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f66800700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3f4c0b29f0 0x7f3f4c0b4f00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55675e1d4b30 tx=0x7f3f600099e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/845514199 shutdown_connections 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1d41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/845514199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1d3870 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.754+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/845514199 >> 172.21.15.12:0/845514199 conn(0x55675e0b4fc0 msgr2=0x55675e1ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.755+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/845514199 shutdown_connections 2024-05-27T07:25:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.755+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/845514199 wait complete. 2024-05-27T07:25:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.755+0000 7f3f7bc01b80 1 -- start start 2024-05-27T07:25:33.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.755+0000 7f3f7bc01b80 1 Processor -- start 2024-05-27T07:25:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.756+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1dd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.756+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1dda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.756+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.756+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55675dff7fa0 con 0x55675e1d9d30 2024-05-27T07:25:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.756+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55675e092760 con 0x55675e1d3db0 2024-05-27T07:25:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.756+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55675e1e07a0 con 0x55675e054e40 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.756+0000 7f3f7bc01b80 10 client.0 _ll_register_callbacks cb 0x55675e09bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1ddf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49510/0 (socket says 172.21.15.12:49510) 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 -- 172.21.15.12:0/4199237937 learned_addr learned my addr 172.21.15.12:0/4199237937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f65e00700 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1dd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f65400700 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 -- 172.21.15.12:0/4199237937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 msgr2=0x55675e1dd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1dd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 -- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 msgr2=0x55675e1dda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1dda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.757+0000 7f3f66800700 1 -- 172.21.15.12:0/4199237937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55675e0418d0 con 0x55675e1d9d30 2024-05-27T07:25:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.758+0000 7f3f66800700 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1ddf90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3f60015000 tx=0x7f3f6000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9ae95fe8360502 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.758+0000 7f3f47e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.758+0000 7f3f47e00700 1 -- 172.21.15.12:0/4199237937 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60007a50 con 0x55675e1d9d30 2024-05-27T07:25:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.758+0000 7f3f47e00700 1 -- 172.21.15.12:0/4199237937 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f3f60007bf0 con 0x55675e1d9d30 2024-05-27T07:25:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.758+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55675e103d90 con 0x55675e1d9d30 2024-05-27T07:25:33.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.758+0000 7f3f47e00700 1 -- 172.21.15.12:0/4199237937 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3f60005040 con 0x55675e1d9d30 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.759+0000 7f3f7bc01b80 10 client.15735 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.759+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55675e0c2060 con 0x55675e1d9d30 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.759+0000 7f3f47e00700 1 -- 172.21.15.12:0/4199237937 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55675e103d90 con 0x55675e1d9d30 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.759+0000 7f3f47e00700 1 -- 172.21.15.12:0/4199237937 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3f60023030 con 0x55675e1d9d30 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.759+0000 7f3f7bc01b80 10 client.15735 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.759+0000 7f3f7bc01b80 10 client.15735 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.759+0000 7f3f7bc01b80 20 client.15735 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.760+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e1ed700 0x55675e1efb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.760+0000 7f3f7bc01b80 4 client.15735 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.760+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55675e0fd570 con 0x55675e1ed700 2024-05-27T07:25:33.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.760+0000 7f3f65e00700 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e1ed700 0x55675e1efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.761+0000 7f3f65e00700 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e1ed700 0x55675e1efb20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83fc06e91e658d47 server_cookie=d4308983ac6974a7 in_seq=0 out_seq=0 2024-05-27T07:25:33.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.761+0000 7f3f47e00700 10 client.15735 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:25:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.764+0000 7f3f47e00700 1 -- 172.21.15.12:0/4199237937 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55675e0fd570 con 0x55675e1ed700 2024-05-27T07:25:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.764+0000 7f3f47e00700 10 client.15735 handle_command_reply: tid=0 2024-05-27T07:25:33.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.764+0000 7f3f7bc01b80 1 client.15735 shutdown 2024-05-27T07:25:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 msgr2=0x55675e1ddf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1ddf90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3f60015000 tx=0x7f3f6000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 shutdown_connections 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e1ed700 0x55675e1efb20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d9d30 0x55675e1ddf90 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1dd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/4199237937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1dda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 >> 172.21.15.12:0/4199237937 conn(0x55675e0b4fc0 msgr2=0x55675e1ce580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f47e00700 0 client.15735 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 shutdown_connections 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.765+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/4199237937 wait complete. 2024-05-27T07:25:33.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.766+0000 7f3f7bc01b80 20 client.15735 trim_cache size 0 max 16384 2024-05-27T07:25:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.789+0000 7f3f7bc01b80 1 Processor -- start 2024-05-27T07:25:33.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.789+0000 7f3f7bc01b80 1 -- start start 2024-05-27T07:25:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d3db0 0x55675e1df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d9d30 0x55675e1e9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f681b5170 con 0x55675e1d3db0 2024-05-27T07:25:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f681b2a00 con 0x55675e1d9d30 2024-05-27T07:25:33.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f681b0850 con 0x55675e054e40 2024-05-27T07:25:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f65400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d3db0 0x55675e1df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f65400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d3db0 0x55675e1df9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49522/0 (socket says 172.21.15.12:49522) 2024-05-27T07:25:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f65400700 1 -- 172.21.15.12:0/2300501281 learned_addr learned my addr 172.21.15.12:0/2300501281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.790+0000 7f3f66800700 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d9d30 0x55675e1e9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f65e00700 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f65400700 1 -- 172.21.15.12:0/2300501281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 msgr2=0x55675e1d3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f65400700 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f65400700 1 -- 172.21.15.12:0/2300501281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d9d30 msgr2=0x55675e1e9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f65400700 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d9d30 0x55675e1e9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f65400700 1 -- 172.21.15.12:0/2300501281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55675e1de300 con 0x55675e1d3db0 2024-05-27T07:25:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f66800700 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d9d30 0x55675e1e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:33.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.791+0000 7f3f65400700 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d3db0 0x55675e1df9b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400dd80 tx=0x7f3f54012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65140fa1fbce1ac1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.792+0000 7f3f52a00700 1 -- 172.21.15.12:0/2300501281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54004030 con 0x55675e1d3db0 2024-05-27T07:25:33.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.792+0000 7f3f52a00700 1 -- 172.21.15.12:0/2300501281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f54003680 con 0x55675e1d3db0 2024-05-27T07:25:33.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.792+0000 7f3f52a00700 1 -- 172.21.15.12:0/2300501281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5401c680 con 0x55675e1d3db0 2024-05-27T07:25:33.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.793+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2300501281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d3db0 msgr2=0x55675e1df9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.793+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d3db0 0x55675e1df9b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400dd80 tx=0x7f3f54012730 comp rx=0 tx=0).stop 2024-05-27T07:25:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.793+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2300501281 shutdown_connections 2024-05-27T07:25:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.793+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d9d30 0x55675e1e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.793+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e054e40 0x55675e1d3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.793+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2300501281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e1d3db0 0x55675e1df9b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.793+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2300501281 >> 172.21.15.12:0/2300501281 conn(0x55675e0b4fc0 msgr2=0x55675e1d7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.794+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2300501281 shutdown_connections 2024-05-27T07:25:33.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.794+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2300501281 wait complete. 2024-05-27T07:25:33.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.794+0000 7f3f7bc01b80 1 -- start start 2024-05-27T07:25:33.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.794+0000 7f3f7bc01b80 1 Processor -- start 2024-05-27T07:25:33.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.796+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e054e40 0x55675e1e5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.796+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1e6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.796+0000 7f3f7bc01b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e1d9d30 0x55675e1f4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.796+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f681b0ec0 con 0x55675e054e40 2024-05-27T07:25:33.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.796+0000 7f3f65e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e054e40 0x55675e1e5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.797+0000 7f3f65e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e054e40 0x55675e1e5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49538/0 (socket says 172.21.15.12:49538) 2024-05-27T07:25:33.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.797+0000 7f3f7bc01b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f681b2330 con 0x55675e1d3db0 2024-05-27T07:25:33.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.797+0000 7f3f65e00700 1 -- 172.21.15.12:0/2900937777 learned_addr learned my addr 172.21.15.12:0/2900937777 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:33.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.797+0000 7f3f65400700 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1e6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.797+0000 7f3f66800700 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e1d9d30 0x55675e1f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.797+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f681b1590 con 0x55675e1d9d30 2024-05-27T07:25:33.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.797+0000 7f3f7bc01b80 10 client.0 _ll_register_callbacks cb 0x55675e09bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f65e00700 1 -- 172.21.15.12:0/2900937777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e1d9d30 msgr2=0x55675e1f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f65e00700 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e1d9d30 0x55675e1f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f65e00700 1 -- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 msgr2=0x55675e1e6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f65e00700 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1e6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f65e00700 1 -- 172.21.15.12:0/2900937777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55675e1f4e10 con 0x55675e054e40 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f65e00700 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e054e40 0x55675e1e5c10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3f5805f180 tx=0x7f3f58000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e752efc24ece683 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f47e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.798+0000 7f3f47e00700 1 -- 172.21.15.12:0/2900937777 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f580e2050 con 0x55675e054e40 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.799+0000 7f3f47e00700 1 -- 172.21.15.12:0/2900937777 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f3f580910b0 con 0x55675e054e40 2024-05-27T07:25:33.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.799+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f54003680 con 0x55675e054e40 2024-05-27T07:25:33.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.799+0000 7f3f47e00700 1 -- 172.21.15.12:0/2900937777 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3f580a8cb0 con 0x55675e054e40 2024-05-27T07:25:33.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.799+0000 7f3f7bc01b80 10 client.15741 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:33.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.799+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55675e260320 con 0x55675e054e40 2024-05-27T07:25:33.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.799+0000 7f3f47e00700 1 -- 172.21.15.12:0/2900937777 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f580a9540 con 0x55675e054e40 2024-05-27T07:25:33.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f47e00700 1 -- 172.21.15.12:0/2900937777 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3f580e8160 con 0x55675e054e40 2024-05-27T07:25:33.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f7bc01b80 10 client.15741 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:33.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f7bc01b80 10 client.15741 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:25:33.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f7bc01b80 20 client.15741 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:33.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e260550 0x55675e262970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f7bc01b80 4 client.15741 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:25:33.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3f54003680 con 0x55675e260550 2024-05-27T07:25:33.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.800+0000 7f3f65400700 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e260550 0x55675e262970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.802+0000 7f3f65400700 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e260550 0x55675e262970 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7efb1aa1333821ae server_cookie=9a74adc4daa873ec in_seq=0 out_seq=0 2024-05-27T07:25:33.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.802+0000 7f3f47e00700 10 client.15741 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:25:33.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f47e00700 1 -- 172.21.15.12:0/2900937777 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3f54003680 con 0x55675e260550 2024-05-27T07:25:33.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f47e00700 10 client.15741 handle_command_reply: tid=0 2024-05-27T07:25:33.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 client.15741 shutdown 2024-05-27T07:25:33.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e054e40 msgr2=0x55675e1e5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e054e40 0x55675e1e5c10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3f5805f180 tx=0x7f3f58000bc0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 shutdown_connections 2024-05-27T07:25:33.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55675e260550 0x55675e262970 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55675e1d9d30 0x55675e1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55675e054e40 0x55675e1e5c10 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 --2- 172.21.15.12:0/2900937777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55675e1d3db0 0x55675e1e6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 >> 172.21.15.12:0/2900937777 conn(0x55675e0b4fc0 msgr2=0x55675e1d8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.803+0000 7f3f47e00700 0 client.15741 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:25:33.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.804+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 shutdown_connections 2024-05-27T07:25:33.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.804+0000 7f3f7bc01b80 1 -- 172.21.15.12:0/2900937777 wait complete. 2024-05-27T07:25:33.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.804+0000 7f3f7bc01b80 20 client.15741 trim_cache size 0 max 16384 2024-05-27T07:25:33.796 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:25:33.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3f4c0b29f0 msgr2=0x7f3f4c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3f4c0b29f0 0x7f3f4c0b4f00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55675e1d4b30 tx=0x7f3f600099e0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f681055a0 msgr2=0x7f3f68103290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f681055a0 0x7f3f68103290 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800d950 tx=0x7f3f5800dd10 comp rx=0 tx=0).stop 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 shutdown_connections 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3f4c0b29f0 0x7f3f4c0b4f00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f6810e980 0x7f3f68103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f681055a0 0x7f3f68103290 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 --2- 172.21.15.12:0/3290667011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f68105ef0 0x7f3f681037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 >> 172.21.15.12:0/3290667011 conn(0x7f3f68078df0 msgr2=0x7f3f68100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.805+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 shutdown_connections 2024-05-27T07:25:33.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:33.806+0000 7f3f6d000700 1 -- 172.21.15.12:0/3290667011 wait complete. 2024-05-27T07:25:34.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:34 smithi012 ceph-mon[31473]: pgmap v98: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 489 KiB/s wr, 86 op/s 2024-05-27T07:25:34.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:34 smithi162 ceph-mon[35676]: pgmap v98: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 489 KiB/s wr, 86 op/s 2024-05-27T07:25:34.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:34 smithi062 ceph-mon[35304]: pgmap v98: 129 pgs: 129 active+clean; 43 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 489 KiB/s wr, 86 op/s 2024-05-27T07:25:34.745 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:25:35.336 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:36.548 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:36 smithi012 ceph-mon[31473]: pgmap v99: 129 pgs: 129 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 670 KiB/s wr, 118 op/s 2024-05-27T07:25:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:36 smithi162 ceph-mon[35676]: pgmap v99: 129 pgs: 129 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 670 KiB/s wr, 118 op/s 2024-05-27T07:25:36.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:36 smithi062 ceph-mon[35304]: pgmap v99: 129 pgs: 129 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 670 KiB/s wr, 118 op/s 2024-05-27T07:25:37.725 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f36df800700 time 2024-05-27T07:25:37.477485+0000 2024-05-27T07:25:37.725 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f36f5214ba6] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: clone() 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: *** Caught signal (Aborted) ** 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: in thread 7f36df800700 thread_name:md_log_replay 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.479+0000 7f36df800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f36df800700 time 2024-05-27T07:25:37.477485+0000 2024-05-27T07:25:37.727 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f36f5214ba6] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.728 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: clone() 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f36f3d91d20] 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: gsignal() 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: abort() 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f36f5214ced] 2024-05-27T07:25:37.729 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.730 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.730 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.730 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.730 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.730 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.730 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.730 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 17: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 19: clone() 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.484+0000 7f36df800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: in thread 7f36df800700 thread_name:md_log_replay 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f36f3d91d20] 2024-05-27T07:25:37.731 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: gsignal() 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: abort() 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f36f5214ced] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.732 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 17: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 19: clone() 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: -1> 2024-05-27T07:25:37.479+0000 7f36df800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f36df800700 time 2024-05-27T07:25:37.477485+0000 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f36f5214ba6] 2024-05-27T07:25:37.733 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.734 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.734 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.734 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.734 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.734 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.734 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.734 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: clone() 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 0> 2024-05-27T07:25:37.484+0000 7f36df800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: in thread 7f36df800700 thread_name:md_log_replay 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.735 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f36f3d91d20] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: gsignal() 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: abort() 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f36f5214ced] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.736 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 17: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 19: clone() 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: -1> 2024-05-27T07:25:37.479+0000 7f36df800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f36df800700 time 2024-05-27T07:25:37.477485+0000 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f36f5214ba6] 2024-05-27T07:25:37.737 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.738 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.739 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: clone() 2024-05-27T07:25:37.739 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.739 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 0> 2024-05-27T07:25:37.484+0000 7f36df800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:37.739 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: in thread 7f36df800700 thread_name:md_log_replay 2024-05-27T07:25:37.739 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.739 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:37.739 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f36f3d91d20] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2: gsignal() 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 3: abort() 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f36f5214ced] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f36f5214dd1] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56239033123d] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56239031c971] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 8: (SnapRealm::check_cache() const+0x13e) [0x56239031afa0] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56239031b651] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5623902b0347] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56238ffac87e] 2024-05-27T07:25:37.740 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56238ffb1e94] 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56238ffbc969] 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 14: (MDLog::_replay_thread()+0x1f38) [0x56239035548c] 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56239035e9cb] 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 16: (Thread::entry_wrapper()+0x43) [0x7f36f51e372d] 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 17: (Thread::_entry_func(void*)+0xd) [0x7f36f51e3749] 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f36f3d871ca] 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 19: clone() 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[66629]: 2024-05-27T07:25:37.741 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:37 smithi162 podman[70818]: 2024-05-27 07:25:37.63844115 +0000 UTC m=+0.037033226 container died ce8edb43337e4d539418b9f61955d3cc9ab7e398a4e42d124eea1f21e14b9bc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:25:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.759+0000 7fafa3e00700 1 -- 172.21.15.12:0/2019602096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 msgr2=0x7fafa4110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.759+0000 7fafa3e00700 1 --2- 172.21.15.12:0/2019602096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa4110510 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7faf90009990 tx=0x7faf90009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.759+0000 7fafa3e00700 1 -- 172.21.15.12:0/2019602096 shutdown_connections 2024-05-27T07:25:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.759+0000 7fafa3e00700 1 --2- 172.21.15.12:0/2019602096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa4102ea0 0x7fafa41032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.759+0000 7fafa3e00700 1 --2- 172.21.15.12:0/2019602096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa4110a50 0x7fafa4112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.759+0000 7fafa3e00700 1 --2- 172.21.15.12:0/2019602096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa4110510 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.759+0000 7fafa3e00700 1 -- 172.21.15.12:0/2019602096 >> 172.21.15.12:0/2019602096 conn(0x7fafa4078ab0 msgr2=0x7fafa4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.760+0000 7fafa3e00700 1 -- 172.21.15.12:0/2019602096 shutdown_connections 2024-05-27T07:25:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.760+0000 7fafa3e00700 1 -- 172.21.15.12:0/2019602096 wait complete. 2024-05-27T07:25:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.761+0000 7fafa3e00700 1 Processor -- start 2024-05-27T07:25:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.761+0000 7fafa3e00700 1 -- start start 2024-05-27T07:25:37.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.761+0000 7fafa3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa4102ea0 0x7fafa419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa3e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa4110a50 0x7fafa41a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41163f0 con 0x7fafa4103880 2024-05-27T07:25:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fafa4116270 con 0x7fafa4102ea0 2024-05-27T07:25:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa3e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fafa4116570 con 0x7fafa4110a50 2024-05-27T07:25:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49566/0 (socket says 172.21.15.12:49566) 2024-05-27T07:25:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa2000700 1 -- 172.21.15.12:0/305341866 learned_addr learned my addr 172.21.15.12:0/305341866 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa3400700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa4110a50 0x7fafa41a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.762+0000 7fafa2a00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa4102ea0 0x7fafa419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa2000700 1 -- 172.21.15.12:0/305341866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa4110a50 msgr2=0x7fafa41a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa2000700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa4110a50 0x7fafa41a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa2000700 1 -- 172.21.15.12:0/305341866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa4102ea0 msgr2=0x7fafa419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa2000700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa4102ea0 0x7fafa419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa2000700 1 -- 172.21.15.12:0/305341866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa41a1d30 con 0x7fafa4103880 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa2000700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa419d400 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faf900098e0 tx=0x7faf90003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa8a00700 1 -- 172.21.15.12:0/305341866 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90027070 con 0x7fafa4103880 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa8a00700 1 -- 172.21.15.12:0/305341866 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf9000fc00 con 0x7fafa4103880 2024-05-27T07:25:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa41a1fc0 con 0x7fafa4103880 2024-05-27T07:25:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa8a00700 1 -- 172.21.15.12:0/305341866 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90022ce0 con 0x7fafa4103880 2024-05-27T07:25:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.763+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa41a24a0 con 0x7fafa4103880 2024-05-27T07:25:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.768+0000 7fafa8a00700 1 -- 172.21.15.12:0/305341866 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7faf9002a070 con 0x7fafa4103880 2024-05-27T07:25:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.769+0000 7fafb893cb80 1 Processor -- start 2024-05-27T07:25:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.769+0000 7fafb893cb80 1 -- start start 2024-05-27T07:25:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.769+0000 7fafa8a00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faf880b29f0 0x7faf880b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.769+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48359ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.769+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.769+0000 7fafa8a00700 1 -- 172.21.15.12:0/305341866 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7faf900e3a30 con 0x7fafa4103880 2024-05-27T07:25:37.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.769+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d48367020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafb893cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558d48288e40 con 0x558d4835a820 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafb893cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558d482789e0 con 0x558d481dbe40 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafb893cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558d4827ca20 con 0x558d4835a410 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafa3400700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faf880b29f0 0x7faf880b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafa3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d48367020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafa3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d48367020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49570/0 (socket says 172.21.15.12:49570) 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafa3400700 1 -- 172.21.15.12:0/1548371821 learned_addr learned my addr 172.21.15.12:0/1548371821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafa2000700 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafa2a00700 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48359ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.770+0000 7fafa3400700 1 -- 172.21.15.12:0/1548371821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 msgr2=0x558d48364ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7fafa3400700 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7fafa3400700 1 -- 172.21.15.12:0/1548371821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 msgr2=0x558d48359ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7fafa3400700 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48359ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7fafa3400700 1 -- 172.21.15.12:0/1548371821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d48241420 con 0x558d4835a820 2024-05-27T07:25:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7fafa3400700 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d48367020 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00ae70 tx=0x7faf9c016550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31624d0213d2fe88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7fafa3400700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faf880b29f0 0x7faf880b4f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0168a0 tx=0x7faf9c01a000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7faf98c00700 1 -- 172.21.15.12:0/1548371821 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c013040 con 0x558d4835a820 2024-05-27T07:25:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7faf98c00700 1 -- 172.21.15.12:0/1548371821 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf9c00a520 con 0x558d4835a820 2024-05-27T07:25:37.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.771+0000 7faf98c00700 1 -- 172.21.15.12:0/1548371821 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c020950 con 0x558d4835a820 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.772+0000 7fafb893cb80 1 -- 172.21.15.12:0/1548371821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 msgr2=0x558d48367020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.772+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d48367020 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00ae70 tx=0x7faf9c016550 comp rx=0 tx=0).stop 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 -- 172.21.15.12:0/1548371821 shutdown_connections 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48359ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d48367020 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1548371821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 -- 172.21.15.12:0/1548371821 >> 172.21.15.12:0/1548371821 conn(0x558d4823bfc0 msgr2=0x558d48355d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 -- 172.21.15.12:0/1548371821 shutdown_connections 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 -- 172.21.15.12:0/1548371821 wait complete. 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.773+0000 7fafb893cb80 1 -- start start 2024-05-27T07:25:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 1 Processor -- start 2024-05-27T07:25:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48364650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558d4817efa0 con 0x558d4835a820 2024-05-27T07:25:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558d48219760 con 0x558d481dbe40 2024-05-27T07:25:37.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558d48367930 con 0x558d4835a410 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.774+0000 7fafb893cb80 10 client.0 _ll_register_callbacks cb 0x558d48222c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49574/0 (socket says 172.21.15.12:49574) 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 -- 172.21.15.12:0/2304544050 learned_addr learned my addr 172.21.15.12:0/2304544050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa2a00700 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48364650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa2000700 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 -- 172.21.15.12:0/2304544050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 msgr2=0x558d48364b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 -- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 msgr2=0x558d48364650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48364650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.775+0000 7fafa3400700 1 -- 172.21.15.12:0/2304544050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d481c88d0 con 0x558d4835a820 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7fafa3400700 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483650d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009470 tx=0x7faf9c0204e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b72ed6f7eb97102 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7faf8d600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7faf8d600700 1 -- 172.21.15.12:0/2304544050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c02e070 con 0x558d4835a820 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf9c00a520 con 0x558d4835a820 2024-05-27T07:25:37.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7faf8d600700 1 -- 172.21.15.12:0/2304544050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7faf9c019be0 con 0x558d4835a820 2024-05-27T07:25:37.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7faf8d600700 1 -- 172.21.15.12:0/2304544050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7faf9c019de0 con 0x558d4835a820 2024-05-27T07:25:37.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7fafb893cb80 10 client.15753 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:37.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.776+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558d48249060 con 0x558d4835a820 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7faf8d600700 1 -- 172.21.15.12:0/2304544050 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf9c00a520 con 0x558d4835a820 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7faf8d600700 1 -- 172.21.15.12:0/2304544050 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7faf9c02a560 con 0x558d4835a820 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7fafb893cb80 10 client.15753 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7fafb893cb80 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7fafb893cb80 20 client.15753 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483748e0 0x558d48376d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7fafb893cb80 4 client.15753 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.777+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d4828ad90 con 0x558d483748e0 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.778+0000 7fafa2a00700 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483748e0 0x558d48376d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.779+0000 7fafa2a00700 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483748e0 0x558d48376d00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d1065a86ac34dbb server_cookie=6d5745eac402edaf in_seq=0 out_seq=0 2024-05-27T07:25:37.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.779+0000 7faf8d600700 10 client.15753 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:37.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.781+0000 7faf8d600700 1 -- 172.21.15.12:0/2304544050 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558d4828ad90 con 0x558d483748e0 2024-05-27T07:25:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.781+0000 7faf8d600700 10 client.15753 handle_command_reply: tid=0 2024-05-27T07:25:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 client.15753 shutdown 2024-05-27T07:25:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 msgr2=0x558d483650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483650d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009470 tx=0x7faf9c0204e0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 shutdown_connections 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483748e0 0x558d48376d00 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d481dbe40 0x558d48364650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483650d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2304544050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d4835a410 0x558d48364b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 >> 172.21.15.12:0/2304544050 conn(0x558d4823bfc0 msgr2=0x558d48354ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7faf8d600700 0 client.15753 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 shutdown_connections 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 1 -- 172.21.15.12:0/2304544050 wait complete. 2024-05-27T07:25:37.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.782+0000 7fafb893cb80 20 client.15753 trim_cache size 0 max 16384 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 Processor -- start 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 -- start start 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 0x558d4835fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a410 0x558d4835e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a820 0x558d483709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b4dd0 con 0x558d4835a410 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b2660 con 0x558d4835a820 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.800+0000 7fafb893cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b04b0 con 0x558d481dbe40 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a410 0x558d4835e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a410 0x558d4835e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49580/0 (socket says 172.21.15.12:49580) 2024-05-27T07:25:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 -- 172.21.15.12:0/2413040428 learned_addr learned my addr 172.21.15.12:0/2413040428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:37.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa3400700 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a820 0x558d483709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 -- 172.21.15.12:0/2413040428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 msgr2=0x558d4835fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 0x558d4835fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 -- 172.21.15.12:0/2413040428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a820 msgr2=0x558d483709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a820 0x558d483709d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 -- 172.21.15.12:0/2413040428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d48357350 con 0x558d4835a410 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.801+0000 7fafa2000700 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a410 0x558d4835e7c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7faf900aa920 tx=0x7faf900cd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ade56954a9a6f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.802+0000 7faf8fe00700 1 -- 172.21.15.12:0/2413040428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf900aa990 con 0x558d4835a410 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.802+0000 7faf8fe00700 1 -- 172.21.15.12:0/2413040428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf900aed40 con 0x558d4835a410 2024-05-27T07:25:37.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.802+0000 7faf8fe00700 1 -- 172.21.15.12:0/2413040428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf900aabf0 con 0x558d4835a410 2024-05-27T07:25:37.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.803+0000 7fafb893cb80 1 -- 172.21.15.12:0/2413040428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a410 msgr2=0x558d4835e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.803+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a410 0x558d4835e7c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7faf900aa920 tx=0x7faf900cd040 comp rx=0 tx=0).stop 2024-05-27T07:25:37.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.803+0000 7fafb893cb80 1 -- 172.21.15.12:0/2413040428 shutdown_connections 2024-05-27T07:25:37.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.803+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a820 0x558d483709d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.803+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 0x558d4835fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.803+0000 7fafb893cb80 1 --2- 172.21.15.12:0/2413040428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a410 0x558d4835e7c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.803+0000 7fafb893cb80 1 -- 172.21.15.12:0/2413040428 >> 172.21.15.12:0/2413040428 conn(0x558d4823bfc0 msgr2=0x558d48362f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:37.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.804+0000 7fafb893cb80 1 -- 172.21.15.12:0/2413040428 shutdown_connections 2024-05-27T07:25:37.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.804+0000 7fafb893cb80 1 -- 172.21.15.12:0/2413040428 wait complete. 2024-05-27T07:25:37.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.804+0000 7fafb893cb80 1 -- start start 2024-05-27T07:25:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 1 Processor -- start 2024-05-27T07:25:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 0x558d48368180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a410 0x558d483686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483d7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b0b20 con 0x558d4835a820 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b1f90 con 0x558d4835a410 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b11f0 con 0x558d481dbe40 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.805+0000 7fafb893cb80 10 client.0 _ll_register_callbacks cb 0x558d48222c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483d7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49592/0 (socket says 172.21.15.12:49592) 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 -- 172.21.15.12:0/1100644767 learned_addr learned my addr 172.21.15.12:0/1100644767 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa2a00700 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 0x558d48368180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 -- 172.21.15.12:0/1100644767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 msgr2=0x558d48368180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 0x558d48368180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 -- 172.21.15.12:0/1100644767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a410 msgr2=0x558d483686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a410 0x558d483686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 -- 172.21.15.12:0/1100644767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d483d7800 con 0x558d4835a820 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7fafa3400700 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483d7290 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00ae70 tx=0x7faf9c02aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba29fd2f4cbec76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.806+0000 7faf8d600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.807+0000 7faf8d600700 1 -- 172.21.15.12:0/1100644767 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c044060 con 0x558d4835a820 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.807+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf900aed40 con 0x558d4835a820 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.807+0000 7faf8d600700 1 -- 172.21.15.12:0/1100644767 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7faf9c019be0 con 0x558d4835a820 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.807+0000 7faf8d600700 1 -- 172.21.15.12:0/1100644767 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7faf9c04b050 con 0x558d4835a820 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.807+0000 7fafb893cb80 10 client.15759 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.807+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558d483deae0 con 0x558d4835a820 2024-05-27T07:25:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7faf8d600700 1 -- 172.21.15.12:0/1100644767 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faf900aed40 con 0x558d4835a820 2024-05-27T07:25:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7faf8d600700 1 -- 172.21.15.12:0/1100644767 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7faf9c03fb40 con 0x558d4835a820 2024-05-27T07:25:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7fafb893cb80 10 client.15759 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7fafb893cb80 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7fafb893cb80 20 client.15759 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483df1f0 0x558d483e1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7fafb893cb80 4 client.15759 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:25:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.808+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558d48284570 con 0x558d483df1f0 2024-05-27T07:25:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.809+0000 7fafa2a00700 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483df1f0 0x558d483e1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.810+0000 7fafa2a00700 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483df1f0 0x558d483e1630 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e73817608a3810e server_cookie=fedcc3b4a41daf in_seq=0 out_seq=0 2024-05-27T07:25:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.810+0000 7faf8d600700 10 client.15759 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7faf8d600700 1 -- 172.21.15.12:0/1100644767 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558d48284570 con 0x558d483df1f0 2024-05-27T07:25:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7faf8d600700 10 client.15759 handle_command_reply: tid=0 2024-05-27T07:25:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 client.15759 shutdown 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 msgr2=0x558d483d7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483d7290 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00ae70 tx=0x7faf9c02aed0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 shutdown_connections 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558d4835a820 0x558d483d7290 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d481dbe40 0x558d48368180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558d483df1f0 0x558d483e1630 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 --2- 172.21.15.12:0/1100644767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d4835a410 0x558d483686c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 >> 172.21.15.12:0/1100644767 conn(0x558d4823bfc0 msgr2=0x558d48363970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7faf8d600700 0 client.15759 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 shutdown_connections 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 1 -- 172.21.15.12:0/1100644767 wait complete. 2024-05-27T07:25:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.812+0000 7fafb893cb80 20 client.15759 trim_cache size 0 max 16384 2024-05-27T07:25:37.805 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:25:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faf880b29f0 msgr2=0x7faf880b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faf880b29f0 0x7faf880b4f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0168a0 tx=0x7faf9c01a000 comp rx=0 tx=0).stop 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 msgr2=0x7fafa419d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa419d400 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faf900098e0 tx=0x7faf90003680 comp rx=0 tx=0).stop 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 shutdown_connections 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faf880b29f0 0x7faf880b4f00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa4102ea0 0x7fafa419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa4110a50 0x7fafa41a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 --2- 172.21.15.12:0/305341866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fafa4103880 0x7fafa419d400 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.813+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 >> 172.21.15.12:0/305341866 conn(0x7fafa4078ab0 msgr2=0x7fafa4111fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.814+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 shutdown_connections 2024-05-27T07:25:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:37.814+0000 7fafa3e00700 1 -- 172.21.15.12:0/305341866 wait complete. 2024-05-27T07:25:38.239 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6a3200700 time 2024-05-27T07:25:37.938484+0000 2024-05-27T07:25:38.239 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.239 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.239 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6b8dadba6] 2024-05-27T07:25:38.239 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.239 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.239 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: clone() 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: *** Caught signal (Aborted) ** 2024-05-27T07:25:38.240 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: in thread 7ff6a3200700 thread_name:md_log_replay 2024-05-27T07:25:38.241 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:37.941+0000 7ff6a3200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6a3200700 time 2024-05-27T07:25:37.938484+0000 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6b8dadba6] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.242 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: clone() 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b792ad20] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: gsignal() 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: abort() 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6b8dadced] 2024-05-27T07:25:38.243 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.245 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.245 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.245 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.245 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.245 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.245 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.245 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 19: clone() 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:37.946+0000 7ff6a3200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: in thread 7ff6a3200700 thread_name:md_log_replay 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b792ad20] 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: gsignal() 2024-05-27T07:25:38.246 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: abort() 2024-05-27T07:25:38.247 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6b8dadced] 2024-05-27T07:25:38.247 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.247 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.247 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.247 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.247 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.247 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.248 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.248 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.248 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.248 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.248 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.248 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.248 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 19: clone() 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: -1> 2024-05-27T07:25:37.941+0000 7ff6a3200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6a3200700 time 2024-05-27T07:25:37.938484+0000 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6b8dadba6] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.249 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.250 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.251 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.251 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.251 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.251 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.251 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: clone() 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 0> 2024-05-27T07:25:37.946+0000 7ff6a3200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: in thread 7ff6a3200700 thread_name:md_log_replay 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b792ad20] 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: gsignal() 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: abort() 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6b8dadced] 2024-05-27T07:25:38.252 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.253 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.254 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.254 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 19: clone() 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: -1> 2024-05-27T07:25:37.941+0000 7ff6a3200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6a3200700 time 2024-05-27T07:25:37.938484+0000 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6b8dadba6] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.255 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.256 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: clone() 2024-05-27T07:25:38.257 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.257 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 0> 2024-05-27T07:25:37.946+0000 7ff6a3200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:38.257 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: in thread 7ff6a3200700 thread_name:md_log_replay 2024-05-27T07:25:38.257 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b792ad20] 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2: gsignal() 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 3: abort() 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6b8dadced] 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6b8daddd1] 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c7c20ce23d] 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c7c20b9971] 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c7c20b7fa0] 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c7c20b8651] 2024-05-27T07:25:38.258 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c7c204d347] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c7c1d4987e] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c7c1d4ee94] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c7c1d59969] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c7c20f248c] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c7c20fb9cb] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6b8d7c72d] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6b8d7c749] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b79201ca] 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 19: clone() 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[65233]: 2024-05-27T07:25:38.259 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:38 smithi062 podman[70365]: 2024-05-27 07:25:38.096424405 +0000 UTC m=+0.040356830 container died ff5ae73d73948dc6eed798250867092a4332530841a9ac50dc91f46d303a9f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:25:38.520 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-mon[35676]: pgmap v100: 129 pgs: 129 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 484 KiB/s wr, 76 op/s 2024-05-27T07:25:38.521 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:38 smithi162 podman[70818]: 2024-05-27 07:25:38.518219892 +0000 UTC m=+0.916811962 container remove ce8edb43337e4d539418b9f61955d3cc9ab7e398a4e42d124eea1f21e14b9bc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc4ffe00700 time 2024-05-27T07:25:38.523494+0000 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc515949ba6] 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.524 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: clone() 2024-05-27T07:25:38.525 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.525+0000 7fc4ffe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc4ffe00700 time 2024-05-27T07:25:38.523494+0000 2024-05-27T07:25:38.527 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:38 smithi062 ceph-mon[35304]: pgmap v100: 129 pgs: 129 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 484 KiB/s wr, 76 op/s 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc515949ba6] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.528 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: clone() 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: *** Caught signal (Aborted) ** 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: in thread 7fc4ffe00700 thread_name:md_log_replay 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc5144c6d20] 2024-05-27T07:25:38.529 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: gsignal() 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: abort() 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc515949ced] 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.530 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 19: clone() 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.530+0000 7fc4ffe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: in thread 7fc4ffe00700 thread_name:md_log_replay 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.531 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc5144c6d20] 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: gsignal() 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: abort() 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc515949ced] 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.532 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 19: clone() 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:38.533 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:38 smithi012 ceph-mon[31473]: pgmap v100: 129 pgs: 129 active+clean; 45 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 484 KiB/s wr, 76 op/s 2024-05-27T07:25:38.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:38 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:25:38.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: -1> 2024-05-27T07:25:38.525+0000 7fc4ffe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc4ffe00700 time 2024-05-27T07:25:38.523494+0000 2024-05-27T07:25:38.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc515949ba6] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: clone() 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 0> 2024-05-27T07:25:38.530+0000 7fc4ffe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: in thread 7fc4ffe00700 thread_name:md_log_replay 2024-05-27T07:25:38.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc5144c6d20] 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: gsignal() 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: abort() 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc515949ced] 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 19: clone() 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: -1> 2024-05-27T07:25:38.525+0000 7fc4ffe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fc4ffe00700 time 2024-05-27T07:25:38.523494+0000 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:38.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc515949ba6] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: clone() 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 0> 2024-05-27T07:25:38.530+0000 7fc4ffe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: in thread 7fc4ffe00700 thread_name:md_log_replay 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc5144c6d20] 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2: gsignal() 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 3: abort() 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc515949ced] 2024-05-27T07:25:38.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc515949dd1] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56319fb1223d] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56319fafd971] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 8: (SnapRealm::check_cache() const+0x13e) [0x56319fafbfa0] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56319fafc651] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56319fa91347] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56319f78d87e] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56319f792e94] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56319f79d969] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 14: (MDLog::_replay_thread()+0x1f38) [0x56319fb3648c] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56319fb3f9cb] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 16: (Thread::entry_wrapper()+0x43) [0x7fc51591872d] 2024-05-27T07:25:38.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 17: (Thread::_entry_func(void*)+0xd) [0x7fc515918749] 2024-05-27T07:25:38.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc5144bc1ca] 2024-05-27T07:25:38.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 19: clone() 2024-05-27T07:25:38.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:38.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[66330]: 2024-05-27T07:25:38.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:38 smithi162 podman[70848]: 2024-05-27 07:25:38.667993932 +0000 UTC m=+0.033525739 container died 6506cb73b5802bfcc5f2990960e0fe055e2012aac24d358877c6401761fae3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:25:39.485 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:39 smithi062 podman[70365]: 2024-05-27 07:25:39.100841931 +0000 UTC m=+1.044774366 container remove ff5ae73d73948dc6eed798250867092a4332530841a9ac50dc91f46d303a9f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:25:39.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:39 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:25:39.715 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:39 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-27T07:25:39.715 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:39 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.470s CPU time. 2024-05-27T07:25:39.716 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:39 smithi162 podman[70848]: 2024-05-27 07:25:39.341091074 +0000 UTC m=+0.706622901 container remove 6506cb73b5802bfcc5f2990960e0fe055e2012aac24d358877c6401761fae3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T07:25:39.716 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:39 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:25:39.799 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:25:39.799 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:25:39.986 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:39 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:25:39.986 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:39 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 6.580s CPU time. 2024-05-27T07:25:40.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:39 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:25:40.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:39 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.620s CPU time. 2024-05-27T07:25:40.290 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:40.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:40 smithi012 ceph-mon[31473]: pgmap v101: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 622 KiB/s wr, 94 op/s 2024-05-27T07:25:40.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:40 smithi162 ceph-mon[35676]: pgmap v101: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 622 KiB/s wr, 94 op/s 2024-05-27T07:25:40.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:40 smithi062 ceph-mon[35304]: pgmap v101: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 622 KiB/s wr, 94 op/s 2024-05-27T07:25:42.431 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:42 smithi012 ceph-mon[31473]: pgmap v102: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 318 KiB/s wr, 49 op/s 2024-05-27T07:25:42.543 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:42 smithi062 ceph-mon[35304]: pgmap v102: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 318 KiB/s wr, 49 op/s 2024-05-27T07:25:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:42 smithi162 ceph-mon[35676]: pgmap v102: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 318 KiB/s wr, 49 op/s 2024-05-27T07:25:42.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.856+0000 7f5e38000700 1 -- 172.21.15.12:0/1131511301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3010e8f0 msgr2=0x7f5e30110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.856+0000 7f5e38000700 1 --2- 172.21.15.12:0/1131511301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3010e8f0 0x7f5e30110db0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800b0a0 tx=0x7f5e2800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.857+0000 7f5e38000700 1 -- 172.21.15.12:0/1131511301 shutdown_connections 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.857+0000 7f5e38000700 1 --2- 172.21.15.12:0/1131511301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e3010e8f0 0x7f5e30110db0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.857+0000 7f5e38000700 1 --2- 172.21.15.12:0/1131511301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e30105510 0x7f5e30105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.857+0000 7f5e38000700 1 --2- 172.21.15.12:0/1131511301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e30105e60 0x7f5e3010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.857+0000 7f5e38000700 1 -- 172.21.15.12:0/1131511301 >> 172.21.15.12:0/1131511301 conn(0x7f5e300fedb0 msgr2=0x7f5e301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.857+0000 7f5e38000700 1 -- 172.21.15.12:0/1131511301 shutdown_connections 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.857+0000 7f5e38000700 1 -- 172.21.15.12:0/1131511301 wait complete. 2024-05-27T07:25:42.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.858+0000 7f5e38000700 1 Processor -- start 2024-05-27T07:25:42.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e38000700 1 -- start start 2024-05-27T07:25:42.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e38000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e30105510 0x7f5e30103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e38000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e30105e60 0x7f5e30103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e38000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e3010e8f0 0x7f5e30103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e38000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5e301142e0 con 0x7f5e30105510 2024-05-27T07:25:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e38000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5e30114160 con 0x7f5e30105e60 2024-05-27T07:25:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e38000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e30114460 con 0x7f5e3010e8f0 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e35600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e30105510 0x7f5e30103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e35600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e30105510 0x7f5e30103200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56078/0 (socket says 172.21.15.12:56078) 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.859+0000 7f5e35600700 1 -- 172.21.15.12:0/2447088399 learned_addr learned my addr 172.21.15.12:0/2447088399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.860+0000 7f5e34c00700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e30105e60 0x7f5e30103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.860+0000 7f5e36000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e3010e8f0 0x7f5e30103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.860+0000 7f5e35600700 1 -- 172.21.15.12:0/2447088399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e3010e8f0 msgr2=0x7f5e30103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.860+0000 7f5e35600700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e3010e8f0 0x7f5e30103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.860+0000 7f5e35600700 1 -- 172.21.15.12:0/2447088399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e30105e60 msgr2=0x7f5e30103740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.860+0000 7f5e35600700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e30105e60 0x7f5e30103740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.860+0000 7f5e35600700 1 -- 172.21.15.12:0/2447088399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e301a9070 con 0x7f5e30105510 2024-05-27T07:25:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.861+0000 7f5e35600700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e30105510 0x7f5e30103200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400d950 tx=0x7f5e2400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.861+0000 7f5e2e000700 1 -- 172.21.15.12:0/2447088399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24016070 con 0x7f5e30105510 2024-05-27T07:25:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.861+0000 7f5e2e000700 1 -- 172.21.15.12:0/2447088399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e24010460 con 0x7f5e30105510 2024-05-27T07:25:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.861+0000 7f5e2e000700 1 -- 172.21.15.12:0/2447088399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2400f620 con 0x7f5e30105510 2024-05-27T07:25:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.861+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e301a9360 con 0x7f5e30105510 2024-05-27T07:25:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.861+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e301a9850 con 0x7f5e30105510 2024-05-27T07:25:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.865+0000 7f5e2e000700 1 -- 172.21.15.12:0/2447088399 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5e24019070 con 0x7f5e30105510 2024-05-27T07:25:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.866+0000 7f5e46a96b80 1 Processor -- start 2024-05-27T07:25:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.866+0000 7f5e46a96b80 1 -- start start 2024-05-27T07:25:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb28d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb2878f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb288240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5560cb1b5ec0 con 0x5560cb28ddb0 2024-05-27T07:25:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5560cb1a5a60 con 0x5560cb108ec0 2024-05-27T07:25:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5560cb1a9aa0 con 0x5560cb287e30 2024-05-27T07:25:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e2e000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5e180b2a70 0x7f5e180b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e34c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb288240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e2e000700 1 -- 172.21.15.12:0/2447088399 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f5e240e25d0 con 0x7f5e30105510 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e36000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb28d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e34c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb288240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:36810/0 (socket says 172.21.15.12:36810) 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.867+0000 7f5e34c00700 1 -- 172.21.15.12:0/2186568051 learned_addr learned my addr 172.21.15.12:0/2186568051 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e35600700 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb2878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e36000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5e180b2a70 0x7f5e180b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e34c00700 1 -- 172.21.15.12:0/2186568051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 msgr2=0x5560cb28d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e34c00700 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb28d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e34c00700 1 -- 172.21.15.12:0/2186568051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 msgr2=0x5560cb2878f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e34c00700 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb2878f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e34c00700 1 -- 172.21.15.12:0/2186568051 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560cb16e4a0 con 0x5560cb287e30 2024-05-27T07:25:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.868+0000 7f5e36000700 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb28d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.869+0000 7f5e36000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5e180b2a70 0x7f5e180b4f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5560cb288bb0 tx=0x7f5e280099e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.869+0000 7f5e34c00700 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb288240 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00dcb0 tx=0x7f5e1c00f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea7caecc9580ef46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.869+0000 7f5e22a00700 1 -- 172.21.15.12:0/2186568051 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00fbe0 con 0x5560cb287e30 2024-05-27T07:25:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.870+0000 7f5e22a00700 1 -- 172.21.15.12:0/2186568051 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e1c00fd80 con 0x5560cb287e30 2024-05-27T07:25:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.870+0000 7f5e22a00700 1 -- 172.21.15.12:0/2186568051 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c01b670 con 0x5560cb287e30 2024-05-27T07:25:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 -- 172.21.15.12:0/2186568051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 msgr2=0x5560cb288240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb288240 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00dcb0 tx=0x7f5e1c00f360 comp rx=0 tx=0).stop 2024-05-27T07:25:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 -- 172.21.15.12:0/2186568051 shutdown_connections 2024-05-27T07:25:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb28d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb288240 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/2186568051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb2878f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 -- 172.21.15.12:0/2186568051 >> 172.21.15.12:0/2186568051 conn(0x5560cb169040 msgr2=0x5560cb282950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 -- 172.21.15.12:0/2186568051 shutdown_connections 2024-05-27T07:25:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.871+0000 7f5e46a96b80 1 -- 172.21.15.12:0/2186568051 wait complete. 2024-05-27T07:25:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.872+0000 7f5e46a96b80 1 -- start start 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 1 Processor -- start 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb291600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb291b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb292080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5560cb0ac020 con 0x5560cb28ddb0 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5560cb1467e0 con 0x5560cb108ec0 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5560cb294880 con 0x5560cb287e30 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.873+0000 7f5e46a96b80 10 client.0 _ll_register_callbacks cb 0x5560cb14fcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb292080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb292080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56084/0 (socket says 172.21.15.12:56084) 2024-05-27T07:25:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 -- 172.21.15.12:0/3028821424 learned_addr learned my addr 172.21.15.12:0/3028821424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:42.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e34c00700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb291b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e35600700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb291600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 -- 172.21.15.12:0/3028821424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 msgr2=0x5560cb291b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb291b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 -- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 msgr2=0x5560cb291600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb291600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e36000700 1 -- 172.21.15.12:0/3028821424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560cb0f5950 con 0x5560cb28ddb0 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.874+0000 7f5e35600700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb291600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e36000700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb292080 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e28007840 tx=0x7f5e2800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee118b994b85ad35 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e17400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e17400700 1 -- 172.21.15.12:0/3028821424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2802f030 con 0x5560cb28ddb0 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e17400700 1 -- 172.21.15.12:0/3028821424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f5e2802a470 con 0x5560cb28ddb0 2024-05-27T07:25:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e1c00fd80 con 0x5560cb28ddb0 2024-05-27T07:25:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e17400700 1 -- 172.21.15.12:0/3028821424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5e28024040 con 0x5560cb28ddb0 2024-05-27T07:25:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e46a96b80 10 client.15771 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.875+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5560cb1760e0 con 0x5560cb28ddb0 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e17400700 1 -- 172.21.15.12:0/3028821424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e1c00fd80 con 0x5560cb28ddb0 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e17400700 1 -- 172.21.15.12:0/3028821424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5e28011070 con 0x5560cb28ddb0 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e46a96b80 10 client.15771 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e46a96b80 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e46a96b80 20 client.15771 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb2a1890 0x5560cb2a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e46a96b80 4 client.15771 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.876+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560cb1b7e10 con 0x5560cb2a1890 2024-05-27T07:25:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.877+0000 7f5e35600700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb2a1890 0x5560cb2a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.878+0000 7f5e35600700 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb2a1890 0x5560cb2a3cd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d05784df024deba8 server_cookie=8347ffc9cab65ef3 in_seq=0 out_seq=0 2024-05-27T07:25:42.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.878+0000 7f5e17400700 10 client.15771 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e17400700 1 -- 172.21.15.12:0/3028821424 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5560cb1b7e10 con 0x5560cb2a1890 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e17400700 10 client.15771 handle_command_reply: tid=0 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 client.15771 shutdown 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 msgr2=0x5560cb292080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb292080 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e28007840 tx=0x7f5e2800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 shutdown_connections 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb2a1890 0x5560cb2a3cd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb108ec0 0x5560cb291600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb28ddb0 0x5560cb292080 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.881+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3028821424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb287e30 0x5560cb291b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.882+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 >> 172.21.15.12:0/3028821424 conn(0x5560cb169040 msgr2=0x5560cb282600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.882+0000 7f5e17400700 0 client.15771 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:42.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.882+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 shutdown_connections 2024-05-27T07:25:42.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.882+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3028821424 wait complete. 2024-05-27T07:25:42.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.882+0000 7f5e46a96b80 20 client.15771 trim_cache size 0 max 16384 2024-05-27T07:25:42.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.906+0000 7f5e46a96b80 1 Processor -- start 2024-05-27T07:25:42.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.906+0000 7f5e46a96b80 1 -- start start 2024-05-27T07:25:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.906+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb108ec0 0x5560cb2878f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.906+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb287e30 0x5560cb293aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb28ddb0 0x5560cb29d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5e301b5050 con 0x5560cb287e30 2024-05-27T07:25:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5e301b28e0 con 0x5560cb28ddb0 2024-05-27T07:25:42.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e301b0730 con 0x5560cb108ec0 2024-05-27T07:25:42.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb287e30 0x5560cb293aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb287e30 0x5560cb293aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56092/0 (socket says 172.21.15.12:56092) 2024-05-27T07:25:42.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 -- 172.21.15.12:0/3524194194 learned_addr learned my addr 172.21.15.12:0/3524194194 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:42.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e35600700 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb108ec0 0x5560cb2878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 -- 172.21.15.12:0/3524194194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb108ec0 msgr2=0x5560cb2878f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb108ec0 0x5560cb2878f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 -- 172.21.15.12:0/3524194194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb28ddb0 msgr2=0x5560cb29d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb28ddb0 0x5560cb29d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.907+0000 7f5e34c00700 1 -- 172.21.15.12:0/3524194194 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560cb28cdc0 con 0x5560cb287e30 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.908+0000 7f5e34c00700 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb287e30 0x5560cb293aa0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00f6b0 tx=0x7f5e1c00d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28b3cea3c6f1427 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.908+0000 7f5e22000700 1 -- 172.21.15.12:0/3524194194 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00fbe0 con 0x5560cb287e30 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.908+0000 7f5e22000700 1 -- 172.21.15.12:0/3524194194 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e1c02a070 con 0x5560cb287e30 2024-05-27T07:25:42.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.908+0000 7f5e22000700 1 -- 172.21.15.12:0/3524194194 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0055d0 con 0x5560cb287e30 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3524194194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb287e30 msgr2=0x5560cb293aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb287e30 0x5560cb293aa0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00f6b0 tx=0x7f5e1c00d820 comp rx=0 tx=0).stop 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3524194194 shutdown_connections 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb28ddb0 0x5560cb29d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb108ec0 0x5560cb2878f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/3524194194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb287e30 0x5560cb293aa0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3524194194 >> 172.21.15.12:0/3524194194 conn(0x5560cb169040 msgr2=0x5560cb2913e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3524194194 shutdown_connections 2024-05-27T07:25:42.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 -- 172.21.15.12:0/3524194194 wait complete. 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.910+0000 7f5e46a96b80 1 -- start start 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 1 Processor -- start 2024-05-27T07:25:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb108ec0 0x5560cb2a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb287e30 0x5560cb2a9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb28ddb0 0x5560cb2a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5e301b0da0 con 0x5560cb108ec0 2024-05-27T07:25:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5e301b2210 con 0x5560cb287e30 2024-05-27T07:25:42.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e301b1470 con 0x5560cb28ddb0 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.911+0000 7f5e46a96b80 10 client.0 _ll_register_callbacks cb 0x5560cb14fcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb108ec0 0x5560cb2a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb108ec0 0x5560cb2a9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56096/0 (socket says 172.21.15.12:56096) 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 -- 172.21.15.12:0/4254050734 learned_addr learned my addr 172.21.15.12:0/4254050734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e36000700 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb28ddb0 0x5560cb2a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e34c00700 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb287e30 0x5560cb2a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 -- 172.21.15.12:0/4254050734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb28ddb0 msgr2=0x5560cb2a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb28ddb0 0x5560cb2a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 -- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb287e30 msgr2=0x5560cb2a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb287e30 0x5560cb2a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.912+0000 7f5e35600700 1 -- 172.21.15.12:0/4254050734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560cb30b4a0 con 0x5560cb108ec0 2024-05-27T07:25:42.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e35600700 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb108ec0 0x5560cb2a9150 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400b2f0 tx=0x7f5e2400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6804766859e5f73 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e17400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e17400700 1 -- 172.21.15.12:0/4254050734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e240c3060 con 0x5560cb108ec0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e17400700 1 -- 172.21.15.12:0/4254050734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f5e240a9700 con 0x5560cb108ec0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e1c02a070 con 0x5560cb108ec0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e17400700 1 -- 172.21.15.12:0/4254050734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5e24020040 con 0x5560cb108ec0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e46a96b80 10 client.15777 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.913+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5560cb30c740 con 0x5560cb108ec0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e17400700 1 -- 172.21.15.12:0/4254050734 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e1c02a070 con 0x5560cb108ec0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e17400700 1 -- 172.21.15.12:0/4254050734 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5e240c8030 con 0x5560cb108ec0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e46a96b80 10 client.15777 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e46a96b80 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e46a96b80 20 client.15777 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb30ce10 0x5560cb30f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e46a96b80 4 client.15777 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.914+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5560cb1b15f0 con 0x5560cb30ce10 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.915+0000 7f5e34c00700 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb30ce10 0x5560cb30f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.916+0000 7f5e34c00700 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb30ce10 0x5560cb30f230 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f5993b1e67a2fdb server_cookie=86a2f2d24d5dfc59 in_seq=0 out_seq=0 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.916+0000 7f5e17400700 10 client.15777 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:42.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e17400700 1 -- 172.21.15.12:0/4254050734 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5560cb1b15f0 con 0x5560cb30ce10 2024-05-27T07:25:42.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e17400700 10 client.15777 handle_command_reply: tid=0 2024-05-27T07:25:42.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 client.15777 shutdown 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb108ec0 msgr2=0x5560cb2a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb108ec0 0x5560cb2a9150 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400b2f0 tx=0x7f5e2400b5b0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 shutdown_connections 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560cb28ddb0 0x5560cb2a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5560cb108ec0 0x5560cb2a9150 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5560cb30ce10 0x5560cb30f230 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 --2- 172.21.15.12:0/4254050734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560cb287e30 0x5560cb2a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e17400700 0 client.15777 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 >> 172.21.15.12:0/4254050734 conn(0x5560cb169040 msgr2=0x5560cb291e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.920+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 shutdown_connections 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.921+0000 7f5e46a96b80 1 -- 172.21.15.12:0/4254050734 wait complete. 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.921+0000 7f5e46a96b80 20 client.15777 trim_cache size 0 max 16384 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "096ae71a-009d-45c4-9152-ec84dbbf437d", 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5e180b2a70 msgr2=0x7f5e180b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5e180b2a70 0x7f5e180b4f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5560cb288bb0 tx=0x7f5e280099e0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e30105510 msgr2=0x7f5e30103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e30105510 0x7f5e30103200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400d950 tx=0x7f5e2400dd10 comp rx=0 tx=0).stop 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 shutdown_connections 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5e180b2a70 0x7f5e180b4f80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e3010e8f0 0x7f5e30103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e30105510 0x7f5e30103200 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 --2- 172.21.15.12:0/2447088399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e30105e60 0x7f5e30103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 >> 172.21.15.12:0/2447088399 conn(0x7f5e300fedb0 msgr2=0x7f5e300ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 shutdown_connections 2024-05-27T07:25:42.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:42.922+0000 7f5e38000700 1 -- 172.21.15.12:0/2447088399 wait complete. 2024-05-27T07:25:44.077 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:25:44.549 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:44.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: pgmap v103: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 318 KiB/s wr, 49 op/s 2024-05-27T07:25:44.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:25:44.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:25:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:25:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:25:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:44 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: pgmap v103: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 318 KiB/s wr, 49 op/s 2024-05-27T07:25:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:25:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:25:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:25:44.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:25:44.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:44 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: pgmap v103: 129 pgs: 129 active+clean; 47 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 318 KiB/s wr, 49 op/s 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:44.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:44 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:45.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:45 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:45.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:45 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:45.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:45 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:46.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:46 smithi012 ceph-mon[31473]: pgmap v104: 129 pgs: 129 active+clean; 49 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 491 KiB/s wr, 99 op/s 2024-05-27T07:25:46.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:46 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:46.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:46 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:46.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:46 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:25:46.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:46 smithi162 ceph-mon[35676]: pgmap v104: 129 pgs: 129 active+clean; 49 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 491 KiB/s wr, 99 op/s 2024-05-27T07:25:46.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:46 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:46.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:46 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:46.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:46 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:25:46.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:46 smithi062 ceph-mon[35304]: pgmap v104: 129 pgs: 129 active+clean; 49 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 491 KiB/s wr, 99 op/s 2024-05-27T07:25:46.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:46 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:46.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:46 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:46.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:46 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:25:47.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 -- 172.21.15.12:0/2845099769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 msgr2=0x7f7cec10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 --2- 172.21.15.12:0/2845099769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec10b320 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80099c0 tx=0x7f7cd8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 -- 172.21.15.12:0/2845099769 shutdown_connections 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 --2- 172.21.15.12:0/2845099769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cec10b860 0x7f7cec1131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 --2- 172.21.15.12:0/2845099769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cec106630 0x7f7cec106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 --2- 172.21.15.12:0/2845099769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec10b320 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 -- 172.21.15.12:0/2845099769 >> 172.21.15.12:0/2845099769 conn(0x7f7cec102020 msgr2=0x7f7cec104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 -- 172.21.15.12:0/2845099769 shutdown_connections 2024-05-27T07:25:47.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.060+0000 7f7ceb200700 1 -- 172.21.15.12:0/2845099769 wait complete. 2024-05-27T07:25:47.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.061+0000 7f7ceb200700 1 Processor -- start 2024-05-27T07:25:47.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ceb200700 1 -- start start 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ceb200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cec106630 0x7f7cec19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ceb200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec19d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ceb200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cec10b860 0x7f7cec1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ceb200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec116370 con 0x7f7cec10af10 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ceb200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1161f0 con 0x7f7cec10b860 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ceb200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1164f0 con 0x7f7cec106630 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ce9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec19d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ce9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec19d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56116/0 (socket says 172.21.15.12:56116) 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.062+0000 7f7ce9400700 1 -- 172.21.15.12:0/736634902 learned_addr learned my addr 172.21.15.12:0/736634902 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:47.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.063+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cec106630 0x7f7cec19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.063+0000 7f7ce9400700 1 -- 172.21.15.12:0/736634902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cec106630 msgr2=0x7f7cec19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.063+0000 7f7ce9400700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cec106630 0x7f7cec19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.063+0000 7f7ce9400700 1 -- 172.21.15.12:0/736634902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cec10b860 msgr2=0x7f7cec1a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.063+0000 7f7ce9400700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cec10b860 0x7f7cec1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.063+0000 7f7ce9400700 1 -- 172.21.15.12:0/736634902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cec1a1c70 con 0x7f7cec10af10 2024-05-27T07:25:47.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.064+0000 7f7ce9400700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec19d340 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800b580 tx=0x7f7cd800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.064+0000 7f7ce2a00700 1 -- 172.21.15.12:0/736634902 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8027070 con 0x7f7cec10af10 2024-05-27T07:25:47.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.064+0000 7f7ce2a00700 1 -- 172.21.15.12:0/736634902 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cd800fda0 con 0x7f7cec10af10 2024-05-27T07:25:47.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.064+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cec1a1f00 con 0x7f7cec10af10 2024-05-27T07:25:47.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.064+0000 7f7ce2a00700 1 -- 172.21.15.12:0/736634902 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80173f0 con 0x7f7cec10af10 2024-05-27T07:25:47.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.064+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cec1a9880 con 0x7f7cec10af10 2024-05-27T07:25:47.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.068+0000 7f7ce2a00700 1 -- 172.21.15.12:0/736634902 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7cd802a070 con 0x7f7cec10af10 2024-05-27T07:25:47.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.070+0000 7f7ce2a00700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7cd00b2a70 0x7f7cd00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.070+0000 7f7ce2a00700 1 -- 172.21.15.12:0/736634902 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f7cd80e3810 con 0x7f7cec10af10 2024-05-27T07:25:47.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.070+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7cd00b2a70 0x7f7cd00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.071+0000 7f7cffb61b80 1 Processor -- start 2024-05-27T07:25:47.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.071+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7cd00b2a70 0x7f7cd00b4f80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc005950 tx=0x7f7cdc0058e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.071+0000 7f7cffb61b80 1 -- start start 2024-05-27T07:25:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890e2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3820 0x55d1890f0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d189011e40 con 0x55d1890e3820 2024-05-27T07:25:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d1890019e0 con 0x55d188f64e40 2024-05-27T07:25:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d189005a20 con 0x55d1890e3410 2024-05-27T07:25:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3820 0x55d1890f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cea800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3820 0x55d1890f0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56128/0 (socket says 172.21.15.12:56128) 2024-05-27T07:25:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7cea800700 1 -- 172.21.15.12:0/1188812929 learned_addr learned my addr 172.21.15.12:0/1188812929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:47.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890e2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.072+0000 7f7ce9400700 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.073+0000 7f7cea800700 1 -- 172.21.15.12:0/1188812929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 msgr2=0x55d1890edad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.073+0000 7f7cea800700 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.073+0000 7f7cea800700 1 -- 172.21.15.12:0/1188812929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 msgr2=0x55d1890e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.073+0000 7f7cea800700 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890e2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.073+0000 7f7cea800700 1 -- 172.21.15.12:0/1188812929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d188fca420 con 0x55d1890e3820 2024-05-27T07:25:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.074+0000 7f7cea800700 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3820 0x55d1890f0020 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009be0 tx=0x7f7ce4009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970cdc17be5a4d6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.074+0000 7f7cd7400700 1 -- 172.21.15.12:0/1188812929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4004bb0 con 0x55d1890e3820 2024-05-27T07:25:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.074+0000 7f7cd7400700 1 -- 172.21.15.12:0/1188812929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ce4004d50 con 0x55d1890e3820 2024-05-27T07:25:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.074+0000 7f7cd7400700 1 -- 172.21.15.12:0/1188812929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4013650 con 0x55d1890e3820 2024-05-27T07:25:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.075+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1188812929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3820 msgr2=0x55d1890f0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.075+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3820 0x55d1890f0020 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009be0 tx=0x7f7ce4009ef0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1188812929 shutdown_connections 2024-05-27T07:25:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890e2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3820 0x55d1890f0020 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1188812929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1188812929 >> 172.21.15.12:0/1188812929 conn(0x55d188fc4fc0 msgr2=0x55d1890ded30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1188812929 shutdown_connections 2024-05-27T07:25:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1188812929 wait complete. 2024-05-27T07:25:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 -- start start 2024-05-27T07:25:47.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.076+0000 7f7cffb61b80 1 Processor -- start 2024-05-27T07:25:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.077+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890ed570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.077+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.077+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890edff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.077+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d188f07fa0 con 0x55d188f64e40 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.077+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d188fa2760 con 0x55d1890e3820 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.077+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d1890f0990 con 0x55d1890e3410 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.077+0000 7f7cffb61b80 10 client.0 _ll_register_callbacks cb 0x55d188fabc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.078+0000 7f7ce9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.078+0000 7f7ce9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890ed570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56144/0 (socket says 172.21.15.12:56144) 2024-05-27T07:25:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.078+0000 7f7ce9e00700 1 -- 172.21.15.12:0/122890154 learned_addr learned my addr 172.21.15.12:0/122890154 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.078+0000 7f7ce9400700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.078+0000 7f7cea800700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ce9e00700 1 -- 172.21.15.12:0/122890154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 msgr2=0x55d1890edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ce9e00700 1 -- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 msgr2=0x55d1890edff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890edff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ce9e00700 1 -- 172.21.15.12:0/122890154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d188f518d0 con 0x55d188f64e40 2024-05-27T07:25:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7cea800700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890ed570 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00ffc0 tx=0x7f7cdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed372061c7ab7d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ccbe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.079+0000 7f7ccbe00700 1 -- 172.21.15.12:0/122890154 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc028670 con 0x55d188f64e40 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.080+0000 7f7ccbe00700 1 -- 172.21.15.12:0/122890154 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f7cdc028810 con 0x55d188f64e40 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.080+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ce4004d50 con 0x55d188f64e40 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.080+0000 7f7ccbe00700 1 -- 172.21.15.12:0/122890154 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7cdc031050 con 0x55d188f64e40 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.080+0000 7f7cffb61b80 10 client.15789 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.080+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d188fd2060 con 0x55d188f64e40 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.080+0000 7f7ccbe00700 1 -- 172.21.15.12:0/122890154 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ce4004d50 con 0x55d188f64e40 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.081+0000 7f7ccbe00700 1 -- 172.21.15.12:0/122890154 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7cdc01e430 con 0x55d188f64e40 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.081+0000 7f7cffb61b80 10 client.15789 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:47.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.081+0000 7f7cffb61b80 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.081+0000 7f7cffb61b80 20 client.15789 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.081+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d1890fd8d0 0x55d1890ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.081+0000 7f7cffb61b80 4 client.15789 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.081+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d189013d90 con 0x55d1890fd8d0 2024-05-27T07:25:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.082+0000 7f7ce9400700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d1890fd8d0 0x55d1890ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.083+0000 7f7ce9400700 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d1890fd8d0 0x55d1890ffd70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2ee46a127079244 server_cookie=67a131dd711e7512 in_seq=0 out_seq=0 2024-05-27T07:25:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.083+0000 7f7ccbe00700 10 client.15789 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7ccbe00700 1 -- 172.21.15.12:0/122890154 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d189013d90 con 0x55d1890fd8d0 2024-05-27T07:25:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7ccbe00700 10 client.15789 handle_command_reply: tid=0 2024-05-27T07:25:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 client.15789 shutdown 2024-05-27T07:25:47.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 msgr2=0x55d1890ed570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890ed570 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00ffc0 tx=0x7f7cdc004750 comp rx=0 tx=0).stop 2024-05-27T07:25:47.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 shutdown_connections 2024-05-27T07:25:47.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d1890fd8d0 0x55d1890ffd70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890ed570 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/122890154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.087+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 >> 172.21.15.12:0/122890154 conn(0x55d188fc4fc0 msgr2=0x55d1890ddff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.088+0000 7f7ccbe00700 0 client.15789 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.088+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 shutdown_connections 2024-05-27T07:25:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.088+0000 7f7cffb61b80 1 -- 172.21.15.12:0/122890154 wait complete. 2024-05-27T07:25:47.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.088+0000 7f7cffb61b80 20 client.15789 trim_cache size 0 max 16384 2024-05-27T07:25:47.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.113+0000 7f7cffb61b80 1 Processor -- start 2024-05-27T07:25:47.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7cffb61b80 1 -- start start 2024-05-27T07:25:47.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3410 0x55d1890f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3820 0x55d1890eda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1b4d80 con 0x55d1890e3410 2024-05-27T07:25:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1b2610 con 0x55d188f64e40 2024-05-27T07:25:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1b0460 con 0x55d1890e3820 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.114+0000 7f7ce9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3410 0x55d1890f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3410 0x55d1890f9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56152/0 (socket says 172.21.15.12:56152) 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 -- 172.21.15.12:0/1352357737 learned_addr learned my addr 172.21.15.12:0/1352357737 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7cea800700 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3820 0x55d1890eda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 -- 172.21.15.12:0/1352357737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3820 msgr2=0x55d1890eda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3820 0x55d1890eda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 -- 172.21.15.12:0/1352357737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 msgr2=0x55d1890ef8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890ef8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 -- 172.21.15.12:0/1352357737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1890ec4d0 con 0x55d1890e3410 2024-05-27T07:25:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.115+0000 7f7ce9400700 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3410 0x55d1890f9a30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80a79e0 tx=0x7f7cd8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5b6756a25085a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.116+0000 7f7cd6a00700 1 -- 172.21.15.12:0/1352357737 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80aa770 con 0x55d1890e3410 2024-05-27T07:25:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.116+0000 7f7cd6a00700 1 -- 172.21.15.12:0/1352357737 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cd80aeb20 con 0x55d1890e3410 2024-05-27T07:25:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.116+0000 7f7cd6a00700 1 -- 172.21.15.12:0/1352357737 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80aac30 con 0x55d1890e3410 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1352357737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3410 msgr2=0x55d1890f9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3410 0x55d1890f9a30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80a79e0 tx=0x7f7cd8009310 comp rx=0 tx=0).stop 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1352357737 shutdown_connections 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d188f64e40 0x55d1890ef8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3820 0x55d1890eda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/1352357737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d1890e3410 0x55d1890f9a30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1352357737 >> 172.21.15.12:0/1352357737 conn(0x55d188fc4fc0 msgr2=0x55d1890ed350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1352357737 shutdown_connections 2024-05-27T07:25:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.117+0000 7f7cffb61b80 1 -- 172.21.15.12:0/1352357737 wait complete. 2024-05-27T07:25:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.118+0000 7f7cffb61b80 1 -- start start 2024-05-27T07:25:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.118+0000 7f7cffb61b80 1 Processor -- start 2024-05-27T07:25:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.118+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890f1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.118+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.118+0000 7f7cffb61b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890f2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.118+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1b0ad0 con 0x55d188f64e40 2024-05-27T07:25:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1b1f40 con 0x55d1890e3820 2024-05-27T07:25:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7cffb61b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7cec1b11a0 con 0x55d1890e3410 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7cffb61b80 10 client.0 _ll_register_callbacks cb 0x55d188fabc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890f1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56164/0 (socket says 172.21.15.12:56164) 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9e00700 1 -- 172.21.15.12:0/70389773 learned_addr learned my addr 172.21.15.12:0/70389773 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9400700 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9e00700 1 -- 172.21.15.12:0/70389773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 msgr2=0x55d1890f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9e00700 1 -- 172.21.15.12:0/70389773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 msgr2=0x55d1890f2390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.119+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890f2390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.120+0000 7f7ce9e00700 1 -- 172.21.15.12:0/70389773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d189167490 con 0x55d188f64e40 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.120+0000 7f7ce9400700 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.120+0000 7f7ce9e00700 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890f1910 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc01e820 tx=0x7f7cdc027610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91c33b1709f1ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.120+0000 7f7ccbe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.120+0000 7f7ccbe00700 1 -- 172.21.15.12:0/70389773 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc01eea0 con 0x55d188f64e40 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.120+0000 7f7ccbe00700 1 -- 172.21.15.12:0/70389773 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9466+0+0 (secure 0 0 0) 0x7f7cdc005040 con 0x55d188f64e40 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.120+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd80aeb20 con 0x55d188f64e40 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.121+0000 7f7ccbe00700 1 -- 172.21.15.12:0/70389773 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7cdc043050 con 0x55d188f64e40 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.121+0000 7f7cffb61b80 10 client.15795 fetch_fsmap learned FSMap version 28 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.121+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d189168690 con 0x55d188f64e40 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.121+0000 7f7ccbe00700 1 -- 172.21.15.12:0/70389773 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cd80aeb20 con 0x55d188f64e40 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.121+0000 7f7ccbe00700 1 -- 172.21.15.12:0/70389773 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7cdc034040 con 0x55d188f64e40 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.122+0000 7f7cffb61b80 10 client.15795 fetch_fsmap finished waiting for FSMap version 28 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.122+0000 7f7cffb61b80 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:47.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.122+0000 7f7cffb61b80 20 client.15795 populate_metadata read hostname 'smithi012' 2024-05-27T07:25:47.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.122+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d189168e20 0x55d18916b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.122+0000 7f7cffb61b80 4 client.15795 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:25:47.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.122+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d18900d570 con 0x55d189168e20 2024-05-27T07:25:47.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.123+0000 7f7ce9400700 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d189168e20 0x55d18916b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.124+0000 7f7ce9400700 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d189168e20 0x55d18916b240 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97fd211a771f8f4 server_cookie=5a9360ab19131795 in_seq=0 out_seq=0 2024-05-27T07:25:47.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.124+0000 7f7ccbe00700 10 client.15795 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7ccbe00700 1 -- 172.21.15.12:0/70389773 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d18900d570 con 0x55d189168e20 2024-05-27T07:25:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7ccbe00700 10 client.15795 handle_command_reply: tid=0 2024-05-27T07:25:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 client.15795 shutdown 2024-05-27T07:25:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 msgr2=0x55d1890f1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890f1910 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc01e820 tx=0x7f7cdc027610 comp rx=0 tx=0).stop 2024-05-27T07:25:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 shutdown_connections 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d189168e20 0x55d18916b240 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d1890e3820 0x55d1890f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d188f64e40 0x55d1890f1910 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 --2- 172.21.15.12:0/70389773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d1890e3410 0x55d1890f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7ccbe00700 0 client.15795 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 >> 172.21.15.12:0/70389773 conn(0x55d188fc4fc0 msgr2=0x55d1890edd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.125+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 shutdown_connections 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.126+0000 7f7cffb61b80 1 -- 172.21.15.12:0/70389773 wait complete. 2024-05-27T07:25:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.126+0000 7f7cffb61b80 20 client.15795 trim_cache size 0 max 16384 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "096ae71a-009d-45c4-9152-ec84dbbf437d": { 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "096ae71a-009d-45c4-9152-ec84dbbf437d", 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:25:47.118 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:25:47.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.126+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7cd00b2a70 msgr2=0x7f7cd00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7cd00b2a70 0x7f7cd00b4f80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc005950 tx=0x7f7cdc0058e0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 msgr2=0x7f7cec19d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec19d340 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800b580 tx=0x7f7cd800f460 comp rx=0 tx=0).stop 2024-05-27T07:25:47.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 shutdown_connections 2024-05-27T07:25:47.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7cd00b2a70 0x7f7cd00b4f80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cec10b860 0x7f7cec1a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cec106630 0x7f7cec19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 --2- 172.21.15.12:0/736634902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cec10af10 0x7f7cec19d340 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 >> 172.21.15.12:0/736634902 conn(0x7f7cec102020 msgr2=0x7f7cec10a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 shutdown_connections 2024-05-27T07:25:47.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:25:47.127+0000 7f7ceb200700 1 -- 172.21.15.12:0/736634902 wait complete. 2024-05-27T07:25:48.321 INFO:tasks.cephfs.filesystem:scrub status for tag:096ae71a-009d-45c4-9152-ec84dbbf437d - {'path': '/', 'tag': '096ae71a-009d-45c4-9152-ec84dbbf437d', 'options': 'recursive,force'} 2024-05-27T07:25:48.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:48 smithi012 ceph-mon[31473]: pgmap v105: 129 pgs: 129 active+clean; 49 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 310 KiB/s wr, 67 op/s 2024-05-27T07:25:48.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:48 smithi162 ceph-mon[35676]: pgmap v105: 129 pgs: 129 active+clean; 49 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 310 KiB/s wr, 67 op/s 2024-05-27T07:25:48.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:48 smithi062 ceph-mon[35304]: pgmap v105: 129 pgs: 129 active+clean; 49 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 310 KiB/s wr, 67 op/s 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: Stopped Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.470s CPU time. 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: Starting Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: Stopped Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.620s CPU time. 2024-05-27T07:25:50.154 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:49 smithi162 systemd[1]: Starting Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:25:50.166 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:49 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:25:50.166 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:49 smithi062 systemd[1]: Stopped Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:25:50.166 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:49 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 6.580s CPU time. 2024-05-27T07:25:50.166 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:49 smithi062 systemd[1]: Starting Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:25:50.467 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:50 smithi162 ceph-mon[35676]: pgmap v106: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 570 KiB/s wr, 118 op/s 2024-05-27T07:25:50.468 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 podman[71128]: 2024-05-27 07:25:50.069401873 +0000 UTC m=+0.043181839 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:25:50.468 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 podman[71128]: 2024-05-27 07:25:50.292999372 +0000 UTC m=+0.266779378 container create 94e559b26219f72357f643f0fccc2d747c759fec43054579b97b34f75179173e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:25:50.468 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 podman[71127]: 2024-05-27 07:25:50.069169766 +0000 UTC m=+0.045131086 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:25:50.468 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 podman[71127]: 2024-05-27 07:25:50.187072494 +0000 UTC m=+0.163033856 container create 4294ab8a8b6369572c8ce900c722907bb50812c1dc505224663c781fe5b151f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:25:50.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:50 smithi062 ceph-mon[35304]: pgmap v106: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 570 KiB/s wr, 118 op/s 2024-05-27T07:25:50.485 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:50 smithi062 podman[70582]: 2024-05-27 07:25:50.081810774 +0000 UTC m=+0.043043709 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:25:50.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:50 smithi012 ceph-mon[31473]: pgmap v106: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 570 KiB/s wr, 118 op/s 2024-05-27T07:25:50.948 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 podman[71128]: 2024-05-27 07:25:50.641926684 +0000 UTC m=+0.615706731 container init 94e559b26219f72357f643f0fccc2d747c759fec43054579b97b34f75179173e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:25:50.948 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 podman[71128]: 2024-05-27 07:25:50.651604292 +0000 UTC m=+0.625384301 container start 94e559b26219f72357f643f0fccc2d747c759fec43054579b97b34f75179173e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 ceph-mds[71163]: -- 172.21.15.162:0/1200889785 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c1be7b3a40 con 0x55c1be7fb600 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: starting mds.c at 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 podman[71127]: 2024-05-27 07:25:50.742110059 +0000 UTC m=+0.718071477 container init 4294ab8a8b6369572c8ce900c722907bb50812c1dc505224663c781fe5b151f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 podman[71127]: 2024-05-27 07:25:50.752190292 +0000 UTC m=+0.728151657 container start 4294ab8a8b6369572c8ce900c722907bb50812c1dc505224663c781fe5b151f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 bash[71127]: 4294ab8a8b6369572c8ce900c722907bb50812c1dc505224663c781fe5b151f3 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 systemd[1]: Started Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 ceph-mds[71165]: -- 172.21.15.162:0/2278619830 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5620fa8cfa40 con 0x5620fa917180 2024-05-27T07:25:50.949 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: starting mds.i at 2024-05-27T07:25:51.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 bash[71128]: 94e559b26219f72357f643f0fccc2d747c759fec43054579b97b34f75179173e 2024-05-27T07:25:51.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:50 smithi162 systemd[1]: Started Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:25:51.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:51 smithi062 ceph-mon[35304]: daemon mds.c restarted 2024-05-27T07:25:51.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:51 smithi062 podman[70582]: 2024-05-27 07:25:51.195420264 +0000 UTC m=+1.156653252 container create fdd929c675ad42374dfb9163c40b16440cde630136d3fbf0ad3a9f3dfcdf2ffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:25:51.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-mon[35676]: daemon mds.c restarted 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7efcd3200700 time 2024-05-27T07:25:51.515327+0000 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efce8cd1ba6] 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.582 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.583 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.584 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.584 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: clone() 2024-05-27T07:25:51.584 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.517+0000 7efcd3200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7efcd3200700 time 2024-05-27T07:25:51.515327+0000 2024-05-27T07:25:51.585 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.585 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.585 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.585 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efce8cd1ba6] 2024-05-27T07:25:51.585 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.585 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.585 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.586 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: clone() 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: *** Caught signal (Aborted) ** 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: in thread 7efcd3200700 thread_name:md_log_replay 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce784ed20] 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: gsignal() 2024-05-27T07:25:51.587 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: abort() 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efce8cd1ced] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.588 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 17: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 19: clone() 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.521+0000 7efcd3200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: in thread 7efcd3200700 thread_name:md_log_replay 2024-05-27T07:25:51.589 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce784ed20] 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: gsignal() 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: abort() 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efce8cd1ced] 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 17: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.591 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 19: clone() 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: -1> 2024-05-27T07:25:51.517+0000 7efcd3200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7efcd3200700 time 2024-05-27T07:25:51.515327+0000 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efce8cd1ba6] 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.592 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.593 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.593 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.594 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.594 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.594 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:51 smithi012 ceph-mon[31473]: daemon mds.c restarted 2024-05-27T07:25:51.595 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.595 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: clone() 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 0> 2024-05-27T07:25:51.521+0000 7efcd3200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:51.596 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: in thread 7efcd3200700 thread_name:md_log_replay 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce784ed20] 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: gsignal() 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: abort() 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efce8cd1ced] 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.597 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 17: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.598 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 19: clone() 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: -1> 2024-05-27T07:25:51.517+0000 7efcd3200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7efcd3200700 time 2024-05-27T07:25:51.515327+0000 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efce8cd1ba6] 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.599 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: clone() 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 0> 2024-05-27T07:25:51.521+0000 7efcd3200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:51.600 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: in thread 7efcd3200700 thread_name:md_log_replay 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce784ed20] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2: gsignal() 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 3: abort() 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efce8cd1ced] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efce8cd1dd1] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620df4f123d] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620df4dc971] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620df4dafa0] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620df4db651] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620df470347] 2024-05-27T07:25:51.601 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620df16c87e] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620df171e94] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620df17c969] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620df51548c] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620df51e9cb] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 16: (Thread::entry_wrapper()+0x43) [0x7efce8ca072d] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 17: (Thread::_entry_func(void*)+0xd) [0x7efce8ca0749] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7efce78441ca] 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 19: clone() 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:51.602 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71159]: 2024-05-27T07:25:51.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:51 smithi162 podman[71237]: 2024-05-27 07:25:51.695022357 +0000 UTC m=+0.047184696 container died 4294ab8a8b6369572c8ce900c722907bb50812c1dc505224663c781fe5b151f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbeafe00700 time 2024-05-27T07:25:51.602791+0000 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbec596fba6] 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: clone() 2024-05-27T07:25:51.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: *** Caught signal (Aborted) ** 2024-05-27T07:25:51.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: in thread 7fbeafe00700 thread_name:md_log_replay 2024-05-27T07:25:51.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.605+0000 7fbeafe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbeafe00700 time 2024-05-27T07:25:51.602791+0000 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbec596fba6] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.978 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: clone() 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbec44ecd20] 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: gsignal() 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: abort() 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbec596fced] 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 19: clone() 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.609+0000 7fbeafe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: in thread 7fbeafe00700 thread_name:md_log_replay 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbec44ecd20] 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: gsignal() 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: abort() 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbec596fced] 2024-05-27T07:25:51.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 19: clone() 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: -1> 2024-05-27T07:25:51.605+0000 7fbeafe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbeafe00700 time 2024-05-27T07:25:51.602791+0000 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbec596fba6] 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: clone() 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 0> 2024-05-27T07:25:51.609+0000 7fbeafe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: in thread 7fbeafe00700 thread_name:md_log_replay 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbec44ecd20] 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: gsignal() 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: abort() 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbec596fced] 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 19: clone() 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: -1> 2024-05-27T07:25:51.605+0000 7fbeafe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbeafe00700 time 2024-05-27T07:25:51.602791+0000 2024-05-27T07:25:51.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbec596fba6] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: clone() 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:51 smithi062 podman[70582]: 2024-05-27 07:25:51.509419779 +0000 UTC m=+1.470652787 container init fdd929c675ad42374dfb9163c40b16440cde630136d3fbf0ad3a9f3dfcdf2ffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:51 smithi062 podman[70582]: 2024-05-27 07:25:51.520406648 +0000 UTC m=+1.481639634 container start fdd929c675ad42374dfb9163c40b16440cde630136d3fbf0ad3a9f3dfcdf2ffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:51 smithi062 bash[70582]: fdd929c675ad42374dfb9163c40b16440cde630136d3fbf0ad3a9f3dfcdf2ffe 2024-05-27T07:25:51.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:51 smithi062 systemd[1]: Started Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:51 smithi062 ceph-mds[70599]: -- 172.21.15.62:0/4207665081 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556d0cec5a40 con 0x556d0cf0d180 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:25:51 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: starting mds.k at 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 0> 2024-05-27T07:25:51.609+0000 7fbeafe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: in thread 7fbeafe00700 thread_name:md_log_replay 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbec44ecd20] 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2: gsignal() 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 3: abort() 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbec596fced] 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbec596fdd1] 2024-05-27T07:25:51.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c19a79623d] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c19a781971] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c19a77ffa0] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c19a780651] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c19a715347] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c19a41187e] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c19a416e94] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c19a421969] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c19a7ba48c] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c19a7c39cb] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 16: (Thread::entry_wrapper()+0x43) [0x7fbec593e72d] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbec593e749] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbec44e21ca] 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 19: clone() 2024-05-27T07:25:51.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:51.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71153]: 2024-05-27T07:25:52.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: pgmap v107: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 433 KiB/s wr, 100 op/s 2024-05-27T07:25:52.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:25:52.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/2068967103,v1:172.21.15.162:6837/2068967103] up:boot 2024-05-27T07:25:52.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: daemon mds.i restarted 2024-05-27T07:25:52.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:25:52.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:25:52.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:25:52.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6838/3307824625,v1:172.21.15.162:6839/3307824625] up:boot 2024-05-27T07:25:52.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:25:52.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:25:52.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:25:52.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:52 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:25:52.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:52 smithi162 podman[71246]: 2024-05-27 07:25:52.479320538 +0000 UTC m=+0.764838738 container died 94e559b26219f72357f643f0fccc2d747c759fec43054579b97b34f75179173e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:25:52.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:52 smithi162 podman[71237]: 2024-05-27 07:25:52.372888665 +0000 UTC m=+0.725051044 container remove 4294ab8a8b6369572c8ce900c722907bb50812c1dc505224663c781fe5b151f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:25:52.726 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:52 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:25:52.726 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f44e1800700 time 2024-05-27T07:25:52.374295+0000 2024-05-27T07:25:52.726 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:52.726 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.726 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f44f732aba6] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: clone() 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.376+0000 7f44e1800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f44e1800700 time 2024-05-27T07:25:52.374295+0000 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: pgmap v107: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 433 KiB/s wr, 100 op/s 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/2068967103,v1:172.21.15.162:6837/2068967103] up:boot 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: daemon mds.i restarted 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:25:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6838/3307824625,v1:172.21.15.162:6839/3307824625] up:boot 2024-05-27T07:25:52.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:25:52.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:25:52.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:25:52.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:52 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f44f732aba6] 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.730 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: clone() 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: *** Caught signal (Aborted) ** 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: in thread 7f44e1800700 thread_name:md_log_replay 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f44f5ea7d20] 2024-05-27T07:25:52.731 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: gsignal() 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: abort() 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f44f732aced] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.732 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 17: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 19: clone() 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.379+0000 7f44e1800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: in thread 7f44e1800700 thread_name:md_log_replay 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f44f5ea7d20] 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: gsignal() 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: abort() 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f44f732aced] 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.733 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 17: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 19: clone() 2024-05-27T07:25:52.734 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: -1> 2024-05-27T07:25:52.376+0000 7f44e1800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f44e1800700 time 2024-05-27T07:25:52.374295+0000 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f44f732aba6] 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.735 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: clone() 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 0> 2024-05-27T07:25:52.379+0000 7f44e1800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: in thread 7f44e1800700 thread_name:md_log_replay 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.736 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f44f5ea7d20] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: gsignal() 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: abort() 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f44f732aced] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.737 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 17: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 19: clone() 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: -1> 2024-05-27T07:25:52.376+0000 7f44e1800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f44e1800700 time 2024-05-27T07:25:52.374295+0000 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f44f732aba6] 2024-05-27T07:25:52.738 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.739 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: clone() 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 0> 2024-05-27T07:25:52.379+0000 7f44e1800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: in thread 7f44e1800700 thread_name:md_log_replay 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f44f5ea7d20] 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2: gsignal() 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 3: abort() 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f44f732aced] 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f44f732add1] 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55839c59923d] 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55839c584971] 2024-05-27T07:25:52.740 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 8: (SnapRealm::check_cache() const+0x13e) [0x55839c582fa0] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55839c583651] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55839c518347] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55839c21487e] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55839c219e94] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55839c224969] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 14: (MDLog::_replay_thread()+0x1f38) [0x55839c5bd48c] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55839c5c69cb] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 16: (Thread::entry_wrapper()+0x43) [0x7f44f72f972d] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 17: (Thread::_entry_func(void*)+0xd) [0x7f44f72f9749] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f44f5e9d1ca] 2024-05-27T07:25:52.741 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 19: clone() 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[66025]: 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: pgmap v107: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 433 KiB/s wr, 100 op/s 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/2068967103,v1:172.21.15.162:6837/2068967103] up:boot 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: daemon mds.i restarted 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:25:52.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:25:52.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:25:52.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6838/3307824625,v1:172.21.15.162:6839/3307824625] up:boot 2024-05-27T07:25:52.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:25:52.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:25:52.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:25:52.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:52 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:25:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:53 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6832/2790830298,v1:172.21.15.62:6833/2790830298] up:boot 2024-05-27T07:25:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:53 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:25:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:53 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:25:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:53 smithi162 ceph-mon[35676]: pgmap v110: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 390 KiB/s wr, 76 op/s 2024-05-27T07:25:53.473 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:53 smithi162 podman[71246]: 2024-05-27 07:25:53.194989995 +0000 UTC m=+1.480508201 container remove 94e559b26219f72357f643f0fccc2d747c759fec43054579b97b34f75179173e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:25:53.474 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:53 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:25:53.474 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:53 smithi162 podman[71261]: 2024-05-27 07:25:53.21354046 +0000 UTC m=+0.790900058 container died ebb2488adfa96edf6a986f3a5fdae35b7f280bde596321f51c981f12b3c6080d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:25:53.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:53 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6832/2790830298,v1:172.21.15.62:6833/2790830298] up:boot 2024-05-27T07:25:53.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:53 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:25:53.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:53 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:25:53.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:53 smithi062 ceph-mon[35304]: pgmap v110: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 390 KiB/s wr, 76 op/s 2024-05-27T07:25:53.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:53 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6832/2790830298,v1:172.21.15.62:6833/2790830298] up:boot 2024-05-27T07:25:53.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:53 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:25:53.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:53 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:25:53.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:53 smithi012 ceph-mon[31473]: pgmap v110: 129 pgs: 129 active+clean; 51 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 390 KiB/s wr, 76 op/s 2024-05-27T07:25:54.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:54 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:25:54.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:25:54 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.670s CPU time. 2024-05-27T07:25:54.223 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:53 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-27T07:25:54.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:25:53 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 1.632s CPU time. 2024-05-27T07:25:54.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:53 smithi162 podman[71270]: 2024-05-27 07:25:53.933591347 +0000 UTC m=+1.465248476 container remove ebb2488adfa96edf6a986f3a5fdae35b7f280bde596321f51c981f12b3c6080d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:25:54.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:53 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:25:54.723 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:54 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:25:54.723 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:25:54 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 3.576s CPU time. 2024-05-27T07:25:55.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:55 smithi012 ceph-mon[31473]: pgmap v111: 129 pgs: 129 active+clean; 54 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 776 KiB/s wr, 137 op/s 2024-05-27T07:25:55.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:55 smithi162 ceph-mon[35676]: pgmap v111: 129 pgs: 129 active+clean; 54 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 776 KiB/s wr, 137 op/s 2024-05-27T07:25:55.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:55 smithi062 ceph-mon[35304]: pgmap v111: 129 pgs: 129 active+clean; 54 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 776 KiB/s wr, 137 op/s 2024-05-27T07:25:56.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:56 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:25:56.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:56 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:25:56.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:56 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:25:56.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:56 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:25:56.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:56 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:25:56.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:56 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:25:56.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:56 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:25:56.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:56 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:25:56.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:56 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:25:57.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:57 smithi012 ceph-mon[31473]: pgmap v112: 129 pgs: 129 active+clean; 54 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 386 KiB/s wr, 60 op/s 2024-05-27T07:25:57.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:57 smithi162 ceph-mon[35676]: pgmap v112: 129 pgs: 129 active+clean; 54 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 386 KiB/s wr, 60 op/s 2024-05-27T07:25:57.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:57 smithi062 ceph-mon[35304]: pgmap v112: 129 pgs: 129 active+clean; 54 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 386 KiB/s wr, 60 op/s 2024-05-27T07:25:59.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:25:59 smithi012 ceph-mon[31473]: pgmap v113: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 780 KiB/s wr, 117 op/s 2024-05-27T07:25:59.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:25:59 smithi162 ceph-mon[35676]: pgmap v113: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 780 KiB/s wr, 117 op/s 2024-05-27T07:25:59.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:25:59 smithi062 ceph-mon[35304]: pgmap v113: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 780 KiB/s wr, 117 op/s 2024-05-27T07:26:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:01 smithi012 ceph-mon[31473]: pgmap v114: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 686 KiB/s wr, 103 op/s 2024-05-27T07:26:01.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:01 smithi162 ceph-mon[35676]: pgmap v114: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 686 KiB/s wr, 103 op/s 2024-05-27T07:26:01.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:01 smithi062 ceph-mon[35304]: pgmap v114: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 686 KiB/s wr, 103 op/s 2024-05-27T07:26:03.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:03 smithi012 ceph-mon[31473]: pgmap v115: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 564 KiB/s wr, 84 op/s 2024-05-27T07:26:03.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:03 smithi162 ceph-mon[35676]: pgmap v115: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 564 KiB/s wr, 84 op/s 2024-05-27T07:26:03.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:03 smithi062 ceph-mon[35304]: pgmap v115: 129 pgs: 129 active+clean; 57 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 564 KiB/s wr, 84 op/s 2024-05-27T07:26:04.243 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:26:04.243 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: Stopped Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:04.243 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.670s CPU time. 2024-05-27T07:26:04.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:03 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:26:04.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:03 smithi162 systemd[1]: Stopped Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:04.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:03 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 1.632s CPU time. 2024-05-27T07:26:04.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: Starting Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:04.608 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: Starting Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:04.608 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:26:04.608 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: Stopped Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:04.608 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 3.576s CPU time. 2024-05-27T07:26:04.608 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: Starting Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:04.608 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 podman[71413]: 2024-05-27 07:26:04.298343287 +0000 UTC m=+0.038277419 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:04.609 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 podman[71413]: 2024-05-27 07:26:04.425429492 +0000 UTC m=+0.165363650 container create a86ed0ef4a70439194bdea083df423c79b186af41f83e144263a7490c6025fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:04.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 podman[71413]: 2024-05-27 07:26:04.712738942 +0000 UTC m=+0.452673088 container init a86ed0ef4a70439194bdea083df423c79b186af41f83e144263a7490c6025fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:26:04.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 podman[71413]: 2024-05-27 07:26:04.72194779 +0000 UTC m=+0.461881955 container start a86ed0ef4a70439194bdea083df423c79b186af41f83e144263a7490c6025fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:26:04.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 bash[71413]: a86ed0ef4a70439194bdea083df423c79b186af41f83e144263a7490c6025fc3 2024-05-27T07:26:04.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 systemd[1]: Started Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:04.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 ceph-mds[71470]: -- 172.21.15.162:0/1996477381 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d0eddda40 con 0x555d0ee25180 2024-05-27T07:26:04.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:26:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: starting mds.i at 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to '0e5c12dfd008efc2848c98108c9237487e91ef35'. 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T07:26:05.124 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T07:26:05.125 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-05-27T07:26:05.131 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at 0e5c12df xfs/522: use reflink instead of crc as test feature 2024-05-27T07:26:05.175 INFO:tasks.workunit.client.0.smithi012.stderr:+ make -j4 2024-05-27T07:26:05.271 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize -cfi 2024-05-27T07:26:05.364 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 podman[71563]: 2024-05-27 07:26:05.243032778 +0000 UTC m=+0.221877431 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:05.365 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 podman[71538]: 2024-05-27 07:26:05.005655824 +0000 UTC m=+0.038362388 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:05.365 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 podman[71538]: 2024-05-27 07:26:05.11363479 +0000 UTC m=+0.146341329 container create f6f0fc672e8b94bbfd58886fc1e32b7795c8f2175f03215e6bc948f57d028419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-27T07:26:05.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:05 smithi162 ceph-mon[35676]: pgmap v116: 129 pgs: 129 active+clean; 60 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 814 KiB/s wr, 129 op/s 2024-05-27T07:26:05.620 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:05 smithi162 ceph-mon[35676]: daemon mds.i restarted 2024-05-27T07:26:05.621 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 podman[71563]: 2024-05-27 07:26:05.374893374 +0000 UTC m=+0.353738020 container create 4644c7c268c7898ef5e2665fb70a2a031d95ff3dc3e80694ce5c842b6c8e857c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True) 2024-05-27T07:26:05.621 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 podman[71538]: 2024-05-27 07:26:05.488775315 +0000 UTC m=+0.521481934 container init f6f0fc672e8b94bbfd58886fc1e32b7795c8f2175f03215e6bc948f57d028419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:26:05.621 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 podman[71538]: 2024-05-27 07:26:05.500050941 +0000 UTC m=+0.532757476 container start f6f0fc672e8b94bbfd58886fc1e32b7795c8f2175f03215e6bc948f57d028419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:26:05.824 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:05 smithi062 ceph-mon[35304]: pgmap v116: 129 pgs: 129 active+clean; 60 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 814 KiB/s wr, 129 op/s 2024-05-27T07:26:05.824 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:05 smithi062 ceph-mon[35304]: daemon mds.i restarted 2024-05-27T07:26:05.825 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9f2fe00700 time 2024-05-27T07:26:05.708727+0000 2024-05-27T07:26:05.825 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f458e1ba6] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.826 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: clone() 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.710+0000 7f9f2fe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9f2fe00700 time 2024-05-27T07:26:05.708727+0000 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f458e1ba6] 2024-05-27T07:26:05.827 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.828 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: clone() 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: *** Caught signal (Aborted) ** 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: in thread 7f9f2fe00700 thread_name:md_log_replay 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f4445ed20] 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: gsignal() 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: abort() 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f458e1ced] 2024-05-27T07:26:05.829 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.830 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 19: clone() 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.713+0000 7f9f2fe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: in thread 7f9f2fe00700 thread_name:md_log_replay 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f4445ed20] 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: gsignal() 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: abort() 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f458e1ced] 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.831 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.832 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 19: clone() 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: -1> 2024-05-27T07:26:05.710+0000 7f9f2fe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9f2fe00700 time 2024-05-27T07:26:05.708727+0000 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f458e1ba6] 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.833 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.834 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.834 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.834 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.834 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.834 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: clone() 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 0> 2024-05-27T07:26:05.713+0000 7f9f2fe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: in thread 7f9f2fe00700 thread_name:md_log_replay 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f4445ed20] 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: gsignal() 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: abort() 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f458e1ced] 2024-05-27T07:26:05.835 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.836 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 19: clone() 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: -1> 2024-05-27T07:26:05.710+0000 7f9f2fe00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9f2fe00700 time 2024-05-27T07:26:05.708727+0000 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f458e1ba6] 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.837 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: clone() 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 0> 2024-05-27T07:26:05.713+0000 7f9f2fe00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: in thread 7f9f2fe00700 thread_name:md_log_replay 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.838 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f4445ed20] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2: gsignal() 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 3: abort() 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f458e1ced] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f458e1dd1] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556cf263823d] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556cf2623971] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 8: (SnapRealm::check_cache() const+0x13e) [0x556cf2621fa0] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556cf2622651] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556cf25b7347] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556cf22b387e] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556cf22b8e94] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556cf22c3969] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 14: (MDLog::_replay_thread()+0x1f38) [0x556cf265c48c] 2024-05-27T07:26:05.839 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556cf26659cb] 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f458b072d] 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f458b0749] 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f444541ca] 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 19: clone() 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70595]: 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:05 smithi062 podman[70630]: 2024-05-27 07:26:05.835428015 +0000 UTC m=+0.034575025 container died fdd929c675ad42374dfb9163c40b16440cde630136d3fbf0ad3a9f3dfcdf2ffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T07:26:05.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:05 smithi012 ceph-mon[31473]: pgmap v116: 129 pgs: 129 active+clean; 60 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 814 KiB/s wr, 129 op/s 2024-05-27T07:26:05.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:05 smithi012 ceph-mon[31473]: daemon mds.i restarted 2024-05-27T07:26:05.935 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 ceph-mds[71582]: -- 172.21.15.162:0/1790970408 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b1ece1fa40 con 0x55b1ece67180 2024-05-27T07:26:05.936 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: starting mds.f at 2024-05-27T07:26:05.936 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 bash[71538]: f6f0fc672e8b94bbfd58886fc1e32b7795c8f2175f03215e6bc948f57d028419 2024-05-27T07:26:05.936 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:05 smithi162 systemd[1]: Started Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:05.936 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 podman[71563]: 2024-05-27 07:26:05.847362201 +0000 UTC m=+0.826206891 container init 4644c7c268c7898ef5e2665fb70a2a031d95ff3dc3e80694ce5c842b6c8e857c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T07:26:05.936 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 podman[71563]: 2024-05-27 07:26:05.857606407 +0000 UTC m=+0.836451023 container start 4644c7c268c7898ef5e2665fb70a2a031d95ff3dc3e80694ce5c842b6c8e857c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:26:06.012 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: putting auxiliary files in '.'. 2024-05-27T07:26:06.012 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file './config.guess' 2024-05-27T07:26:06.038 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file './config.sub' 2024-05-27T07:26:06.066 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file './install-sh' 2024-05-27T07:26:06.092 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file './ltmain.sh' 2024-05-27T07:26:06.221 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-05-27T07:26:06.221 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-05-27T07:26:06.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 bash[71563]: 4644c7c268c7898ef5e2665fb70a2a031d95ff3dc3e80694ce5c842b6c8e857c 2024-05-27T07:26:06.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 systemd[1]: Started Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:06.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 ceph-mds[71607]: -- 172.21.15.162:0/102903966 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f04bb63a40 con 0x55f04bbaba80 2024-05-27T07:26:06.224 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: starting mds.c at 2024-05-27T07:26:06.258 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-05-27T07:26:06.292 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-05-27T07:26:06.327 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-05-27T07:26:06.363 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-05-27T07:26:06.391 INFO:tasks.workunit.client.0.smithi012.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-05-27T07:26:06.397 INFO:tasks.workunit.client.0.smithi012.stdout:cp include/install-sh . 2024-05-27T07:26:06.637 INFO:tasks.workunit.client.0.smithi012.stdout:aclocal -I m4 2024-05-27T07:26:06.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:06 smithi162 ceph-mon[35676]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:26:06.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:06 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6834/2607883669,v1:172.21.15.162:6835/2607883669] up:boot 2024-05-27T07:26:06.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:06 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:06.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:06 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:26:06.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:06 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:06.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:06 smithi162 ceph-mon[35676]: daemon mds.f restarted 2024-05-27T07:26:06.821 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe331a00700 time 2024-05-27T07:26:06.819706+0000 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe347473ba6] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:06.822 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: clone() 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:06.820+0000 7fe331a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe331a00700 time 2024-05-27T07:26:06.819706+0000 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:06.823 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe347473ba6] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:06.824 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: clone() 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: *** Caught signal (Aborted) ** 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: in thread 7fe331a00700 thread_name:md_log_replay 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe345ff0d20] 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: gsignal() 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: abort() 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe347473ced] 2024-05-27T07:26:06.825 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:06.826 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:06.826 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:06.826 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:06.826 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:06.826 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:06.826 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 19: clone() 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:06.823+0000 7fe331a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: in thread 7fe331a00700 thread_name:md_log_replay 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:06.827 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe345ff0d20] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: gsignal() 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: abort() 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe347473ced] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:06.828 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:06.829 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:06.829 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:06.829 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:06.829 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 19: clone() 2024-05-27T07:26:06.829 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:06.829 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:06.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:06 smithi012 ceph-mon[31473]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:26:06.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:06 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6834/2607883669,v1:172.21.15.162:6835/2607883669] up:boot 2024-05-27T07:26:06.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:06 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:06.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:06 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:26:06.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:06 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:06.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:06 smithi012 ceph-mon[31473]: daemon mds.f restarted 2024-05-27T07:26:06.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:06 smithi062 ceph-mon[35304]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:26:06.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:06 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6834/2607883669,v1:172.21.15.162:6835/2607883669] up:boot 2024-05-27T07:26:06.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:06 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:06.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:06 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:26:06.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:06 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:06.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:06 smithi062 ceph-mon[35304]: daemon mds.f restarted 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: -1> 2024-05-27T07:26:06.820+0000 7fe331a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe331a00700 time 2024-05-27T07:26:06.819706+0000 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe347473ba6] 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:07.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: clone() 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 0> 2024-05-27T07:26:06.823+0000 7fe331a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:07.225 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: in thread 7fe331a00700 thread_name:md_log_replay 2024-05-27T07:26:07.226 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.226 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.226 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe345ff0d20] 2024-05-27T07:26:07.226 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: gsignal() 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: abort() 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe347473ced] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:07.227 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 19: clone() 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: -1> 2024-05-27T07:26:06.820+0000 7fe331a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe331a00700 time 2024-05-27T07:26:06.819706+0000 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe347473ba6] 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:07.228 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: clone() 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 0> 2024-05-27T07:26:06.823+0000 7fe331a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:07.229 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: in thread 7fe331a00700 thread_name:md_log_replay 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe345ff0d20] 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2: gsignal() 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 3: abort() 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe347473ced] 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe347473dd1] 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b1b52d623d] 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b1b52c1971] 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b1b52bffa0] 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b1b52c0651] 2024-05-27T07:26:07.230 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b1b5255347] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b1b4f5187e] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b1b4f56e94] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b1b4f61969] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b1b52fa48c] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b1b53039cb] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 16: (Thread::entry_wrapper()+0x43) [0x7fe34744272d] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe347442749] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe345fe61ca] 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 19: clone() 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71574]: 2024-05-27T07:26:07.231 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:06 smithi162 podman[71650]: 2024-05-27 07:26:06.955149624 +0000 UTC m=+0.044771492 container died f6f0fc672e8b94bbfd58886fc1e32b7795c8f2175f03215e6bc948f57d028419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:26:07.774 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:07 smithi162 podman[71650]: 2024-05-27 07:26:07.751177348 +0000 UTC m=+0.840799236 container remove f6f0fc672e8b94bbfd58886fc1e32b7795c8f2175f03215e6bc948f57d028419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:26:07.774 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:07 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: pgmap v118: 129 pgs: 129 active+clean; 60 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 668 KiB/s wr, 106 op/s 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: daemon mds.c restarted 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/1861764323,v1:172.21.15.162:6837/1861764323] up:boot 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6838/3450040148,v1:172.21.15.162:6839/3450040148] up:boot 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:07.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbbc9a00700 time 2024-05-27T07:26:07.523811+0000 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbbdf49aba6] 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.777 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: clone() 2024-05-27T07:26:07.778 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: *** Caught signal (Aborted) ** 2024-05-27T07:26:07.779 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: in thread 7fbbc9a00700 thread_name:md_log_replay 2024-05-27T07:26:07.779 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.524+0000 7fbbc9a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbbc9a00700 time 2024-05-27T07:26:07.523811+0000 2024-05-27T07:26:07.779 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:07.779 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.779 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.779 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbbdf49aba6] 2024-05-27T07:26:07.779 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.780 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: clone() 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbbde017d20] 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: gsignal() 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: abort() 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbbdf49aced] 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.781 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.782 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 19: clone() 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.528+0000 7fbbc9a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: in thread 7fbbc9a00700 thread_name:md_log_replay 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbbde017d20] 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: gsignal() 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: abort() 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbbdf49aced] 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.783 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 19: clone() 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.784 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: -1> 2024-05-27T07:26:07.524+0000 7fbbc9a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbbc9a00700 time 2024-05-27T07:26:07.523811+0000 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbbdf49aba6] 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.785 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: clone() 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.786 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 0> 2024-05-27T07:26:07.528+0000 7fbbc9a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: in thread 7fbbc9a00700 thread_name:md_log_replay 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbbde017d20] 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: gsignal() 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: abort() 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbbdf49aced] 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.787 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 19: clone() 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: -1> 2024-05-27T07:26:07.524+0000 7fbbc9a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbbc9a00700 time 2024-05-27T07:26:07.523811+0000 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:07.788 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbbdf49aba6] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.789 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: clone() 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 0> 2024-05-27T07:26:07.528+0000 7fbbc9a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: in thread 7fbbc9a00700 thread_name:md_log_replay 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.790 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbbde017d20] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2: gsignal() 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 3: abort() 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbbdf49aced] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbbdf49add1] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f01137423d] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f01135f971] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f01135dfa0] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f01135e651] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f0112f3347] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f010fef87e] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55f010ff4e94] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f010fff969] 2024-05-27T07:26:07.791 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f01139848c] 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f0113a19cb] 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 16: (Thread::entry_wrapper()+0x43) [0x7fbbdf46972d] 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbbdf469749] 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbbde00d1ca] 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 19: clone() 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71595]: 2024-05-27T07:26:07.792 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:07 smithi162 podman[71662]: 2024-05-27 07:26:07.770555625 +0000 UTC m=+0.149875024 container died 4644c7c268c7898ef5e2665fb70a2a031d95ff3dc3e80694ce5c842b6c8e857c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: pgmap v118: 129 pgs: 129 active+clean; 60 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 668 KiB/s wr, 106 op/s 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: daemon mds.c restarted 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/1861764323,v1:172.21.15.162:6837/1861764323] up:boot 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6838/3450040148,v1:172.21.15.162:6839/3450040148] up:boot 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:07 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:07.985 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:07 smithi062 podman[70630]: 2024-05-27 07:26:07.571939692 +0000 UTC m=+1.771086745 container remove fdd929c675ad42374dfb9163c40b16440cde630136d3fbf0ad3a9f3dfcdf2ffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:26:07.985 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:07 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: pgmap v118: 129 pgs: 129 active+clean; 60 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 668 KiB/s wr, 106 op/s 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: daemon mds.c restarted 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/1861764323,v1:172.21.15.162:6837/1861764323] up:boot 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6838/3450040148,v1:172.21.15.162:6839/3450040148] up:boot 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:07.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:07.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:07 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:08.485 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:08 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:26:08.485 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:08 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.745s CPU time. 2024-05-27T07:26:08.944 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:08 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:26:08.945 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:08 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.677s CPU time. 2024-05-27T07:26:08.945 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:08 smithi162 podman[71662]: 2024-05-27 07:26:08.483616866 +0000 UTC m=+0.862936243 container remove 4644c7c268c7898ef5e2665fb70a2a031d95ff3dc3e80694ce5c842b6c8e857c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:26:08.945 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:08 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:09.473 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:08 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:26:09.473 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:08 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.146s CPU time. 2024-05-27T07:26:09.477 INFO:tasks.workunit.client.0.smithi012.stdout:autoheader 2024-05-27T07:26:09.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:09 smithi012 ceph-mon[31473]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-05-27T07:26:09.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:09 smithi162 ceph-mon[35676]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-05-27T07:26:09.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:09 smithi062 ceph-mon[35304]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 155 op/s 2024-05-27T07:26:10.342 INFO:tasks.workunit.client.0.smithi012.stdout:autoconf 2024-05-27T07:26:11.045 INFO:tasks.workunit.client.0.smithi012.stdout:./configure \ 2024-05-27T07:26:11.046 INFO:tasks.workunit.client.0.smithi012.stdout: --libexecdir=/usr/lib \ 2024-05-27T07:26:11.046 INFO:tasks.workunit.client.0.smithi012.stdout: --exec_prefix=/var/lib 2024-05-27T07:26:11.716 INFO:tasks.workunit.client.0.smithi012.stdout:checking build system type... x86_64-pc-linux-gnu 2024-05-27T07:26:11.717 INFO:tasks.workunit.client.0.smithi012.stdout:checking host system type... x86_64-pc-linux-gnu 2024-05-27T07:26:11.724 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to print strings... printf 2024-05-27T07:26:11.726 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gcc... gcc 2024-05-27T07:26:11.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:11 smithi012 ceph-mon[31473]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 489 KiB/s wr, 53 op/s 2024-05-27T07:26:11.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:11 smithi162 ceph-mon[35676]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 489 KiB/s wr, 53 op/s 2024-05-27T07:26:11.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:11 smithi062 ceph-mon[35304]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 489 KiB/s wr, 53 op/s 2024-05-27T07:26:12.191 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether the C compiler works... yes 2024-05-27T07:26:12.192 INFO:tasks.workunit.client.0.smithi012.stdout:checking for C compiler default output file name... a.out 2024-05-27T07:26:12.408 INFO:tasks.workunit.client.0.smithi012.stdout:checking for suffix of executables... 2024-05-27T07:26:12.610 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether we are cross compiling... no 2024-05-27T07:26:12.808 INFO:tasks.workunit.client.0.smithi012.stdout:checking for suffix of object files... o 2024-05-27T07:26:12.935 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether we are using the GNU C compiler... yes 2024-05-27T07:26:13.061 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether gcc accepts -g... yes 2024-05-27T07:26:13.360 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gcc option to accept ISO C89... none needed 2024-05-27T07:26:13.387 INFO:tasks.workunit.client.0.smithi012.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-05-27T07:26:13.405 INFO:tasks.workunit.client.0.smithi012.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-05-27T07:26:13.410 INFO:tasks.workunit.client.0.smithi012.stdout:checking for egrep... /usr/bin/grep -E 2024-05-27T07:26:13.417 INFO:tasks.workunit.client.0.smithi012.stdout:checking for fgrep... /usr/bin/grep -F 2024-05-27T07:26:13.432 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ld used by gcc... /usr/bin/ld 2024-05-27T07:26:13.440 INFO:tasks.workunit.client.0.smithi012.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-05-27T07:26:13.467 INFO:tasks.workunit.client.0.smithi012.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-05-27T07:26:13.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:13 smithi012 ceph-mon[31473]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 425 KiB/s wr, 46 op/s 2024-05-27T07:26:13.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:13 smithi162 ceph-mon[35676]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 425 KiB/s wr, 46 op/s 2024-05-27T07:26:13.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:13 smithi062 ceph-mon[35304]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 425 KiB/s wr, 46 op/s 2024-05-27T07:26:14.692 INFO:tasks.workunit.client.0.smithi012.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-05-27T07:26:14.694 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether ln -s works... yes 2024-05-27T07:26:14.726 INFO:tasks.workunit.client.0.smithi012.stdout:checking the maximum length of command line arguments... 1572864 2024-05-27T07:26:14.739 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-05-27T07:26:14.740 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-05-27T07:26:14.742 INFO:tasks.workunit.client.0.smithi012.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-05-27T07:26:14.745 INFO:tasks.workunit.client.0.smithi012.stdout:checking for objdump... objdump 2024-05-27T07:26:14.747 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to recognize dependent libraries... pass_all 2024-05-27T07:26:14.749 INFO:tasks.workunit.client.0.smithi012.stdout:checking for dlltool... no 2024-05-27T07:26:14.751 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-05-27T07:26:14.753 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ar... ar 2024-05-27T07:26:15.109 INFO:tasks.workunit.client.0.smithi012.stdout:checking for archiver @FILE support... @ 2024-05-27T07:26:15.113 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strip... strip 2024-05-27T07:26:15.118 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ranlib... ranlib 2024-05-27T07:26:15.122 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gawk... gawk 2024-05-27T07:26:15.708 INFO:tasks.workunit.client.0.smithi012.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-05-27T07:26:15.715 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sysroot... no 2024-05-27T07:26:15.787 INFO:tasks.workunit.client.0.smithi012.stdout:checking for a working dd... /usr/bin/dd 2024-05-27T07:26:15.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:15 smithi012 ceph-mon[31473]: pgmap v124: 129 pgs: 129 active+clean; 64 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 658 KiB/s wr, 61 op/s 2024-05-27T07:26:15.859 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-05-27T07:26:15.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:15 smithi162 ceph-mon[35676]: pgmap v124: 129 pgs: 129 active+clean; 64 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 658 KiB/s wr, 61 op/s 2024-05-27T07:26:15.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:15 smithi062 ceph-mon[35304]: pgmap v124: 129 pgs: 129 active+clean; 64 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 658 KiB/s wr, 61 op/s 2024-05-27T07:26:16.039 INFO:tasks.workunit.client.0.smithi012.stdout:checking for mt... no 2024-05-27T07:26:16.118 INFO:tasks.workunit.client.0.smithi012.stdout:checking if : is a manifest tool... no 2024-05-27T07:26:16.399 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to run the C preprocessor... gcc -E 2024-05-27T07:26:17.279 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ANSI C header files... yes 2024-05-27T07:26:17.462 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/types.h... yes 2024-05-27T07:26:17.713 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/stat.h... yes 2024-05-27T07:26:17.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:17 smithi012 ceph-mon[31473]: pgmap v125: 129 pgs: 129 active+clean; 64 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 532 KiB/s wr, 49 op/s 2024-05-27T07:26:17.911 INFO:tasks.workunit.client.0.smithi012.stdout:checking for stdlib.h... yes 2024-05-27T07:26:17.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:17 smithi162 ceph-mon[35676]: pgmap v125: 129 pgs: 129 active+clean; 64 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 532 KiB/s wr, 49 op/s 2024-05-27T07:26:17.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:17 smithi062 ceph-mon[35304]: pgmap v125: 129 pgs: 129 active+clean; 64 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 532 KiB/s wr, 49 op/s 2024-05-27T07:26:18.129 INFO:tasks.workunit.client.0.smithi012.stdout:checking for string.h... yes 2024-05-27T07:26:18.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:26:18.339 INFO:tasks.workunit.client.0.smithi012.stdout:checking for memory.h... yes 2024-05-27T07:26:18.507 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strings.h... yes 2024-05-27T07:26:18.664 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:18 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:26:18.664 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:18 smithi062 systemd[1]: Stopped Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:18.664 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:18 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.745s CPU time. 2024-05-27T07:26:18.664 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:18 smithi062 systemd[1]: Starting Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:18.697 INFO:tasks.workunit.client.0.smithi012.stdout:checking for inttypes.h... yes 2024-05-27T07:26:18.821 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:18.862 INFO:tasks.workunit.client.0.smithi012.stdout:checking for stdint.h... yes 2024-05-27T07:26:18.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:18 smithi062 podman[70722]: 2024-05-27 07:26:18.579003322 +0000 UTC m=+0.040233063 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:18.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:18 smithi062 podman[70722]: 2024-05-27 07:26:18.715120577 +0000 UTC m=+0.176350313 container create 5844e1f769c408305ee8d799193780d95096572fb4d1e10aa496364c00b841b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:26:19.032 INFO:tasks.workunit.client.0.smithi012.stdout:checking for unistd.h... yes 2024-05-27T07:26:19.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:18 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:26:19.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:18 smithi162 systemd[1]: Stopped Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:19.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:18 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.677s CPU time. 2024-05-27T07:26:19.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:18 smithi162 systemd[1]: Starting Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:19.173 INFO:tasks.workunit.client.0.smithi012.stdout:checking for dlfcn.h... yes 2024-05-27T07:26:19.195 INFO:tasks.workunit.client.0.smithi012.stdout:checking for objdir... .libs 2024-05-27T07:26:19.397 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:19 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:26:19.397 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:19 smithi162 systemd[1]: Stopped Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:19.397 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:19 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.146s CPU time. 2024-05-27T07:26:19.397 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:19 smithi162 systemd[1]: Starting Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:19.397 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 podman[71796]: 2024-05-27 07:26:19.036117859 +0000 UTC m=+0.032547773 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:19.397 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 podman[71796]: 2024-05-27 07:26:19.167593024 +0000 UTC m=+0.164022989 container create d1d3958eb7817eb638d393438ad437be941d004d29fb971a2eb1c6035c53c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:26:19.397 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 podman[71796]: 2024-05-27 07:26:19.407281672 +0000 UTC m=+0.403711613 container init d1d3958eb7817eb638d393438ad437be941d004d29fb971a2eb1c6035c53c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:26:19.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 podman[70722]: 2024-05-27 07:26:19.028131943 +0000 UTC m=+0.489361722 container init 5844e1f769c408305ee8d799193780d95096572fb4d1e10aa496364c00b841b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:26:19.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 podman[70722]: 2024-05-27 07:26:19.040828675 +0000 UTC m=+0.502058424 container start 5844e1f769c408305ee8d799193780d95096572fb4d1e10aa496364c00b841b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:26:19.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-mds[70740]: -- 172.21.15.62:0/1638449680 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555da8cc5a40 con 0x555da8d0d180 2024-05-27T07:26:19.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: starting mds.k at 2024-05-27T07:26:19.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 bash[70722]: 5844e1f769c408305ee8d799193780d95096572fb4d1e10aa496364c00b841b3 2024-05-27T07:26:19.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 systemd[1]: Started Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:19.681 INFO:tasks.workunit.client.0.smithi012.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-05-27T07:26:19.685 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-05-27T07:26:19.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:19 smithi162 ceph-mon[35676]: pgmap v126: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 562 KiB/s wr, 61 op/s 2024-05-27T07:26:19.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:19 smithi162 ceph-mon[35676]: daemon mds.k restarted 2024-05-27T07:26:19.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 podman[71796]: 2024-05-27 07:26:19.419835984 +0000 UTC m=+0.416265912 container start d1d3958eb7817eb638d393438ad437be941d004d29fb971a2eb1c6035c53c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:19.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 bash[71796]: d1d3958eb7817eb638d393438ad437be941d004d29fb971a2eb1c6035c53c578 2024-05-27T07:26:19.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 systemd[1]: Started Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:19.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-mds[71839]: -- 172.21.15.162:0/4178517327 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55853f6a1a40 con 0x55853f6e9180 2024-05-27T07:26:19.727 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: starting mds.f at 2024-05-27T07:26:19.821 INFO:tasks.workunit.client.0.smithi012.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-05-27T07:26:19.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:19 smithi012 ceph-mon[31473]: pgmap v126: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 562 KiB/s wr, 61 op/s 2024-05-27T07:26:19.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:19 smithi012 ceph-mon[31473]: daemon mds.k restarted 2024-05-27T07:26:19.878 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:19 smithi062 ceph-mon[35304]: pgmap v126: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 562 KiB/s wr, 61 op/s 2024-05-27T07:26:19.878 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:19 smithi062 ceph-mon[35304]: daemon mds.k restarted 2024-05-27T07:26:19.878 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd35d200700 time 2024-05-27T07:26:19.760732+0000 2024-05-27T07:26:19.878 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:19.878 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd372cc8ba6] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.879 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.880 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.880 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.880 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.880 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: clone() 2024-05-27T07:26:19.880 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.761+0000 7fd35d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd35d200700 time 2024-05-27T07:26:19.760732+0000 2024-05-27T07:26:19.880 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd372cc8ba6] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.881 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: clone() 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: *** Caught signal (Aborted) ** 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: in thread 7fd35d200700 thread_name:md_log_replay 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.882 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd371845d20] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: gsignal() 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: abort() 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd372cc8ced] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.883 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 19: clone() 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.764+0000 7fd35d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: in thread 7fd35d200700 thread_name:md_log_replay 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd371845d20] 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: gsignal() 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: abort() 2024-05-27T07:26:19.884 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd372cc8ced] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.885 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 19: clone() 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: -1> 2024-05-27T07:26:19.761+0000 7fd35d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd35d200700 time 2024-05-27T07:26:19.760732+0000 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd372cc8ba6] 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.886 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.887 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.887 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.887 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.887 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.887 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.887 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: clone() 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 0> 2024-05-27T07:26:19.764+0000 7fd35d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: in thread 7fd35d200700 thread_name:md_log_replay 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd371845d20] 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: gsignal() 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: abort() 2024-05-27T07:26:19.888 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd372cc8ced] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.889 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 19: clone() 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: -1> 2024-05-27T07:26:19.761+0000 7fd35d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd35d200700 time 2024-05-27T07:26:19.760732+0000 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd372cc8ba6] 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: clone() 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 0> 2024-05-27T07:26:19.764+0000 7fd35d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: in thread 7fd35d200700 thread_name:md_log_replay 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd371845d20] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2: gsignal() 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 3: abort() 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd372cc8ced] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd372cc8dd1] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d9fe2c23d] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d9fe17971] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d9fe15fa0] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d9fe16651] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d9fdab347] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d9faa787e] 2024-05-27T07:26:19.892 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d9faace94] 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d9fab7969] 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d9fe5048c] 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d9fe599cb] 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 16: (Thread::entry_wrapper()+0x43) [0x7fd372c9772d] 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd372c97749] 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd37183b1ca] 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 19: clone() 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70736]: 2024-05-27T07:26:19.893 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:19 smithi062 podman[70771]: 2024-05-27 07:26:19.888651112 +0000 UTC m=+0.033541736 container died 5844e1f769c408305ee8d799193780d95096572fb4d1e10aa496364c00b841b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T07:26:19.977 INFO:tasks.workunit.client.0.smithi012.stdout:checking if gcc static flag -static works... no 2024-05-27T07:26:20.016 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:19 smithi162 podman[71880]: 2024-05-27 07:26:19.651809612 +0000 UTC m=+0.038152220 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:20.016 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:19 smithi162 podman[71880]: 2024-05-27 07:26:19.763464206 +0000 UTC m=+0.149806827 container create daf006a776c28e868fd164a30fce7344617f9802361d76c7f4da5cc5f1ea6c03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:26:20.016 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:20 smithi162 podman[71880]: 2024-05-27 07:26:20.026047757 +0000 UTC m=+0.412390433 container init daf006a776c28e868fd164a30fce7344617f9802361d76c7f4da5cc5f1ea6c03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f502a800700 time 2024-05-27T07:26:19.777025+0000 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f504028eba6] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.017 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: clone() 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:19.777+0000 7f502a800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f502a800700 time 2024-05-27T07:26:19.777025+0000 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:20.018 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f504028eba6] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.019 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: clone() 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: *** Caught signal (Aborted) ** 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: in thread 7f502a800700 thread_name:md_log_replay 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f503ee0bd20] 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: gsignal() 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: abort() 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f504028eced] 2024-05-27T07:26:20.020 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.021 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 19: clone() 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:19.780+0000 7f502a800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: in thread 7f502a800700 thread_name:md_log_replay 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.022 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f503ee0bd20] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: gsignal() 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: abort() 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f504028eced] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.023 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 19: clone() 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: -1> 2024-05-27T07:26:19.777+0000 7f502a800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f502a800700 time 2024-05-27T07:26:19.777025+0000 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f504028eba6] 2024-05-27T07:26:20.024 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.025 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.025 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.025 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.025 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.025 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.025 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.025 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.026 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.026 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.026 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.026 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.026 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.026 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: clone() 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 0> 2024-05-27T07:26:19.780+0000 7f502a800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: in thread 7f502a800700 thread_name:md_log_replay 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f503ee0bd20] 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: gsignal() 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: abort() 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f504028eced] 2024-05-27T07:26:20.027 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.028 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 19: clone() 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: -1> 2024-05-27T07:26:19.777+0000 7f502a800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f502a800700 time 2024-05-27T07:26:19.777025+0000 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f504028eba6] 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.029 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: clone() 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 0> 2024-05-27T07:26:19.780+0000 7f502a800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: in thread 7f502a800700 thread_name:md_log_replay 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:20.030 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f503ee0bd20] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2: gsignal() 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 3: abort() 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f504028eced] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f504028edd1] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851180423d] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5585117ef971] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5585117edfa0] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5585117ee651] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558511783347] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55851147f87e] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558511484e94] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55851148f969] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851182848c] 2024-05-27T07:26:20.031 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5585118319cb] 2024-05-27T07:26:20.032 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 16: (Thread::entry_wrapper()+0x43) [0x7f504025d72d] 2024-05-27T07:26:20.032 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f504025d749] 2024-05-27T07:26:20.032 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f503ee011ca] 2024-05-27T07:26:20.032 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 19: clone() 2024-05-27T07:26:20.032 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:20.032 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[71824]: 2024-05-27T07:26:20.285 INFO:tasks.workunit.client.0.smithi012.stdout:checking if gcc supports -c -o file.o... yes 2024-05-27T07:26:20.287 INFO:tasks.workunit.client.0.smithi012.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-05-27T07:26:20.315 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-05-27T07:26:20.473 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:20 smithi162 podman[71905]: 2024-05-27 07:26:20.263813843 +0000 UTC m=+0.373615976 container died d1d3958eb7817eb638d393438ad437be941d004d29fb971a2eb1c6035c53c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:26:20.473 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:20 smithi162 podman[71880]: 2024-05-27 07:26:20.040070911 +0000 UTC m=+0.426413553 container start daf006a776c28e868fd164a30fce7344617f9802361d76c7f4da5cc5f1ea6c03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T07:26:20.474 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mds[71923]: -- 172.21.15.162:0/356707554 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd2bdeba40 con 0x55bd2be33a80 2024-05-27T07:26:20.474 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: starting mds.c at 2024-05-27T07:26:20.474 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:20 smithi162 bash[71880]: daf006a776c28e868fd164a30fce7344617f9802361d76c7f4da5cc5f1ea6c03 2024-05-27T07:26:20.474 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:20 smithi162 systemd[1]: Started Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:20.551 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether -lc should be explicitly linked in... no 2024-05-27T07:26:20.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:26:20.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6832/1671331939,v1:172.21.15.62:6833/1671331939] up:boot 2024-05-27T07:26:20.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: daemon mds.f restarted 2024-05-27T07:26:20.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:20.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:20.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:26:20.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/3551084943,v1:172.21.15.162:6837/3551084943] up:boot 2024-05-27T07:26:20.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:20.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:20.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:20 smithi012 ceph-mon[31473]: daemon mds.c restarted 2024-05-27T07:26:20.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:26:20.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6832/1671331939,v1:172.21.15.62:6833/1671331939] up:boot 2024-05-27T07:26:20.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: daemon mds.f restarted 2024-05-27T07:26:20.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:20.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:20.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:26:20.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/3551084943,v1:172.21.15.162:6837/3551084943] up:boot 2024-05-27T07:26:20.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:20.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:20.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:20 smithi062 ceph-mon[35304]: daemon mds.c restarted 2024-05-27T07:26:20.818 INFO:tasks.workunit.client.0.smithi012.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-05-27T07:26:20.819 INFO:tasks.workunit.client.0.smithi012.stdout:checking how to hardcode library paths into programs... immediate 2024-05-27T07:26:20.826 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether stripping libraries is possible... yes 2024-05-27T07:26:20.827 INFO:tasks.workunit.client.0.smithi012.stdout:checking if libtool supports shared libraries... yes 2024-05-27T07:26:20.827 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether to build shared libraries... yes 2024-05-27T07:26:20.828 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether to build static libraries... yes 2024-05-27T07:26:20.849 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gcc... (cached) gcc 2024-05-27T07:26:20.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:26:20.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6832/1671331939,v1:172.21.15.62:6833/1671331939] up:boot 2024-05-27T07:26:20.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: daemon mds.f restarted 2024-05-27T07:26:20.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:20.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:20.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:26:20.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/3551084943,v1:172.21.15.162:6837/3551084943] up:boot 2024-05-27T07:26:20.958 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:20.958 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:20.958 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:20 smithi162 ceph-mon[35676]: daemon mds.c restarted 2024-05-27T07:26:21.004 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-05-27T07:26:21.006 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether gcc accepts -g... (cached) yes 2024-05-27T07:26:21.008 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-05-27T07:26:21.011 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gmake... /usr/bin/gmake 2024-05-27T07:26:21.012 INFO:tasks.workunit.client.0.smithi012.stderr:./configure: line 12082: test: $(SHELL): binary operator expected 2024-05-27T07:26:21.015 INFO:tasks.workunit.client.0.smithi012.stdout:checking for tar... /bin/tar 2024-05-27T07:26:21.018 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gzip... /bin/gzip 2024-05-27T07:26:21.090 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether gcc -MM is supported... yes 2024-05-27T07:26:21.094 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sort... /bin/sort 2024-05-27T07:26:21.096 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether ln -s works... yes 2024-05-27T07:26:21.099 INFO:tasks.workunit.client.0.smithi012.stdout:checking for rpm... /bin/rpm 2024-05-27T07:26:21.113 INFO:tasks.workunit.client.0.smithi012.stdout:checking for rpmbuild... no 2024-05-27T07:26:21.223 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:20 smithi162 podman[71905]: 2024-05-27 07:26:20.964879992 +0000 UTC m=+1.074682025 container remove d1d3958eb7817eb638d393438ad437be941d004d29fb971a2eb1c6035c53c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:26:21.223 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:20 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:21.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:20 smithi062 podman[70771]: 2024-05-27 07:26:20.802274585 +0000 UTC m=+0.947165253 container remove 5844e1f769c408305ee8d799193780d95096572fb4d1e10aa496364c00b841b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T07:26:21.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:20 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 -- 172.21.15.12:0/999973498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb401045e0 msgr2=0x7feb401049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 --2- 172.21.15.12:0/999973498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb401045e0 0x7feb401049f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7feb30009920 tx=0x7feb30009c30 comp rx=0 tx=0).stop 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 -- 172.21.15.12:0/999973498 shutdown_connections 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 --2- 172.21.15.12:0/999973498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb40109860 0x7feb401111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 --2- 172.21.15.12:0/999973498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb401045e0 0x7feb401049f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 --2- 172.21.15.12:0/999973498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb40108f10 0x7feb40109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 -- 172.21.15.12:0/999973498 >> 172.21.15.12:0/999973498 conn(0x7feb40100030 msgr2=0x7feb40102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:21.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 -- 172.21.15.12:0/999973498 shutdown_connections 2024-05-27T07:26:21.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.288+0000 7feb45a00700 1 -- 172.21.15.12:0/999973498 wait complete. 2024-05-27T07:26:21.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.289+0000 7feb45a00700 1 Processor -- start 2024-05-27T07:26:21.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb45a00700 1 -- start start 2024-05-27T07:26:21.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb45a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb401045e0 0x7feb4019d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb45a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb40108f10 0x7feb4019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb45a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb40109860 0x7feb401a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb45a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb40114390 con 0x7feb40108f10 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb45a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feb40114210 con 0x7feb40109860 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb45a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb40114510 con 0x7feb401045e0 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.290+0000 7feb3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb40108f10 0x7feb4019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb40108f10 0x7feb4019d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36412/0 (socket says 172.21.15.12:36412) 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3e000700 1 -- 172.21.15.12:0/3407479295 learned_addr learned my addr 172.21.15.12:0/3407479295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3ea00700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb401045e0 0x7feb4019d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3f400700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb40109860 0x7feb401a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3e000700 1 -- 172.21.15.12:0/3407479295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb401045e0 msgr2=0x7feb4019d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3e000700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb401045e0 0x7feb4019d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3e000700 1 -- 172.21.15.12:0/3407479295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb40109860 msgr2=0x7feb401a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3e000700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb40109860 0x7feb401a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.291+0000 7feb3e000700 1 -- 172.21.15.12:0/3407479295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb401a1f90 con 0x7feb40108f10 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.292+0000 7feb3e000700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb40108f10 0x7feb4019d660 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00e9f0 tx=0x7feb2c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.292+0000 7feb37400700 1 -- 172.21.15.12:0/3407479295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00c9c0 con 0x7feb40108f10 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.292+0000 7feb37400700 1 -- 172.21.15.12:0/3407479295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb2c010430 con 0x7feb40108f10 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.292+0000 7feb37400700 1 -- 172.21.15.12:0/3407479295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c018680 con 0x7feb40108f10 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.292+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb401a2280 con 0x7feb40108f10 2024-05-27T07:26:21.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.292+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb401a9ba0 con 0x7feb40108f10 2024-05-27T07:26:21.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 Processor -- start 2024-05-27T07:26:21.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 -- start start 2024-05-27T07:26:21.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04576b9f0 0x55d0457fe330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d0457fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0457fee70 0x55d04580b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d04572ce40 con 0x55d04576b9f0 2024-05-27T07:26:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d04571c9e0 con 0x55d04567fe40 2024-05-27T07:26:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.298+0000 7feb5451db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d045720a20 con 0x55d0457fee70 2024-05-27T07:26:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.299+0000 7feb37400700 1 -- 172.21.15.12:0/3407479295 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7feb2c0108e0 con 0x7feb40108f10 2024-05-27T07:26:21.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.299+0000 7feb3f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04576b9f0 0x55d0457fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.299+0000 7feb3e000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0457fee70 0x55d04580b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.299+0000 7feb3f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04576b9f0 0x55d0457fe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36426/0 (socket says 172.21.15.12:36426) 2024-05-27T07:26:21.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.299+0000 7feb3f400700 1 -- 172.21.15.12:0/945488936 learned_addr learned my addr 172.21.15.12:0/945488936 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:21.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.299+0000 7feb3ea00700 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d0457fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.300+0000 7feb3e000700 1 -- 172.21.15.12:0/945488936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 msgr2=0x55d0457fe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.300+0000 7feb3e000700 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d0457fe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.300+0000 7feb3e000700 1 -- 172.21.15.12:0/945488936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04576b9f0 msgr2=0x55d0457fe330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.300+0000 7feb3e000700 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04576b9f0 0x55d0457fe330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.300+0000 7feb3e000700 1 -- 172.21.15.12:0/945488936 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0456e5420 con 0x55d0457fee70 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.300+0000 7feb3f400700 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04576b9f0 0x55d0457fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.300+0000 7feb3e000700 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0457fee70 0x55d04580b110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0dff10 tx=0x7feb2c0eb740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=188cb41118d11abf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.301+0000 7feb2be00700 1 -- 172.21.15.12:0/945488936 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0f4640 con 0x55d0457fee70 2024-05-27T07:26:21.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.301+0000 7feb2be00700 1 -- 172.21.15.12:0/945488936 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb2c0e88d0 con 0x55d0457fee70 2024-05-27T07:26:21.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.301+0000 7feb2be00700 1 -- 172.21.15.12:0/945488936 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0e7690 con 0x55d0457fee70 2024-05-27T07:26:21.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.301+0000 7feb37400700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7feb240b2a70 0x7feb240b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.301+0000 7feb37400700 1 -- 172.21.15.12:0/3407479295 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9466+0+0 (secure 0 0 0) 0x7feb2c0e2bd0 con 0x7feb40108f10 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.301+0000 7feb3f400700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7feb240b2a70 0x7feb240b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 -- 172.21.15.12:0/945488936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0457fee70 msgr2=0x55d04580b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0457fee70 0x55d04580b110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0dff10 tx=0x7feb2c0eb740 comp rx=0 tx=0).stop 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 -- 172.21.15.12:0/945488936 shutdown_connections 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0457fee70 0x55d04580b110 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04576b9f0 0x55d0457fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 --2- 172.21.15.12:0/945488936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d0457fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 -- 172.21.15.12:0/945488936 >> 172.21.15.12:0/945488936 conn(0x55d045719390 msgr2=0x55d0457fa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb3f400700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7feb240b2a70 0x7feb240b4f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb38009fd0 tx=0x7feb38009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 -- 172.21.15.12:0/945488936 shutdown_connections 2024-05-27T07:26:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.302+0000 7feb5451db80 1 -- 172.21.15.12:0/945488936 wait complete. 2024-05-27T07:26:21.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.303+0000 7feb5451db80 1 -- start start 2024-05-27T07:26:21.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.303+0000 7feb5451db80 1 Processor -- start 2024-05-27T07:26:21.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.304+0000 7feb5451db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d045808600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.304+0000 7feb5451db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d045808b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.304+0000 7feb5451db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0457fee70 0x55d045809080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.304+0000 7feb5451db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d045622fa0 con 0x55d0457fee70 2024-05-27T07:26:21.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.304+0000 7feb5451db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d0456bd760 con 0x55d04567fe40 2024-05-27T07:26:21.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.304+0000 7feb5451db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d04580bb80 con 0x55d04576b9f0 2024-05-27T07:26:21.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.304+0000 7feb5451db80 10 client.0 _ll_register_callbacks cb 0x55d0456c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0457fee70 0x55d045809080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0457fee70 0x55d045809080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36438/0 (socket says 172.21.15.12:36438) 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 -- 172.21.15.12:0/3306361702 learned_addr learned my addr 172.21.15.12:0/3306361702 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3ea00700 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d045808600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 -- 172.21.15.12:0/3306361702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 msgr2=0x55d045808b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d045808b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 -- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 msgr2=0x55d045808600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d045808600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.305+0000 7feb3f400700 1 -- 172.21.15.12:0/3306361702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d04566c8d0 con 0x55d0457fee70 2024-05-27T07:26:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.306+0000 7feb3f400700 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0457fee70 0x55d045809080 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feb38016e30 tx=0x7feb3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6455feed028224e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.306+0000 7feb28c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.306+0000 7feb28c00700 1 -- 172.21.15.12:0/3306361702 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38027030 con 0x55d0457fee70 2024-05-27T07:26:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.306+0000 7feb28c00700 1 -- 172.21.15.12:0/3306361702 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9466+0+0 (secure 0 0 0) 0x7feb380090b0 con 0x55d0457fee70 2024-05-27T07:26:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.306+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb2c0e88d0 con 0x55d0457fee70 2024-05-27T07:26:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.307+0000 7feb28c00700 1 -- 172.21.15.12:0/3306361702 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7feb38007dd0 con 0x55d0457fee70 2024-05-27T07:26:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.307+0000 7feb5451db80 10 client.15849 fetch_fsmap learned FSMap version 40 2024-05-27T07:26:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.307+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d0456ed060 con 0x55d0457fee70 2024-05-27T07:26:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.307+0000 7feb28c00700 1 -- 172.21.15.12:0/3306361702 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb2c0e88d0 con 0x55d0457fee70 2024-05-27T07:26:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb28c00700 1 -- 172.21.15.12:0/3306361702 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7feb38004330 con 0x55d0457fee70 2024-05-27T07:26:21.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb5451db80 10 client.15849 fetch_fsmap finished waiting for FSMap version 40 2024-05-27T07:26:21.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb5451db80 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:21.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb5451db80 20 client.15849 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:21.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb5451db80 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d045818b40 0x55d04581b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb5451db80 4 client.15849 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:21.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d04572ed90 con 0x55d045818b40 2024-05-27T07:26:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.308+0000 7feb3ea00700 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d045818b40 0x55d04581b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.310+0000 7feb3ea00700 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d045818b40 0x55d04581b010 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dedfd29bf57c2a32 server_cookie=bc5886923e76ac3c in_seq=0 out_seq=0 2024-05-27T07:26:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.310+0000 7feb28c00700 10 client.15849 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.311+0000 7feb28c00700 1 -- 172.21.15.12:0/3306361702 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d04572ed90 con 0x55d045818b40 2024-05-27T07:26:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.311+0000 7feb28c00700 10 client.15849 handle_command_reply: tid=0 2024-05-27T07:26:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.311+0000 7feb5451db80 1 client.15849 shutdown 2024-05-27T07:26:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0457fee70 msgr2=0x55d045809080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0457fee70 0x55d045809080 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feb38016e30 tx=0x7feb3800b040 comp rx=0 tx=0).stop 2024-05-27T07:26:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 shutdown_connections 2024-05-27T07:26:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d045818b40 0x55d04581b010 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04567fe40 0x55d045808600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d0457fee70 0x55d045809080 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 --2- 172.21.15.12:0/3306361702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d045808b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 >> 172.21.15.12:0/3306361702 conn(0x55d045719390 msgr2=0x55d0457fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:21.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb28c00700 0 client.15849 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:21.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 shutdown_connections 2024-05-27T07:26:21.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.312+0000 7feb5451db80 1 -- 172.21.15.12:0/3306361702 wait complete. 2024-05-27T07:26:21.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.313+0000 7feb5451db80 20 client.15849 trim_cache size 0 max 16384 2024-05-27T07:26:21.322 INFO:tasks.workunit.client.0.smithi012.stdout:checking assert.h usability... yes 2024-05-27T07:26:21.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.337+0000 7feb5451db80 1 Processor -- start 2024-05-27T07:26:21.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb5451db80 1 -- start start 2024-05-27T07:26:21.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb5451db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d045719390 0x55d045807920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb5451db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04567fe40 0x55d045803df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb5451db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d0457fe720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb5451db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb401b5360 con 0x55d04567fe40 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb5451db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feb401b2b90 con 0x55d045719390 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb5451db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb401b09e0 con 0x55d04576b9f0 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04567fe40 0x55d045803df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04567fe40 0x55d045803df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36446/0 (socket says 172.21.15.12:36446) 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.338+0000 7feb3e000700 1 -- 172.21.15.12:0/2240562647 learned_addr learned my addr 172.21.15.12:0/2240562647 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3ea00700 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d045719390 0x55d045807920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3f400700 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d0457fe720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3e000700 1 -- 172.21.15.12:0/2240562647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 msgr2=0x55d0457fe720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3e000700 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d0457fe720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3e000700 1 -- 172.21.15.12:0/2240562647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d045719390 msgr2=0x55d045807920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3e000700 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d045719390 0x55d045807920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3e000700 1 -- 172.21.15.12:0/2240562647 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0457f6240 con 0x55d04567fe40 2024-05-27T07:26:21.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3f400700 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d0457fe720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.339+0000 7feb3e000700 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04567fe40 0x55d045803df0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0a9a30 tx=0x7feb2c009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3413d76f3531295 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.340+0000 7feb2b400700 1 -- 172.21.15.12:0/2240562647 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0a9870 con 0x55d04567fe40 2024-05-27T07:26:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.340+0000 7feb2b400700 1 -- 172.21.15.12:0/2240562647 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb2c0a95b0 con 0x55d04567fe40 2024-05-27T07:26:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.340+0000 7feb2b400700 1 -- 172.21.15.12:0/2240562647 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0a5230 con 0x55d04567fe40 2024-05-27T07:26:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 -- 172.21.15.12:0/2240562647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04567fe40 msgr2=0x55d045803df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04567fe40 0x55d045803df0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0a9a30 tx=0x7feb2c009290 comp rx=0 tx=0).stop 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 -- 172.21.15.12:0/2240562647 shutdown_connections 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d045719390 0x55d045807920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04576b9f0 0x55d0457fe720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 --2- 172.21.15.12:0/2240562647 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d04567fe40 0x55d045803df0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 -- 172.21.15.12:0/2240562647 >> 172.21.15.12:0/2240562647 conn(0x55d04581b360 msgr2=0x55d0458085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 -- 172.21.15.12:0/2240562647 shutdown_connections 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.341+0000 7feb5451db80 1 -- 172.21.15.12:0/2240562647 wait complete. 2024-05-27T07:26:21.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.342+0000 7feb5451db80 1 -- start start 2024-05-27T07:26:21.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.342+0000 7feb5451db80 1 Processor -- start 2024-05-27T07:26:21.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.342+0000 7feb5451db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04567fe40 0x55d0458023b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.342+0000 7feb5451db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d045719390 0x55d0458028f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.342+0000 7feb5451db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04576b9f0 0x55d045802e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb5451db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb401b1050 con 0x55d045719390 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb5451db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feb401b24c0 con 0x55d04576b9f0 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb5451db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb401b1720 con 0x55d04567fe40 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb5451db80 10 client.0 _ll_register_callbacks cb 0x55d0456c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d045719390 0x55d0458028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d045719390 0x55d0458028f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36458/0 (socket says 172.21.15.12:36458) 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3e000700 1 -- 172.21.15.12:0/3163099434 learned_addr learned my addr 172.21.15.12:0/3163099434 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3ea00700 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04567fe40 0x55d0458023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3f400700 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04576b9f0 0x55d045802e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3e000700 1 -- 172.21.15.12:0/3163099434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04567fe40 msgr2=0x55d0458023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3e000700 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04567fe40 0x55d0458023b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3e000700 1 -- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04576b9f0 msgr2=0x55d045802e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.343+0000 7feb3e000700 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04576b9f0 0x55d045802e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.344+0000 7feb3e000700 1 -- 172.21.15.12:0/3163099434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d04580ff10 con 0x55d045719390 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.344+0000 7feb3e000700 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d045719390 0x55d0458028f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feb2c010810 tx=0x7feb2c010840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487d36750710edc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.344+0000 7feb28c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.344+0000 7feb28c00700 1 -- 172.21.15.12:0/3163099434 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0a9210 con 0x55d045719390 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.344+0000 7feb28c00700 1 -- 172.21.15.12:0/3163099434 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9466+0+0 (secure 0 0 0) 0x7feb2c0a6090 con 0x55d045719390 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.344+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb2c0a95b0 con 0x55d045719390 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.345+0000 7feb28c00700 1 -- 172.21.15.12:0/3163099434 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7feb2c0a93f0 con 0x55d045719390 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.345+0000 7feb5451db80 10 client.15855 fetch_fsmap learned FSMap version 40 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.345+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d045811160 con 0x55d045719390 2024-05-27T07:26:21.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.345+0000 7feb28c00700 1 -- 172.21.15.12:0/3163099434 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb2c0a95b0 con 0x55d045719390 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.345+0000 7feb28c00700 1 -- 172.21.15.12:0/3163099434 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7feb2c0f2070 con 0x55d045719390 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.346+0000 7feb5451db80 10 client.15855 fetch_fsmap finished waiting for FSMap version 40 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.346+0000 7feb5451db80 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.346+0000 7feb5451db80 20 client.15855 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.346+0000 7feb5451db80 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d0457fee70 0x55d045811470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.346+0000 7feb5451db80 4 client.15855 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.346+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d045728570 con 0x55d0457fee70 2024-05-27T07:26:21.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.346+0000 7feb3ea00700 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d0457fee70 0x55d045811470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:21.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.348+0000 7feb3ea00700 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d0457fee70 0x55d045811470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba43ae5f79ce125b server_cookie=955d5d6673cad7e2 in_seq=0 out_seq=0 2024-05-27T07:26:21.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.348+0000 7feb28c00700 10 client.15855 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:21.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.350+0000 7feb28c00700 1 -- 172.21.15.12:0/3163099434 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d045728570 con 0x55d0457fee70 2024-05-27T07:26:21.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.350+0000 7feb28c00700 10 client.15855 handle_command_reply: tid=0 2024-05-27T07:26:21.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.350+0000 7feb5451db80 1 client.15855 shutdown 2024-05-27T07:26:21.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.350+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d045719390 msgr2=0x55d0458028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d045719390 0x55d0458028f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feb2c010810 tx=0x7feb2c010840 comp rx=0 tx=0).stop 2024-05-27T07:26:21.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 shutdown_connections 2024-05-27T07:26:21.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d0457fee70 0x55d045811470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d04576b9f0 0x55d045802e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04567fe40 0x55d0458023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 --2- 172.21.15.12:0/3163099434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d045719390 0x55d0458028f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 >> 172.21.15.12:0/3163099434 conn(0x55d04581b360 msgr2=0x55d045807680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:21.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb28c00700 0 client.15855 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:21.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 shutdown_connections 2024-05-27T07:26:21.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 1 -- 172.21.15.12:0/3163099434 wait complete. 2024-05-27T07:26:21.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.351+0000 7feb5451db80 20 client.15855 trim_cache size 0 max 16384 2024-05-27T07:26:21.342 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:26:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:26:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:26:21.342 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:26:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.352+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7feb240b2a70 msgr2=0x7feb240b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.352+0000 7feb45a00700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7feb240b2a70 0x7feb240b4f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb38009fd0 tx=0x7feb38009040 comp rx=0 tx=0).stop 2024-05-27T07:26:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.352+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb40108f10 msgr2=0x7feb4019d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.352+0000 7feb45a00700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb40108f10 0x7feb4019d660 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00e9f0 tx=0x7feb2c00ed00 comp rx=0 tx=0).stop 2024-05-27T07:26:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.352+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 shutdown_connections 2024-05-27T07:26:21.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.353+0000 7feb45a00700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7feb240b2a70 0x7feb240b4f80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.353+0000 7feb45a00700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb40109860 0x7feb401a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.353+0000 7feb45a00700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb401045e0 0x7feb4019d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.353+0000 7feb45a00700 1 --2- 172.21.15.12:0/3407479295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb40108f10 0x7feb4019d660 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.353+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 >> 172.21.15.12:0/3407479295 conn(0x7feb40100030 msgr2=0x7feb4010a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.353+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 shutdown_connections 2024-05-27T07:26:21.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:21.353+0000 7feb45a00700 1 -- 172.21.15.12:0/3407479295 wait complete. 2024-05-27T07:26:21.387 INFO:tasks.workunit.client.0.smithi012.stdout:checking assert.h presence... yes 2024-05-27T07:26:21.388 INFO:tasks.workunit.client.0.smithi012.stdout:checking for assert.h... yes 2024-05-27T07:26:21.644 INFO:tasks.workunit.client.0.smithi012.stdout:checking bstring.h usability... no 2024-05-27T07:26:21.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-mon[35676]: pgmap v129: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 464 KiB/s wr, 51 op/s 2024-05-27T07:26:21.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-mon[35676]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:26:21.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/1055586154,v1:172.21.15.162:6837/1055586154] up:boot 2024-05-27T07:26:21.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:21.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:21.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:21.711 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:21 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:26:21.711 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:21 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.620s CPU time. 2024-05-27T07:26:21.711 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82e7600700 time 2024-05-27T07:26:21.575500+0000 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f82fd0cfba6] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.712 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.713 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.713 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.713 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.713 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.713 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: clone() 2024-05-27T07:26:21.713 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.576+0000 7f82e7600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82e7600700 time 2024-05-27T07:26:21.575500+0000 2024-05-27T07:26:21.713 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f82fd0cfba6] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.714 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: clone() 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: *** Caught signal (Aborted) ** 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: in thread 7f82e7600700 thread_name:md_log_replay 2024-05-27T07:26:21.715 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.716 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82fbc4cd20] 2024-05-27T07:26:21.716 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: gsignal() 2024-05-27T07:26:21.716 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: abort() 2024-05-27T07:26:21.716 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f82fd0cfced] 2024-05-27T07:26:21.716 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.717 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 17: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 19: clone() 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.578+0000 7f82e7600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: in thread 7f82e7600700 thread_name:md_log_replay 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.718 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82fbc4cd20] 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: gsignal() 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: abort() 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f82fd0cfced] 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.719 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 17: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 19: clone() 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: -1> 2024-05-27T07:26:21.576+0000 7f82e7600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82e7600700 time 2024-05-27T07:26:21.575500+0000 2024-05-27T07:26:21.720 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f82fd0cfba6] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.721 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: clone() 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 0> 2024-05-27T07:26:21.578+0000 7f82e7600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: in thread 7f82e7600700 thread_name:md_log_replay 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.722 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82fbc4cd20] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: gsignal() 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: abort() 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f82fd0cfced] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 17: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 19: clone() 2024-05-27T07:26:21.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: -1> 2024-05-27T07:26:21.576+0000 7f82e7600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82e7600700 time 2024-05-27T07:26:21.575500+0000 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f82fd0cfba6] 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.725 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: clone() 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 0> 2024-05-27T07:26:21.578+0000 7f82e7600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: in thread 7f82e7600700 thread_name:md_log_replay 2024-05-27T07:26:21.726 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82fbc4cd20] 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2: gsignal() 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 3: abort() 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f82fd0cfced] 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f82fd0cfdd1] 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bceda9e23d] 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bceda89971] 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bceda87fa0] 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bceda88651] 2024-05-27T07:26:21.727 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bceda1d347] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bced71987e] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bced71ee94] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bced729969] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bcedac248c] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bcedacb9cb] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 16: (Thread::entry_wrapper()+0x43) [0x7f82fd09e72d] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 17: (Thread::_entry_func(void*)+0xd) [0x7f82fd09e749] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f82fbc421ca] 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 19: clone() 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[71907]: 2024-05-27T07:26:21.728 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:21 smithi162 podman[71983]: 2024-05-27 07:26:21.720777632 +0000 UTC m=+0.037408963 container died daf006a776c28e868fd164a30fce7344617f9802361d76c7f4da5cc5f1ea6c03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:26:21.735 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:21 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:26:21.735 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:21 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.785s CPU time. 2024-05-27T07:26:21.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:21 smithi062 ceph-mon[35304]: pgmap v129: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 464 KiB/s wr, 51 op/s 2024-05-27T07:26:21.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:21 smithi062 ceph-mon[35304]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:26:21.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:21 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/1055586154,v1:172.21.15.162:6837/1055586154] up:boot 2024-05-27T07:26:21.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:21 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:21.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:21 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:21.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:21 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:21.803 INFO:tasks.workunit.client.0.smithi012.stdout:checking bstring.h presence... no 2024-05-27T07:26:21.805 INFO:tasks.workunit.client.0.smithi012.stdout:checking for bstring.h... no 2024-05-27T07:26:21.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:21 smithi012 ceph-mon[31473]: pgmap v129: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 464 KiB/s wr, 51 op/s 2024-05-27T07:26:21.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:21 smithi012 ceph-mon[31473]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:26:21.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:21 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/1055586154,v1:172.21.15.162:6837/1055586154] up:boot 2024-05-27T07:26:21.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:21 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:21.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:21 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:21.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:21 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:21.992 INFO:tasks.workunit.client.0.smithi012.stdout:checking libgen.h usability... yes 2024-05-27T07:26:22.080 INFO:tasks.workunit.client.0.smithi012.stdout:checking libgen.h presence... yes 2024-05-27T07:26:22.082 INFO:tasks.workunit.client.0.smithi012.stdout:checking for libgen.h... yes 2024-05-27T07:26:22.357 INFO:tasks.workunit.client.0.smithi012.stdout:checking dirent.h usability... yes 2024-05-27T07:26:22.411 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:26:22.411 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:26:22.442 INFO:tasks.workunit.client.0.smithi012.stdout:checking dirent.h presence... yes 2024-05-27T07:26:22.450 INFO:tasks.workunit.client.0.smithi012.stdout:checking for dirent.h... yes 2024-05-27T07:26:22.659 INFO:tasks.workunit.client.0.smithi012.stdout:checking errno.h usability... yes 2024-05-27T07:26:22.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:22 smithi162 podman[71983]: 2024-05-27 07:26:22.454316934 +0000 UTC m=+0.770948267 container remove daf006a776c28e868fd164a30fce7344617f9802361d76c7f4da5cc5f1ea6c03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:26:22.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:22 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:22.742 INFO:tasks.workunit.client.0.smithi012.stdout:checking errno.h presence... yes 2024-05-27T07:26:22.742 INFO:tasks.workunit.client.0.smithi012.stdout:checking for errno.h... yes 2024-05-27T07:26:22.916 INFO:tasks.workunit.client.0.smithi012.stdout:checking malloc.h usability... yes 2024-05-27T07:26:22.926 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:22.990 INFO:tasks.workunit.client.0.smithi012.stdout:checking malloc.h presence... yes 2024-05-27T07:26:22.990 INFO:tasks.workunit.client.0.smithi012.stdout:checking for malloc.h... yes 2024-05-27T07:26:23.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:22 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:26:23.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:22 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.163s CPU time. 2024-05-27T07:26:23.246 INFO:tasks.workunit.client.0.smithi012.stdout:checking uuid.h usability... no 2024-05-27T07:26:23.402 INFO:tasks.workunit.client.0.smithi012.stdout:checking uuid.h presence... no 2024-05-27T07:26:23.405 INFO:tasks.workunit.client.0.smithi012.stdout:checking for uuid.h... no 2024-05-27T07:26:23.631 INFO:tasks.workunit.client.0.smithi012.stdout:checking uuid/uuid.h usability... yes 2024-05-27T07:26:23.731 INFO:tasks.workunit.client.0.smithi012.stdout:checking uuid/uuid.h presence... yes 2024-05-27T07:26:23.732 INFO:tasks.workunit.client.0.smithi012.stdout:checking for uuid/uuid.h... yes 2024-05-27T07:26:23.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:23 smithi012 ceph-mon[31473]: pgmap v131: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 231 KiB/s wr, 40 op/s 2024-05-27T07:26:23.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:23 smithi162 ceph-mon[35676]: pgmap v131: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 231 KiB/s wr, 40 op/s 2024-05-27T07:26:23.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:23 smithi062 ceph-mon[35304]: pgmap v131: 129 pgs: 129 active+clean; 65 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 231 KiB/s wr, 40 op/s 2024-05-27T07:26:23.991 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/uuid.h usability... no 2024-05-27T07:26:24.146 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/uuid.h presence... no 2024-05-27T07:26:24.148 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/uuid.h... no 2024-05-27T07:26:24.389 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/file.h usability... yes 2024-05-27T07:26:24.490 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/file.h presence... yes 2024-05-27T07:26:24.493 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/file.h... yes 2024-05-27T07:26:24.777 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fcntl.h usability... yes 2024-05-27T07:26:24.864 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fcntl.h presence... yes 2024-05-27T07:26:24.865 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/fcntl.h... yes 2024-05-27T07:26:25.279 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/syssgi.h usability... no 2024-05-27T07:26:25.454 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/syssgi.h presence... no 2024-05-27T07:26:25.455 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/syssgi.h... no 2024-05-27T07:26:25.853 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/param.h usability... yes 2024-05-27T07:26:25.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:25 smithi012 ceph-mon[31473]: pgmap v132: 129 pgs: 129 active+clean; 67 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 274 KiB/s wr, 57 op/s 2024-05-27T07:26:25.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:25 smithi162 ceph-mon[35676]: pgmap v132: 129 pgs: 129 active+clean; 67 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 274 KiB/s wr, 57 op/s 2024-05-27T07:26:25.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:25 smithi062 ceph-mon[35304]: pgmap v132: 129 pgs: 129 active+clean; 67 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 274 KiB/s wr, 57 op/s 2024-05-27T07:26:26.162 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/param.h presence... yes 2024-05-27T07:26:26.163 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/param.h... yes 2024-05-27T07:26:26.235 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/stat.h... (cached) yes 2024-05-27T07:26:26.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 -- 172.21.15.12:0/3151503678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 msgr2=0x7fa72c103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 --2- 172.21.15.12:0/3151503678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c103500 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009990 tx=0x7fa71c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 -- 172.21.15.12:0/3151503678 shutdown_connections 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 --2- 172.21.15.12:0/3151503678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c110ca0 0x7fa72c1130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 --2- 172.21.15.12:0/3151503678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c103500 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 --2- 172.21.15.12:0/3151503678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c103a40 0x7fa72c110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 -- 172.21.15.12:0/3151503678 >> 172.21.15.12:0/3151503678 conn(0x7fa72c0fedb0 msgr2=0x7fa72c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 -- 172.21.15.12:0/3151503678 shutdown_connections 2024-05-27T07:26:26.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.292+0000 7fa732e00700 1 -- 172.21.15.12:0/3151503678 wait complete. 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 Processor -- start 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 -- start start 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c103a40 0x7fa72c19d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c110ca0 0x7fa72c1a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c116550 con 0x7fa72c1030f0 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c1163d0 con 0x7fa72c110ca0 2024-05-27T07:26:26.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.294+0000 7fa732e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c1166d0 con 0x7fa72c103a40 2024-05-27T07:26:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c19cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39940/0 (socket says 172.21.15.12:39940) 2024-05-27T07:26:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 -- 172.21.15.12:0/1031630353 learned_addr learned my addr 172.21.15.12:0/1031630353 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72b400700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c103a40 0x7fa72c19d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa730e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c110ca0 0x7fa72c1a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 -- 172.21.15.12:0/1031630353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c103a40 msgr2=0x7fa72c19d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c103a40 0x7fa72c19d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 -- 172.21.15.12:0/1031630353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c110ca0 msgr2=0x7fa72c1a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c110ca0 0x7fa72c1a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa72be00700 1 -- 172.21.15.12:0/1031630353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72c1a1e30 con 0x7fa72c1030f0 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.295+0000 7fa730e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c110ca0 0x7fa72c1a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.296+0000 7fa72be00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c19cf60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000c00 tx=0x7fa71c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.296+0000 7fa728c00700 1 -- 172.21.15.12:0/1031630353 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c027070 con 0x7fa72c1030f0 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.296+0000 7fa728c00700 1 -- 172.21.15.12:0/1031630353 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa71c00fb80 con 0x7fa72c1030f0 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.296+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72c1a20c0 con 0x7fa72c1030f0 2024-05-27T07:26:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.296+0000 7fa728c00700 1 -- 172.21.15.12:0/1031630353 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c022c30 con 0x7fa72c1030f0 2024-05-27T07:26:26.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.296+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72c1a25a0 con 0x7fa72c1030f0 2024-05-27T07:26:26.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.298+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa72c0512b0 con 0x7fa72c1030f0 2024-05-27T07:26:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.300+0000 7fa728c00700 1 -- 172.21.15.12:0/1031630353 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa71c02a070 con 0x7fa72c1030f0 2024-05-27T07:26:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.302+0000 7fa728c00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa7100b29f0 0x7fa7100b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.302+0000 7fa728c00700 1 -- 172.21.15.12:0/1031630353 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9466+0+0 (secure 0 0 0) 0x7fa71c0e3c90 con 0x7fa72c1030f0 2024-05-27T07:26:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.302+0000 7fa72b400700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa7100b29f0 0x7fa7100b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.303+0000 7fa72b400700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa7100b29f0 0x7fa7100b4f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa718005950 tx=0x7fa7180058e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.305+0000 7fa728c00700 1 -- 172.21.15.12:0/1031630353 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa71c05fdf0 con 0x7fa72c1030f0 2024-05-27T07:26:26.406 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/statvfs.h usability... yes 2024-05-27T07:26:26.482 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/statvfs.h presence... yes 2024-05-27T07:26:26.486 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/statvfs.h... yes 2024-05-27T07:26:26.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.617+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa72c02e5e0 con 0x7fa72c1030f0 2024-05-27T07:26:26.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.620+0000 7fa728c00700 1 -- 172.21.15.12:0/1031630353 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11874 (secure 0 0 0) 0x7fa71c013070 con 0x7fa72c1030f0 2024-05-27T07:26:26.610 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:26:26.610 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":25489,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2607883669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2607883669},{"type":"v1","addr":"172.21.15.162:6835","nonce":2607883669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:26:20.639044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_25510":{"gid":25510,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/3551084943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3551084943},{"type":"v1","addr":"172.21.15.162:6837","nonce":3551084943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25516":{"gid":25516,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6837/1055586154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1055586154},{"type":"v1","addr":"172.21.15.162:6837","nonce":1055586154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_35447":{"gid":35447,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6833/1671331939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1671331939},{"type":"v1","addr":"172.21.15.62:6833","nonce":1671331939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34475,34415,34439]},"id":1}]} 2024-05-27T07:26:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa7100b29f0 msgr2=0x7fa7100b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa7100b29f0 0x7fa7100b4f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa718005950 tx=0x7fa7180058e0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 msgr2=0x7fa72c19cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c19cf60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000c00 tx=0x7fa71c005ef0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 shutdown_connections 2024-05-27T07:26:26.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa7100b29f0 0x7fa7100b4f00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c110ca0 0x7fa72c1a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa72c1030f0 0x7fa72c19cf60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 --2- 172.21.15.12:0/1031630353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c103a40 0x7fa72c19d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 >> 172.21.15.12:0/1031630353 conn(0x7fa72c0fedb0 msgr2=0x7fa72c1128e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 shutdown_connections 2024-05-27T07:26:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:26.623+0000 7fa732e00700 1 -- 172.21.15.12:0/1031630353 wait complete. 2024-05-27T07:26:26.614 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 40 2024-05-27T07:26:26.703 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/time.h usability... yes 2024-05-27T07:26:26.781 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/time.h presence... yes 2024-05-27T07:26:26.784 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/time.h... yes 2024-05-27T07:26:27.012 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/ioctl.h usability... yes 2024-05-27T07:26:27.090 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/ioctl.h presence... yes 2024-05-27T07:26:27.091 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/ioctl.h... yes 2024-05-27T07:26:27.349 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/wait.h usability... yes 2024-05-27T07:26:27.442 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/wait.h presence... yes 2024-05-27T07:26:27.444 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/wait.h... yes 2024-05-27T07:26:27.542 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/types.h... (cached) yes 2024-05-27T07:26:27.591 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:26:27.614 INFO:tasks.workunit.client.0.smithi012.stdout:checking for strings.h... (cached) yes 2024-05-27T07:26:27.810 INFO:tasks.workunit.client.0.smithi012.stdout:checking err.h usability... yes 2024-05-27T07:26:27.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:27 smithi012 ceph-mon[31473]: pgmap v133: 129 pgs: 129 active+clean; 67 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 240 KiB/s wr, 49 op/s 2024-05-27T07:26:27.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:27 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1031630353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:27.887 INFO:tasks.workunit.client.0.smithi012.stdout:checking err.h presence... yes 2024-05-27T07:26:27.888 INFO:tasks.workunit.client.0.smithi012.stdout:checking for err.h... yes 2024-05-27T07:26:27.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:27 smithi162 ceph-mon[35676]: pgmap v133: 129 pgs: 129 active+clean; 67 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 240 KiB/s wr, 49 op/s 2024-05-27T07:26:27.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:27 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1031630353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:27.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:27 smithi062 ceph-mon[35304]: pgmap v133: 129 pgs: 129 active+clean; 67 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 240 KiB/s wr, 49 op/s 2024-05-27T07:26:27.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:27 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1031630353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:28.087 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/falloc.h usability... yes 2024-05-27T07:26:28.152 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/falloc.h presence... yes 2024-05-27T07:26:28.153 INFO:tasks.workunit.client.0.smithi012.stdout:checking for linux/falloc.h... yes 2024-05-27T07:26:28.433 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-05-27T07:26:28.571 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-05-27T07:26:28.573 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/fs/xfs_fsops.h... no 2024-05-27T07:26:28.782 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fs/xfs_itable.h usability... no 2024-05-27T07:26:28.923 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/fs/xfs_itable.h presence... no 2024-05-27T07:26:28.926 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/fs/xfs_itable.h... no 2024-05-27T07:26:29.137 INFO:tasks.workunit.client.0.smithi012.stdout:checking xfs/platform_defs.h usability... no 2024-05-27T07:26:29.283 INFO:tasks.workunit.client.0.smithi012.stdout:checking xfs/platform_defs.h presence... no 2024-05-27T07:26:29.285 INFO:tasks.workunit.client.0.smithi012.stdout:checking for xfs/platform_defs.h... no 2024-05-27T07:26:29.495 INFO:tasks.workunit.client.0.smithi012.stdout:checking btrfs/ioctl.h usability... no 2024-05-27T07:26:29.651 INFO:tasks.workunit.client.0.smithi012.stdout:checking btrfs/ioctl.h presence... no 2024-05-27T07:26:29.653 INFO:tasks.workunit.client.0.smithi012.stdout:checking for btrfs/ioctl.h... no 2024-05-27T07:26:29.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:29 smithi012 ceph-mon[31473]: pgmap v134: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 491 KiB/s wr, 65 op/s 2024-05-27T07:26:29.854 INFO:tasks.workunit.client.0.smithi012.stdout:checking cifs/ioctl.h usability... no 2024-05-27T07:26:29.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:29 smithi162 ceph-mon[35676]: pgmap v134: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 491 KiB/s wr, 65 op/s 2024-05-27T07:26:29.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:29 smithi062 ceph-mon[35304]: pgmap v134: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 491 KiB/s wr, 65 op/s 2024-05-27T07:26:30.008 INFO:tasks.workunit.client.0.smithi012.stdout:checking cifs/ioctl.h presence... no 2024-05-27T07:26:30.010 INFO:tasks.workunit.client.0.smithi012.stdout:checking for cifs/ioctl.h... no 2024-05-27T07:26:30.200 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/mman.h usability... yes 2024-05-27T07:26:30.284 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/mman.h presence... yes 2024-05-27T07:26:30.286 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/mman.h... yes 2024-05-27T07:26:30.525 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/ext4.h usability... no 2024-05-27T07:26:30.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 -- 172.21.15.12:0/3332493946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 msgr2=0x7f1158075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 --2- 172.21.15.12:0/3332493946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f1158075f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1148009990 tx=0x7f1148009ca0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 -- 172.21.15.12:0/3332493946 shutdown_connections 2024-05-27T07:26:30.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 --2- 172.21.15.12:0/3332493946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1158115360 0x7f1158117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 --2- 172.21.15.12:0/3332493946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f1158075f60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 --2- 172.21.15.12:0/3332493946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11580764a0 0x7f1158114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 -- 172.21.15.12:0/3332493946 >> 172.21.15.12:0/3332493946 conn(0x7f1158100030 msgr2=0x7f1158102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:30.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.574+0000 7f115d000700 1 -- 172.21.15.12:0/3332493946 shutdown_connections 2024-05-27T07:26:30.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.575+0000 7f115d000700 1 -- 172.21.15.12:0/3332493946 wait complete. 2024-05-27T07:26:30.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.576+0000 7f115d000700 1 Processor -- start 2024-05-27T07:26:30.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.576+0000 7f115d000700 1 -- start start 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f115d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f11581a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f115d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11580764a0 0x7f11581a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f115d000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1158115360 0x7f11581a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f115d000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f115811a910 con 0x7f1158075b50 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f115d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f115811a790 con 0x7f11580764a0 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f115d000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f115811aa90 con 0x7f1158115360 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f1155e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f11581a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f1155e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f11581a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39948/0 (socket says 172.21.15.12:39948) 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f1155e00700 1 -- 172.21.15.12:0/3960941918 learned_addr learned my addr 172.21.15.12:0/3960941918 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.577+0000 7f1155400700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11580764a0 0x7f11581a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1156800700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1158115360 0x7f11581a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1155e00700 1 -- 172.21.15.12:0/3960941918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1158115360 msgr2=0x7f11581a5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1155e00700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1158115360 0x7f11581a5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1155e00700 1 -- 172.21.15.12:0/3960941918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11580764a0 msgr2=0x7f11581a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1155e00700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11580764a0 0x7f11581a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1155e00700 1 -- 172.21.15.12:0/3960941918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11581a6120 con 0x7f1158075b50 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1155400700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11580764a0 0x7f11581a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1156800700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1158115360 0x7f11581a5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.578+0000 7f1155e00700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f11581a1300 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1148000c00 tx=0x7f1148009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.579+0000 7f114ea00700 1 -- 172.21.15.12:0/3960941918 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148027070 con 0x7f1158075b50 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.579+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11581a63b0 con 0x7f1158075b50 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.579+0000 7f114ea00700 1 -- 172.21.15.12:0/3960941918 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f114800fc70 con 0x7f1158075b50 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.579+0000 7f114ea00700 1 -- 172.21.15.12:0/3960941918 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148022d40 con 0x7f1158075b50 2024-05-27T07:26:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.579+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581a6890 con 0x7f1158075b50 2024-05-27T07:26:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.582+0000 7f114ea00700 1 -- 172.21.15.12:0/3960941918 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f114802a070 con 0x7f1158075b50 2024-05-27T07:26:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.584+0000 7f114ea00700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f113c0b29f0 0x7f113c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.584+0000 7f116ba8db80 1 Processor -- start 2024-05-27T07:26:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.584+0000 7f114ea00700 1 -- 172.21.15.12:0/3960941918 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9466+0+0 (secure 0 0 0) 0x7f11480e37f0 con 0x7f1158075b50 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.584+0000 7f116ba8db80 1 -- start start 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.584+0000 7f1155400700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f113c0b29f0 0x7f113c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.585+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee797f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.585+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee73870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.585+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee741c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.585+0000 7f116ba8db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56033eda1e40 con 0x56033ee79d30 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.585+0000 7f116ba8db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56033ed919e0 con 0x56033ecf4e40 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.585+0000 7f116ba8db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56033ed95a20 con 0x56033ee73db0 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee73870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f113c0b29f0 0x7f113c0b4f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1144005fd0 tx=0x7f1144005e20 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee797f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee73870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39954/0 (socket says 172.21.15.12:39954) 2024-05-27T07:26:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 -- 172.21.15.12:0/4023170546 learned_addr learned my addr 172.21.15.12:0/4023170546 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1156800700 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee741c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 -- 172.21.15.12:0/4023170546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 msgr2=0x56033ee741c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee741c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 -- 172.21.15.12:0/4023170546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 msgr2=0x56033ee797f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee797f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.586+0000 7f1155400700 1 -- 172.21.15.12:0/4023170546 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56033ed5a420 con 0x56033ee79d30 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.587+0000 7f1155400700 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee73870 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f114400dc30 tx=0x7f1144010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7617e6745e2883 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.587+0000 7f1143400700 1 -- 172.21.15.12:0/4023170546 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144020070 con 0x56033ee79d30 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.587+0000 7f1143400700 1 -- 172.21.15.12:0/4023170546 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f114400fdc0 con 0x56033ee79d30 2024-05-27T07:26:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.587+0000 7f1143400700 1 -- 172.21.15.12:0/4023170546 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114401ce30 con 0x56033ee79d30 2024-05-27T07:26:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 -- 172.21.15.12:0/4023170546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 msgr2=0x56033ee73870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee73870 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f114400dc30 tx=0x7f1144010000 comp rx=0 tx=0).stop 2024-05-27T07:26:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 -- 172.21.15.12:0/4023170546 shutdown_connections 2024-05-27T07:26:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee797f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 --2- 172.21.15.12:0/4023170546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee73870 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 -- 172.21.15.12:0/4023170546 >> 172.21.15.12:0/4023170546 conn(0x56033ed54fc0 msgr2=0x56033ee6e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 -- 172.21.15.12:0/4023170546 shutdown_connections 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.588+0000 7f116ba8db80 1 -- 172.21.15.12:0/4023170546 wait complete. 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.589+0000 7f116ba8db80 1 -- start start 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.589+0000 7f116ba8db80 1 Processor -- start 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.590+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee7d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.590+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee73db0 0x56033ee7d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.590+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee79d30 0x56033ee7de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.590+0000 7f116ba8db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56033ec97fa0 con 0x56033ee73db0 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.590+0000 7f116ba8db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56033ed32760 con 0x56033ee79d30 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.590+0000 7f116ba8db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56033ee80800 con 0x56033ecf4e40 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.590+0000 7f116ba8db80 10 client.0 _ll_register_callbacks cb 0x56033ed3bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee73db0 0x56033ee7d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee73db0 0x56033ee7d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39968/0 (socket says 172.21.15.12:39968) 2024-05-27T07:26:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 -- 172.21.15.12:0/3411565283 learned_addr learned my addr 172.21.15.12:0/3411565283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1156800700 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee79d30 0x56033ee7de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155e00700 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee7d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 -- 172.21.15.12:0/3411565283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 msgr2=0x56033ee7d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee7d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 -- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee79d30 msgr2=0x56033ee7de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee79d30 0x56033ee7de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.591+0000 7f1155400700 1 -- 172.21.15.12:0/3411565283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56033ece18d0 con 0x56033ee73db0 2024-05-27T07:26:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.592+0000 7f1155400700 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee73db0 0x56033ee7d950 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1144008750 tx=0x7f114401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d251c716a487783a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.592+0000 7f1137e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.592+0000 7f1137e00700 1 -- 172.21.15.12:0/3411565283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144033060 con 0x56033ee73db0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.592+0000 7f1137e00700 1 -- 172.21.15.12:0/3411565283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9466+0+0 (secure 0 0 0) 0x7f114400f940 con 0x56033ee73db0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.592+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f114400fdc0 con 0x56033ee73db0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f1137e00700 1 -- 172.21.15.12:0/3411565283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f114403a050 con 0x56033ee73db0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f116ba8db80 10 client.15873 fetch_fsmap learned FSMap version 40 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56033ed62060 con 0x56033ee73db0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f1137e00700 1 -- 172.21.15.12:0/3411565283 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f114400fdc0 con 0x56033ee73db0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f1137e00700 1 -- 172.21.15.12:0/3411565283 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f1144004780 con 0x56033ee73db0 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f116ba8db80 10 client.15873 fetch_fsmap finished waiting for FSMap version 40 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f116ba8db80 10 client.15873 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.593+0000 7f116ba8db80 20 client.15873 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.594+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033ee8d5a0 0x56033ee8f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.594+0000 7f116ba8db80 4 client.15873 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.594+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56033eda3d90 con 0x56033ee8d5a0 2024-05-27T07:26:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.594+0000 7f1155e00700 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033ee8d5a0 0x56033ee8f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.596+0000 7f1155e00700 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033ee8d5a0 0x56033ee8f9c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c897507369f1a1 server_cookie=701e2a07d83bf990 in_seq=0 out_seq=0 2024-05-27T07:26:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.597+0000 7f1137e00700 10 client.15873 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:26:30.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f1137e00700 1 -- 172.21.15.12:0/3411565283 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56033eda3d90 con 0x56033ee8d5a0 2024-05-27T07:26:30.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f1137e00700 10 client.15873 handle_command_reply: tid=0 2024-05-27T07:26:30.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 client.15873 shutdown 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee73db0 msgr2=0x56033ee7d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee73db0 0x56033ee7d950 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1144008750 tx=0x7f114401cc60 comp rx=0 tx=0).stop 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 shutdown_connections 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033ee8d5a0 0x56033ee8f9c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee79d30 0x56033ee7de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee7d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3411565283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee73db0 0x56033ee7d950 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 >> 172.21.15.12:0/3411565283 conn(0x56033ed54fc0 msgr2=0x56033ee6e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f1137e00700 0 client.15873 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 shutdown_connections 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.601+0000 7f116ba8db80 1 -- 172.21.15.12:0/3411565283 wait complete. 2024-05-27T07:26:30.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.602+0000 7f116ba8db80 20 client.15873 trim_cache size 0 max 16384 2024-05-27T07:26:30.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.626+0000 7f116ba8db80 1 Processor -- start 2024-05-27T07:26:30.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.626+0000 7f116ba8db80 1 -- start start 2024-05-27T07:26:30.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee73870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee73db0 0x56033ee89a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee7f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f116ba8db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f11581b93d0 con 0x56033ee79d30 2024-05-27T07:26:30.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f116ba8db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f11581b6c00 con 0x56033ee73db0 2024-05-27T07:26:30.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f116ba8db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11581b4a50 con 0x56033ecf4e40 2024-05-27T07:26:30.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f1156800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee7f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f1156800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee7f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39982/0 (socket says 172.21.15.12:39982) 2024-05-27T07:26:30.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f1156800700 1 -- 172.21.15.12:0/1253676787 learned_addr learned my addr 172.21.15.12:0/1253676787 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:30.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.627+0000 7f1155e00700 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee73870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.628+0000 7f1155400700 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee73db0 0x56033ee89a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.628+0000 7f1156800700 1 -- 172.21.15.12:0/1253676787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 msgr2=0x56033ee73870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.628+0000 7f1156800700 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee73870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.628+0000 7f1156800700 1 -- 172.21.15.12:0/1253676787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee73db0 msgr2=0x56033ee89a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.628+0000 7f1156800700 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee73db0 0x56033ee89a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.628+0000 7f1156800700 1 -- 172.21.15.12:0/1253676787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56033ee70330 con 0x56033ee79d30 2024-05-27T07:26:30.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.628+0000 7f1156800700 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee7f8b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1150009fa0 tx=0x7f11500077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6c1f68f554813e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.630+0000 7f1142a00700 1 -- 172.21.15.12:0/1253676787 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115001b070 con 0x56033ee79d30 2024-05-27T07:26:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.630+0000 7f1142a00700 1 -- 172.21.15.12:0/1253676787 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11500178d0 con 0x56033ee79d30 2024-05-27T07:26:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.630+0000 7f1142a00700 1 -- 172.21.15.12:0/1253676787 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11500168a0 con 0x56033ee79d30 2024-05-27T07:26:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 -- 172.21.15.12:0/1253676787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 msgr2=0x56033ee7f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee7f8b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1150009fa0 tx=0x7f11500077e0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 -- 172.21.15.12:0/1253676787 shutdown_connections 2024-05-27T07:26:30.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033ee7f8b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ecf4e40 0x56033ee73870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 --2- 172.21.15.12:0/1253676787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ee73db0 0x56033ee89a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 -- 172.21.15.12:0/1253676787 >> 172.21.15.12:0/1253676787 conn(0x56033ed54fc0 msgr2=0x56033ee7bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:30.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 -- 172.21.15.12:0/1253676787 shutdown_connections 2024-05-27T07:26:30.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.631+0000 7f116ba8db80 1 -- 172.21.15.12:0/1253676787 wait complete. 2024-05-27T07:26:30.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.632+0000 7f116ba8db80 1 -- start start 2024-05-27T07:26:30.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.632+0000 7f116ba8db80 1 Processor -- start 2024-05-27T07:26:30.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee8fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f116ba8db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033eef5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f116ba8db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f11581b50c0 con 0x56033ee79d30 2024-05-27T07:26:30.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f116ba8db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f11581b6530 con 0x56033ecf4e40 2024-05-27T07:26:30.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f116ba8db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11581b5790 con 0x56033ee73db0 2024-05-27T07:26:30.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f116ba8db80 10 client.0 _ll_register_callbacks cb 0x56033ed3bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:30.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f1156800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033eef5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f1156800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033eef5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39996/0 (socket says 172.21.15.12:39996) 2024-05-27T07:26:30.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f1156800700 1 -- 172.21.15.12:0/3895457434 learned_addr learned my addr 172.21.15.12:0/3895457434 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:30.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f1155e00700 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee8fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.633+0000 7f1155400700 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.634+0000 7f1156800700 1 -- 172.21.15.12:0/3895457434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 msgr2=0x56033ee900f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.634+0000 7f1156800700 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee900f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.634+0000 7f1156800700 1 -- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 msgr2=0x56033ee8fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:30.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.634+0000 7f1156800700 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee8fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.634+0000 7f1156800700 1 -- 172.21.15.12:0/3895457434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56033eef6340 con 0x56033ee79d30 2024-05-27T07:26:30.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.634+0000 7f1156800700 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033eef5e00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1150000f80 tx=0x7f1150004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada6e45e991d1e21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:30.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.634+0000 7f1137e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:30.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.635+0000 7f1137e00700 1 -- 172.21.15.12:0/3895457434 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115001b030 con 0x56033ee79d30 2024-05-27T07:26:30.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.635+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11500178d0 con 0x56033ee79d30 2024-05-27T07:26:30.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.635+0000 7f1137e00700 1 -- 172.21.15.12:0/3895457434 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9466+0+0 (secure 0 0 0) 0x7f1150016520 con 0x56033ee79d30 2024-05-27T07:26:30.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.635+0000 7f1137e00700 1 -- 172.21.15.12:0/3895457434 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1150031050 con 0x56033ee79d30 2024-05-27T07:26:30.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.635+0000 7f116ba8db80 10 client.15879 fetch_fsmap learned FSMap version 40 2024-05-27T07:26:30.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.635+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56033eef74f0 con 0x56033ee79d30 2024-05-27T07:26:30.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f1137e00700 1 -- 172.21.15.12:0/3895457434 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11500178d0 con 0x56033ee79d30 2024-05-27T07:26:30.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f1137e00700 1 -- 172.21.15.12:0/3895457434 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f1150020830 con 0x56033ee79d30 2024-05-27T07:26:30.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f116ba8db80 10 client.15879 fetch_fsmap finished waiting for FSMap version 40 2024-05-27T07:26:30.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f116ba8db80 10 client.15879 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:26:30.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f116ba8db80 20 client.15879 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:30.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033eef7ad0 0x56033eef9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:30.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f116ba8db80 4 client.15879 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:30.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.636+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56033ed9d570 con 0x56033eef7ad0 2024-05-27T07:26:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.637+0000 7f1155e00700 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033eef7ad0 0x56033eef9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.638+0000 7f1155e00700 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033eef7ad0 0x56033eef9ef0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d80fcb200985855d server_cookie=911ced6f3fdf1120 in_seq=0 out_seq=0 2024-05-27T07:26:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.638+0000 7f1137e00700 10 client.15879 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:26:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f1137e00700 1 -- 172.21.15.12:0/3895457434 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56033ed9d570 con 0x56033eef7ad0 2024-05-27T07:26:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f1137e00700 10 client.15879 handle_command_reply: tid=0 2024-05-27T07:26:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 client.15879 shutdown 2024-05-27T07:26:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 msgr2=0x56033eef5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033eef5e00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1150000f80 tx=0x7f1150004db0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 shutdown_connections 2024-05-27T07:26:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56033eef7ad0 0x56033eef9ef0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56033ecf4e40 0x56033ee8fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56033ee79d30 0x56033eef5e00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 --2- 172.21.15.12:0/3895457434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56033ee73db0 0x56033ee900f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 >> 172.21.15.12:0/3895457434 conn(0x56033ed54fc0 msgr2=0x56033ee779a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f1137e00700 0 client.15879 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:26:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 shutdown_connections 2024-05-27T07:26:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.639+0000 7f116ba8db80 1 -- 172.21.15.12:0/3895457434 wait complete. 2024-05-27T07:26:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.640+0000 7f116ba8db80 20 client.15879 trim_cache size 0 max 16384 2024-05-27T07:26:30.637 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:26:30.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.640+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f113c0b29f0 msgr2=0x7f113c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.640+0000 7f115d000700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f113c0b29f0 0x7f113c0b4f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1144005fd0 tx=0x7f1144005e20 comp rx=0 tx=0).stop 2024-05-27T07:26:30.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.640+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 msgr2=0x7f11581a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:30.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.640+0000 7f115d000700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f11581a1300 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1148000c00 tx=0x7f1148009f90 comp rx=0 tx=0).stop 2024-05-27T07:26:30.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 shutdown_connections 2024-05-27T07:26:30.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f113c0b29f0 0x7f113c0b4f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1158115360 0x7f11581a5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1158075b50 0x7f11581a1300 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 --2- 172.21.15.12:0/3960941918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11580764a0 0x7f11581a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:30.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 >> 172.21.15.12:0/3960941918 conn(0x7f1158100030 msgr2=0x7f1158100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:30.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 shutdown_connections 2024-05-27T07:26:30.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:30.641+0000 7f115d000700 1 -- 172.21.15.12:0/3960941918 wait complete. 2024-05-27T07:26:30.666 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/ext4.h presence... no 2024-05-27T07:26:30.667 INFO:tasks.workunit.client.0.smithi012.stdout:checking for linux/ext4.h... no 2024-05-27T07:26:30.791 INFO:tasks.workunit.client.0.smithi012.stdout:checking for xfs/xfs_log_format.h... no 2024-05-27T07:26:31.045 INFO:tasks.workunit.client.0.smithi012.stdout:checking for uuid_compare... no 2024-05-27T07:26:31.280 INFO:tasks.workunit.client.0.smithi012.stdout:checking for uuid_compare in -luuid... yes 2024-05-27T07:26:31.683 INFO:tasks.workunit.client.0.smithi012.stdout:checking for xfs/xfs.h... yes 2024-05-27T07:26:31.692 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:26:31.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:31 smithi012 ceph-mon[31473]: pgmap v135: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 434 KiB/s wr, 57 op/s 2024-05-27T07:26:31.807 INFO:tasks.workunit.client.0.smithi012.stdout:checking for xfs/libxfs.h... no 2024-05-27T07:26:31.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:31 smithi162 ceph-mon[35676]: pgmap v135: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 434 KiB/s wr, 57 op/s 2024-05-27T07:26:31.875 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:31 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:26:31.875 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:31 smithi162 systemd[1]: Stopped Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:31.875 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:31 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.620s CPU time. 2024-05-27T07:26:31.875 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:31 smithi162 systemd[1]: Starting Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:31.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:31 smithi062 ceph-mon[35304]: pgmap v135: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 434 KiB/s wr, 57 op/s 2024-05-27T07:26:31.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:31 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:26:31.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:31 smithi062 systemd[1]: Stopped Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:31.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:31 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.785s CPU time. 2024-05-27T07:26:31.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:31 smithi062 systemd[1]: Starting Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:31.919 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether xlog_assign_lsn is declared... no 2024-05-27T07:26:32.041 INFO:tasks.workunit.client.0.smithi012.stdout:checking for xfs/xqm.h... yes 2024-05-27T07:26:32.181 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:31 smithi162 podman[72077]: 2024-05-27 07:26:31.789599635 +0000 UTC m=+0.037544679 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:32.181 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:31 smithi162 podman[72077]: 2024-05-27 07:26:31.921068793 +0000 UTC m=+0.169013873 container create e32c11f3a5bf2ef7a3bd8c046145c9a9030c4f31657b1977c2496d8a1fafef50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:26:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:32.220 INFO:tasks.workunit.client.0.smithi012.stdout:checking xfsctl from xfs/xfs.h... ok 2024-05-27T07:26:32.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:31 smithi062 podman[70858]: 2024-05-27 07:26:31.824971854 +0000 UTC m=+0.032878260 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:32.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:31 smithi062 podman[70858]: 2024-05-27 07:26:31.953522863 +0000 UTC m=+0.161429306 container create dd72b32a020c9433a1856110f9468b1f27b853353d30d616c4297320cd81451e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T07:26:32.362 INFO:tasks.workunit.client.0.smithi012.stdout:checking xfs/handle.h usability... yes 2024-05-27T07:26:32.432 INFO:tasks.workunit.client.0.smithi012.stdout:checking xfs/handle.h presence... yes 2024-05-27T07:26:32.433 INFO:tasks.workunit.client.0.smithi012.stdout:checking for xfs/handle.h... yes 2024-05-27T07:26:32.474 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 podman[72077]: 2024-05-27 07:26:32.191687703 +0000 UTC m=+0.439632847 container init e32c11f3a5bf2ef7a3bd8c046145c9a9030c4f31657b1977c2496d8a1fafef50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:26:32.474 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 podman[72077]: 2024-05-27 07:26:32.201702064 +0000 UTC m=+0.449647122 container start e32c11f3a5bf2ef7a3bd8c046145c9a9030c4f31657b1977c2496d8a1fafef50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:26:32.474 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 bash[72077]: e32c11f3a5bf2ef7a3bd8c046145c9a9030c4f31657b1977c2496d8a1fafef50 2024-05-27T07:26:32.474 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 systemd[1]: Started Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:32.474 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-mds[72096]: -- 172.21.15.162:0/2362043783 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee58ef5a40 con 0x55ee58f3d600 2024-05-27T07:26:32.474 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: starting mds.f at 2024-05-27T07:26:32.643 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 podman[70858]: 2024-05-27 07:26:32.268920132 +0000 UTC m=+0.476826601 container init dd72b32a020c9433a1856110f9468b1f27b853353d30d616c4297320cd81451e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:26:32.643 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 podman[70858]: 2024-05-27 07:26:32.283711425 +0000 UTC m=+0.491617888 container start dd72b32a020c9433a1856110f9468b1f27b853353d30d616c4297320cd81451e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T07:26:32.643 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 bash[70858]: dd72b32a020c9433a1856110f9468b1f27b853353d30d616c4297320cd81451e 2024-05-27T07:26:32.643 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 systemd[1]: Started Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:32.643 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-mds[70877]: -- 172.21.15.62:0/3498404207 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558ab69eba40 con 0x558ab6a33600 2024-05-27T07:26:32.643 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: starting mds.k at 2024-05-27T07:26:32.662 INFO:tasks.workunit.client.0.smithi012.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-05-27T07:26:32.815 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/acl.h usability... yes 2024-05-27T07:26:32.881 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/acl.h presence... yes 2024-05-27T07:26:32.882 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/acl.h... yes 2024-05-27T07:26:32.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:32 smithi162 ceph-mon[35676]: daemon mds.f restarted 2024-05-27T07:26:32.891 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f484b200700 time 2024-05-27T07:26:32.853432+0000 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4860c81ba6] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.892 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: clone() 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: *** Caught signal (Aborted) ** 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: in thread 7f484b200700 thread_name:md_log_replay 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.855+0000 7f484b200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f484b200700 time 2024-05-27T07:26:32.853432+0000 2024-05-27T07:26:32.893 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4860c81ba6] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.894 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: clone() 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485f7fed20] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: gsignal() 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: abort() 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4860c81ced] 2024-05-27T07:26:32.895 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.896 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.896 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.896 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.896 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.896 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.896 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 19: clone() 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.858+0000 7f484b200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: in thread 7f484b200700 thread_name:md_log_replay 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.897 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485f7fed20] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: gsignal() 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: abort() 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4860c81ced] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.898 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 19: clone() 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: -1> 2024-05-27T07:26:32.855+0000 7f484b200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f484b200700 time 2024-05-27T07:26:32.853432+0000 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4860c81ba6] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: clone() 2024-05-27T07:26:32.900 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 0> 2024-05-27T07:26:32.858+0000 7f484b200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: in thread 7f484b200700 thread_name:md_log_replay 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485f7fed20] 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: gsignal() 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: abort() 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4860c81ced] 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.901 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 19: clone() 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.902 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: -1> 2024-05-27T07:26:32.855+0000 7f484b200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f484b200700 time 2024-05-27T07:26:32.853432+0000 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4860c81ba6] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.903 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: clone() 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 0> 2024-05-27T07:26:32.858+0000 7f484b200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: in thread 7f484b200700 thread_name:md_log_replay 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.904 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f485f7fed20] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2: gsignal() 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 3: abort() 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4860c81ced] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4860c81dd1] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ee53f0c23d] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ee53ef7971] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ee53ef5fa0] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ee53ef6651] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ee53e8b347] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ee53b8787e] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ee53b8ce94] 2024-05-27T07:26:32.905 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ee53b97969] 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ee53f3048c] 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ee53f399cb] 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 16: (Thread::entry_wrapper()+0x43) [0x7f4860c5072d] 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4860c50749] 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f485f7f41ca] 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 19: clone() 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:32.906 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72092]: 2024-05-27T07:26:32.910 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:32 smithi062 ceph-mon[35304]: daemon mds.f restarted 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f70600700 time 2024-05-27T07:26:32.799203+0000 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f861e8ba6] 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.911 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: clone() 2024-05-27T07:26:32.912 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.799+0000 7f6f70600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f70600700 time 2024-05-27T07:26:32.799203+0000 2024-05-27T07:26:32.913 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.913 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f861e8ba6] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.914 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.915 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.915 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.915 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.915 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: clone() 2024-05-27T07:26:32.915 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.916 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: *** Caught signal (Aborted) ** 2024-05-27T07:26:32.916 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: in thread 7f6f70600700 thread_name:md_log_replay 2024-05-27T07:26:32.916 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.916 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f84d65d20] 2024-05-27T07:26:32.916 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: gsignal() 2024-05-27T07:26:32.916 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: abort() 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f861e8ced] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.917 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 19: clone() 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.802+0000 7f6f70600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: in thread 7f6f70600700 thread_name:md_log_replay 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f84d65d20] 2024-05-27T07:26:32.918 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: gsignal() 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: abort() 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f861e8ced] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.919 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 19: clone() 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: -1> 2024-05-27T07:26:32.799+0000 7f6f70600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f70600700 time 2024-05-27T07:26:32.799203+0000 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f861e8ba6] 2024-05-27T07:26:32.920 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.921 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: clone() 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 0> 2024-05-27T07:26:32.802+0000 7f6f70600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: in thread 7f6f70600700 thread_name:md_log_replay 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f84d65d20] 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: gsignal() 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: abort() 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f861e8ced] 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.922 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.923 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.924 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.924 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.924 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.925 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 19: clone() 2024-05-27T07:26:32.925 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:32.925 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.925 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: -1> 2024-05-27T07:26:32.799+0000 7f6f70600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f70600700 time 2024-05-27T07:26:32.799203+0000 2024-05-27T07:26:32.925 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:32.925 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.925 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f861e8ba6] 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.926 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: clone() 2024-05-27T07:26:32.927 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 0> 2024-05-27T07:26:32.802+0000 7f6f70600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: in thread 7f6f70600700 thread_name:md_log_replay 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f84d65d20] 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2: gsignal() 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 3: abort() 2024-05-27T07:26:32.928 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f861e8ced] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f861e8dd1] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558a9283e23d] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558a92829971] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 8: (SnapRealm::check_cache() const+0x13e) [0x558a92827fa0] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558a92828651] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558a927bd347] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558a924b987e] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558a924bee94] 2024-05-27T07:26:32.929 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558a924c9969] 2024-05-27T07:26:32.930 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 14: (MDLog::_replay_thread()+0x1f38) [0x558a9286248c] 2024-05-27T07:26:32.930 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558a9286b9cb] 2024-05-27T07:26:32.930 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f861b772d] 2024-05-27T07:26:32.930 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f861b7749] 2024-05-27T07:26:32.930 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f84d5b1ca] 2024-05-27T07:26:32.930 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 19: clone() 2024-05-27T07:26:32.930 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:32.931 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[70873]: 2024-05-27T07:26:32.931 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:32 smithi062 podman[70908]: 2024-05-27 07:26:32.920919293 +0000 UTC m=+0.034156168 container died dd72b32a020c9433a1856110f9468b1f27b853353d30d616c4297320cd81451e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:26:33.035 INFO:tasks.workunit.client.0.smithi012.stdout:checking acl/libacl.h usability... yes 2024-05-27T07:26:33.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:32 smithi012 ceph-mon[31473]: daemon mds.f restarted 2024-05-27T07:26:33.120 INFO:tasks.workunit.client.0.smithi012.stdout:checking acl/libacl.h presence... yes 2024-05-27T07:26:33.122 INFO:tasks.workunit.client.0.smithi012.stdout:checking for acl/libacl.h... yes 2024-05-27T07:26:33.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:32 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:26:33.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:32 smithi162 systemd[1]: Stopped Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:33.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:32 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.163s CPU time. 2024-05-27T07:26:33.224 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:33 smithi162 systemd[1]: Starting Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:33.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:32 smithi162 podman[72127]: 2024-05-27 07:26:32.971150242 +0000 UTC m=+0.031768701 container died e32c11f3a5bf2ef7a3bd8c046145c9a9030c4f31657b1977c2496d8a1fafef50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:26:33.372 INFO:tasks.workunit.client.0.smithi012.stdout:checking for acl_init in -lacl... yes 2024-05-27T07:26:33.569 INFO:tasks.workunit.client.0.smithi012.stdout:checking gdbm-ndbm.h usability... no 2024-05-27T07:26:33.714 INFO:tasks.workunit.client.0.smithi012.stdout:checking gdbm-ndbm.h presence... no 2024-05-27T07:26:33.715 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gdbm-ndbm.h... no 2024-05-27T07:26:33.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: pgmap v136: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 366 KiB/s wr, 48 op/s 2024-05-27T07:26:33.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/3013462645,v1:172.21.15.162:6837/3013462645] up:boot 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: daemon mds.k restarted 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6832/4195865533,v1:172.21.15.62:6833/4195865533] up:boot 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:33.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:33 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:33.867 INFO:tasks.workunit.client.0.smithi012.stdout:checking gdbm/ndbm.h usability... no 2024-05-27T07:26:33.972 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:33 smithi162 podman[72127]: 2024-05-27 07:26:33.713024093 +0000 UTC m=+0.773642574 container remove e32c11f3a5bf2ef7a3bd8c046145c9a9030c4f31657b1977c2496d8a1fafef50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:26:33.972 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:33 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:33.972 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: pgmap v136: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 366 KiB/s wr, 48 op/s 2024-05-27T07:26:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:26:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/3013462645,v1:172.21.15.162:6837/3013462645] up:boot 2024-05-27T07:26:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: daemon mds.k restarted 2024-05-27T07:26:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:26:33.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6832/4195865533,v1:172.21.15.62:6833/4195865533] up:boot 2024-05-27T07:26:33.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:33.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:33 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:33.974 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: pgmap v136: 129 pgs: 129 active+clean; 69 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 366 KiB/s wr, 48 op/s 2024-05-27T07:26:33.975 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:26:33.975 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/3013462645,v1:172.21.15.162:6837/3013462645] up:boot 2024-05-27T07:26:33.975 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: daemon mds.k restarted 2024-05-27T07:26:33.975 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:33.975 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:33.975 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:26:33.976 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6832/4195865533,v1:172.21.15.62:6833/4195865533] up:boot 2024-05-27T07:26:33.976 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:33.976 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:33 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:34.009 INFO:tasks.workunit.client.0.smithi012.stdout:checking gdbm/ndbm.h presence... no 2024-05-27T07:26:34.011 INFO:tasks.workunit.client.0.smithi012.stdout:checking for gdbm/ndbm.h... no 2024-05-27T07:26:34.166 INFO:tasks.workunit.client.0.smithi012.stdout:checking ndbm.h usability... no 2024-05-27T07:26:34.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:33 smithi162 podman[72201]: 2024-05-27 07:26:33.886737317 +0000 UTC m=+0.031179587 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:34.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:33 smithi162 podman[72201]: 2024-05-27 07:26:33.989014746 +0000 UTC m=+0.133457052 container create 08bef980ea11cbbb3561bdc3a25c18e87ab490b46ea760e2981a96847fa5ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:26:34.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:33 smithi062 podman[70908]: 2024-05-27 07:26:33.982467194 +0000 UTC m=+1.095704135 container remove dd72b32a020c9433a1856110f9468b1f27b853353d30d616c4297320cd81451e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-05-27T07:26:34.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:33 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:34.308 INFO:tasks.workunit.client.0.smithi012.stdout:checking ndbm.h presence... no 2024-05-27T07:26:34.310 INFO:tasks.workunit.client.0.smithi012.stdout:checking for ndbm.h... no 2024-05-27T07:26:34.427 INFO:tasks.workunit.client.0.smithi012.stdout:checking libaio.h usability... yes 2024-05-27T07:26:34.505 INFO:tasks.workunit.client.0.smithi012.stdout:checking libaio.h presence... yes 2024-05-27T07:26:34.506 INFO:tasks.workunit.client.0.smithi012.stdout:checking for libaio.h... yes 2024-05-27T07:26:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 -- 172.21.15.12:0/1852875475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 msgr2=0x7f95b0117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 --2- 172.21.15.12:0/1852875475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b0117780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f95a800b0a0 tx=0x7f95a800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 -- 172.21.15.12:0/1852875475 shutdown_connections 2024-05-27T07:26:34.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 --2- 172.21.15.12:0/1852875475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95b0075b50 0x7f95b0075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 --2- 172.21.15.12:0/1852875475 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b0117780 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 --2- 172.21.15.12:0/1852875475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95b00764a0 0x7f95b0114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 -- 172.21.15.12:0/1852875475 >> 172.21.15.12:0/1852875475 conn(0x7f95b0100030 msgr2=0x7f95b0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:34.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 -- 172.21.15.12:0/1852875475 shutdown_connections 2024-05-27T07:26:34.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.532+0000 7f95b5800700 1 -- 172.21.15.12:0/1852875475 wait complete. 2024-05-27T07:26:34.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 Processor -- start 2024-05-27T07:26:34.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 -- start start 2024-05-27T07:26:34.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95b0075b50 0x7f95b01a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95b00764a0 0x7f95b01a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b01a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f95b011a920 con 0x7f95b0115360 2024-05-27T07:26:34.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f95b011a7a0 con 0x7f95b0075b50 2024-05-27T07:26:34.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.534+0000 7f95b5800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f95b011aaa0 con 0x7f95b00764a0 2024-05-27T07:26:34.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95af400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b01a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95af400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b01a5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59434/0 (socket says 172.21.15.12:59434) 2024-05-27T07:26:34.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95af400700 1 -- 172.21.15.12:0/4263641910 learned_addr learned my addr 172.21.15.12:0/4263641910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:34.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95aea00700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95b0075b50 0x7f95b01a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95ae000700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95b00764a0 0x7f95b01a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95af400700 1 -- 172.21.15.12:0/4263641910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95b00764a0 msgr2=0x7f95b01a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95af400700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95b00764a0 0x7f95b01a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95af400700 1 -- 172.21.15.12:0/4263641910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95b0075b50 msgr2=0x7f95b01a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.535+0000 7f95af400700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95b0075b50 0x7f95b01a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.536+0000 7f95af400700 1 -- 172.21.15.12:0/4263641910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b01a61c0 con 0x7f95b0115360 2024-05-27T07:26:34.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.536+0000 7f95af400700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b01a5b20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f95a800b830 tx=0x7f95a8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:34.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.536+0000 7f95a7400700 1 -- 172.21.15.12:0/4263641910 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8021070 con 0x7f95b0115360 2024-05-27T07:26:34.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.536+0000 7f95a7400700 1 -- 172.21.15.12:0/4263641910 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95a8007b80 con 0x7f95b0115360 2024-05-27T07:26:34.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.536+0000 7f95a7400700 1 -- 172.21.15.12:0/4263641910 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a80052a0 con 0x7f95b0115360 2024-05-27T07:26:34.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.536+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b01a6450 con 0x7f95b0115360 2024-05-27T07:26:34.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.537+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b01a68e0 con 0x7f95b0115360 2024-05-27T07:26:34.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.541+0000 7f95a7400700 1 -- 172.21.15.12:0/4263641910 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f95a8024070 con 0x7f95b0115360 2024-05-27T07:26:34.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.542+0000 7f95c43d3b80 1 Processor -- start 2024-05-27T07:26:34.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.542+0000 7f95c43d3b80 1 -- start start 2024-05-27T07:26:34.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x557041a047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x5570419fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x5570419ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55704192ce40 con 0x557041a04d30 2024-05-27T07:26:34.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55704191c9e0 con 0x5570419fedb0 2024-05-27T07:26:34.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557041920a20 con 0x55704187fe40 2024-05-27T07:26:34.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95aea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x5570419fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95ae000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x557041a047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95ae000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x557041a047f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:41562/0 (socket says 172.21.15.12:41562) 2024-05-27T07:26:34.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95ae000700 1 -- 172.21.15.12:0/144620197 learned_addr learned my addr 172.21.15.12:0/144620197 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:34.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95ae000700 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x5570419ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.543+0000 7f95a7400700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f95940b29f0 0x7f95940b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.544+0000 7f95a7400700 1 -- 172.21.15.12:0/4263641910 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f95a800f020 con 0x7f95b0115360 2024-05-27T07:26:34.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.544+0000 7f95af400700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f95940b29f0 0x7f95940b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.544+0000 7f95aea00700 1 -- 172.21.15.12:0/144620197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 msgr2=0x557041a047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.544+0000 7f95aea00700 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x557041a047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.544+0000 7f95aea00700 1 -- 172.21.15.12:0/144620197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 msgr2=0x5570419ff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.544+0000 7f95aea00700 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x5570419ff1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.544+0000 7f95aea00700 1 -- 172.21.15.12:0/144620197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570418e5420 con 0x557041a04d30 2024-05-27T07:26:34.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.545+0000 7f95aea00700 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x5570419fe870 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f95a000bc80 tx=0x7f95a000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facb11afe8c40043 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:34.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.545+0000 7f95af400700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f95940b29f0 0x7f95940b4f00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95a80e1350 tx=0x7f95a8009880 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:34.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.545+0000 7f959be00700 1 -- 172.21.15.12:0/144620197 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0010ca0 con 0x557041a04d30 2024-05-27T07:26:34.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.545+0000 7f959be00700 1 -- 172.21.15.12:0/144620197 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95a0010e40 con 0x557041a04d30 2024-05-27T07:26:34.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.545+0000 7f959be00700 1 -- 172.21.15.12:0/144620197 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0005020 con 0x557041a04d30 2024-05-27T07:26:34.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 -- 172.21.15.12:0/144620197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 msgr2=0x5570419fe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x5570419fe870 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f95a000bc80 tx=0x7f95a000bf90 comp rx=0 tx=0).stop 2024-05-27T07:26:34.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 -- 172.21.15.12:0/144620197 shutdown_connections 2024-05-27T07:26:34.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x5570419ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x557041a047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/144620197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x5570419fe870 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 -- 172.21.15.12:0/144620197 >> 172.21.15.12:0/144620197 conn(0x5570418dffc0 msgr2=0x5570419f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:34.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.546+0000 7f95c43d3b80 1 -- 172.21.15.12:0/144620197 shutdown_connections 2024-05-27T07:26:34.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.547+0000 7f95c43d3b80 1 -- 172.21.15.12:0/144620197 wait complete. 2024-05-27T07:26:34.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.547+0000 7f95c43d3b80 1 -- start start 2024-05-27T07:26:34.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.547+0000 7f95c43d3b80 1 Processor -- start 2024-05-27T07:26:34.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a084e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a08a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a08f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557041822fa0 con 0x55704187fe40 2024-05-27T07:26:34.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5570418bd760 con 0x5570419fedb0 2024-05-27T07:26:34.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557041a0b800 con 0x557041a04d30 2024-05-27T07:26:34.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95c43d3b80 10 client.0 _ll_register_callbacks cb 0x5570418c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:34.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.548+0000 7f95ae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a084e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59452/0 (socket says 172.21.15.12:59452) 2024-05-27T07:26:34.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 -- 172.21.15.12:0/2182836058 learned_addr learned my addr 172.21.15.12:0/2182836058 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:34.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95aea00700 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a08a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a08f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 -- 172.21.15.12:0/2182836058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 msgr2=0x557041a08f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a08f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 -- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 msgr2=0x557041a08a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a08a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.549+0000 7f95ae000700 1 -- 172.21.15.12:0/2182836058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55704186c8d0 con 0x55704187fe40 2024-05-27T07:26:34.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f95ae000700 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a084e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f959c0036d0 tx=0x7f959c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c8750d77db888c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:34.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f9598c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:34.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f9598c00700 1 -- 172.21.15.12:0/2182836058 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c012770 con 0x55704187fe40 2024-05-27T07:26:34.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f9598c00700 1 -- 172.21.15.12:0/2182836058 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f959c012910 con 0x55704187fe40 2024-05-27T07:26:34.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95a0010e40 con 0x55704187fe40 2024-05-27T07:26:34.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f9598c00700 1 -- 172.21.15.12:0/2182836058 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f959c012b10 con 0x55704187fe40 2024-05-27T07:26:34.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f95c43d3b80 10 client.15900 fetch_fsmap learned FSMap version 42 2024-05-27T07:26:34.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.550+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5570418ed060 con 0x55704187fe40 2024-05-27T07:26:34.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f9598c00700 1 -- 172.21.15.12:0/2182836058 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95a0010e40 con 0x55704187fe40 2024-05-27T07:26:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f9598c00700 1 -- 172.21.15.12:0/2182836058 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7f959c021ba0 con 0x55704187fe40 2024-05-27T07:26:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f95c43d3b80 10 client.15900 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T07:26:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f95c43d3b80 10 client.15900 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:26:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f95c43d3b80 20 client.15900 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a18710 0x557041a1ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f95c43d3b80 4 client.15900 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:34.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.551+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55704192ed90 con 0x557041a18710 2024-05-27T07:26:34.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.552+0000 7f95aea00700 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a18710 0x557041a1ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.553+0000 7f95aea00700 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a18710 0x557041a1ab70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96e53ed5ed6e122f server_cookie=af250db916be6352 in_seq=0 out_seq=0 2024-05-27T07:26:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.553+0000 7f9598c00700 10 client.15900 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:26:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.555+0000 7f9598c00700 1 -- 172.21.15.12:0/2182836058 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55704192ed90 con 0x557041a18710 2024-05-27T07:26:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.555+0000 7f9598c00700 10 client.15900 handle_command_reply: tid=0 2024-05-27T07:26:34.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.555+0000 7f95c43d3b80 1 client.15900 shutdown 2024-05-27T07:26:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.555+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 msgr2=0x557041a084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.555+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a084e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f959c0036d0 tx=0x7f959c004750 comp rx=0 tx=0).stop 2024-05-27T07:26:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 shutdown_connections 2024-05-27T07:26:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a18710 0x557041a1ab70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a08f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a084e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2182836058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a08a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f9598c00700 0 client.15900 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:26:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 >> 172.21.15.12:0/2182836058 conn(0x5570418dffc0 msgr2=0x5570419f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 shutdown_connections 2024-05-27T07:26:34.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2182836058 wait complete. 2024-05-27T07:26:34.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.556+0000 7f95c43d3b80 20 client.15900 trim_cache size 0 max 16384 2024-05-27T07:26:34.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.574+0000 7f95c43d3b80 1 Processor -- start 2024-05-27T07:26:34.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.575+0000 7f95c43d3b80 1 -- start start 2024-05-27T07:26:34.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.575+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x5570419fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.575+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a02920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.575+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x557041a148d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.575+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01b94a0 con 0x557041a04d30 2024-05-27T07:26:34.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.575+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01b6cd0 con 0x5570419fedb0 2024-05-27T07:26:34.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.575+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01b4b20 con 0x55704187fe40 2024-05-27T07:26:34.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x557041a148d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a02920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x557041a148d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59466/0 (socket says 172.21.15.12:59466) 2024-05-27T07:26:34.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 -- 172.21.15.12:0/2894750022 learned_addr learned my addr 172.21.15.12:0/2894750022 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:34.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x5570419fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 -- 172.21.15.12:0/2894750022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 msgr2=0x5570419fe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x5570419fe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 -- 172.21.15.12:0/2894750022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 msgr2=0x557041a02920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a02920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.576+0000 7f95ae000700 1 -- 172.21.15.12:0/2894750022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557041a092d0 con 0x557041a04d30 2024-05-27T07:26:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.577+0000 7f95ae000700 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x557041a148d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f959c00bce0 tx=0x7f959c004450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc95bd09232e8c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.577+0000 7f959b400700 1 -- 172.21.15.12:0/2894750022 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c010720 con 0x557041a04d30 2024-05-27T07:26:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.577+0000 7f959b400700 1 -- 172.21.15.12:0/2894750022 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f959c012770 con 0x557041a04d30 2024-05-27T07:26:34.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.577+0000 7f959b400700 1 -- 172.21.15.12:0/2894750022 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c005070 con 0x557041a04d30 2024-05-27T07:26:34.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2894750022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 msgr2=0x557041a148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x557041a148d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f959c00bce0 tx=0x7f959c004450 comp rx=0 tx=0).stop 2024-05-27T07:26:34.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2894750022 shutdown_connections 2024-05-27T07:26:34.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557041a04d30 0x557041a148d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704187fe40 0x5570419fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/2894750022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a02920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2894750022 >> 172.21.15.12:0/2894750022 conn(0x5570418dffc0 msgr2=0x557041a02210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2894750022 shutdown_connections 2024-05-27T07:26:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.578+0000 7f95c43d3b80 1 -- 172.21.15.12:0/2894750022 wait complete. 2024-05-27T07:26:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.579+0000 7f95c43d3b80 1 -- start start 2024-05-27T07:26:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.579+0000 7f95c43d3b80 1 Processor -- start 2024-05-27T07:26:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.580+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a10ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.580+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a11000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.580+0000 7f95c43d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a1f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.580+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01b5190 con 0x55704187fe40 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.580+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01b6600 con 0x5570419fedb0 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.580+0000 7f95c43d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01b5860 con 0x557041a04d30 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.580+0000 7f95c43d3b80 10 client.0 _ll_register_callbacks cb 0x5570418c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a10ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a11000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a10ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59480/0 (socket says 172.21.15.12:59480) 2024-05-27T07:26:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 -- 172.21.15.12:0/4204138131 learned_addr learned my addr 172.21.15.12:0/4204138131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a1f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 -- 172.21.15.12:0/4204138131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 msgr2=0x557041a1f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a1f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 -- 172.21.15.12:0/4204138131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 msgr2=0x557041a11000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a11000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.581+0000 7f95ae000700 1 -- 172.21.15.12:0/4204138131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557041a20d40 con 0x55704187fe40 2024-05-27T07:26:34.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.582+0000 7f95ae000700 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a10ac0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ba70 tx=0x7f959c003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49a16cc0e37c169 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:34.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.582+0000 7f9598c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:34.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.582+0000 7f9598c00700 1 -- 172.21.15.12:0/4204138131 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c011ea0 con 0x55704187fe40 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.582+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f959c012770 con 0x55704187fe40 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.582+0000 7f9598c00700 1 -- 172.21.15.12:0/4204138131 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9466+0+0 (secure 0 0 0) 0x7f959c0109d0 con 0x55704187fe40 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f9598c00700 1 -- 172.21.15.12:0/4204138131 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f959c010bd0 con 0x55704187fe40 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f95c43d3b80 10 client.15906 fetch_fsmap learned FSMap version 42 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x557041a825c0 con 0x55704187fe40 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f9598c00700 1 -- 172.21.15.12:0/4204138131 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f959c012770 con 0x55704187fe40 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f9598c00700 1 -- 172.21.15.12:0/4204138131 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7f959c004760 con 0x55704187fe40 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f95c43d3b80 10 client.15906 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f95c43d3b80 10 client.15906 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:26:34.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.583+0000 7f95c43d3b80 20 client.15906 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:34.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.584+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a82c50 0x557041a85070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:34.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.584+0000 7f95c43d3b80 4 client.15906 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:34.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.584+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557041928570 con 0x557041a82c50 2024-05-27T07:26:34.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.585+0000 7f95aea00700 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a82c50 0x557041a85070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:34.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.586+0000 7f95aea00700 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a82c50 0x557041a85070 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21f0b1cffa95136 server_cookie=d215bef95b06489b in_seq=0 out_seq=0 2024-05-27T07:26:34.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.586+0000 7f9598c00700 10 client.15906 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:26:34.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.586+0000 7f9598c00700 1 -- 172.21.15.12:0/4204138131 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557041928570 con 0x557041a82c50 2024-05-27T07:26:34.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.586+0000 7f9598c00700 10 client.15906 handle_command_reply: tid=0 2024-05-27T07:26:34.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.586+0000 7f95c43d3b80 1 client.15906 shutdown 2024-05-27T07:26:34.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 msgr2=0x557041a10ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a10ac0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ba70 tx=0x7f959c003d70 comp rx=0 tx=0).stop 2024-05-27T07:26:34.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 shutdown_connections 2024-05-27T07:26:34.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x557041a82c50 0x557041a85070 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f9598c00700 0 client.15906 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:26:34.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557041a04d30 0x557041a1f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55704187fe40 0x557041a10ac0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 --2- 172.21.15.12:0/4204138131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570419fedb0 0x557041a11000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 >> 172.21.15.12:0/4204138131 conn(0x5570418dffc0 msgr2=0x557041a082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:34.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 shutdown_connections 2024-05-27T07:26:34.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.587+0000 7f95c43d3b80 1 -- 172.21.15.12:0/4204138131 wait complete. 2024-05-27T07:26:34.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.588+0000 7f95c43d3b80 20 client.15906 trim_cache size 0 max 16384 2024-05-27T07:26:34.578 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:26:34.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.588+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f95940b29f0 msgr2=0x7f95940b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.588+0000 7f95b5800700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f95940b29f0 0x7f95940b4f00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95a80e1350 tx=0x7f95a8009880 comp rx=0 tx=0).stop 2024-05-27T07:26:34.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.588+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 msgr2=0x7f95b01a5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:34.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.588+0000 7f95b5800700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b01a5b20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f95a800b830 tx=0x7f95a8004d00 comp rx=0 tx=0).stop 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 shutdown_connections 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f95940b29f0 0x7f95940b4f00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95b0075b50 0x7f95b01a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95b0115360 0x7f95b01a5b20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 --2- 172.21.15.12:0/4263641910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95b00764a0 0x7f95b01a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 >> 172.21.15.12:0/4263641910 conn(0x7f95b0100030 msgr2=0x7f95b01006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 shutdown_connections 2024-05-27T07:26:34.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:34.589+0000 7f95b5800700 1 -- 172.21.15.12:0/4263641910 wait complete. 2024-05-27T07:26:34.684 INFO:tasks.workunit.client.0.smithi012.stdout:checking liburing.h usability... no 2024-05-27T07:26:34.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:34 smithi062 ceph-mon[35304]: daemon mds.c restarted 2024-05-27T07:26:34.736 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:34 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:26:34.736 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:34 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.651s CPU time. 2024-05-27T07:26:34.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:34 smithi162 ceph-mon[35676]: daemon mds.c restarted 2024-05-27T07:26:34.808 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:34 smithi162 podman[72201]: 2024-05-27 07:26:34.501690267 +0000 UTC m=+0.646132580 container init 08bef980ea11cbbb3561bdc3a25c18e87ab490b46ea760e2981a96847fa5ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:26:34.808 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:34 smithi162 podman[72201]: 2024-05-27 07:26:34.511687184 +0000 UTC m=+0.656129520 container start 08bef980ea11cbbb3561bdc3a25c18e87ab490b46ea760e2981a96847fa5ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:26:34.808 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:34 smithi162 bash[72201]: 08bef980ea11cbbb3561bdc3a25c18e87ab490b46ea760e2981a96847fa5ee99 2024-05-27T07:26:34.808 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:34 smithi162 systemd[1]: Started Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:34.808 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:34 smithi162 ceph-mds[72219]: -- 172.21.15.162:0/1465470842 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56459ead1a40 con 0x56459eb19a80 2024-05-27T07:26:34.809 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: starting mds.c at 2024-05-27T07:26:34.812 INFO:tasks.workunit.client.0.smithi012.stdout:checking liburing.h presence... no 2024-05-27T07:26:34.813 INFO:tasks.workunit.client.0.smithi012.stdout:checking for liburing.h... no 2024-05-27T07:26:34.968 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/capability.h usability... no 2024-05-27T07:26:35.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:34 smithi012 ceph-mon[31473]: daemon mds.c restarted 2024-05-27T07:26:35.129 INFO:tasks.workunit.client.0.smithi012.stdout:checking sys/capability.h presence... no 2024-05-27T07:26:35.131 INFO:tasks.workunit.client.0.smithi012.stdout:checking for sys/capability.h... no 2024-05-27T07:26:35.223 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:34 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:26:35.223 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:34 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.623s CPU time. 2024-05-27T07:26:35.278 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/fiemap.h usability... yes 2024-05-27T07:26:35.371 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/fiemap.h presence... yes 2024-05-27T07:26:35.373 INFO:tasks.workunit.client.0.smithi012.stdout:checking for linux/fiemap.h... yes 2024-05-27T07:26:35.630 INFO:tasks.workunit.client.0.smithi012.stdout:checking for fallocate... yes 2024-05-27T07:26:35.766 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:26:35.837 INFO:tasks.workunit.client.0.smithi012.stdout:checking for open_by_handle_at... yes 2024-05-27T07:26:35.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-mon[35676]: pgmap v139: 129 pgs: 129 active+clean; 71 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 552 KiB/s wr, 72 op/s 2024-05-27T07:26:35.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-mon[35676]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:26:35.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/4155146909,v1:172.21.15.162:6837/4155146909] up:boot 2024-05-27T07:26:35.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:35.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:35.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:35.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7299e00700 time 2024-05-27T07:26:35.551317+0000 2024-05-27T07:26:35.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f72af8cdba6] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.976 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: clone() 2024-05-27T07:26:35.977 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.553+0000 7f7299e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7299e00700 time 2024-05-27T07:26:35.551317+0000 2024-05-27T07:26:35.977 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/fs.h usability... yes 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f72af8cdba6] 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.979 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: clone() 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.980 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: *** Caught signal (Aborted) ** 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: in thread 7f7299e00700 thread_name:md_log_replay 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f72ae44ad20] 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: gsignal() 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: abort() 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f72af8cdced] 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.981 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 17: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 19: clone() 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.556+0000 7f7299e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: in thread 7f7299e00700 thread_name:md_log_replay 2024-05-27T07:26:35.982 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f72ae44ad20] 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: gsignal() 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: abort() 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f72af8cdced] 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.983 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 17: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 19: clone() 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.984 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: -1> 2024-05-27T07:26:35.553+0000 7f7299e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7299e00700 time 2024-05-27T07:26:35.551317+0000 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f72af8cdba6] 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.985 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:35 smithi062 ceph-mon[35304]: pgmap v139: 129 pgs: 129 active+clean; 71 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 552 KiB/s wr, 72 op/s 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:35 smithi062 ceph-mon[35304]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:35 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/4155146909,v1:172.21.15.162:6837/4155146909] up:boot 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:35 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:35 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:35 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.986 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: clone() 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 0> 2024-05-27T07:26:35.556+0000 7f7299e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: in thread 7f7299e00700 thread_name:md_log_replay 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f72ae44ad20] 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: gsignal() 2024-05-27T07:26:35.987 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: abort() 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f72af8cdced] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.988 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 17: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 19: clone() 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: -1> 2024-05-27T07:26:35.553+0000 7f7299e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7299e00700 time 2024-05-27T07:26:35.551317+0000 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f72af8cdba6] 2024-05-27T07:26:35.989 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.990 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: clone() 2024-05-27T07:26:35.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 0> 2024-05-27T07:26:35.556+0000 7f7299e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:35.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: in thread 7f7299e00700 thread_name:md_log_replay 2024-05-27T07:26:35.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:35.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f72ae44ad20] 2024-05-27T07:26:35.991 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2: gsignal() 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 3: abort() 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f72af8cdced] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f72af8cddd1] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564598bb223d] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564598b9d971] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 8: (SnapRealm::check_cache() const+0x13e) [0x564598b9bfa0] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564598b9c651] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564598b31347] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56459882d87e] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564598832e94] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56459883d969] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 14: (MDLog::_replay_thread()+0x1f38) [0x564598bd648c] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564598bdf9cb] 2024-05-27T07:26:35.992 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 16: (Thread::entry_wrapper()+0x43) [0x7f72af89c72d] 2024-05-27T07:26:35.993 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 17: (Thread::_entry_func(void*)+0xd) [0x7f72af89c749] 2024-05-27T07:26:35.993 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f72ae4401ca] 2024-05-27T07:26:35.993 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 19: clone() 2024-05-27T07:26:35.993 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:35.993 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72215]: 2024-05-27T07:26:35.993 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:35 smithi162 podman[72265]: 2024-05-27 07:26:35.674033678 +0000 UTC m=+0.032826225 container died 08bef980ea11cbbb3561bdc3a25c18e87ab490b46ea760e2981a96847fa5ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T07:26:36.053 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/fs.h presence... yes 2024-05-27T07:26:36.054 INFO:tasks.workunit.client.0.smithi012.stdout:checking for linux/fs.h... yes 2024-05-27T07:26:36.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:35 smithi012 ceph-mon[31473]: pgmap v139: 129 pgs: 129 active+clean; 71 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 552 KiB/s wr, 72 op/s 2024-05-27T07:26:36.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:35 smithi012 ceph-mon[31473]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:26:36.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:35 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/4155146909,v1:172.21.15.162:6837/4155146909] up:boot 2024-05-27T07:26:36.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:35 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:36.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:35 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:36.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:35 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:36.220 INFO:tasks.workunit.client.0.smithi012.stdout:checking btrfsutil.h usability... no 2024-05-27T07:26:36.224 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:36.344 INFO:tasks.workunit.client.0.smithi012.stdout:checking btrfsutil.h presence... no 2024-05-27T07:26:36.346 INFO:tasks.workunit.client.0.smithi012.stdout:checking for btrfsutil.h... no 2024-05-27T07:26:36.590 INFO:tasks.workunit.client.0.smithi012.stdout:checking for copy_file_range... yes 2024-05-27T07:26:36.794 INFO:tasks.workunit.client.0.smithi012.stdout:checking for SEEK_DATA... yes 2024-05-27T07:26:37.006 INFO:tasks.workunit.client.0.smithi012.stdout:checking for BMV_OF_SHARED... yes 2024-05-27T07:26:37.190 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:36 smithi162 podman[72265]: 2024-05-27 07:26:36.757255498 +0000 UTC m=+1.116048089 container remove 08bef980ea11cbbb3561bdc3a25c18e87ab490b46ea760e2981a96847fa5ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:37.190 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:36 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:37.264 INFO:tasks.workunit.client.0.smithi012.stdout:checking for nftw... yes 2024-05-27T07:26:37.473 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:37 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:26:37.473 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:37 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.099s CPU time. 2024-05-27T07:26:37.503 INFO:tasks.workunit.client.0.smithi012.stdout:checking for RLIMIT_NOFILE... yes 2024-05-27T07:26:37.708 INFO:tasks.workunit.client.0.smithi012.stdout:checking for XFS_IOC_EXCHANGE_RANGE... no 2024-05-27T07:26:37.910 INFO:tasks.workunit.client.0.smithi012.stdout:checking for FICLONE... yes 2024-05-27T07:26:37.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:37 smithi162 ceph-mon[35676]: pgmap v141: 129 pgs: 129 active+clean; 71 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 287 KiB/s wr, 57 op/s 2024-05-27T07:26:37.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:37 smithi062 ceph-mon[35304]: pgmap v141: 129 pgs: 129 active+clean; 71 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 287 KiB/s wr, 57 op/s 2024-05-27T07:26:38.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:37 smithi012 ceph-mon[31473]: pgmap v141: 129 pgs: 129 active+clean; 71 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 287 KiB/s wr, 57 op/s 2024-05-27T07:26:38.137 INFO:tasks.workunit.client.0.smithi012.stdout:checking for renameat2... yes 2024-05-27T07:26:38.362 INFO:tasks.workunit.client.0.smithi012.stdout:checking for reallocarray... yes 2024-05-27T07:26:38.629 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct mount_attr... yes 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.710+0000 7fe7ed000700 1 -- 172.21.15.12:0/2610703844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8110ca0 msgr2=0x7fe7e81130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.710+0000 7fe7ed000700 1 --2- 172.21.15.12:0/2610703844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8110ca0 0x7fe7e81130c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000b0a0 tx=0x7fe7e000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.711+0000 7fe7ed000700 1 -- 172.21.15.12:0/2610703844 shutdown_connections 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.711+0000 7fe7ed000700 1 --2- 172.21.15.12:0/2610703844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e81030f0 0x7fe7e8103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.711+0000 7fe7ed000700 1 --2- 172.21.15.12:0/2610703844 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8110ca0 0x7fe7e81130c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.711+0000 7fe7ed000700 1 --2- 172.21.15.12:0/2610703844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7e8103a40 0x7fe7e8110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.711+0000 7fe7ed000700 1 -- 172.21.15.12:0/2610703844 >> 172.21.15.12:0/2610703844 conn(0x7fe7e80fedb0 msgr2=0x7fe7e81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:38.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.711+0000 7fe7ed000700 1 -- 172.21.15.12:0/2610703844 shutdown_connections 2024-05-27T07:26:38.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.711+0000 7fe7ed000700 1 -- 172.21.15.12:0/2610703844 wait complete. 2024-05-27T07:26:38.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.712+0000 7fe7ed000700 1 Processor -- start 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.712+0000 7fe7ed000700 1 -- start start 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7ed000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e81030f0 0x7fe7e819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7ed000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8103a40 0x7fe7e819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7ed000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7e8110ca0 0x7fe7e81a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7ed000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e8116510 con 0x7fe7e8103a40 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7ed000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e8116390 con 0x7fe7e81030f0 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7ed000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e8116690 con 0x7fe7e8110ca0 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7e5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8103a40 0x7fe7e819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7e5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8103a40 0x7fe7e819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59500/0 (socket says 172.21.15.12:59500) 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7e5400700 1 -- 172.21.15.12:0/1337790524 learned_addr learned my addr 172.21.15.12:0/1337790524 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.713+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e81030f0 0x7fe7e819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e6800700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7e8110ca0 0x7fe7e81a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e5400700 1 -- 172.21.15.12:0/1337790524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7e8110ca0 msgr2=0x7fe7e81a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e5400700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7e8110ca0 0x7fe7e81a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e5400700 1 -- 172.21.15.12:0/1337790524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e81030f0 msgr2=0x7fe7e819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e5400700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e81030f0 0x7fe7e819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e5400700 1 -- 172.21.15.12:0/1337790524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e81a1d40 con 0x7fe7e8103a40 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e6800700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7e8110ca0 0x7fe7e81a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.714+0000 7fe7e5400700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8103a40 0x7fe7e819d460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400d980 tx=0x7fe7d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.715+0000 7fe7dea00700 1 -- 172.21.15.12:0/1337790524 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4017030 con 0x7fe7e8103a40 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.715+0000 7fe7dea00700 1 -- 172.21.15.12:0/1337790524 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7d40044b0 con 0x7fe7e8103a40 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.715+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e81a2030 con 0x7fe7e8103a40 2024-05-27T07:26:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.715+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e81a9950 con 0x7fe7e8103a40 2024-05-27T07:26:38.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.715+0000 7fe7dea00700 1 -- 172.21.15.12:0/1337790524 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d40038e0 con 0x7fe7e8103a40 2024-05-27T07:26:38.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.720+0000 7fe7fbbbfb80 1 Processor -- start 2024-05-27T07:26:38.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.720+0000 7fe7fbbbfb80 1 -- start start 2024-05-27T07:26:38.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e464e40 0x561d9e5e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5e3e70 0x561d9e5f0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561d9e511e40 con 0x561d9e5e3e70 2024-05-27T07:26:38.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d9e5019e0 con 0x561d9e464e40 2024-05-27T07:26:38.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d9e505a20 con 0x561d9e5509f0 2024-05-27T07:26:38.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7e6800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7e6800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:41628/0 (socket says 172.21.15.12:41628) 2024-05-27T07:26:38.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7e6800700 1 -- 172.21.15.12:0/2367125385 learned_addr learned my addr 172.21.15.12:0/2367125385 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:38.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.721+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e464e40 0x561d9e5e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.722+0000 7fe7e6800700 1 -- 172.21.15.12:0/2367125385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e464e40 msgr2=0x561d9e5e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.722+0000 7fe7e6800700 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e464e40 0x561d9e5e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.722+0000 7fe7e6800700 1 -- 172.21.15.12:0/2367125385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5e3e70 msgr2=0x561d9e5f0110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:26:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.722+0000 7fe7e6800700 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5e3e70 0x561d9e5f0110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.722+0000 7fe7e6800700 1 -- 172.21.15.12:0/2367125385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d9e4ca420 con 0x561d9e5509f0 2024-05-27T07:26:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.722+0000 7fe7dea00700 1 -- 172.21.15.12:0/1337790524 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe7d401a070 con 0x7fe7e8103a40 2024-05-27T07:26:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.723+0000 7fe7e6800700 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e3330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0006130 tx=0x7fe7e0009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e13eecffcf7af989 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:38.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.723+0000 7fe7d3400700 1 -- 172.21.15.12:0/2367125385 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0022070 con 0x561d9e5509f0 2024-05-27T07:26:38.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.723+0000 7fe7d3400700 1 -- 172.21.15.12:0/2367125385 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7e0009450 con 0x561d9e5509f0 2024-05-27T07:26:38.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.723+0000 7fe7d3400700 1 -- 172.21.15.12:0/2367125385 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e001c440 con 0x561d9e5509f0 2024-05-27T07:26:38.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.724+0000 7fe7dea00700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7cc0b29f0 0x7fe7cc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7cc0b29f0 0x7fe7cc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/2367125385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 msgr2=0x561d9e5e3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e3330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0006130 tx=0x7fe7e0009dd0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7dea00700 1 -- 172.21.15.12:0/1337790524 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fe7d40e22b0 con 0x7fe7e8103a40 2024-05-27T07:26:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/2367125385 shutdown_connections 2024-05-27T07:26:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5e3e70 0x561d9e5f0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e3330 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/2367125385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e464e40 0x561d9e5e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/2367125385 >> 172.21.15.12:0/2367125385 conn(0x561d9e4fe390 msgr2=0x561d9e5df4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:38.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/2367125385 shutdown_connections 2024-05-27T07:26:38.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.725+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/2367125385 wait complete. 2024-05-27T07:26:38.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.726+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7cc0b29f0 0x7fe7cc0b4f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x561d9e5e0790 tx=0x7fe7d800b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:38.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.726+0000 7fe7fbbbfb80 1 -- start start 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.726+0000 7fe7fbbbfb80 1 Processor -- start 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e464e40 0x561d9e5ed650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5509f0 0x561d9e5edb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561d9e407fa0 con 0x561d9e5509f0 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d9e4a2760 con 0x561d9e5e3e70 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d9e5f0b20 con 0x561d9e464e40 2024-05-27T07:26:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7fbbbfb80 10 client.0 _ll_register_callbacks cb 0x561d9e4abc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.727+0000 7fe7e5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5509f0 0x561d9e5edb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5509f0 0x561d9e5edb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59502/0 (socket says 172.21.15.12:59502) 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5400700 1 -- 172.21.15.12:0/4122588394 learned_addr learned my addr 172.21.15.12:0/4122588394 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e464e40 0x561d9e5ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e6800700 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5400700 1 -- 172.21.15.12:0/4122588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e464e40 msgr2=0x561d9e5ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5400700 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e464e40 0x561d9e5ed650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5400700 1 -- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 msgr2=0x561d9e5ee0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5400700 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5ee0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.728+0000 7fe7e5400700 1 -- 172.21.15.12:0/4122588394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d9e4518d0 con 0x561d9e5509f0 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7e5400700 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5509f0 0x561d9e5edb90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40a5260 tx=0x7fe7d40a8cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba5870331f1361e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7c7e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:38.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7c7e00700 1 -- 172.21.15.12:0/4122588394 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d40a91e0 con 0x561d9e5509f0 2024-05-27T07:26:38.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7c7e00700 1 -- 172.21.15.12:0/4122588394 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fe7d40c3660 con 0x561d9e5509f0 2024-05-27T07:26:38.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7e0009450 con 0x561d9e5509f0 2024-05-27T07:26:38.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7c7e00700 1 -- 172.21.15.12:0/4122588394 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe7d40ad590 con 0x561d9e5509f0 2024-05-27T07:26:38.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7fbbbfb80 10 client.15921 fetch_fsmap learned FSMap version 44 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.729+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561d9e4d2060 con 0x561d9e5509f0 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7c7e00700 1 -- 172.21.15.12:0/4122588394 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7e0009450 con 0x561d9e5509f0 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7c7e00700 1 -- 172.21.15.12:0/4122588394 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fe7d4024060 con 0x561d9e5509f0 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7fbbbfb80 10 client.15921 fetch_fsmap finished waiting for FSMap version 44 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7fbbbfb80 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7fbbbfb80 20 client.15921 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e5fd9e0 0x561d9e5ffe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7fbbbfb80 4 client.15921 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.730+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d9e513d90 con 0x561d9e5fd9e0 2024-05-27T07:26:38.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.731+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e5fd9e0 0x561d9e5ffe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.732+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e5fd9e0 0x561d9e5ffe00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45c19ecc3e7924c server_cookie=4e89c2163224b384 in_seq=0 out_seq=0 2024-05-27T07:26:38.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.732+0000 7fe7c7e00700 10 client.15921 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:38.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.736+0000 7fe7c7e00700 1 -- 172.21.15.12:0/4122588394 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561d9e513d90 con 0x561d9e5fd9e0 2024-05-27T07:26:38.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.736+0000 7fe7c7e00700 10 client.15921 handle_command_reply: tid=0 2024-05-27T07:26:38.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.736+0000 7fe7fbbbfb80 1 client.15921 shutdown 2024-05-27T07:26:38.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5509f0 msgr2=0x561d9e5edb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5509f0 0x561d9e5edb90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40a5260 tx=0x7fe7d40a8cf0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 shutdown_connections 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e5fd9e0 0x561d9e5ffe00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5ee0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e464e40 0x561d9e5ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/4122588394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e5509f0 0x561d9e5edb90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 >> 172.21.15.12:0/4122588394 conn(0x561d9e4fe390 msgr2=0x561d9e5e3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7c7e00700 0 client.15921 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:38.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 shutdown_connections 2024-05-27T07:26:38.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/4122588394 wait complete. 2024-05-27T07:26:38.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.737+0000 7fe7fbbbfb80 20 client.15921 trim_cache size 0 max 16384 2024-05-27T07:26:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.757+0000 7fe7fbbbfb80 1 Processor -- start 2024-05-27T07:26:38.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.757+0000 7fe7fbbbfb80 1 -- start start 2024-05-27T07:26:38.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5df460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e81b4ff0 con 0x561d9e464e40 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e81b2820 con 0x561d9e5e3e70 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e81b0670 con 0x561d9e5509f0 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7e5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7e5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59508/0 (socket says 172.21.15.12:59508) 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.758+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3658138112 learned_addr learned my addr 172.21.15.12:0/3658138112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5400700 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5df460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e6800700 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3658138112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 msgr2=0x561d9e5df460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5df460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3658138112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 msgr2=0x561d9e5e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3658138112 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d9e5db240 con 0x561d9e464e40 2024-05-27T07:26:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5400700 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5df460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e6800700 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.759+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e8e30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8003bd0 tx=0x7fe7d8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3d0219cbb7a1d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:38.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.760+0000 7fe7d2a00700 1 -- 172.21.15.12:0/3658138112 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d801e970 con 0x561d9e464e40 2024-05-27T07:26:38.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.760+0000 7fe7d2a00700 1 -- 172.21.15.12:0/3658138112 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7d802f070 con 0x561d9e464e40 2024-05-27T07:26:38.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.760+0000 7fe7d2a00700 1 -- 172.21.15.12:0/3658138112 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d801d410 con 0x561d9e464e40 2024-05-27T07:26:38.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3658138112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 msgr2=0x561d9e5e8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e8e30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8003bd0 tx=0x7fe7d8019000 comp rx=0 tx=0).stop 2024-05-27T07:26:38.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3658138112 shutdown_connections 2024-05-27T07:26:38.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e8e30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3658138112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5df460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3658138112 >> 172.21.15.12:0/3658138112 conn(0x561d9e4fe390 msgr2=0x561d9e5ded50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3658138112 shutdown_connections 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.761+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3658138112 wait complete. 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.762+0000 7fe7fbbbfb80 1 -- start start 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.762+0000 7fe7fbbbfb80 1 Processor -- start 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.762+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.762+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.762+0000 7fe7fbbbfb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e81b0ce0 con 0x561d9e464e40 2024-05-27T07:26:38.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e81b2150 con 0x561d9e5e3e70 2024-05-27T07:26:38.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7fbbbfb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e81b13b0 con 0x561d9e5509f0 2024-05-27T07:26:38.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7fbbbfb80 10 client.0 _ll_register_callbacks cb 0x561d9e4abc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:38.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59512/0 (socket says 172.21.15.12:59512) 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3092445190 learned_addr learned my addr 172.21.15.12:0/3092445190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5400700 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e6800700 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3092445190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 msgr2=0x561d9e5e7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 msgr2=0x561d9e5e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.763+0000 7fe7e5e00700 1 -- 172.21.15.12:0/3092445190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d9e5f4de0 con 0x561d9e464e40 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.764+0000 7fe7e5e00700 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e72f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800b4c0 tx=0x7fe7d801ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c1d4cc27eedac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.764+0000 7fe7c7e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.764+0000 7fe7c7e00700 1 -- 172.21.15.12:0/3092445190 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d802a670 con 0x561d9e464e40 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.764+0000 7fe7c7e00700 1 -- 172.21.15.12:0/3092445190 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fe7d802a810 con 0x561d9e464e40 2024-05-27T07:26:38.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.764+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7d802f070 con 0x561d9e464e40 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.764+0000 7fe7c7e00700 1 -- 172.21.15.12:0/3092445190 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe7d8045050 con 0x561d9e464e40 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.764+0000 7fe7fbbbfb80 10 client.15927 fetch_fsmap learned FSMap version 44 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561d9e5f6040 con 0x561d9e464e40 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7c7e00700 1 -- 172.21.15.12:0/3092445190 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7d802f070 con 0x561d9e464e40 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7c7e00700 1 -- 172.21.15.12:0/3092445190 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fe7d8034c00 con 0x561d9e464e40 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7fbbbfb80 10 client.15927 fetch_fsmap finished waiting for FSMap version 44 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7fbbbfb80 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7fbbbfb80 20 client.15927 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e668450 0x561d9e66a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7fbbbfb80 4 client.15927 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.765+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d9e50d570 con 0x561d9e668450 2024-05-27T07:26:38.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.766+0000 7fe7e5400700 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e668450 0x561d9e66a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:38.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.767+0000 7fe7e5400700 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e668450 0x561d9e66a870 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=354d56d5fbe05e server_cookie=dbadaae55eaa48aa in_seq=0 out_seq=0 2024-05-27T07:26:38.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.767+0000 7fe7c7e00700 10 client.15927 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:38.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.770+0000 7fe7c7e00700 1 -- 172.21.15.12:0/3092445190 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d9e50d570 con 0x561d9e668450 2024-05-27T07:26:38.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.770+0000 7fe7c7e00700 10 client.15927 handle_command_reply: tid=0 2024-05-27T07:26:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.770+0000 7fe7fbbbfb80 1 client.15927 shutdown 2024-05-27T07:26:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 msgr2=0x561d9e5e72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e72f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800b4c0 tx=0x7fe7d801ea20 comp rx=0 tx=0).stop 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 shutdown_connections 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561d9e668450 0x561d9e66a870 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d9e5e3e70 0x561d9e5e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561d9e464e40 0x561d9e5e72f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 --2- 172.21.15.12:0/3092445190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d9e5509f0 0x561d9e5e7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 >> 172.21.15.12:0/3092445190 conn(0x561d9e4fe390 msgr2=0x561d9e5e4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7c7e00700 0 client.15927 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 shutdown_connections 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 1 -- 172.21.15.12:0/3092445190 wait complete. 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.771+0000 7fe7fbbbfb80 20 client.15927 trim_cache size 0 max 16384 2024-05-27T07:26:38.761 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:26:38.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7cc0b29f0 msgr2=0x7fe7cc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7cc0b29f0 0x7fe7cc0b4f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x561d9e5e0790 tx=0x7fe7d800b410 comp rx=0 tx=0).stop 2024-05-27T07:26:38.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8103a40 msgr2=0x7fe7e819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:38.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8103a40 0x7fe7e819d460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400d980 tx=0x7fe7d400dc90 comp rx=0 tx=0).stop 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 shutdown_connections 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7cc0b29f0 0x7fe7cc0b4f00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e81030f0 0x7fe7e819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7e8110ca0 0x7fe7e81a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 --2- 172.21.15.12:0/1337790524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7e8103a40 0x7fe7e819d460 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.772+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 >> 172.21.15.12:0/1337790524 conn(0x7fe7e80fedb0 msgr2=0x7fe7e81127d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.773+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 shutdown_connections 2024-05-27T07:26:38.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:38.773+0000 7fe7ed000700 1 -- 172.21.15.12:0/1337790524 wait complete. 2024-05-27T07:26:38.899 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_qgroup_limit... yes 2024-05-27T07:26:39.209 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_qgroup_inherit... yes 2024-05-27T07:26:39.511 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_ioctl_vol_args... yes 2024-05-27T07:26:39.816 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_ioctl_vol_args_v2... yes 2024-05-27T07:26:39.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:39 smithi162 ceph-mon[35676]: pgmap v142: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 569 KiB/s wr, 124 op/s 2024-05-27T07:26:39.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:39 smithi062 ceph-mon[35304]: pgmap v142: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 569 KiB/s wr, 124 op/s 2024-05-27T07:26:40.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:39 smithi012 ceph-mon[31473]: pgmap v142: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 569 KiB/s wr, 124 op/s 2024-05-27T07:26:40.105 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_ioctl_ino_lookup_args... yes 2024-05-27T07:26:40.406 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_ioctl_ino_lookup_user_args... yes 2024-05-27T07:26:40.709 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_ioctl_get_subvol_rootref_args... yes 2024-05-27T07:26:40.777 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:26:40.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:26:40.880 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/btrfs.h usability... yes 2024-05-27T07:26:40.955 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/btrfs.h presence... yes 2024-05-27T07:26:40.957 INFO:tasks.workunit.client.0.smithi012.stdout:checking for linux/btrfs.h... yes 2024-05-27T07:26:41.167 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/btrfs_tree.h usability... yes 2024-05-27T07:26:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:41.231 INFO:tasks.workunit.client.0.smithi012.stdout:checking linux/btrfs_tree.h presence... yes 2024-05-27T07:26:41.231 INFO:tasks.workunit.client.0.smithi012.stdout:checking for linux/btrfs_tree.h... yes 2024-05-27T07:26:41.384 INFO:tasks.workunit.client.0.smithi012.stdout:checking for struct btrfs_ioctl_vol_args_v2.subvolid... yes 2024-05-27T07:26:41.526 INFO:tasks.workunit.client.0.smithi012.stdout:checking whether BTRFS_IOC_SNAP_DESTROY_V2 is declared... yes 2024-05-27T07:26:41.603 INFO:tasks.workunit.client.0.smithi012.stdout:configure: creating ./config.status 2024-05-27T07:26:41.667 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:41 smithi012 ceph-mon[31473]: pgmap v143: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 436 KiB/s wr, 95 op/s 2024-05-27T07:26:41.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:41 smithi162 ceph-mon[35676]: pgmap v143: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 436 KiB/s wr, 95 op/s 2024-05-27T07:26:41.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:41 smithi062 ceph-mon[35304]: pgmap v143: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 436 KiB/s wr, 95 op/s 2024-05-27T07:26:42.987 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: creating include/builddefs 2024-05-27T07:26:43.149 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: creating include/config.h 2024-05-27T07:26:43.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 -- 172.21.15.12:0/3333156423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb14810c5b0 msgr2=0x7fb14810c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3333156423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb14810c5b0 0x7fb14810c9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb138009990 tx=0x7fb138009ca0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 -- 172.21.15.12:0/3333156423 shutdown_connections 2024-05-27T07:26:43.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3333156423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb148106ec0 0x7fb148113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3333156423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb14810c5b0 0x7fb14810c9c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3333156423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb148106570 0x7fb148106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 -- 172.21.15.12:0/3333156423 >> 172.21.15.12:0/3333156423 conn(0x7fb148102070 msgr2=0x7fb1481044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:43.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 -- 172.21.15.12:0/3333156423 shutdown_connections 2024-05-27T07:26:43.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.673+0000 7fb14ea00700 1 -- 172.21.15.12:0/3333156423 wait complete. 2024-05-27T07:26:43.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 Processor -- start 2024-05-27T07:26:43.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 -- start start 2024-05-27T07:26:43.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb148106570 0x7fb14819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb148106ec0 0x7fb14819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb14810c5b0 0x7fb1481a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481165c0 con 0x7fb148106ec0 2024-05-27T07:26:43.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb148116440 con 0x7fb148106570 2024-05-27T07:26:43.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.675+0000 7fb14ea00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb148116740 con 0x7fb14810c5b0 2024-05-27T07:26:43.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb148106ec0 0x7fb14819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb148106ec0 0x7fb14819d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56902/0 (socket says 172.21.15.12:56902) 2024-05-27T07:26:43.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 -- 172.21.15.12:0/3946375529 learned_addr learned my addr 172.21.15.12:0/3946375529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:43.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb147e00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb148106570 0x7fb14819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb14ca00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb14810c5b0 0x7fb1481a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 -- 172.21.15.12:0/3946375529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb14810c5b0 msgr2=0x7fb1481a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb14810c5b0 0x7fb1481a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 -- 172.21.15.12:0/3946375529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb148106570 msgr2=0x7fb14819cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb148106570 0x7fb14819cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.676+0000 7fb13fe00700 1 -- 172.21.15.12:0/3946375529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1481a1d90 con 0x7fb148106ec0 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.677+0000 7fb13fe00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb148106ec0 0x7fb14819d510 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c9b0 tx=0x7fb13000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.677+0000 7fb145600700 1 -- 172.21.15.12:0/3946375529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130007aa0 con 0x7fb148106ec0 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.677+0000 7fb145600700 1 -- 172.21.15.12:0/3946375529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb130007c40 con 0x7fb148106ec0 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.677+0000 7fb145600700 1 -- 172.21.15.12:0/3946375529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130018660 con 0x7fb148106ec0 2024-05-27T07:26:43.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.677+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1481a2080 con 0x7fb148106ec0 2024-05-27T07:26:43.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.677+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1481a9a50 con 0x7fb148106ec0 2024-05-27T07:26:43.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.682+0000 7fb145600700 1 -- 172.21.15.12:0/3946375529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb13000f450 con 0x7fb148106ec0 2024-05-27T07:26:43.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.683+0000 7fb15d5d0b80 1 Processor -- start 2024-05-27T07:26:43.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.683+0000 7fb15d5d0b80 1 -- start start 2024-05-27T07:26:43.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.684+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b3b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.684+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b358f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.684+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b36240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.684+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5562d3a63ec0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.684+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5562d3a53a60 con 0x5562d3b35e30 2024-05-27T07:26:43.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.684+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5562d3a57aa0 con 0x5562d39b6ec0 2024-05-27T07:26:43.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.684+0000 7fb147e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb147e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b358f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56906/0 (socket says 172.21.15.12:56906) 2024-05-27T07:26:43.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb147e00700 1 -- 172.21.15.12:0/3993289827 learned_addr learned my addr 172.21.15.12:0/3993289827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:43.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb14ca00700 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b3b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb145600700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb12c0b2a70 0x7fb12c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb13fe00700 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b36240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb147e00700 1 -- 172.21.15.12:0/3993289827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 msgr2=0x5562d3b3b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb147e00700 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b3b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb147e00700 1 -- 172.21.15.12:0/3993289827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 msgr2=0x5562d3b36240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb147e00700 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b36240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb147e00700 1 -- 172.21.15.12:0/3993289827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562d3a1c4a0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.685+0000 7fb145600700 1 -- 172.21.15.12:0/3946375529 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb1300e2890 con 0x7fb148106ec0 2024-05-27T07:26:43.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.686+0000 7fb147e00700 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b358f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb138015000 tx=0x7fb138003720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa9712f89e7bbe2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:43.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.686+0000 7fb14ca00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb12c0b2a70 0x7fb12c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.686+0000 7fb13d600700 1 -- 172.21.15.12:0/3993289827 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138027070 con 0x5562d3b3bdb0 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.687+0000 7fb13d600700 1 -- 172.21.15.12:0/3993289827 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1380037c0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.687+0000 7fb13d600700 1 -- 172.21.15.12:0/3993289827 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13800ba30 con 0x5562d3b3bdb0 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.687+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3993289827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 msgr2=0x5562d3b358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.687+0000 7fb14ca00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb12c0b2a70 0x7fb12c0b4f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5562d3b36bb0 tx=0x7fb14000b040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.687+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b358f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb138015000 tx=0x7fb138003720 comp rx=0 tx=0).stop 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3993289827 shutdown_connections 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b36240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b3b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3993289827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b358f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3993289827 >> 172.21.15.12:0/3993289827 conn(0x5562d3a17040 msgr2=0x5562d3b30950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:43.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3993289827 shutdown_connections 2024-05-27T07:26:43.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3993289827 wait complete. 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.688+0000 7fb15d5d0b80 1 -- start start 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 1 Processor -- start 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d39b6ec0 0x5562d3b3f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d3b35e30 0x5562d3b3fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b3ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5562d395a020 con 0x5562d3b3bdb0 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5562d39f47e0 con 0x5562d39b6ec0 2024-05-27T07:26:43.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5562d3b42820 con 0x5562d3b35e30 2024-05-27T07:26:43.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.689+0000 7fb15d5d0b80 10 client.0 _ll_register_callbacks cb 0x5562d39fdcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:43.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b3ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b3ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56916/0 (socket says 172.21.15.12:56916) 2024-05-27T07:26:43.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 -- 172.21.15.12:0/1270693652 learned_addr learned my addr 172.21.15.12:0/1270693652 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:43.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb147e00700 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d39b6ec0 0x5562d3b3f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb13fe00700 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d3b35e30 0x5562d3b3fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 -- 172.21.15.12:0/1270693652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d3b35e30 msgr2=0x5562d3b3fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d3b35e30 0x5562d3b3fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 -- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d39b6ec0 msgr2=0x5562d3b3f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d39b6ec0 0x5562d3b3f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.690+0000 7fb14ca00700 1 -- 172.21.15.12:0/1270693652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562d39a3950 con 0x5562d3b3bdb0 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.691+0000 7fb14ca00700 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b3ff80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb14001be50 tx=0x7fb140004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ef3b611a8c2e46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.691+0000 7fb136000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.691+0000 7fb136000700 1 -- 172.21.15.12:0/1270693652 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140028060 con 0x5562d3b3bdb0 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.691+0000 7fb136000700 1 -- 172.21.15.12:0/1270693652 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb140019b40 con 0x5562d3b3bdb0 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.691+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1380037c0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.691+0000 7fb136000700 1 -- 172.21.15.12:0/1270693652 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb14002f050 con 0x5562d3b3bdb0 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 10 client.15939 fetch_fsmap learned FSMap version 44 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5562d3a240e0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb136000700 1 -- 172.21.15.12:0/1270693652 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1380037c0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb136000700 1 -- 172.21.15.12:0/1270693652 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb1400223b0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 10 client.15939 fetch_fsmap finished waiting for FSMap version 44 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 20 client.15939 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3b4f740 0x5562d3b51bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 4 client.15939 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.692+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562d3a65e10 con 0x5562d3b4f740 2024-05-27T07:26:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.693+0000 7fb147e00700 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3b4f740 0x5562d3b51bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.695+0000 7fb147e00700 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3b4f740 0x5562d3b51bc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f3bc51d1d2781e8 server_cookie=28aa69e675743d24 in_seq=0 out_seq=0 2024-05-27T07:26:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.695+0000 7fb136000700 10 client.15939 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb136000700 1 -- 172.21.15.12:0/1270693652 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5562d3a65e10 con 0x5562d3b4f740 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb136000700 10 client.15939 handle_command_reply: tid=0 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 client.15939 shutdown 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 msgr2=0x5562d3b3ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b3ff80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb14001be50 tx=0x7fb140004ad0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 shutdown_connections 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3b4f740 0x5562d3b51bc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d39b6ec0 0x5562d3b3f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b3ff80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/1270693652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d3b35e30 0x5562d3b3fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb136000700 0 client.15939 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 >> 172.21.15.12:0/1270693652 conn(0x5562d3a17040 msgr2=0x5562d3b30600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 shutdown_connections 2024-05-27T07:26:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.697+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/1270693652 wait complete. 2024-05-27T07:26:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.698+0000 7fb15d5d0b80 20 client.15939 trim_cache size 0 max 16384 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.721+0000 7fb15d5d0b80 1 Processor -- start 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.721+0000 7fb15d5d0b80 1 -- start start 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b358f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b39bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b4b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481b4fc0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481b27f0 con 0x5562d3b35e30 2024-05-27T07:26:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481b0640 con 0x5562d39b6ec0 2024-05-27T07:26:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb14ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b4b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb14ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b4b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56928/0 (socket says 172.21.15.12:56928) 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb14ca00700 1 -- 172.21.15.12:0/2674793884 learned_addr learned my addr 172.21.15.12:0/2674793884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb147e00700 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.722+0000 7fb13fe00700 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b39bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb14ca00700 1 -- 172.21.15.12:0/2674793884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 msgr2=0x5562d3b358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb14ca00700 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b358f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb14ca00700 1 -- 172.21.15.12:0/2674793884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 msgr2=0x5562d3b39bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb14ca00700 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b39bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb14ca00700 1 -- 172.21.15.12:0/2674793884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562d3b3b200 con 0x5562d3b3bdb0 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb14ca00700 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b4b8c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb1400227d0 tx=0x7fb140031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6bd37e5a67ab81 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb13cc00700 1 -- 172.21.15.12:0/2674793884 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140037020 con 0x5562d3b3bdb0 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb13cc00700 1 -- 172.21.15.12:0/2674793884 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb140013050 con 0x5562d3b3bdb0 2024-05-27T07:26:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.723+0000 7fb13cc00700 1 -- 172.21.15.12:0/2674793884 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140022de0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/2674793884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 msgr2=0x5562d3b4b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b4b8c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb1400227d0 tx=0x7fb140031e20 comp rx=0 tx=0).stop 2024-05-27T07:26:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/2674793884 shutdown_connections 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b4b8c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b358f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/2674793884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b39bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/2674793884 >> 172.21.15.12:0/2674793884 conn(0x5562d3a17040 msgr2=0x5562d3b3fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/2674793884 shutdown_connections 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.724+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/2674793884 wait complete. 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.725+0000 7fb15d5d0b80 1 -- start start 2024-05-27T07:26:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.725+0000 7fb15d5d0b80 1 Processor -- start 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b47cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b481f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb15d5d0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b44a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481b0cb0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481b2120 con 0x5562d3b35e30 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb15d5d0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481b1380 con 0x5562d39b6ec0 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb15d5d0b80 10 client.0 _ll_register_callbacks cb 0x5562d39fdcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb14ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b44a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb14ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b44a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56944/0 (socket says 172.21.15.12:56944) 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb14ca00700 1 -- 172.21.15.12:0/3227030214 learned_addr learned my addr 172.21.15.12:0/3227030214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb147e00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b47cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.726+0000 7fb13fe00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb14ca00700 1 -- 172.21.15.12:0/3227030214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 msgr2=0x5562d3b47cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb14ca00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b47cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb14ca00700 1 -- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 msgr2=0x5562d3b481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb14ca00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b481f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb14ca00700 1 -- 172.21.15.12:0/3227030214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562d3b450f0 con 0x5562d3b3bdb0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb13fe00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb14ca00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b44a80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb1400226d0 tx=0x7fb140004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2272fe9e4d891a4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb136000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb136000700 1 -- 172.21.15.12:0/3227030214 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140037030 con 0x5562d3b3bdb0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb136000700 1 -- 172.21.15.12:0/3227030214 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9466+0+0 (secure 0 0 0) 0x7fb140003a00 con 0x5562d3b3bdb0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.727+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb140013050 con 0x5562d3b3bdb0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb136000700 1 -- 172.21.15.12:0/3227030214 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb14004a050 con 0x5562d3b3bdb0 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 10 client.15945 fetch_fsmap learned FSMap version 44 2024-05-27T07:26:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5562d3bc3390 con 0x5562d3b3bdb0 2024-05-27T07:26:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb136000700 1 -- 172.21.15.12:0/3227030214 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb140013050 con 0x5562d3b3bdb0 2024-05-27T07:26:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb136000700 1 -- 172.21.15.12:0/3227030214 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb140004030 con 0x5562d3b3bdb0 2024-05-27T07:26:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 10 client.15945 fetch_fsmap finished waiting for FSMap version 44 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 20 client.15945 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3bc3980 0x5562d3bc5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 4 client.15945 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.728+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5562d3a5f5f0 con 0x5562d3bc3980 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.729+0000 7fb147e00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3bc3980 0x5562d3bc5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.730+0000 7fb147e00700 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3bc3980 0x5562d3bc5da0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6535552f8c37eea server_cookie=dad1f92fb33908d2 in_seq=0 out_seq=0 2024-05-27T07:26:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.730+0000 7fb136000700 10 client.15945 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:43.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb136000700 1 -- 172.21.15.12:0/3227030214 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5562d3a5f5f0 con 0x5562d3bc3980 2024-05-27T07:26:43.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb136000700 10 client.15945 handle_command_reply: tid=0 2024-05-27T07:26:43.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 client.15945 shutdown 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 msgr2=0x5562d3b44a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b44a80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb1400226d0 tx=0x7fb140004a90 comp rx=0 tx=0).stop 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 shutdown_connections 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562d3b3bdb0 0x5562d3b44a80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562d39b6ec0 0x5562d3b47cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562d3bc3980 0x5562d3bc5da0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 --2- 172.21.15.12:0/3227030214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562d3b35e30 0x5562d3b481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 >> 172.21.15.12:0/3227030214 conn(0x5562d3a17040 msgr2=0x5562d3b3c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.734+0000 7fb136000700 0 client.15945 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.735+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 shutdown_connections 2024-05-27T07:26:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.735+0000 7fb15d5d0b80 1 -- 172.21.15.12:0/3227030214 wait complete. 2024-05-27T07:26:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.735+0000 7fb15d5d0b80 20 client.15945 trim_cache size 0 max 16384 2024-05-27T07:26:43.725 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:26:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:26:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "8360b20f-d4d7-47bb-a7b3-fcfd4cce2495", 2024-05-27T07:26:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:26:43.725 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb12c0b2a70 msgr2=0x7fb12c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb12c0b2a70 0x7fb12c0b4f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5562d3b36bb0 tx=0x7fb14000b040 comp rx=0 tx=0).stop 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb148106ec0 msgr2=0x7fb14819d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb148106ec0 0x7fb14819d510 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c9b0 tx=0x7fb13000ccc0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 shutdown_connections 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb12c0b2a70 0x7fb12c0b4f80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb148106570 0x7fb14819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb14810c5b0 0x7fb1481a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 --2- 172.21.15.12:0/3946375529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb148106ec0 0x7fb14819d510 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:43.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 >> 172.21.15.12:0/3946375529 conn(0x7fb148102070 msgr2=0x7fb148104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:43.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 shutdown_connections 2024-05-27T07:26:43.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:43.736+0000 7fb14ea00700 1 -- 172.21.15.12:0/3946375529 wait complete. 2024-05-27T07:26:43.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:43 smithi162 ceph-mon[35676]: pgmap v144: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 348 KiB/s wr, 76 op/s 2024-05-27T07:26:43.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:43 smithi062 ceph-mon[35304]: pgmap v144: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 348 KiB/s wr, 76 op/s 2024-05-27T07:26:44.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:43 smithi012 ceph-mon[31473]: pgmap v144: 129 pgs: 129 active+clean; 72 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 348 KiB/s wr, 76 op/s 2024-05-27T07:26:44.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:44 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:26:44.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:44 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:44.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:44 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:26:44.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:44 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:26:44.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:44 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:44 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:44 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:44 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:44 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:44 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:44 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:44 smithi062 systemd[1]: Stopped Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:44 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.651s CPU time. 2024-05-27T07:26:44.894 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:44 smithi062 systemd[1]: Starting Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:44 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:44 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:44 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:44 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:44 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:45.220 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:44 smithi062 podman[70997]: 2024-05-27 07:26:44.809280032 +0000 UTC m=+0.038447075 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:45.220 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:44 smithi062 podman[70997]: 2024-05-27 07:26:44.967338931 +0000 UTC m=+0.196505998 container create ecfb63a9ed8733775995c3aa4d4694271b4caff7ee528ca49a5a122b55941edc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:26:45.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:26:45.374 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:44 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:26:45.374 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:44 smithi162 systemd[1]: Stopped Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:45.374 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:44 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.623s CPU time. 2024-05-27T07:26:45.374 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 systemd[1]: Starting Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:45.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 podman[70997]: 2024-05-27 07:26:45.229555487 +0000 UTC m=+0.458722577 container init ecfb63a9ed8733775995c3aa4d4694271b4caff7ee528ca49a5a122b55941edc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-27T07:26:45.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 podman[70997]: 2024-05-27 07:26:45.241134111 +0000 UTC m=+0.470301158 container start ecfb63a9ed8733775995c3aa4d4694271b4caff7ee528ca49a5a122b55941edc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:26:45.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-mds[71015]: -- 172.21.15.62:0/2434397388 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5634bd88c4e0 con 0x5634bd8d7a80 2024-05-27T07:26:45.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-mds[71015]: -- 172.21.15.62:0/2434397388 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634bd88fa40 con 0x5634bd8d7a80 2024-05-27T07:26:45.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: starting mds.k at 2024-05-27T07:26:45.487 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 bash[70997]: ecfb63a9ed8733775995c3aa4d4694271b4caff7ee528ca49a5a122b55941edc 2024-05-27T07:26:45.487 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 systemd[1]: Started Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:45.586 INFO:tasks.workunit.client.0.smithi012.stdout:config.status: executing libtool commands 2024-05-27T07:26:45.660 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 podman[72355]: 2024-05-27 07:26:45.289477274 +0000 UTC m=+0.033896543 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:45.660 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 podman[72355]: 2024-05-27 07:26:45.41608856 +0000 UTC m=+0.160507870 container create 1ea2ef81099caa07a772f5cfd002a97d5a09937192acaff357ab314a41453620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:45.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:45 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:26:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:45 smithi162 ceph-mon[35676]: pgmap v145: 129 pgs: 129 active+clean; 73 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 331 KiB/s wr, 71 op/s 2024-05-27T07:26:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:45 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:45 smithi162 ceph-mon[35676]: daemon mds.k restarted 2024-05-27T07:26:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:45 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:45.974 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 podman[72355]: 2024-05-27 07:26:45.670220334 +0000 UTC m=+0.414639687 container init 1ea2ef81099caa07a772f5cfd002a97d5a09937192acaff357ab314a41453620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:45.975 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 podman[72355]: 2024-05-27 07:26:45.681256976 +0000 UTC m=+0.425676250 container start 1ea2ef81099caa07a772f5cfd002a97d5a09937192acaff357ab314a41453620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:26:45.975 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 bash[72355]: 1ea2ef81099caa07a772f5cfd002a97d5a09937192acaff357ab314a41453620 2024-05-27T07:26:45.975 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 systemd[1]: Started Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:45.975 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 ceph-mds[72373]: -- 172.21.15.162:0/1303482465 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b8a16da40 con 0x558b8a1b5600 2024-05-27T07:26:45.975 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: starting mds.f at 2024-05-27T07:26:45.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:45 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:26:45.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:45 smithi062 ceph-mon[35304]: pgmap v145: 129 pgs: 129 active+clean; 73 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 331 KiB/s wr, 71 op/s 2024-05-27T07:26:45.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:45 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:45.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:45 smithi062 ceph-mon[35304]: daemon mds.k restarted 2024-05-27T07:26:45.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:45 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:45.986 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff2b4600700 time 2024-05-27T07:26:45.872452+0000 2024-05-27T07:26:45.986 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:45.986 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.986 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff2ca19aba6] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:45.987 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: clone() 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.874+0000 7ff2b4600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff2b4600700 time 2024-05-27T07:26:45.872452+0000 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff2ca19aba6] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:45.988 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:45.989 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: clone() 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: *** Caught signal (Aborted) ** 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: in thread 7ff2b4600700 thread_name:md_log_replay 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff2c8d17d20] 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: gsignal() 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: abort() 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff2ca19aced] 2024-05-27T07:26:45.990 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:45.991 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:45.991 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:45.991 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:45.991 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:45.991 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:45.991 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 19: clone() 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.877+0000 7ff2b4600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: in thread 7ff2b4600700 thread_name:md_log_replay 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.992 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff2c8d17d20] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: gsignal() 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: abort() 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff2ca19aced] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:45.993 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 19: clone() 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: -1> 2024-05-27T07:26:45.874+0000 7ff2b4600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff2b4600700 time 2024-05-27T07:26:45.872452+0000 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff2ca19aba6] 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:45.994 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:45.995 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: clone() 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 0> 2024-05-27T07:26:45.877+0000 7ff2b4600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: in thread 7ff2b4600700 thread_name:md_log_replay 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff2c8d17d20] 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: gsignal() 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: abort() 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff2ca19aced] 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:45.996 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 19: clone() 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: -1> 2024-05-27T07:26:45.874+0000 7ff2b4600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff2b4600700 time 2024-05-27T07:26:45.872452+0000 2024-05-27T07:26:45.997 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff2ca19aba6] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:45.998 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: clone() 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 0> 2024-05-27T07:26:45.877+0000 7ff2b4600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: in thread 7ff2b4600700 thread_name:md_log_replay 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff2c8d17d20] 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2: gsignal() 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 3: abort() 2024-05-27T07:26:45.999 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff2ca19aced] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff2ca19add1] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b9bf223d] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b9bdd971] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b9bdbfa0] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b9bdc651] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9b71347] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b986d87e] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9872e94] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b987d969] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b9c1648c] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b9c1f9cb] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 16: (Thread::entry_wrapper()+0x43) [0x7ff2ca16972d] 2024-05-27T07:26:46.000 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff2ca169749] 2024-05-27T07:26:46.001 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff2c8d0d1ca] 2024-05-27T07:26:46.001 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 19: clone() 2024-05-27T07:26:46.001 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:46.001 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71011]: 2024-05-27T07:26:46.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:45 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:26:46.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:45 smithi012 ceph-mon[31473]: pgmap v145: 129 pgs: 129 active+clean; 73 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 331 KiB/s wr, 71 op/s 2024-05-27T07:26:46.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:45 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:46.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:45 smithi012 ceph-mon[31473]: daemon mds.k restarted 2024-05-27T07:26:46.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:45 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:46.215 INFO:tasks.workunit.client.0.smithi012.stdout:Building lib 2024-05-27T07:26:46.216 INFO:tasks.workunit.client.0.smithi012.stdout:Building tests 2024-05-27T07:26:46.216 INFO:tasks.workunit.client.0.smithi012.stdout:Building include 2024-05-27T07:26:46.275 INFO:tasks.workunit.client.0.smithi012.stdout: [LTDEP] 2024-05-27T07:26:46.375 INFO:tasks.workunit.client.0.smithi012.stdout:Building ceph 2024-05-27T07:26:46.376 INFO:tasks.workunit.client.0.smithi012.stdout:Building btrfs 2024-05-27T07:26:46.376 INFO:tasks.workunit.client.0.smithi012.stdout:Building cifs 2024-05-27T07:26:46.473 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/ceph/group.list 2024-05-27T07:26:46.475 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/cifs/group.list 2024-05-27T07:26:46.485 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:46 smithi062 podman[71046]: 2024-05-27 07:26:46.009012858 +0000 UTC m=+0.039567171 container died ecfb63a9ed8733775995c3aa4d4694271b4caff7ee528ca49a5a122b55941edc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:26:46.640 INFO:tasks.workunit.client.0.smithi012.stdout:Building ext4 2024-05-27T07:26:46.715 INFO:tasks.workunit.client.0.smithi012.stdout:Building f2fs 2024-05-27T07:26:46.780 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/ext4/group.list 2024-05-27T07:26:46.807 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/f2fs/group.list 2024-05-27T07:26:46.878 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/btrfs/group.list 2024-05-27T07:26:46.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:26:46.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:26:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6832/379794041,v1:172.21.15.62:6833/379794041] up:boot 2024-05-27T07:26:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:46.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: daemon mds.f restarted 2024-05-27T07:26:46.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:46 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:46.976 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff725800700 time 2024-05-27T07:26:46.869583+0000 2024-05-27T07:26:46.976 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff73b2c1ba6] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:46.977 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: clone() 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: *** Caught signal (Aborted) ** 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: in thread 7ff725800700 thread_name:md_log_replay 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.871+0000 7ff725800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff725800700 time 2024-05-27T07:26:46.869583+0000 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:46.978 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff73b2c1ba6] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:46.979 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: clone() 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff739e3ed20] 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: gsignal() 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: abort() 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff73b2c1ced] 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:46.980 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:46.981 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 19: clone() 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.874+0000 7ff725800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: in thread 7ff725800700 thread_name:md_log_replay 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff739e3ed20] 2024-05-27T07:26:46.982 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: gsignal() 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: abort() 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff73b2c1ced] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:46.983 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 19: clone() 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:46.984 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: -1> 2024-05-27T07:26:46.871+0000 7ff725800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff725800700 time 2024-05-27T07:26:46.869583+0000 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff73b2c1ba6] 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:46.985 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:46.986 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: clone() 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 0> 2024-05-27T07:26:46.874+0000 7ff725800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: in thread 7ff725800700 thread_name:md_log_replay 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff739e3ed20] 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: gsignal() 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: abort() 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff73b2c1ced] 2024-05-27T07:26:46.987 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:46.988 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:46.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:26:46.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:46.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:26:46.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6832/379794041,v1:172.21.15.62:6833/379794041] up:boot 2024-05-27T07:26:46.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:46.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:46.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:46.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: daemon mds.f restarted 2024-05-27T07:26:46.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:46 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:46.991 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:46.991 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:46.992 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 19: clone() 2024-05-27T07:26:46.992 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:46.992 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.992 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: -1> 2024-05-27T07:26:46.871+0000 7ff725800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff725800700 time 2024-05-27T07:26:46.869583+0000 2024-05-27T07:26:46.992 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:46.993 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.993 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.993 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff73b2c1ba6] 2024-05-27T07:26:46.994 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:46.994 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:46.995 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:46.995 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:46.995 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:46.995 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:46.995 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:46.996 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:46.996 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:46.996 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:46.996 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:46.996 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:46.997 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:46.997 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:46.997 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: clone() 2024-05-27T07:26:46.997 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.997 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 0> 2024-05-27T07:26:46.874+0000 7ff725800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:46.998 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: in thread 7ff725800700 thread_name:md_log_replay 2024-05-27T07:26:46.998 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:46.998 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:46.999 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff739e3ed20] 2024-05-27T07:26:46.999 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2: gsignal() 2024-05-27T07:26:46.999 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 3: abort() 2024-05-27T07:26:46.999 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff73b2c1ced] 2024-05-27T07:26:46.999 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff73b2c1dd1] 2024-05-27T07:26:47.000 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558b4d34923d] 2024-05-27T07:26:47.000 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558b4d334971] 2024-05-27T07:26:47.000 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 8: (SnapRealm::check_cache() const+0x13e) [0x558b4d332fa0] 2024-05-27T07:26:47.000 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558b4d333651] 2024-05-27T07:26:47.000 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558b4d2c8347] 2024-05-27T07:26:47.001 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558b4cfc487e] 2024-05-27T07:26:47.001 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558b4cfc9e94] 2024-05-27T07:26:47.001 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558b4cfd4969] 2024-05-27T07:26:47.001 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 14: (MDLog::_replay_thread()+0x1f38) [0x558b4d36d48c] 2024-05-27T07:26:47.001 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558b4d3769cb] 2024-05-27T07:26:47.001 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 16: (Thread::entry_wrapper()+0x43) [0x7ff73b29072d] 2024-05-27T07:26:47.002 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff73b290749] 2024-05-27T07:26:47.002 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff739e341ca] 2024-05-27T07:26:47.002 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 19: clone() 2024-05-27T07:26:47.002 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:47.003 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72369]: 2024-05-27T07:26:47.014 INFO:tasks.workunit.client.0.smithi012.stdout:Building generic 2024-05-27T07:26:47.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:26:47.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:47.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:26:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6832/379794041,v1:172.21.15.62:6833/379794041] up:boot 2024-05-27T07:26:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: daemon mds.f restarted 2024-05-27T07:26:47.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:46 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:47.253 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:47 smithi162 podman[72470]: 2024-05-27 07:26:47.013794003 +0000 UTC m=+0.035040278 container died 1ea2ef81099caa07a772f5cfd002a97d5a09937192acaff357ab314a41453620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:26:47.254 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:47 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:26:47.254 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:47 smithi162 systemd[1]: Stopped Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:47.254 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:47 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.099s CPU time. 2024-05-27T07:26:47.254 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:47 smithi162 systemd[1]: Starting Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:47.300 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/generic/group.list 2024-05-27T07:26:47.450 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:47 smithi062 podman[71046]: 2024-05-27 07:26:47.053937672 +0000 UTC m=+1.084492015 container remove ecfb63a9ed8733775995c3aa4d4694271b4caff7ee528ca49a5a122b55941edc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:47.450 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:47 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:47.726 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] dataascii.lo 2024-05-27T07:26:47.735 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:47 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:26:47.735 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:47 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.592s CPU time. 2024-05-27T07:26:47.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:47 smithi062 ceph-mon[35304]: pgmap v147: 129 pgs: 129 active+clean; 73 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 331 KiB/s wr, 71 op/s 2024-05-27T07:26:47.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:47 smithi062 ceph-mon[35304]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:26:47.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:47 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/2499401485,v1:172.21.15.162:6837/2499401485] up:boot 2024-05-27T07:26:47.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:47 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:47.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:47 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:47.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:47 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:47.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:47 smithi162 ceph-mon[35676]: pgmap v147: 129 pgs: 129 active+clean; 73 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 331 KiB/s wr, 71 op/s 2024-05-27T07:26:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:47 smithi162 ceph-mon[35676]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:26:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:47 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/2499401485,v1:172.21.15.162:6837/2499401485] up:boot 2024-05-27T07:26:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:47 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:47 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:47 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:47.979 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:47 smithi012 ceph-mon[31473]: pgmap v147: 129 pgs: 129 active+clean; 73 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 331 KiB/s wr, 71 op/s 2024-05-27T07:26:47.979 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:47 smithi012 ceph-mon[31473]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:26:47.979 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:47 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/2499401485,v1:172.21.15.162:6837/2499401485] up:boot 2024-05-27T07:26:47.980 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:47 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:47.980 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:47 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:26:47.980 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:47 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:48.215 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] databin.lo 2024-05-27T07:26:48.223 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:47 smithi162 podman[72470]: 2024-05-27 07:26:47.924587646 +0000 UTC m=+0.945833921 container remove 1ea2ef81099caa07a772f5cfd002a97d5a09937192acaff357ab314a41453620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-27T07:26:48.223 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:47 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:48.248 INFO:tasks.workunit.client.0.smithi012.stdout:Building nfs 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.333+0000 7f4a44200700 1 -- 172.21.15.12:0/704301251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c1045e0 msgr2=0x7f4a3c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.333+0000 7f4a44200700 1 --2- 172.21.15.12:0/704301251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1049f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009fc0 tx=0x7f4a2c00b330 comp rx=0 tx=0).stop 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.335+0000 7f4a44200700 1 -- 172.21.15.12:0/704301251 shutdown_connections 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.335+0000 7f4a44200700 1 --2- 172.21.15.12:0/704301251 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3c109860 0x7f4a3c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.335+0000 7f4a44200700 1 --2- 172.21.15.12:0/704301251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1049f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.335+0000 7f4a44200700 1 --2- 172.21.15.12:0/704301251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c108f10 0x7f4a3c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.335+0000 7f4a44200700 1 -- 172.21.15.12:0/704301251 >> 172.21.15.12:0/704301251 conn(0x7f4a3c100030 msgr2=0x7f4a3c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.336+0000 7f4a44200700 1 -- 172.21.15.12:0/704301251 shutdown_connections 2024-05-27T07:26:48.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.336+0000 7f4a44200700 1 -- 172.21.15.12:0/704301251 wait complete. 2024-05-27T07:26:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.337+0000 7f4a44200700 1 Processor -- start 2024-05-27T07:26:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.338+0000 7f4a44200700 1 -- start start 2024-05-27T07:26:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.338+0000 7f4a44200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1a2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.338+0000 7f4a44200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3c108f10 0x7f4a3c1a2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.338+0000 7f4a44200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c109860 0x7f4a3c1ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.338+0000 7f4a44200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c114340 con 0x7f4a3c108f10 2024-05-27T07:26:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.338+0000 7f4a44200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1141c0 con 0x7f4a3c1045e0 2024-05-27T07:26:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.338+0000 7f4a44200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1144c0 con 0x7f4a3c109860 2024-05-27T07:26:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.339+0000 7f4a40e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3c108f10 0x7f4a3c1a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.339+0000 7f4a41800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1a2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.339+0000 7f4a41800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1a2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:45654/0 (socket says 172.21.15.12:45654) 2024-05-27T07:26:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.339+0000 7f4a41800700 1 -- 172.21.15.12:0/3445855188 learned_addr learned my addr 172.21.15.12:0/3445855188 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.339+0000 7f4a42200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c109860 0x7f4a3c1ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.340+0000 7f4a41800700 1 -- 172.21.15.12:0/3445855188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c109860 msgr2=0x7f4a3c1ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.340+0000 7f4a41800700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c109860 0x7f4a3c1ad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.340+0000 7f4a41800700 1 -- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3c108f10 msgr2=0x7f4a3c1a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.340+0000 7f4a41800700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3c108f10 0x7f4a3c1a2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.340+0000 7f4a41800700 1 -- 172.21.15.12:0/3445855188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c1ad5e0 con 0x7f4a3c1045e0 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.341+0000 7f4a40e00700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3c108f10 0x7f4a3c1a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.341+0000 7f4a42200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c109860 0x7f4a3c1ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.341+0000 7f4a41800700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1a2020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00ccb0 tx=0x7f4a2c00cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.342+0000 7f4a3a000700 1 -- 172.21.15.12:0/3445855188 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c021070 con 0x7f4a3c1045e0 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.342+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c06d1d0 con 0x7f4a3c1045e0 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.342+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c06d730 con 0x7f4a3c1045e0 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.344+0000 7f4a3a000700 1 -- 172.21.15.12:0/3445855188 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a2c007500 con 0x7f4a3c1045e0 2024-05-27T07:26:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.344+0000 7f4a3a000700 1 -- 172.21.15.12:0/3445855188 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c01dc60 con 0x7f4a3c1045e0 2024-05-27T07:26:48.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.348+0000 7f4a52e23b80 1 Processor -- start 2024-05-27T07:26:48.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.348+0000 7f4a52e23b80 1 -- start start 2024-05-27T07:26:48.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.348+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5642630a99f0 0x56426313c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.348+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564262fbde40 0x56426313c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56426313ce70 0x564263149110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56426306ae40 con 0x564262fbde40 2024-05-27T07:26:48.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56426305a9e0 con 0x5642630a99f0 2024-05-27T07:26:48.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56426305ea20 con 0x56426313ce70 2024-05-27T07:26:48.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a42200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5642630a99f0 0x56426313c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a40e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56426313ce70 0x564263149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a42200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5642630a99f0 0x56426313c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:45662/0 (socket says 172.21.15.12:45662) 2024-05-27T07:26:48.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.349+0000 7f4a42200700 1 -- 172.21.15.12:0/3778844165 learned_addr learned my addr 172.21.15.12:0/3778844165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:48.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.350+0000 7f4a40e00700 1 -- 172.21.15.12:0/3778844165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5642630a99f0 msgr2=0x56426313c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.350+0000 7f4a40e00700 1 --2- 172.21.15.12:0/3778844165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5642630a99f0 0x56426313c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.350+0000 7f4a40e00700 1 -- 172.21.15.12:0/3778844165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564262fbde40 msgr2=0x56426313c870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:26:48.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.350+0000 7f4a40e00700 1 --2- 172.21.15.12:0/3778844165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564262fbde40 0x56426313c870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.350+0000 7f4a40e00700 1 -- 172.21.15.12:0/3778844165 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564263023420 con 0x56426313ce70 2024-05-27T07:26:48.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.351+0000 7f4a40e00700 1 --2- 172.21.15.12:0/3778844165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56426313ce70 0x564263149110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c1116d0 tx=0x7f4a280117a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5f32e5de9093f2e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.352+0000 7f4a32a00700 1 -- 172.21.15.12:0/3778844165 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28004230 con 0x56426313ce70 2024-05-27T07:26:48.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.352+0000 7f4a32a00700 1 -- 172.21.15.12:0/3778844165 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a280043d0 con 0x56426313ce70 2024-05-27T07:26:48.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.352+0000 7f4a32a00700 1 -- 172.21.15.12:0/3778844165 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2800e900 con 0x56426313ce70 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.353+0000 7f4a52e23b80 1 -- 172.21.15.12:0/3778844165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56426313ce70 msgr2=0x564263149110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.353+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/3778844165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56426313ce70 0x564263149110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c1116d0 tx=0x7f4a280117a0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.353+0000 7f4a52e23b80 1 -- 172.21.15.12:0/3778844165 shutdown_connections 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.353+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/3778844165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5642630a99f0 0x56426313c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.353+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/3778844165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56426313ce70 0x564263149110 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c1116d0 tx=0x7f4a280117a0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.353+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/3778844165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564262fbde40 0x56426313c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.353+0000 7f4a52e23b80 1 -- 172.21.15.12:0/3778844165 >> 172.21.15.12:0/3778844165 conn(0x564263057390 msgr2=0x5642631384c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.354+0000 7f4a3a000700 1 -- 172.21.15.12:0/3445855188 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4a2c024070 con 0x7f4a3c1045e0 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.354+0000 7f4a52e23b80 1 -- 172.21.15.12:0/3778844165 shutdown_connections 2024-05-27T07:26:48.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.355+0000 7f4a52e23b80 1 -- 172.21.15.12:0/3778844165 wait complete. 2024-05-27T07:26:48.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.356+0000 7f4a52e23b80 1 -- start start 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.356+0000 7f4a52e23b80 1 Processor -- start 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.357+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263146790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.357+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631470e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.357+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263150f30 0x564263153350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.357+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564262f60fa0 con 0x564263146cd0 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.357+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564262ffb760 con 0x564263150f30 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.357+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564263149b80 con 0x5642630a99f0 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.357+0000 7f4a52e23b80 10 client.0 _ll_register_callbacks cb 0x564263004c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.358+0000 7f4a40e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263150f30 0x564263153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.358+0000 7f4a40e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263150f30 0x564263153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:45664/0 (socket says 172.21.15.12:45664) 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.358+0000 7f4a40e00700 1 -- 172.21.15.12:0/4024104254 learned_addr learned my addr 172.21.15.12:0/4024104254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.358+0000 7f4a42200700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a41800700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a40e00700 1 -- 172.21.15.12:0/4024104254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 msgr2=0x564263146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a40e00700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263146790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a40e00700 1 -- 172.21.15.12:0/4024104254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 msgr2=0x5642631470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a40e00700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a40e00700 1 -- 172.21.15.12:0/4024104254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564262faa8d0 con 0x564263150f30 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a42200700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a41800700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a40e00700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263150f30 0x564263153350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4a280097c0 tx=0x7f4a28005320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a62b7b4cc8ac8140 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.359+0000 7f4a27400700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.360+0000 7f4a3a000700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a200b2980 0x7f4a200b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.360+0000 7f4a3a000700 1 -- 172.21.15.12:0/3445855188 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9466+0+0 (secure 0 0 0) 0x7f4a2c00f020 con 0x7f4a3c1045e0 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.361+0000 7f4a42200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a200b2980 0x7f4a200b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.361+0000 7f4a27400700 1 -- 172.21.15.12:0/4024104254 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28027070 con 0x564263150f30 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.361+0000 7f4a27400700 1 -- 172.21.15.12:0/4024104254 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9466+0+0 (secure 0 0 0) 0x7f4a28004950 con 0x564263150f30 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.361+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a280043d0 con 0x564263150f30 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.362+0000 7f4a27400700 1 -- 172.21.15.12:0/4024104254 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4a28031050 con 0x564263150f30 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.362+0000 7f4a52e23b80 10 client.35585 fetch_fsmap learned FSMap version 49 2024-05-27T07:26:48.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.362+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56426302b060 con 0x564263150f30 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.363+0000 7f4a42200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a200b2980 0x7f4a200b4e90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005fd0 tx=0x7f4a34009310 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.363+0000 7f4a27400700 1 -- 172.21.15.12:0/4024104254 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a280043d0 con 0x564263150f30 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.364+0000 7f4a27400700 1 -- 172.21.15.12:0/4024104254 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f4a28023460 con 0x564263150f30 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.364+0000 7f4a52e23b80 10 client.35585 fetch_fsmap finished waiting for FSMap version 49 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.364+0000 7f4a52e23b80 10 client.35585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.364+0000 7f4a52e23b80 20 client.35585 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.364+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564263157470 0x564263159890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.364+0000 7f4a52e23b80 4 client.35585 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.364+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56426306cd90 con 0x564263157470 2024-05-27T07:26:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.365+0000 7f4a41800700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564263157470 0x564263159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.366+0000 7f4a41800700 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564263157470 0x564263159890 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c87a0e04688d851 server_cookie=79a65481c754b89b in_seq=0 out_seq=0 2024-05-27T07:26:48.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.366+0000 7f4a27400700 10 client.35585 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:48.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.367+0000 7f4a27400700 1 -- 172.21.15.12:0/4024104254 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56426306cd90 con 0x564263157470 2024-05-27T07:26:48.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.367+0000 7f4a27400700 10 client.35585 handle_command_reply: tid=0 2024-05-27T07:26:48.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.367+0000 7f4a52e23b80 1 client.35585 shutdown 2024-05-27T07:26:48.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263150f30 msgr2=0x564263153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263150f30 0x564263153350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4a280097c0 tx=0x7f4a28005320 comp rx=0 tx=0).stop 2024-05-27T07:26:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 shutdown_connections 2024-05-27T07:26:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564263157470 0x564263159890 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263150f30 0x564263153350 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4a280097c0 tx=0x7f4a28005320 comp rx=0 tx=0).stop 2024-05-27T07:26:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/4024104254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a27400700 0 client.35585 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:48.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.368+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 >> 172.21.15.12:0/4024104254 conn(0x564263057390 msgr2=0x56426313c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.369+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 shutdown_connections 2024-05-27T07:26:48.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.369+0000 7f4a52e23b80 1 -- 172.21.15.12:0/4024104254 wait complete. 2024-05-27T07:26:48.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.370+0000 7f4a52e23b80 20 client.35585 trim_cache size 0 max 16384 2024-05-27T07:26:48.363 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/nfs/group.list 2024-05-27T07:26:48.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.403+0000 7f4a52e23b80 1 Processor -- start 2024-05-27T07:26:48.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.403+0000 7f4a52e23b80 1 -- start start 2024-05-27T07:26:48.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.404+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263137e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.404+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.404+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56426315ef90 0x56426315f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.404+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c0117d0 con 0x564263146cd0 2024-05-27T07:26:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.404+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c00efd0 con 0x56426315ef90 2024-05-27T07:26:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.404+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c00ce20 con 0x5642630a99f0 2024-05-27T07:26:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.404+0000 7f4a40e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.405+0000 7f4a42200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56426315ef90 0x56426315f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.405+0000 7f4a42200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56426315ef90 0x56426315f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:45678/0 (socket says 172.21.15.12:45678) 2024-05-27T07:26:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.405+0000 7f4a42200700 1 -- 172.21.15.12:0/2792563422 learned_addr learned my addr 172.21.15.12:0/2792563422 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.406+0000 7f4a41800700 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263137e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.406+0000 7f4a42200700 1 -- 172.21.15.12:0/2792563422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 msgr2=0x564263137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.406+0000 7f4a42200700 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263137e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.406+0000 7f4a42200700 1 -- 172.21.15.12:0/2792563422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 msgr2=0x5642631383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.406+0000 7f4a42200700 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631383c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.406+0000 7f4a42200700 1 -- 172.21.15.12:0/2792563422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564263139000 con 0x56426315ef90 2024-05-27T07:26:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.407+0000 7f4a42200700 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56426315ef90 0x56426315f3a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400ff10 tx=0x7f4a34017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=245d5e7eca58b4aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.408+0000 7f4a32000700 1 -- 172.21.15.12:0/2792563422 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34009a00 con 0x56426315ef90 2024-05-27T07:26:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.408+0000 7f4a32000700 1 -- 172.21.15.12:0/2792563422 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a34020c50 con 0x56426315ef90 2024-05-27T07:26:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.408+0000 7f4a32000700 1 -- 172.21.15.12:0/2792563422 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34028400 con 0x56426315ef90 2024-05-27T07:26:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.409+0000 7f4a52e23b80 1 -- 172.21.15.12:0/2792563422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56426315ef90 msgr2=0x56426315f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.409+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56426315ef90 0x56426315f3a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400ff10 tx=0x7f4a34017000 comp rx=0 tx=0).stop 2024-05-27T07:26:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.410+0000 7f4a52e23b80 1 -- 172.21.15.12:0/2792563422 shutdown_connections 2024-05-27T07:26:48.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.410+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56426315ef90 0x56426315f3a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.410+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x564263137e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.410+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/2792563422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564263146cd0 0x5642631383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.410+0000 7f4a52e23b80 1 -- 172.21.15.12:0/2792563422 >> 172.21.15.12:0/2792563422 conn(0x564263057390 msgr2=0x564263145080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.411+0000 7f4a52e23b80 1 -- 172.21.15.12:0/2792563422 shutdown_connections 2024-05-27T07:26:48.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.411+0000 7f4a52e23b80 1 -- 172.21.15.12:0/2792563422 wait complete. 2024-05-27T07:26:48.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.411+0000 7f4a52e23b80 1 -- start start 2024-05-27T07:26:48.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.412+0000 7f4a52e23b80 1 Processor -- start 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x56426314a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263146cd0 0x56426314a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a52e23b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56426315ef90 0x56426314ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c00d490 con 0x56426315ef90 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c00e900 con 0x564263146cd0 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a52e23b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c00db60 con 0x5642630a99f0 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a52e23b80 10 client.0 _ll_register_callbacks cb 0x564263004c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.413+0000 7f4a40e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263146cd0 0x56426314a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.414+0000 7f4a40e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263146cd0 0x56426314a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:45692/0 (socket says 172.21.15.12:45692) 2024-05-27T07:26:48.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.414+0000 7f4a40e00700 1 -- 172.21.15.12:0/1399944659 learned_addr learned my addr 172.21.15.12:0/1399944659 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:26:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.414+0000 7f4a42200700 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56426315ef90 0x56426314ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.414+0000 7f4a42200700 1 -- 172.21.15.12:0/1399944659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 msgr2=0x56426314a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.414+0000 7f4a41800700 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x56426314a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.415+0000 7f4a42200700 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x56426314a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.415+0000 7f4a42200700 1 -- 172.21.15.12:0/1399944659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263146cd0 msgr2=0x56426314a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.415+0000 7f4a42200700 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263146cd0 0x56426314a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.415+0000 7f4a42200700 1 -- 172.21.15.12:0/1399944659 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56426314ea30 con 0x56426315ef90 2024-05-27T07:26:48.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.415+0000 7f4a42200700 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56426315ef90 0x56426314ad00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4a3402a040 tx=0x7f4a34012fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b38b5dc838be9a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.415+0000 7f4a27400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:26:48.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.416+0000 7f4a27400700 1 -- 172.21.15.12:0/1399944659 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34037060 con 0x56426315ef90 2024-05-27T07:26:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.416+0000 7f4a27400700 1 -- 172.21.15.12:0/1399944659 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9466+0+0 (secure 0 0 0) 0x7f4a34025050 con 0x56426315ef90 2024-05-27T07:26:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.416+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a34020c50 con 0x56426315ef90 2024-05-27T07:26:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.417+0000 7f4a27400700 1 -- 172.21.15.12:0/1399944659 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4a34012460 con 0x56426315ef90 2024-05-27T07:26:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.417+0000 7f4a52e23b80 10 client.15966 fetch_fsmap learned FSMap version 49 2024-05-27T07:26:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.417+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5642631c2260 con 0x56426315ef90 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.417+0000 7f4a27400700 1 -- 172.21.15.12:0/1399944659 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a34020c50 con 0x56426315ef90 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a27400700 1 -- 172.21.15.12:0/1399944659 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f4a34004c00 con 0x56426315ef90 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a52e23b80 10 client.15966 fetch_fsmap finished waiting for FSMap version 49 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a52e23b80 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a52e23b80 20 client.15966 populate_metadata read hostname 'smithi012' 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5642631c28d0 0x5642631c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a52e23b80 4 client.15966 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564263066570 con 0x5642631c28d0 2024-05-27T07:26:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.418+0000 7f4a41800700 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5642631c28d0 0x5642631c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.419+0000 7f4a27400700 10 client.15966 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.419+0000 7f4a41800700 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5642631c28d0 0x5642631c4cf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2fa831a4d82e7e1 server_cookie=2e8863023e17579b in_seq=0 out_seq=0 2024-05-27T07:26:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.420+0000 7f4a27400700 1 -- 172.21.15.12:0/1399944659 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564263066570 con 0x5642631c28d0 2024-05-27T07:26:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.420+0000 7f4a27400700 10 client.15966 handle_command_reply: tid=0 2024-05-27T07:26:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.420+0000 7f4a52e23b80 1 client.15966 shutdown 2024-05-27T07:26:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.420+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56426315ef90 msgr2=0x56426314ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.420+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56426315ef90 0x56426314ad00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4a3402a040 tx=0x7f4a34012fb0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.420+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 shutdown_connections 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56426315ef90 0x56426314ad00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5642630a99f0 0x56426314a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5642631c28d0 0x5642631c4cf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a27400700 0 client.15966 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a52e23b80 1 --2- 172.21.15.12:0/1399944659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564263146cd0 0x56426314a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 >> 172.21.15.12:0/1399944659 conn(0x564263057390 msgr2=0x564263144ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 shutdown_connections 2024-05-27T07:26:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.421+0000 7f4a52e23b80 1 -- 172.21.15.12:0/1399944659 wait complete. 2024-05-27T07:26:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.422+0000 7f4a52e23b80 20 client.15966 trim_cache size 0 max 16384 2024-05-27T07:26:48.413 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:26:48.413 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:26:48.413 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:26:48.413 INFO:teuthology.orchestra.run.smithi012.stdout: "8360b20f-d4d7-47bb-a7b3-fcfd4cce2495": { 2024-05-27T07:26:48.413 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:26:48.414 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "8360b20f-d4d7-47bb-a7b3-fcfd4cce2495", 2024-05-27T07:26:48.414 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:26:48.414 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:26:48.414 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:26:48.414 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a200b2980 msgr2=0x7f4a200b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a200b2980 0x7f4a200b4e90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005fd0 tx=0x7f4a34009310 comp rx=0 tx=0).stop 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c1045e0 msgr2=0x7f4a3c1a2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1a2020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00ccb0 tx=0x7f4a2c00cce0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 shutdown_connections 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a200b2980 0x7f4a200b4e90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a3c1045e0 0x7f4a3c1a2020 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a3c109860 0x7f4a3c1ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 --2- 172.21.15.12:0/3445855188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a3c108f10 0x7f4a3c1a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 >> 172.21.15.12:0/3445855188 conn(0x7f4a3c100030 msgr2=0x7f4a3c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.424+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 shutdown_connections 2024-05-27T07:26:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:26:48.425+0000 7f4a44200700 1 -- 172.21.15.12:0/3445855188 wait complete. 2024-05-27T07:26:48.577 INFO:tasks.workunit.client.0.smithi012.stdout:Building ocfs2 2024-05-27T07:26:48.585 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 podman[72579]: 2024-05-27 07:26:48.164953828 +0000 UTC m=+0.037278746 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:48.585 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 podman[72579]: 2024-05-27 07:26:48.309950696 +0000 UTC m=+0.182275715 container create 6623981c381adf8773d4c27e1996b74d00f0443b7a076c574eb60d501307c0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:48.663 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/ocfs2/group.list 2024-05-27T07:26:48.833 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] datapid.lo 2024-05-27T07:26:48.888 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] file_lock.lo 2024-05-27T07:26:48.973 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:48 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:26:48.973 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:48 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.541s CPU time. 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:48 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:48 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:48 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:48 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:48 smithi162 ceph-mon[35676]: daemon mds.c restarted 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 podman[72579]: 2024-05-27 07:26:48.596377758 +0000 UTC m=+0.468702691 container init 6623981c381adf8773d4c27e1996b74d00f0443b7a076c574eb60d501307c0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 podman[72579]: 2024-05-27 07:26:48.604223188 +0000 UTC m=+0.476548116 container start 6623981c381adf8773d4c27e1996b74d00f0443b7a076c574eb60d501307c0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 bash[72579]: 6623981c381adf8773d4c27e1996b74d00f0443b7a076c574eb60d501307c0b8 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 systemd[1]: Started Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:48.974 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 ceph-mds[72642]: -- 172.21.15.162:0/2436470720 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564935e25a40 con 0x564935e6d600 2024-05-27T07:26:48.975 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: starting mds.c at 2024-05-27T07:26:49.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:48 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:48 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:48 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:48 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:26:49.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:48 smithi012 ceph-mon[31473]: daemon mds.c restarted 2024-05-27T07:26:49.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:48 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:48 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:48 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:48 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:26:49.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:48 smithi062 ceph-mon[35304]: daemon mds.c restarted 2024-05-27T07:26:49.358 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] forker.lo 2024-05-27T07:26:49.511 INFO:tasks.cephfs.filesystem:scrub status for tag:8360b20f-d4d7-47bb-a7b3-fcfd4cce2495 - {'path': '/', 'tag': '8360b20f-d4d7-47bb-a7b3-fcfd4cce2495', 'options': 'recursive,force'} 2024-05-27T07:26:49.541 INFO:tasks.workunit.client.0.smithi012.stdout:Building overlay 2024-05-27T07:26:49.660 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/overlay/group.list 2024-05-27T07:26:49.904 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] pattern.lo 2024-05-27T07:26:50.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:49 smithi162 ceph-mon[35676]: pgmap v149: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 286 KiB/s wr, 53 op/s 2024-05-27T07:26:50.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:49 smithi162 ceph-mon[35676]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:26:50.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:49 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/2300593989,v1:172.21.15.162:6837/2300593989] up:boot 2024-05-27T07:26:50.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:49 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:50.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:49 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:50.057 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:49 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:50.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:49 smithi012 ceph-mon[31473]: pgmap v149: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 286 KiB/s wr, 53 op/s 2024-05-27T07:26:50.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:49 smithi012 ceph-mon[31473]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:26:50.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:49 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/2300593989,v1:172.21.15.162:6837/2300593989] up:boot 2024-05-27T07:26:50.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:49 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:50.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:49 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:50.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:49 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:50.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:49 smithi062 ceph-mon[35304]: pgmap v149: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 286 KiB/s wr, 53 op/s 2024-05-27T07:26:50.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:49 smithi062 ceph-mon[35304]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:26:50.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:49 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/2300593989,v1:172.21.15.162:6837/2300593989] up:boot 2024-05-27T07:26:50.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:49 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:50.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:49 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:26:50.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:49 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:50.318 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] open_flags.lo 2024-05-27T07:26:50.402 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6c7000700 time 2024-05-27T07:26:50.066699+0000 2024-05-27T07:26:50.402 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6dca7dba6] 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.403 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: clone() 2024-05-27T07:26:50.404 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.068+0000 7ff6c7000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6c7000700 time 2024-05-27T07:26:50.066699+0000 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6dca7dba6] 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.405 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.406 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: clone() 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: *** Caught signal (Aborted) ** 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: in thread 7ff6c7000700 thread_name:md_log_replay 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6db5fad20] 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: gsignal() 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: abort() 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6dca7dced] 2024-05-27T07:26:50.407 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.408 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 19: clone() 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.071+0000 7ff6c7000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: in thread 7ff6c7000700 thread_name:md_log_replay 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6db5fad20] 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: gsignal() 2024-05-27T07:26:50.409 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: abort() 2024-05-27T07:26:50.410 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6dca7dced] 2024-05-27T07:26:50.410 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.410 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.410 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.410 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.410 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.410 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.411 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.411 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.411 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.411 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 19: clone() 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: -1> 2024-05-27T07:26:50.068+0000 7ff6c7000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6c7000700 time 2024-05-27T07:26:50.066699+0000 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.412 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6dca7dba6] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.413 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: clone() 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 0> 2024-05-27T07:26:50.071+0000 7ff6c7000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: in thread 7ff6c7000700 thread_name:md_log_replay 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.414 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6db5fad20] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: gsignal() 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: abort() 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6dca7dced] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.415 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.416 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 19: clone() 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: -1> 2024-05-27T07:26:50.068+0000 7ff6c7000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff6c7000700 time 2024-05-27T07:26:50.066699+0000 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff6dca7dba6] 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.417 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: clone() 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 0> 2024-05-27T07:26:50.071+0000 7ff6c7000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:50.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: in thread 7ff6c7000700 thread_name:md_log_replay 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6db5fad20] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2: gsignal() 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 3: abort() 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff6dca7dced] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff6dca7ddd1] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56490159323d] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56490157e971] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 8: (SnapRealm::check_cache() const+0x13e) [0x56490157cfa0] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56490157d651] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564901512347] 2024-05-27T07:26:50.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56490120e87e] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564901213e94] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56490121e969] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649015b748c] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649015c09cb] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 16: (Thread::entry_wrapper()+0x43) [0x7ff6dca4c72d] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff6dca4c749] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff6db5f01ca] 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 19: clone() 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c[72606]: 2024-05-27T07:26:50.420 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 podman[72676]: 2024-05-27 07:26:50.178066756 +0000 UTC m=+0.035720787 container died 6623981c381adf8773d4c27e1996b74d00f0443b7a076c574eb60d501307c0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:26:50.884 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] random_range.lo 2024-05-27T07:26:51.223 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 podman[72676]: 2024-05-27 07:26:50.852051389 +0000 UTC m=+0.709705434 container remove 6623981c381adf8773d4c27e1996b74d00f0443b7a076c574eb60d501307c0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:26:51.224 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:50 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:26:51.315 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] string_to_tokens.lo 2024-05-27T07:26:51.485 INFO:tasks.workunit.client.0.smithi012.stdout:Building perf 2024-05-27T07:26:51.562 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/perf/group.list 2024-05-27T07:26:51.694 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] str_to_bytes.lo 2024-05-27T07:26:51.696 INFO:tasks.workunit.client.0.smithi012.stdout:Building selftest 2024-05-27T07:26:51.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:51 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:26:51.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:26:51 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.098s CPU time. 2024-05-27T07:26:51.774 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/selftest/group.list 2024-05-27T07:26:51.958 INFO:tasks.workunit.client.0.smithi012.stdout:Building shared 2024-05-27T07:26:52.034 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/shared/group.list 2024-05-27T07:26:52.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:51 smithi012 ceph-mon[31473]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 112 KiB/s wr, 18 op/s 2024-05-27T07:26:52.103 INFO:tasks.workunit.client.0.smithi012.stdout:Building tmpfs 2024-05-27T07:26:52.178 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/tmpfs/group.list 2024-05-27T07:26:52.199 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] tlibio.lo 2024-05-27T07:26:52.214 INFO:tasks.workunit.client.0.smithi012.stdout:Building udf 2024-05-27T07:26:52.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:51 smithi162 ceph-mon[35676]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 112 KiB/s wr, 18 op/s 2024-05-27T07:26:52.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:51 smithi062 ceph-mon[35304]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 112 KiB/s wr, 18 op/s 2024-05-27T07:26:52.291 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/udf/group.list 2024-05-27T07:26:52.317 INFO:tasks.workunit.client.0.smithi012.stdout:Building xfs 2024-05-27T07:26:52.425 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] write_log.lo 2024-05-27T07:26:52.684 INFO:tasks.workunit.client.0.smithi012.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/xfs/group.list 2024-05-27T07:26:52.925 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] random.lo 2024-05-27T07:26:53.303 INFO:tasks.workunit.client.0.smithi012.stdout: [LD] 2024-05-27T07:26:53.894 INFO:tasks.workunit.client.0.smithi012.stdout:Building ltp 2024-05-27T07:26:53.895 INFO:tasks.workunit.client.0.smithi012.stdout:Building src 2024-05-27T07:26:53.940 INFO:tasks.workunit.client.0.smithi012.stdout: [DEP] 2024-05-27T07:26:54.047 INFO:tasks.workunit.client.0.smithi012.stdout: [DEP] 2024-05-27T07:26:54.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:53 smithi012 ceph-mon[31473]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 99 KiB/s wr, 16 op/s 2024-05-27T07:26:54.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:53 smithi162 ceph-mon[35676]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 99 KiB/s wr, 16 op/s 2024-05-27T07:26:54.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:53 smithi062 ceph-mon[35304]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 99 KiB/s wr, 16 op/s 2024-05-27T07:26:56.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:55 smithi012 ceph-mon[31473]: pgmap v153: 129 pgs: 129 active+clean; 76 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 349 KiB/s wr, 39 op/s 2024-05-27T07:26:56.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:55 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:26:56.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:55 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:26:56.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:55 smithi162 ceph-mon[35676]: pgmap v153: 129 pgs: 129 active+clean; 76 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 349 KiB/s wr, 39 op/s 2024-05-27T07:26:56.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:55 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:26:56.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:55 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:26:56.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:55 smithi062 ceph-mon[35304]: pgmap v153: 129 pgs: 129 active+clean; 76 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 349 KiB/s wr, 39 op/s 2024-05-27T07:26:56.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:55 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:26:56.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:55 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:26:57.001 INFO:tasks.workunit.client.0.smithi012.stdout:Building m4 2024-05-27T07:26:57.055 INFO:tasks.workunit.client.0.smithi012.stdout:Building common 2024-05-27T07:26:57.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:56 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:26:57.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:56 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:26:57.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:56 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:26:57.773 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:57 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:26:57.773 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:57 smithi062 systemd[1]: Stopped Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:57.773 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:57 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.592s CPU time. 2024-05-27T07:26:57.773 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:57 smithi062 systemd[1]: Starting Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:58.081 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:57 smithi062 ceph-mon[35304]: pgmap v154: 129 pgs: 129 active+clean; 76 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 286 KiB/s wr, 32 op/s 2024-05-27T07:26:58.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:57 smithi062 podman[71260]: 2024-05-27 07:26:57.811056897 +0000 UTC m=+0.043226394 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:58.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:57 smithi062 podman[71260]: 2024-05-27 07:26:57.923250244 +0000 UTC m=+0.155419798 container create 65ff447d720668b0f707561701e73576cab84d09dd3da5e9b214af2631187683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:26:58.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:57 smithi012 ceph-mon[31473]: pgmap v154: 129 pgs: 129 active+clean; 76 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 286 KiB/s wr, 32 op/s 2024-05-27T07:26:58.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:57 smithi162 ceph-mon[35676]: pgmap v154: 129 pgs: 129 active+clean; 76 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 286 KiB/s wr, 32 op/s 2024-05-27T07:26:58.400 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] doio 2024-05-27T07:26:58.401 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fsstress 2024-05-27T07:26:58.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 podman[71260]: 2024-05-27 07:26:58.19536498 +0000 UTC m=+0.427534484 container init 65ff447d720668b0f707561701e73576cab84d09dd3da5e9b214af2631187683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:26:58.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 podman[71260]: 2024-05-27 07:26:58.207236536 +0000 UTC m=+0.439406033 container start 65ff447d720668b0f707561701e73576cab84d09dd3da5e9b214af2631187683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:26:58.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 bash[71260]: 65ff447d720668b0f707561701e73576cab84d09dd3da5e9b214af2631187683 2024-05-27T07:26:58.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 systemd[1]: Started Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:58.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-mds[71279]: -- 172.21.15.62:0/788974624 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a60d5f9a40 con 0x55a60d641a80 2024-05-27T07:26:58.486 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: starting mds.k at 2024-05-27T07:26:59.071 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:58 smithi062 ceph-mon[35304]: daemon mds.k restarted 2024-05-27T07:26:59.071 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8748200700 time 2024-05-27T07:26:58.959706+0000 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f875dd68ba6] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.072 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: clone() 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:58.962+0000 7f8748200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8748200700 time 2024-05-27T07:26:58.959706+0000 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:59.073 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f875dd68ba6] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.074 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: clone() 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: *** Caught signal (Aborted) ** 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: in thread 7f8748200700 thread_name:md_log_replay 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875c8e5d20] 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: gsignal() 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: abort() 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f875dd68ced] 2024-05-27T07:26:59.075 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 17: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.076 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 19: clone() 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:58.965+0000 7f8748200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: in thread 7f8748200700 thread_name:md_log_replay 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875c8e5d20] 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: gsignal() 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: abort() 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f875dd68ced] 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.077 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.078 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.078 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.078 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.078 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 17: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 19: clone() 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: -1> 2024-05-27T07:26:58.962+0000 7f8748200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8748200700 time 2024-05-27T07:26:58.959706+0000 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f875dd68ba6] 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.079 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: clone() 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.080 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 0> 2024-05-27T07:26:58.965+0000 7f8748200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: in thread 7f8748200700 thread_name:md_log_replay 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875c8e5d20] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: gsignal() 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: abort() 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f875dd68ced] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.081 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 17: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 19: clone() 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: -1> 2024-05-27T07:26:58.962+0000 7f8748200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8748200700 time 2024-05-27T07:26:58.959706+0000 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.082 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f875dd68ba6] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.083 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: clone() 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 0> 2024-05-27T07:26:58.965+0000 7f8748200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: in thread 7f8748200700 thread_name:md_log_replay 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f875c8e5d20] 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2: gsignal() 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 3: abort() 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f875dd68ced] 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f875dd68dd1] 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a5f784623d] 2024-05-27T07:26:59.084 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a5f7831971] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a5f782ffa0] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a5f7830651] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a5f77c5347] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a5f74c187e] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a5f74c6e94] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a5f74d1969] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a5f786a48c] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a5f78739cb] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 16: (Thread::entry_wrapper()+0x43) [0x7f875dd3772d] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 17: (Thread::_entry_func(void*)+0xd) [0x7f875dd37749] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f875c8db1ca] 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 19: clone() 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:26:59.085 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k[71275]: 2024-05-27T07:26:59.086 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:26:59 smithi062 podman[71310]: 2024-05-27 07:26:59.081952347 +0000 UTC m=+0.033549934 container died 65ff447d720668b0f707561701e73576cab84d09dd3da5e9b214af2631187683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:26:59.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:58 smithi012 ceph-mon[31473]: daemon mds.k restarted 2024-05-27T07:26:59.128 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:58 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:26:59.128 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:58 smithi162 systemd[1]: Stopped Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:59.128 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:58 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.541s CPU time. 2024-05-27T07:26:59.128 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:58 smithi162 systemd[1]: Starting Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:26:59.129 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:58 smithi162 ceph-mon[35676]: daemon mds.k restarted 2024-05-27T07:26:59.462 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 podman[72777]: 2024-05-27 07:26:59.044938974 +0000 UTC m=+0.035004046 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:26:59.463 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 podman[72777]: 2024-05-27 07:26:59.173423095 +0000 UTC m=+0.163488196 container create 7fa0021491d4bcc9a298859ee21b4208aca3c2e27cc914c3e114c144655aadd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:26:59.723 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 podman[72777]: 2024-05-27 07:26:59.473129947 +0000 UTC m=+0.463195040 container init 7fa0021491d4bcc9a298859ee21b4208aca3c2e27cc914c3e114c144655aadd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:26:59.724 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 podman[72777]: 2024-05-27 07:26:59.483118619 +0000 UTC m=+0.473183727 container start 7fa0021491d4bcc9a298859ee21b4208aca3c2e27cc914c3e114c144655aadd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T07:26:59.724 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 bash[72777]: 7fa0021491d4bcc9a298859ee21b4208aca3c2e27cc914c3e114c144655aadd6 2024-05-27T07:26:59.724 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 systemd[1]: Started Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:26:59.724 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mds[72796]: -- 172.21.15.162:0/68742901 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b1a3045a40 con 0x55b1a308d600 2024-05-27T07:26:59.724 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: starting mds.f at 2024-05-27T07:26:59.781 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:59 smithi012 ceph-mon[31473]: pgmap v155: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 529 KiB/s wr, 26 op/s 2024-05-27T07:26:59.781 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:59 smithi012 ceph-mon[31473]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:26:59.781 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:59 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.62:6832/2562374457,v1:172.21.15.62:6833/2562374457] up:boot 2024-05-27T07:26:59.781 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:59 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:26:59.782 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:59 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:26:59.782 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:59 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:26:59.782 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:26:59 smithi012 ceph-mon[31473]: daemon mds.f restarted 2024-05-27T07:26:59.933 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fsx 2024-05-27T07:27:00.108 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mon[35676]: pgmap v155: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 529 KiB/s wr, 26 op/s 2024-05-27T07:27:00.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mon[35676]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:27:00.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.62:6832/2562374457,v1:172.21.15.62:6833/2562374457] up:boot 2024-05-27T07:27:00.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:27:00.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:27:00.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:27:00.109 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:26:59 smithi162 ceph-mon[35676]: daemon mds.f restarted 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd597800700 time 2024-05-27T07:26:59.988161+0000 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd5ad409ba6] 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.110 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: clone() 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:26:59.989+0000 7fd597800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd597800700 time 2024-05-27T07:26:59.988161+0000 2024-05-27T07:27:00.111 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd5ad409ba6] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.112 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: clone() 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: *** Caught signal (Aborted) ** 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: in thread 7fd597800700 thread_name:md_log_replay 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5abf86d20] 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: gsignal() 2024-05-27T07:27:00.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: abort() 2024-05-27T07:27:00.114 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd5ad409ced] 2024-05-27T07:27:00.114 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.115 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 19: clone() 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:26:59.991+0000 7fd597800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: in thread 7fd597800700 thread_name:md_log_replay 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5abf86d20] 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: gsignal() 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: abort() 2024-05-27T07:27:00.116 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd5ad409ced] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.117 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 19: clone() 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:26:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: -1> 2024-05-27T07:26:59.989+0000 7fd597800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd597800700 time 2024-05-27T07:26:59.988161+0000 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd5ad409ba6] 2024-05-27T07:27:00.118 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.119 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: clone() 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 0> 2024-05-27T07:26:59.991+0000 7fd597800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: in thread 7fd597800700 thread_name:md_log_replay 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5abf86d20] 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: gsignal() 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: abort() 2024-05-27T07:27:00.120 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd5ad409ced] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.121 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 19: clone() 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: -1> 2024-05-27T07:26:59.989+0000 7fd597800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd597800700 time 2024-05-27T07:26:59.988161+0000 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd5ad409ba6] 2024-05-27T07:27:00.122 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.123 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: clone() 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 0> 2024-05-27T07:26:59.991+0000 7fd597800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: in thread 7fd597800700 thread_name:md_log_replay 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd5abf86d20] 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2: gsignal() 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 3: abort() 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd5ad409ced] 2024-05-27T07:27:00.124 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd5ad409dd1] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b18d41a23d] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b18d405971] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b18d403fa0] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b18d404651] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b18d399347] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b18d09587e] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b18d09ae94] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b18d0a5969] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b18d43e48c] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b18d4479cb] 2024-05-27T07:27:00.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 16: (Thread::entry_wrapper()+0x43) [0x7fd5ad3d872d] 2024-05-27T07:27:00.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd5ad3d8749] 2024-05-27T07:27:00.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd5abf7c1ca] 2024-05-27T07:27:00.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 19: clone() 2024-05-27T07:27:00.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:00.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f[72792]: 2024-05-27T07:27:00.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 podman[72827]: 2024-05-27 07:27:00.119255712 +0000 UTC m=+0.034340762 container died 7fa0021491d4bcc9a298859ee21b4208aca3c2e27cc914c3e114c144655aadd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:27:00.179 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:59 smithi062 ceph-mon[35304]: pgmap v155: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 529 KiB/s wr, 26 op/s 2024-05-27T07:27:00.180 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:59 smithi062 ceph-mon[35304]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:27:00.180 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:59 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.62:6832/2562374457,v1:172.21.15.62:6833/2562374457] up:boot 2024-05-27T07:27:00.180 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:59 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:27:00.180 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:59 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:27:00.180 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:59 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:27:00.180 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:26:59 smithi062 ceph-mon[35304]: daemon mds.f restarted 2024-05-27T07:27:00.783 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:00 smithi062 podman[71310]: 2024-05-27 07:27:00.525761698 +0000 UTC m=+1.477359327 container remove 65ff447d720668b0f707561701e73576cab84d09dd3da5e9b214af2631187683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:27:00.783 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:00 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:01.069 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] growfiles 2024-05-27T07:27:01.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:00 smithi012 ceph-mon[31473]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:27:01.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:00 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6836/601836076,v1:172.21.15.162:6837/601836076] up:boot 2024-05-27T07:27:01.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:00 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:27:01.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:00 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:27:01.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:00 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:27:01.223 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 podman[72827]: 2024-05-27 07:27:00.878505983 +0000 UTC m=+0.793591029 container remove 7fa0021491d4bcc9a298859ee21b4208aca3c2e27cc914c3e114c144655aadd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:27:01.224 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:00 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:00 smithi162 ceph-mon[35676]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:27:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:00 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6836/601836076,v1:172.21.15.162:6837/601836076] up:boot 2024-05-27T07:27:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:00 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:27:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:00 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:27:01.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:00 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:27:01.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:01 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:27:01.235 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:01 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.677s CPU time. 2024-05-27T07:27:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:00 smithi062 ceph-mon[35304]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:27:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:00 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6836/601836076,v1:172.21.15.162:6837/601836076] up:boot 2024-05-27T07:27:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:00 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-05-27T07:27:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:00 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:27:01.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:00 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-27T07:27:01.446 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] iogen 2024-05-27T07:27:01.682 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-stress 2024-05-27T07:27:01.723 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:27:01.723 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.698s CPU time. 2024-05-27T07:27:01.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:27:01.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: Stopped Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:01.723 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.098s CPU time. 2024-05-27T07:27:01.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Start request repeated too quickly. 2024-05-27T07:27:01.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:27:01.724 INFO:journalctl@ceph.mds.c.smithi162.stdout:May 27 07:27:01 smithi162 systemd[1]: Failed to start Ceph mds.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:02.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:01 smithi012 ceph-mon[31473]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 661 KiB/s wr, 33 op/s 2024-05-27T07:27:02.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:01 smithi162 ceph-mon[35676]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 661 KiB/s wr, 33 op/s 2024-05-27T07:27:02.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:01 smithi062 ceph-mon[35304]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 661 KiB/s wr, 33 op/s 2024-05-27T07:27:04.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:03 smithi012 ceph-mon[31473]: pgmap v159: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 396 KiB/s wr, 7 op/s 2024-05-27T07:27:04.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:03 smithi162 ceph-mon[35676]: pgmap v159: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 396 KiB/s wr, 7 op/s 2024-05-27T07:27:04.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:03 smithi062 ceph-mon[35304]: pgmap v159: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 396 KiB/s wr, 7 op/s 2024-05-27T07:27:06.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:05 smithi162 ceph-mon[35676]: pgmap v160: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 726 KiB/s wr, 48 op/s 2024-05-27T07:27:06.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:05 smithi062 ceph-mon[35304]: pgmap v160: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 726 KiB/s wr, 48 op/s 2024-05-27T07:27:06.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:05 smithi012 ceph-mon[31473]: pgmap v160: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 726 KiB/s wr, 48 op/s 2024-05-27T07:27:08.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:07 smithi162 ceph-mon[35676]: pgmap v161: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 329 KiB/s wr, 40 op/s 2024-05-27T07:27:08.171 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:07 smithi162 ceph-mon[35676]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:08.171 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:07 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:27:08.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:07 smithi062 ceph-mon[35304]: pgmap v161: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 329 KiB/s wr, 40 op/s 2024-05-27T07:27:08.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:07 smithi062 ceph-mon[35304]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:08.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:07 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:27:08.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:07 smithi012 ceph-mon[31473]: pgmap v161: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 329 KiB/s wr, 40 op/s 2024-05-27T07:27:08.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:07 smithi012 ceph-mon[31473]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:08.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:07 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:27:08.475 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe29ca00700 time 2024-05-27T07:27:08.181505+0000 2024-05-27T07:27:08.475 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:08.475 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.475 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe2b2512ba6] 2024-05-27T07:27:08.475 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.475 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.475 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.476 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.477 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.477 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: clone() 2024-05-27T07:27:08.477 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.182+0000 7fe29ca00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe29ca00700 time 2024-05-27T07:27:08.181505+0000 2024-05-27T07:27:08.477 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:08.477 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.477 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.477 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe2b2512ba6] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.478 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: clone() 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: *** Caught signal (Aborted) ** 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: in thread 7fe29ca00700 thread_name:md_log_replay 2024-05-27T07:27:08.479 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.480 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe2b108fd20] 2024-05-27T07:27:08.480 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: gsignal() 2024-05-27T07:27:08.480 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: abort() 2024-05-27T07:27:08.480 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe2b2512ced] 2024-05-27T07:27:08.480 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.481 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.481 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.481 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.481 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.481 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.481 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 19: clone() 2024-05-27T07:27:08.482 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.185+0000 7fe29ca00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: in thread 7fe29ca00700 thread_name:md_log_replay 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe2b108fd20] 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: gsignal() 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: abort() 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe2b2512ced] 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.483 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 19: clone() 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:08.484 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: -1> 2024-05-27T07:27:08.182+0000 7fe29ca00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe29ca00700 time 2024-05-27T07:27:08.181505+0000 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe2b2512ba6] 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.485 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: clone() 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 0> 2024-05-27T07:27:08.185+0000 7fe29ca00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:08.486 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: in thread 7fe29ca00700 thread_name:md_log_replay 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe2b108fd20] 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: gsignal() 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: abort() 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe2b2512ced] 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.487 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.488 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.488 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.488 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.488 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.488 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.488 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.488 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 19: clone() 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: -1> 2024-05-27T07:27:08.182+0000 7fe29ca00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe29ca00700 time 2024-05-27T07:27:08.181505+0000 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.489 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe2b2512ba6] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.490 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: clone() 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 0> 2024-05-27T07:27:08.185+0000 7fe29ca00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: in thread 7fe29ca00700 thread_name:md_log_replay 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:08.491 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe2b108fd20] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2: gsignal() 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 3: abort() 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe2b2512ced] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe2b2512dd1] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d060c223d] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d060ad971] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d060abfa0] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d060ac651] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d06041347] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d05d3d87e] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d05d42e94] 2024-05-27T07:27:08.492 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d05d4d969] 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d060e648c] 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d060ef9cb] 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 16: (Thread::entry_wrapper()+0x43) [0x7fe2b24e172d] 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe2b24e1749] 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe2b10851ca] 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 19: clone() 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[71462]: 2024-05-27T07:27:08.493 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:08 smithi162 podman[72881]: 2024-05-27 07:27:08.295404011 +0000 UTC m=+0.032298430 container died a86ed0ef4a70439194bdea083df423c79b186af41f83e144263a7490c6025fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:27:09.435 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:09 smithi162 podman[72881]: 2024-05-27 07:27:09.008853532 +0000 UTC m=+0.745747995 container remove a86ed0ef4a70439194bdea083df423c79b186af41f83e144263a7490c6025fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:27:09.435 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:09 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:09.723 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:09 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-27T07:27:09.723 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:09 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.233s CPU time. 2024-05-27T07:27:10.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:09 smithi162 ceph-mon[35676]: pgmap v162: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 281 KiB/s wr, 36 op/s 2024-05-27T07:27:10.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:09 smithi062 ceph-mon[35304]: pgmap v162: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 281 KiB/s wr, 36 op/s 2024-05-27T07:27:10.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:09 smithi012 ceph-mon[31473]: pgmap v162: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 281 KiB/s wr, 36 op/s 2024-05-27T07:27:11.735 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:11 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:27:11.735 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:11 smithi062 systemd[1]: Stopped Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:11.736 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:11 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.677s CPU time. 2024-05-27T07:27:11.736 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:11 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Start request repeated too quickly. 2024-05-27T07:27:11.736 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:11 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:27:11.736 INFO:journalctl@ceph.mds.k.smithi062.stdout:May 27 07:27:11 smithi062 systemd[1]: Failed to start Ceph mds.k for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:11.864 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:11 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:27:11.864 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:11 smithi162 systemd[1]: Stopped Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:11.864 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:11 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.698s CPU time. 2024-05-27T07:27:11.864 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:11 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-05-27T07:27:11.865 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:11 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:27:11.865 INFO:journalctl@ceph.mds.f.smithi162.stdout:May 27 07:27:11 smithi162 systemd[1]: Failed to start Ceph mds.f for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:12.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:11 smithi162 ceph-mon[35676]: pgmap v163: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 254 KiB/s wr, 33 op/s 2024-05-27T07:27:12.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:11 smithi062 ceph-mon[35304]: pgmap v163: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 254 KiB/s wr, 33 op/s 2024-05-27T07:27:12.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:11 smithi012 ceph-mon[31473]: pgmap v163: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 254 KiB/s wr, 33 op/s 2024-05-27T07:27:14.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:13 smithi162 ceph-mon[35676]: pgmap v164: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 226 KiB/s wr, 29 op/s 2024-05-27T07:27:14.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:13 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:27:14.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:13 smithi062 ceph-mon[35304]: pgmap v164: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 226 KiB/s wr, 29 op/s 2024-05-27T07:27:14.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:13 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:27:14.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:13 smithi012 ceph-mon[31473]: pgmap v164: 129 pgs: 129 active+clean; 81 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 226 KiB/s wr, 29 op/s 2024-05-27T07:27:14.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:13 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-05-27T07:27:16.135 INFO:tasks.workunit.client.0.smithi012.stdout: [DEP] 2024-05-27T07:27:16.136 INFO:tasks.workunit.client.0.smithi012.stdout: [DEP] 2024-05-27T07:27:16.147 INFO:tasks.workunit.client.0.smithi012.stdout: [DEP] 2024-05-27T07:27:16.150 INFO:tasks.workunit.client.0.smithi012.stdout: [DEP] 2024-05-27T07:27:16.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:15 smithi162 ceph-mon[35676]: pgmap v165: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 258 KiB/s wr, 32 op/s 2024-05-27T07:27:16.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:15 smithi062 ceph-mon[35304]: pgmap v165: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 258 KiB/s wr, 32 op/s 2024-05-27T07:27:16.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:15 smithi012 ceph-mon[31473]: pgmap v165: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 258 KiB/s wr, 32 op/s 2024-05-27T07:27:17.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe738e00700 time 2024-05-27T07:27:17.369095+0000 2024-05-27T07:27:17.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe74e9f2ba6] 2024-05-27T07:27:17.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: clone() 2024-05-27T07:27:17.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.369+0000 7fe738e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe738e00700 time 2024-05-27T07:27:17.369095+0000 2024-05-27T07:27:17.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe74e9f2ba6] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: clone() 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: *** Caught signal (Aborted) ** 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: in thread 7fe738e00700 thread_name:md_log_replay 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe74d56fd20] 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: gsignal() 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: abort() 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe74e9f2ced] 2024-05-27T07:27:17.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 19: clone() 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.372+0000 7fe738e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: in thread 7fe738e00700 thread_name:md_log_replay 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe74d56fd20] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: gsignal() 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: abort() 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe74e9f2ced] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 19: clone() 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: -1> 2024-05-27T07:27:17.369+0000 7fe738e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe738e00700 time 2024-05-27T07:27:17.369095+0000 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe74e9f2ba6] 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: clone() 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 0> 2024-05-27T07:27:17.372+0000 7fe738e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: in thread 7fe738e00700 thread_name:md_log_replay 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe74d56fd20] 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: gsignal() 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: abort() 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe74e9f2ced] 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 19: clone() 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: -1> 2024-05-27T07:27:17.369+0000 7fe738e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe738e00700 time 2024-05-27T07:27:17.369095+0000 2024-05-27T07:27:17.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe74e9f2ba6] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: clone() 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 0> 2024-05-27T07:27:17.372+0000 7fe738e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: in thread 7fe738e00700 thread_name:md_log_replay 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe74d56fd20] 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2: gsignal() 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 3: abort() 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe74e9f2ced] 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe74e9f2dd1] 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56390a14a23d] 2024-05-27T07:27:17.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56390a135971] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 8: (SnapRealm::check_cache() const+0x13e) [0x56390a133fa0] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56390a134651] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56390a0c9347] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563909dc587e] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563909dcae94] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563909dd5969] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 14: (MDLog::_replay_thread()+0x1f38) [0x56390a16e48c] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56390a1779cb] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 16: (Thread::entry_wrapper()+0x43) [0x7fe74e9c172d] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe74e9c1749] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe74d5651ca] 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 19: clone() 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:17.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[65707]: 2024-05-27T07:27:17.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:17 smithi162 podman[72934]: 2024-05-27 07:27:17.486750199 +0000 UTC m=+0.032248008 container died e113b963d71c3b70e6706dca4fb32c103f93a75ac379fb0e06ac7bc04171b6d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f28ee400700 time 2024-05-27T07:27:17.436396+0000 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2903e70ba6] 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: clone() 2024-05-27T07:27:17.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.438+0000 7f28ee400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f28ee400700 time 2024-05-27T07:27:17.436396+0000 2024-05-27T07:27:17.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2903e70ba6] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: clone() 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: *** Caught signal (Aborted) ** 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: in thread 7f28ee400700 thread_name:md_log_replay 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29029edd20] 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: gsignal() 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: abort() 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2903e70ced] 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 19: clone() 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.441+0000 7f28ee400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: in thread 7f28ee400700 thread_name:md_log_replay 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29029edd20] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: gsignal() 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: abort() 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2903e70ced] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 19: clone() 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: -1> 2024-05-27T07:27:17.438+0000 7f28ee400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f28ee400700 time 2024-05-27T07:27:17.436396+0000 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2903e70ba6] 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.852 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: clone() 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 0> 2024-05-27T07:27:17.441+0000 7f28ee400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: in thread 7f28ee400700 thread_name:md_log_replay 2024-05-27T07:27:17.853 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29029edd20] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: gsignal() 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: abort() 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2903e70ced] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.854 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 19: clone() 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.855 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: -1> 2024-05-27T07:27:17.438+0000 7f28ee400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f28ee400700 time 2024-05-27T07:27:17.436396+0000 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2903e70ba6] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.856 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: clone() 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 0> 2024-05-27T07:27:17.441+0000 7f28ee400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:17.857 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: in thread 7f28ee400700 thread_name:md_log_replay 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f29029edd20] 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2: gsignal() 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 3: abort() 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2903e70ced] 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2903e70dd1] 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e31a9a223d] 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e31a98d971] 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e31a98bfa0] 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e31a98c651] 2024-05-27T07:27:17.858 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e31a921347] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e31a61d87e] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e31a622e94] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e31a62d969] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e31a9c648c] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e31a9cf9cb] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 16: (Thread::entry_wrapper()+0x43) [0x7f2903e3f72d] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 17: (Thread::_entry_func(void*)+0xd) [0x7f2903e3f749] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f29029e31ca] 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 19: clone() 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[73732]: 2024-05-27T07:27:17.859 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:17 smithi012 podman[104991]: 2024-05-27 07:27:17.572125761 +0000 UTC m=+0.038178593 container died de5f65ec6cc62b0b87a48521091354d24146bf437c44ade4c2d820a3036d39a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T07:27:18.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:17 smithi162 ceph-mon[35676]: pgmap v166: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 38 KiB/s wr, 4 op/s 2024-05-27T07:27:18.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:17 smithi162 ceph-mon[35676]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:18.199 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:17 smithi162 ceph-mon[35676]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:18.199 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:17 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:18.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:17 smithi062 ceph-mon[35304]: pgmap v166: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 38 KiB/s wr, 4 op/s 2024-05-27T07:27:18.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:17 smithi062 ceph-mon[35304]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:18.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:17 smithi062 ceph-mon[35304]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:18.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:17 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:18.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-mon[31473]: pgmap v166: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 38 KiB/s wr, 4 op/s 2024-05-27T07:27:18.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-mon[31473]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:18.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-mon[31473]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:18.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:17 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:18.473 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:18 smithi162 podman[72934]: 2024-05-27 07:27:18.205106303 +0000 UTC m=+0.750604132 container remove e113b963d71c3b70e6706dca4fb32c103f93a75ac379fb0e06ac7bc04171b6d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-27T07:27:18.473 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:18 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:18.792 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:18 smithi012 podman[104991]: 2024-05-27 07:27:18.378192765 +0000 UTC m=+0.844245642 container remove de5f65ec6cc62b0b87a48521091354d24146bf437c44ade4c2d820a3036d39a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:27:18.792 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:18 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:18.928 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] dirstress 2024-05-27T07:27:18.929 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fill 2024-05-27T07:27:18.929 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fill2 2024-05-27T07:27:18.931 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] getpagesize 2024-05-27T07:27:19.089 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:18 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:27:19.089 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:18 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 2.029s CPU time. 2024-05-27T07:27:19.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:18 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:27:19.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:18 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.085s CPU time. 2024-05-27T07:27:19.513 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:27:19.873 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:19 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:27:19.873 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:19 smithi162 systemd[1]: Stopped Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:19.873 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:19 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.233s CPU time. 2024-05-27T07:27:19.874 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:19 smithi162 systemd[1]: Starting Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:20.008 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] holes 2024-05-27T07:27:20.008 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] lstat64 2024-05-27T07:27:20.128 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:20.155 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:19 smithi162 ceph-mon[35676]: pgmap v167: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 43 KiB/s wr, 8 op/s 2024-05-27T07:27:20.155 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:19 smithi162 podman[73025]: 2024-05-27 07:27:19.788948324 +0000 UTC m=+0.033743260 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:20.155 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:19 smithi162 podman[73025]: 2024-05-27 07:27:19.99068635 +0000 UTC m=+0.235481323 container create f9fe1941d95ccbf4e92d18990320f3906284a7d8860ec92bbde4cb3423ff15bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T07:27:20.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:19 smithi062 ceph-mon[35304]: pgmap v167: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 43 KiB/s wr, 8 op/s 2024-05-27T07:27:20.256 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] nametest 2024-05-27T07:27:20.288 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:19 smithi012 ceph-mon[31473]: pgmap v167: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 43 KiB/s wr, 8 op/s 2024-05-27T07:27:20.290 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] permname 2024-05-27T07:27:20.651 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:20 smithi162 podman[73025]: 2024-05-27 07:27:20.278284851 +0000 UTC m=+0.523079848 container init f9fe1941d95ccbf4e92d18990320f3906284a7d8860ec92bbde4cb3423ff15bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:27:20.651 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:20 smithi162 podman[73025]: 2024-05-27 07:27:20.290388092 +0000 UTC m=+0.535183098 container start f9fe1941d95ccbf4e92d18990320f3906284a7d8860ec92bbde4cb3423ff15bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:27:20.651 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:20 smithi162 ceph-mds[73043]: -- 172.21.15.162:0/574571236 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648b9421a40 con 0x5648b9469180 2024-05-27T07:27:20.651 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: starting mds.i at 2024-05-27T07:27:20.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:20 smithi012 ceph-mon[31473]: daemon mds.i restarted 2024-05-27T07:27:20.906 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:20 smithi162 bash[73025]: f9fe1941d95ccbf4e92d18990320f3906284a7d8860ec92bbde4cb3423ff15bf 2024-05-27T07:27:20.906 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:20 smithi162 systemd[1]: Started Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:20.910 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] randholes 2024-05-27T07:27:20.958 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] runas 2024-05-27T07:27:21.209 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] truncfile 2024-05-27T07:27:21.222 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:20 smithi162 ceph-mon[35676]: daemon mds.i restarted 2024-05-27T07:27:21.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:20 smithi062 ceph-mon[35304]: daemon mds.i restarted 2024-05-27T07:27:21.259 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] usemem 2024-05-27T07:27:21.804 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] mmapcat 2024-05-27T07:27:21.944 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] append_reader 2024-05-27T07:27:21.990 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] append_writer 2024-05-27T07:27:22.030 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] dirperf 2024-05-27T07:27:22.140 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:21 smithi012 ceph-mon[31473]: pgmap v168: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 37 KiB/s wr, 5 op/s 2024-05-27T07:27:22.140 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:21 smithi012 ceph-mon[31473]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:27:22.140 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:21 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/3414756554,v1:172.21.15.162:6833/3414756554] up:boot 2024-05-27T07:27:22.140 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:21 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:22.140 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:21 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:27:22.140 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:21 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:22.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:21 smithi162 ceph-mon[35676]: pgmap v168: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 37 KiB/s wr, 5 op/s 2024-05-27T07:27:22.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:21 smithi162 ceph-mon[35676]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:27:22.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:21 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/3414756554,v1:172.21.15.162:6833/3414756554] up:boot 2024-05-27T07:27:22.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:21 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:21 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:21 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6281200700 time 2024-05-27T07:27:21.784767+0000 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6296d52ba6] 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: clone() 2024-05-27T07:27:22.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:21.786+0000 7f6281200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6281200700 time 2024-05-27T07:27:21.784767+0000 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6296d52ba6] 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: clone() 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: *** Caught signal (Aborted) ** 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: in thread 7f6281200700 thread_name:md_log_replay 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62958cfd20] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: gsignal() 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: abort() 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6296d52ced] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 19: clone() 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:21.789+0000 7f6281200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: in thread 7f6281200700 thread_name:md_log_replay 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62958cfd20] 2024-05-27T07:27:22.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: gsignal() 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: abort() 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6296d52ced] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 19: clone() 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: -1> 2024-05-27T07:27:21.786+0000 7f6281200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6281200700 time 2024-05-27T07:27:21.784767+0000 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6296d52ba6] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: clone() 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 0> 2024-05-27T07:27:21.789+0000 7f6281200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: in thread 7f6281200700 thread_name:md_log_replay 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62958cfd20] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: gsignal() 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: abort() 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6296d52ced] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 19: clone() 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: -1> 2024-05-27T07:27:21.786+0000 7f6281200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6281200700 time 2024-05-27T07:27:21.784767+0000 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6296d52ba6] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: clone() 2024-05-27T07:27:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:21 smithi062 ceph-mon[35304]: pgmap v168: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 37 KiB/s wr, 5 op/s 2024-05-27T07:27:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:21 smithi062 ceph-mon[35304]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:27:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:21 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/3414756554,v1:172.21.15.162:6833/3414756554] up:boot 2024-05-27T07:27:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:21 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:21 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:27:22.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:21 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 0> 2024-05-27T07:27:21.789+0000 7f6281200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: in thread 7f6281200700 thread_name:md_log_replay 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f62958cfd20] 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2: gsignal() 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 3: abort() 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6296d52ced] 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6296d52dd1] 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564881ee823d] 2024-05-27T07:27:22.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564881ed3971] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 8: (SnapRealm::check_cache() const+0x13e) [0x564881ed1fa0] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564881ed2651] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564881e67347] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564881b6387e] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564881b68e94] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564881b73969] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 14: (MDLog::_replay_thread()+0x1f38) [0x564881f0c48c] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564881f159cb] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 16: (Thread::entry_wrapper()+0x43) [0x7f6296d2172d] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6296d21749] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f62958c51ca] 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 19: clone() 2024-05-27T07:27:22.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:22.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73039]: 2024-05-27T07:27:22.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:21 smithi162 podman[73074]: 2024-05-27 07:27:21.900242273 +0000 UTC m=+0.031469986 container died f9fe1941d95ccbf4e92d18990320f3906284a7d8860ec92bbde4cb3423ff15bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T07:27:22.477 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] metaperf 2024-05-27T07:27:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.539+0000 7f9bdf800700 1 -- 172.21.15.12:0/2671200698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd8107de0 msgr2=0x7f9bd81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.539+0000 7f9bdf800700 1 --2- 172.21.15.12:0/2671200698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd8107de0 0x7f9bd81081f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40099c0 tx=0x7f9bc4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.541+0000 7f9bdf800700 1 -- 172.21.15.12:0/2671200698 shutdown_connections 2024-05-27T07:27:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.541+0000 7f9bdf800700 1 --2- 172.21.15.12:0/2671200698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd81034b0 0x7f9bd81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.541+0000 7f9bdf800700 1 --2- 172.21.15.12:0/2671200698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bd8108730 0x7f9bd8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.541+0000 7f9bdf800700 1 --2- 172.21.15.12:0/2671200698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd8107de0 0x7f9bd81081f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.541+0000 7f9bdf800700 1 -- 172.21.15.12:0/2671200698 >> 172.21.15.12:0/2671200698 conn(0x7f9bd80fedb0 msgr2=0x7f9bd81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.541+0000 7f9bdf800700 1 -- 172.21.15.12:0/2671200698 shutdown_connections 2024-05-27T07:27:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.542+0000 7f9bdf800700 1 -- 172.21.15.12:0/2671200698 wait complete. 2024-05-27T07:27:22.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.543+0000 7f9bdf800700 1 Processor -- start 2024-05-27T07:27:22.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.543+0000 7f9bdf800700 1 -- start start 2024-05-27T07:27:22.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.544+0000 7f9bdf800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd81034b0 0x7f9bd81a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.544+0000 7f9bdf800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd8107de0 0x7f9bd81a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.544+0000 7f9bdf800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bd8108730 0x7f9bd81a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.544+0000 7f9bdf800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd8113240 con 0x7f9bd81034b0 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.544+0000 7f9bdf800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81130c0 con 0x7f9bd8107de0 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.544+0000 7f9bdf800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd81133c0 con 0x7f9bd8108730 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd81034b0 0x7f9bd81a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd81034b0 0x7f9bd81a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60594/0 (socket says 172.21.15.12:60594) 2024-05-27T07:27:22.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 -- 172.21.15.12:0/3850592123 learned_addr learned my addr 172.21.15.12:0/3850592123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd8107de0 0x7f9bd81a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdd800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bd8108730 0x7f9bd81a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 -- 172.21.15.12:0/3850592123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bd8108730 msgr2=0x7f9bd81a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bd8108730 0x7f9bd81a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 -- 172.21.15.12:0/3850592123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd8107de0 msgr2=0x7f9bd81a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd8107de0 0x7f9bd81a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.545+0000 7f9bdce00700 1 -- 172.21.15.12:0/3850592123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd81a61e0 con 0x7f9bd81034b0 2024-05-27T07:27:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.546+0000 7f9bdce00700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd81034b0 0x7f9bd81a1320 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00d950 tx=0x7f9bcc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.547+0000 7f9bd5600700 1 -- 172.21.15.12:0/3850592123 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc016070 con 0x7f9bd81034b0 2024-05-27T07:27:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.547+0000 7f9bd5600700 1 -- 172.21.15.12:0/3850592123 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bcc010460 con 0x7f9bd81034b0 2024-05-27T07:27:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.547+0000 7f9bd5600700 1 -- 172.21.15.12:0/3850592123 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00f620 con 0x7f9bd81034b0 2024-05-27T07:27:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.547+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd81a64d0 con 0x7f9bd81034b0 2024-05-27T07:27:22.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.547+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd81addf0 con 0x7f9bd81034b0 2024-05-27T07:27:22.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.551+0000 7f9bd5600700 1 -- 172.21.15.12:0/3850592123 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9bcc019070 con 0x7f9bd81034b0 2024-05-27T07:27:22.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.553+0000 7f9bee2e2b80 1 Processor -- start 2024-05-27T07:27:22.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.553+0000 7f9bee2e2b80 1 -- start start 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bd5600700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9bc00b2af0 0x7f9bc00b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bd5600700 1 -- 172.21.15.12:0/3850592123 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9466+0+0 (secure 0 0 0) 0x7f9bcc0e24c0 con 0x7f9bd81034b0 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f60ea3e40 0x561f610397d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f61033880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61033dc0 0x561f610341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561f60f50e40 con 0x561f61039d10 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561f60f409e0 con 0x561f61033dc0 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.554+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561f60f44a20 con 0x561f60ea3e40 2024-05-27T07:27:22.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.555+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f60ea3e40 0x561f610397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.555+0000 7f9bd7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f61033880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.555+0000 7f9bd7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f61033880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60608/0 (socket says 172.21.15.12:60608) 2024-05-27T07:27:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.555+0000 7f9bd7e00700 1 -- 172.21.15.12:0/1409295294 learned_addr learned my addr 172.21.15.12:0/1409295294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.556+0000 7f9bdd800700 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61033dc0 0x561f610341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.556+0000 7f9bd7e00700 1 -- 172.21.15.12:0/1409295294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f60ea3e40 msgr2=0x561f610397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.556+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f60ea3e40 0x561f610397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.556+0000 7f9bd7e00700 1 -- 172.21.15.12:0/1409295294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61033dc0 msgr2=0x561f610341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.556+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61033dc0 0x561f610341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.556+0000 7f9bd7e00700 1 -- 172.21.15.12:0/1409295294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f60f09420 con 0x561f61039d10 2024-05-27T07:27:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.557+0000 7f9bdd800700 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61033dc0 0x561f610341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.557+0000 7f9bdd800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9bc00b2af0 0x7f9bc00b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.557+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f61033880 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4006010 tx=0x7f9bc4005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946ec0a9949cd14b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.557+0000 7f9bca000700 1 -- 172.21.15.12:0/1409295294 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4025070 con 0x561f61039d10 2024-05-27T07:27:22.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.557+0000 7f9bca000700 1 -- 172.21.15.12:0/1409295294 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bc400b790 con 0x561f61039d10 2024-05-27T07:27:22.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.557+0000 7f9bca000700 1 -- 172.21.15.12:0/1409295294 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40038e0 con 0x561f61039d10 2024-05-27T07:27:22.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.558+0000 7f9bdd800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9bc00b2af0 0x7f9bc00b5000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009710 tx=0x7f9bd0009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.559+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/1409295294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 msgr2=0x561f61033880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.559+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f61033880 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4006010 tx=0x7f9bc4005f00 comp rx=0 tx=0).stop 2024-05-27T07:27:22.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.559+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/1409295294 shutdown_connections 2024-05-27T07:27:22.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.559+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61033dc0 0x561f610341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.559+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f60ea3e40 0x561f610397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.559+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/1409295294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f61033880 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.559+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/1409295294 >> 172.21.15.12:0/1409295294 conn(0x561f60f03fc0 msgr2=0x561f6101d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.560+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/1409295294 shutdown_connections 2024-05-27T07:27:22.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.560+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/1409295294 wait complete. 2024-05-27T07:27:22.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.560+0000 7f9bee2e2b80 1 -- start start 2024-05-27T07:27:22.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.561+0000 7f9bee2e2b80 1 Processor -- start 2024-05-27T07:27:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.561+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f60ea3e40 0x561f6103d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.561+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f6103db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.561+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61039d10 0x561f6103e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.561+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561f60e46fa0 con 0x561f60ea3e40 2024-05-27T07:27:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.561+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561f60ee1760 con 0x561f61039d10 2024-05-27T07:27:22.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.561+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561f610407b0 con 0x561f61033dc0 2024-05-27T07:27:22.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.562+0000 7f9bee2e2b80 10 client.0 _ll_register_callbacks cb 0x561f60eeac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:22.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.562+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f60ea3e40 0x561f6103d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.562+0000 7f9bd7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f6103db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.562+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f60ea3e40 0x561f6103d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60612/0 (socket says 172.21.15.12:60612) 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.562+0000 7f9bdce00700 1 -- 172.21.15.12:0/3009863320 learned_addr learned my addr 172.21.15.12:0/3009863320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdd800700 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61039d10 0x561f6103e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdce00700 1 -- 172.21.15.12:0/3009863320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 msgr2=0x561f6103db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdce00700 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f6103db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdce00700 1 -- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61039d10 msgr2=0x561f6103e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdce00700 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61039d10 0x561f6103e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdce00700 1 -- 172.21.15.12:0/3009863320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f60e908d0 con 0x561f60ea3e40 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdd800700 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61039d10 0x561f6103e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bdce00700 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f60ea3e40 0x561f6103d640 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009520 tx=0x7f9bcc0a98d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f377cfceafff9ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.563+0000 7f9bbea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:22.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.564+0000 7f9bbea00700 1 -- 172.21.15.12:0/3009863320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc020040 con 0x561f60ea3e40 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.564+0000 7f9bbea00700 1 -- 172.21.15.12:0/3009863320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9466+0+0 (secure 0 0 0) 0x7f9bcc009b40 con 0x561f60ea3e40 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.564+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bc400b790 con 0x561f60ea3e40 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.564+0000 7f9bbea00700 1 -- 172.21.15.12:0/3009863320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f9bcc0a8ea0 con 0x561f60ea3e40 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.564+0000 7f9bee2e2b80 10 client.16002 fetch_fsmap learned FSMap version 60 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.564+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x561f60f11060 con 0x561f60ea3e40 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bbea00700 1 -- 172.21.15.12:0/3009863320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bc400b790 con 0x561f60ea3e40 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bbea00700 1 -- 172.21.15.12:0/3009863320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 60) ==== 6547+0+0 (secure 0 0 0) 0x7f9bcc0a4d00 con 0x561f60ea3e40 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bee2e2b80 10 client.16002 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bee2e2b80 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bee2e2b80 20 client.16002 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f6104d9f0 0x561f6104fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bee2e2b80 4 client.16002 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:22.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.565+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f60f52d90 con 0x561f6104d9f0 2024-05-27T07:27:22.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.566+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f6104d9f0 0x561f6104fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.567+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f6104d9f0 0x561f6104fea0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d728062cacbe3d32 server_cookie=de70019a1d35fad1 in_seq=0 out_seq=0 2024-05-27T07:27:22.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.567+0000 7f9bbea00700 10 client.16002 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:22.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bbea00700 1 -- 172.21.15.12:0/3009863320 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561f60f52d90 con 0x561f6104d9f0 2024-05-27T07:27:22.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bbea00700 10 client.16002 handle_command_reply: tid=0 2024-05-27T07:27:22.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 client.16002 shutdown 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f60ea3e40 msgr2=0x561f6103d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f60ea3e40 0x561f6103d640 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009520 tx=0x7f9bcc0a98d0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 shutdown_connections 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f6104d9f0 0x561f6104fea0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f61039d10 0x561f6103e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f60ea3e40 0x561f6103d640 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/3009863320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f6103db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.569+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 >> 172.21.15.12:0/3009863320 conn(0x561f60f03fc0 msgr2=0x561f6101d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.570+0000 7f9bbea00700 0 client.16002 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.570+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 shutdown_connections 2024-05-27T07:27:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.570+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/3009863320 wait complete. 2024-05-27T07:27:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.570+0000 7f9bee2e2b80 20 client.16002 trim_cache size 0 max 16384 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 Processor -- start 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 -- start start 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61021860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f61021da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f610222e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb8010790 con 0x561f61039d10 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb800f260 con 0x561f60ea3e40 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.602+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb800f930 con 0x561f61033dc0 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.603+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61021860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.603+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61021860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:48006/0 (socket says 172.21.15.12:48006) 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.603+0000 7f9bdce00700 1 -- 172.21.15.12:0/4280173926 learned_addr learned my addr 172.21.15.12:0/4280173926 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.603+0000 7f9bdd800700 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f610222e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.603+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f61021da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.604+0000 7f9bdce00700 1 -- 172.21.15.12:0/4280173926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 msgr2=0x561f61021da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.604+0000 7f9bdce00700 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f61021da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.604+0000 7f9bdce00700 1 -- 172.21.15.12:0/4280173926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 msgr2=0x561f610222e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.604+0000 7f9bdce00700 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f610222e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.604+0000 7f9bdce00700 1 -- 172.21.15.12:0/4280173926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f61050430 con 0x561f60ea3e40 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.604+0000 7f9bdd800700 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f610222e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.604+0000 7f9bdce00700 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61021860 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0a9040 tx=0x7f9bcc00df10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52905455b8e1be26 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.605+0000 7f9bc9600700 1 -- 172.21.15.12:0/4280173926 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc020070 con 0x561f60ea3e40 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.605+0000 7f9bc9600700 1 -- 172.21.15.12:0/4280173926 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bcc0a9a90 con 0x561f60ea3e40 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.605+0000 7f9bc9600700 1 -- 172.21.15.12:0/4280173926 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc0a85f0 con 0x561f60ea3e40 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.606+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/4280173926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 msgr2=0x561f61021860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.606+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61021860 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0a9040 tx=0x7f9bcc00df10 comp rx=0 tx=0).stop 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.606+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/4280173926 shutdown_connections 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.606+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61021860 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.606+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61039d10 0x561f610222e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.606+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/4280173926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61033dc0 0x561f61021da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.606+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/4280173926 >> 172.21.15.12:0/4280173926 conn(0x561f60f03fc0 msgr2=0x561f61038780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.607+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/4280173926 shutdown_connections 2024-05-27T07:27:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.607+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/4280173926 wait complete. 2024-05-27T07:27:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.608+0000 7f9bee2e2b80 1 -- start start 2024-05-27T07:27:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.608+0000 7f9bee2e2b80 1 Processor -- start 2024-05-27T07:27:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.610+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61046060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.610+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61033dc0 0x561f61054d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.610+0000 7f9bee2e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61039d10 0x561f610552b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.610+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb800d950 con 0x561f61033dc0 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb8010060 con 0x561f60ea3e40 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bee2e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb800cc10 con 0x561f61039d10 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bd7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61033dc0 0x561f61054d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61046060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bdce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61046060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:48018/0 (socket says 172.21.15.12:48018) 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bdce00700 1 -- 172.21.15.12:0/2923085452 learned_addr learned my addr 172.21.15.12:0/2923085452 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61033dc0 0x561f61054d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60630/0 (socket says 172.21.15.12:60630) 2024-05-27T07:27:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bdd800700 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61039d10 0x561f610552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.611+0000 7f9bee2e2b80 10 client.0 _ll_register_callbacks cb 0x561f6101e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.612+0000 7f9bdce00700 1 -- 172.21.15.12:0/2923085452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61039d10 msgr2=0x561f610552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.612+0000 7f9bdce00700 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61039d10 0x561f610552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.612+0000 7f9bdce00700 1 -- 172.21.15.12:0/2923085452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61033dc0 msgr2=0x561f61054d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.612+0000 7f9bdce00700 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61033dc0 0x561f61054d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.612+0000 7f9bdce00700 1 -- 172.21.15.12:0/2923085452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6103e430 con 0x561f60ea3e40 2024-05-27T07:27:22.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.613+0000 7f9bdce00700 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61046060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0a9090 tx=0x7f9bcc009d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ef7c9d02022ddd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.613+0000 7f9bbea00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:27:22.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.614+0000 7f9bbea00700 1 -- 172.21.15.12:0/2923085452 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc020070 con 0x561f60ea3e40 2024-05-27T07:27:22.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.614+0000 7f9bbea00700 1 -- 172.21.15.12:0/2923085452 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9466+0+0 (secure 0 0 0) 0x7f9bcc0a85f0 con 0x561f60ea3e40 2024-05-27T07:27:22.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.614+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bcc0a9a90 con 0x561f60ea3e40 2024-05-27T07:27:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.614+0000 7f9bbea00700 1 -- 172.21.15.12:0/2923085452 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f9bcc0a5250 con 0x561f60ea3e40 2024-05-27T07:27:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.614+0000 7f9bee2e2b80 10 client.35621 fetch_fsmap learned FSMap version 60 2024-05-27T07:27:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.615+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x561f6103e430 con 0x561f60ea3e40 2024-05-27T07:27:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.615+0000 7f9bbea00700 1 -- 172.21.15.12:0/2923085452 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bcc0a9a90 con 0x561f60ea3e40 2024-05-27T07:27:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.615+0000 7f9bbea00700 1 -- 172.21.15.12:0/2923085452 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 60) ==== 6547+0+0 (secure 0 0 0) 0x7f9bcc0ca040 con 0x561f60ea3e40 2024-05-27T07:27:22.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.616+0000 7f9bee2e2b80 10 client.35621 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:27:22.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.616+0000 7f9bee2e2b80 10 client.35621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:22.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.616+0000 7f9bee2e2b80 20 client.35621 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:22.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.616+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f610b8350 0x561f610ba770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.616+0000 7f9bee2e2b80 4 client.35621 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:27:22.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.616+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561f60f4c570 con 0x561f610b8350 2024-05-27T07:27:22.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.617+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f610b8350 0x561f610ba770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.618+0000 7f9bbea00700 10 client.35621 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.618+0000 7f9bd7e00700 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f610b8350 0x561f610ba770 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=283a76ac8ba13fed server_cookie=ea133e1c025c0a55 in_seq=0 out_seq=0 2024-05-27T07:27:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.618+0000 7f9bbea00700 1 -- 172.21.15.12:0/2923085452 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561f60f4c570 con 0x561f610b8350 2024-05-27T07:27:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.618+0000 7f9bbea00700 10 client.35621 handle_command_reply: tid=0 2024-05-27T07:27:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.618+0000 7f9bee2e2b80 1 client.35621 shutdown 2024-05-27T07:27:22.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 msgr2=0x561f61046060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61046060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0a9090 tx=0x7f9bcc009d50 comp rx=0 tx=0).stop 2024-05-27T07:27:22.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 shutdown_connections 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561f610b8350 0x561f610ba770 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f60ea3e40 0x561f61046060 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0a9090 tx=0x7f9bcc009d50 comp rx=0 tx=0).stop 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f61039d10 0x561f610552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 --2- 172.21.15.12:0/2923085452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561f61033dc0 0x561f61054d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 >> 172.21.15.12:0/2923085452 conn(0x561f60f03fc0 msgr2=0x561f61037260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bbea00700 0 client.35621 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 shutdown_connections 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.619+0000 7f9bee2e2b80 1 -- 172.21.15.12:0/2923085452 wait complete. 2024-05-27T07:27:22.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.620+0000 7f9bee2e2b80 20 client.35621 trim_cache size 0 max 16384 2024-05-27T07:27:22.611 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:27:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:27:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:27:22.611 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:27:22.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9bc00b2af0 msgr2=0x7f9bc00b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9bc00b2af0 0x7f9bc00b5000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009710 tx=0x7f9bd0009040 comp rx=0 tx=0).stop 2024-05-27T07:27:22.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd81034b0 msgr2=0x7f9bd81a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd81034b0 0x7f9bd81a1320 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00d950 tx=0x7f9bcc00dd10 comp rx=0 tx=0).stop 2024-05-27T07:27:22.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 shutdown_connections 2024-05-27T07:27:22.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9bc00b2af0 0x7f9bc00b5000 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bd8108730 0x7f9bd81a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bd81034b0 0x7f9bd81a1320 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 --2- 172.21.15.12:0/3850592123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd8107de0 0x7f9bd81a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 >> 172.21.15.12:0/3850592123 conn(0x7f9bd80fedb0 msgr2=0x7f9bd806b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.621+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 shutdown_connections 2024-05-27T07:27:22.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:22.622+0000 7f9bdf800700 1 -- 172.21.15.12:0/3850592123 wait complete. 2024-05-27T07:27:22.760 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] devzero 2024-05-27T07:27:22.797 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] feature 2024-05-27T07:27:22.966 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] alloc 2024-05-27T07:27:22.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:22 smithi162 podman[73074]: 2024-05-27 07:27:22.633828175 +0000 UTC m=+0.765055934 container remove f9fe1941d95ccbf4e92d18990320f3906284a7d8860ec92bbde4cb3423ff15bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:27:22.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:22 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:23.441 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fault 2024-05-27T07:27:23.473 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:23 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-27T07:27:23.473 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:23 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.087s CPU time. 2024-05-27T07:27:23.497 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:27:23.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:27:23.617 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fstest 2024-05-27T07:27:23.701 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_access_root 2024-05-27T07:27:23.808 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] godown 2024-05-27T07:27:24.004 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:24.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:23 smithi012 ceph-mon[31473]: pgmap v170: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 44 KiB/s wr, 6 op/s 2024-05-27T07:27:24.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:23 smithi162 ceph-mon[35676]: pgmap v170: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 44 KiB/s wr, 6 op/s 2024-05-27T07:27:24.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:23 smithi062 ceph-mon[35304]: pgmap v170: 129 pgs: 129 active+clean; 81 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 44 KiB/s wr, 6 op/s 2024-05-27T07:27:24.260 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] resvtest 2024-05-27T07:27:24.434 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] writemod 2024-05-27T07:27:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] writev_on_pagefault 2024-05-27T07:27:24.588 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] makeextents 2024-05-27T07:27:25.150 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] itrash 2024-05-27T07:27:25.174 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] rename 2024-05-27T07:27:25.245 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] multi_open_unlink 2024-05-27T07:27:25.453 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] unwritten_sync 2024-05-27T07:27:25.918 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] genhashnames 2024-05-27T07:27:25.925 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_holes 2024-05-27T07:27:26.000 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_writev 2024-05-27T07:27:26.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:25 smithi012 ceph-mon[31473]: pgmap v171: 129 pgs: 129 active+clean; 82 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 81 KiB/s wr, 31 op/s 2024-05-27T07:27:26.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:25 smithi162 ceph-mon[35676]: pgmap v171: 129 pgs: 129 active+clean; 82 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 81 KiB/s wr, 31 op/s 2024-05-27T07:27:26.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:25 smithi062 ceph-mon[35304]: pgmap v171: 129 pgs: 129 active+clean; 82 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 81 KiB/s wr, 31 op/s 2024-05-27T07:27:26.271 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_truncate_cmtime 2024-05-27T07:27:26.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.579+0000 7fea2ca00700 1 -- 172.21.15.12:0/1834656410 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c003af0 con 0x7fea281045e0 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.579+0000 7fea30800700 1 -- 172.21.15.12:0/1834656410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 msgr2=0x7fea281049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.579+0000 7fea30800700 1 --2- 172.21.15.12:0/1834656410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea281049f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0099c0 tx=0x7fea1c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.580+0000 7fea30800700 1 -- 172.21.15.12:0/1834656410 shutdown_connections 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.580+0000 7fea30800700 1 --2- 172.21.15.12:0/1834656410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea28109860 0x7fea281111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.580+0000 7fea30800700 1 --2- 172.21.15.12:0/1834656410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea281049f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.580+0000 7fea30800700 1 --2- 172.21.15.12:0/1834656410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea28108f10 0x7fea28109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.580+0000 7fea30800700 1 -- 172.21.15.12:0/1834656410 >> 172.21.15.12:0/1834656410 conn(0x7fea28100030 msgr2=0x7fea28102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.581+0000 7fea30800700 1 -- 172.21.15.12:0/1834656410 shutdown_connections 2024-05-27T07:27:26.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.581+0000 7fea30800700 1 -- 172.21.15.12:0/1834656410 wait complete. 2024-05-27T07:27:26.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 Processor -- start 2024-05-27T07:27:26.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 -- start start 2024-05-27T07:27:26.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea28074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea28108f10 0x7fea28075470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea28109860 0x7fea2806f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fea2806d560 con 0x7fea281045e0 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fea2806d3e0 con 0x7fea28109860 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.582+0000 7fea30800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fea2806d6e0 con 0x7fea28108f10 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.583+0000 7fea2de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea28074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.583+0000 7fea2de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea28074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60636/0 (socket says 172.21.15.12:60636) 2024-05-27T07:27:26.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.583+0000 7fea2de00700 1 -- 172.21.15.12:0/3137802500 learned_addr learned my addr 172.21.15.12:0/3137802500 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:26.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.583+0000 7fea2d400700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea28108f10 0x7fea28075470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:26.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.584+0000 7fea2de00700 1 -- 172.21.15.12:0/3137802500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea28108f10 msgr2=0x7fea28075470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:26.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.584+0000 7fea2de00700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea28108f10 0x7fea28075470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.584+0000 7fea2de00700 1 -- 172.21.15.12:0/3137802500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea28109860 msgr2=0x7fea2806f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:26.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.584+0000 7fea2de00700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea28109860 0x7fea2806f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.584+0000 7fea2de00700 1 -- 172.21.15.12:0/3137802500 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2806f9f0 con 0x7fea281045e0 2024-05-27T07:27:26.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.584+0000 7fea2de00700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea28074f10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000c00 tx=0x7fea1c00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:26.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.584+0000 7fea22a00700 1 -- 172.21.15.12:0/3137802500 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c00f930 con 0x7fea281045e0 2024-05-27T07:27:26.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.585+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2806fc80 con 0x7fea281045e0 2024-05-27T07:27:26.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.585+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea28075bb0 con 0x7fea281045e0 2024-05-27T07:27:26.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.585+0000 7fea22a00700 1 -- 172.21.15.12:0/3137802500 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea1c01f4d0 con 0x7fea281045e0 2024-05-27T07:27:26.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.585+0000 7fea22a00700 1 -- 172.21.15.12:0/3137802500 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c017650 con 0x7fea281045e0 2024-05-27T07:27:26.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.587+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea0c005330 con 0x7fea281045e0 2024-05-27T07:27:26.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.590+0000 7fea22a00700 1 -- 172.21.15.12:0/3137802500 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fea1c00fad0 con 0x7fea281045e0 2024-05-27T07:27:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.592+0000 7fea22a00700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fea100b2a70 0x7fea100b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.592+0000 7fea22a00700 1 -- 172.21.15.12:0/3137802500 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9466+0+0 (secure 0 0 0) 0x7fea1c0e7e10 con 0x7fea281045e0 2024-05-27T07:27:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.593+0000 7fea2d400700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fea100b2a70 0x7fea100b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.594+0000 7fea2d400700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fea100b2a70 0x7fea100b4f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea18005950 tx=0x7fea180058e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.599+0000 7fea22a00700 1 -- 172.21.15.12:0/3137802500 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fea1c0eb030 con 0x7fea281045e0 2024-05-27T07:27:26.753 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] dirhash_collide 2024-05-27T07:27:26.753 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_rename_overwrite 2024-05-27T07:27:26.794 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] holetest 2024-05-27T07:27:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.991+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea0c0058f0 con 0x7fea281045e0 2024-05-27T07:27:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.992+0000 7fea22a00700 1 -- 172.21.15.12:0/3137802500 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+9394 (secure 0 0 0) 0x7fea1c013070 con 0x7fea281045e0 2024-05-27T07:27:26.982 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:27:26.983 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/2663671830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":2663671830},{"type":"v1","addr":"172.21.15.12:6839","nonce":2663671830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":60,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:27:20.935948+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6841/3587822997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":3587822997},{"type":"v1","addr":"172.21.15.12:6841","nonce":3587822997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/1597502189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1597502189},{"type":"v1","addr":"172.21.15.162:6833","nonce":1597502189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25654":{"gid":25654,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3414756554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3414756554},{"type":"v1","addr":"172.21.15.162:6833","nonce":3414756554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:27:26.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.995+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fea100b2a70 msgr2=0x7fea100b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:26.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.995+0000 7fea30800700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fea100b2a70 0x7fea100b4f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea18005950 tx=0x7fea180058e0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.995+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 msgr2=0x7fea28074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:26.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.995+0000 7fea30800700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea28074f10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000c00 tx=0x7fea1c00f510 comp rx=0 tx=0).stop 2024-05-27T07:27:26.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 shutdown_connections 2024-05-27T07:27:26.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fea100b2a70 0x7fea100b4f80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea28109860 0x7fea2806f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea281045e0 0x7fea28074f10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 --2- 172.21.15.12:0/3137802500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea28108f10 0x7fea28075470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:26.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 >> 172.21.15.12:0/3137802500 conn(0x7fea28100030 msgr2=0x7fea28102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:26.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 shutdown_connections 2024-05-27T07:27:26.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:26.996+0000 7fea30800700 1 -- 172.21.15.12:0/3137802500 wait complete. 2024-05-27T07:27:26.987 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 60 2024-05-27T07:27:27.012 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_truncate_self 2024-05-27T07:27:27.572 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] af_unix 2024-05-27T07:27:27.610 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_stale_pmd 2024-05-27T07:27:27.696 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_cow_race 2024-05-27T07:27:27.709 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_fallocate 2024-05-27T07:27:27.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:27:28.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:27 smithi162 ceph-mon[35676]: pgmap v172: 129 pgs: 129 active+clean; 82 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 81 KiB/s wr, 31 op/s 2024-05-27T07:27:28.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:27 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3137802500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:28.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:27 smithi062 ceph-mon[35304]: pgmap v172: 129 pgs: 129 active+clean; 82 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 81 KiB/s wr, 31 op/s 2024-05-27T07:27:28.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:27 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3137802500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:28.279 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fsync-err 2024-05-27T07:27:28.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:27 smithi012 ceph-mon[31473]: pgmap v172: 129 pgs: 129 active+clean; 82 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 81 KiB/s wr, 31 op/s 2024-05-27T07:27:28.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:27 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3137802500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:28.388 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_write_ro 2024-05-27T07:27:28.477 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_ext4_dax_journal_corruption 2024-05-27T07:27:28.503 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:28.528 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_ext4_dax_inline_corruption 2024-05-27T07:27:29.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:28 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:27:29.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:28 smithi012 systemd[1]: Stopped Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:29.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:28 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 2.029s CPU time. 2024-05-27T07:27:29.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:28 smithi012 systemd[1]: Starting Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:29.384 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:28 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:27:29.384 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:28 smithi162 systemd[1]: Stopped Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:29.384 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:28 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.085s CPU time. 2024-05-27T07:27:29.384 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 systemd[1]: Starting Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:29.692 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 podman[73166]: 2024-05-27 07:27:29.299567464 +0000 UTC m=+0.036370779 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:29.692 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 podman[73166]: 2024-05-27 07:27:29.401031846 +0000 UTC m=+0.137835207 container create 1d3cd80af82dd8593c34241af6d4167e002128c7a50943cc5b60da48512e4e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T07:27:29.692 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 podman[73166]: 2024-05-27 07:27:29.702381203 +0000 UTC m=+0.439184538 container init 1d3cd80af82dd8593c34241af6d4167e002128c7a50943cc5b60da48512e4e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:27:29.783 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_collision 2024-05-27T07:27:29.784 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] mmap-write-concurrent 2024-05-27T07:27:29.784 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_get_file_time 2024-05-27T07:27:29.784 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_ofd_locks 2024-05-27T07:27:29.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:29 smithi162 ceph-mon[35676]: pgmap v173: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 182 KiB/s wr, 68 op/s 2024-05-27T07:27:29.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:29 smithi162 ceph-mon[35676]: daemon mds.l restarted 2024-05-27T07:27:29.973 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 podman[73166]: 2024-05-27 07:27:29.712366209 +0000 UTC m=+0.449169517 container start 1d3cd80af82dd8593c34241af6d4167e002128c7a50943cc5b60da48512e4e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T07:27:29.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 bash[73166]: 1d3cd80af82dd8593c34241af6d4167e002128c7a50943cc5b60da48512e4e83 2024-05-27T07:27:29.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 systemd[1]: Started Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:29.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 ceph-mds[73185]: -- 172.21.15.162:0/2777766906 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x563f55c444e0 con 0x563f55c8f600 2024-05-27T07:27:29.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 ceph-mds[73185]: -- 172.21.15.162:0/2777766906 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f55c47a40 con 0x563f55c8f600 2024-05-27T07:27:29.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: starting mds.l at 2024-05-27T07:27:30.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:29 smithi012 ceph-mon[31473]: pgmap v173: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 182 KiB/s wr, 68 op/s 2024-05-27T07:27:30.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:29 smithi012 ceph-mon[31473]: daemon mds.l restarted 2024-05-27T07:27:30.178 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faa87600700 time 2024-05-27T07:27:30.127108+0000 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faa9d084ba6] 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.179 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: clone() 2024-05-27T07:27:30.180 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.128+0000 7faa87600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faa87600700 time 2024-05-27T07:27:30.127108+0000 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faa9d084ba6] 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.181 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: clone() 2024-05-27T07:27:30.182 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: *** Caught signal (Aborted) ** 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: in thread 7faa87600700 thread_name:md_log_replay 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faa9bc01d20] 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: gsignal() 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: abort() 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faa9d084ced] 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.183 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 19: clone() 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.131+0000 7faa87600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: in thread 7faa87600700 thread_name:md_log_replay 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faa9bc01d20] 2024-05-27T07:27:30.184 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: gsignal() 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: abort() 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faa9d084ced] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.185 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 19: clone() 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: -1> 2024-05-27T07:27:30.128+0000 7faa87600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faa87600700 time 2024-05-27T07:27:30.127108+0000 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faa9d084ba6] 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.186 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: clone() 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 0> 2024-05-27T07:27:30.131+0000 7faa87600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: in thread 7faa87600700 thread_name:md_log_replay 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faa9bc01d20] 2024-05-27T07:27:30.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: gsignal() 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: abort() 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faa9d084ced] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 19: clone() 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: -1> 2024-05-27T07:27:30.128+0000 7faa87600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faa87600700 time 2024-05-27T07:27:30.127108+0000 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faa9d084ba6] 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.191 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: clone() 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 0> 2024-05-27T07:27:30.131+0000 7faa87600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: in thread 7faa87600700 thread_name:md_log_replay 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faa9bc01d20] 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2: gsignal() 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 3: abort() 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faa9d084ced] 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faa9d084dd1] 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56532063223d] 2024-05-27T07:27:30.192 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56532061d971] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x56532061bfa0] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56532061c651] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5653205b1347] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5653202ad87e] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5653202b2e94] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5653202bd969] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x56532065648c] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56532065f9cb] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7faa9d05372d] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7faa9d053749] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faa9bbf71ca] 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 19: clone() 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:30.193 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[73321]: 2024-05-27T07:27:30.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:29 smithi062 ceph-mon[35304]: pgmap v173: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 182 KiB/s wr, 68 op/s 2024-05-27T07:27:30.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:29 smithi062 ceph-mon[35304]: daemon mds.l restarted 2024-05-27T07:27:30.475 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3f2de00700 time 2024-05-27T07:27:30.082271+0000 2024-05-27T07:27:30.475 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.475 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.475 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3f4385fba6] 2024-05-27T07:27:30.475 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.476 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: clone() 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.084+0000 7f3f2de00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3f2de00700 time 2024-05-27T07:27:30.082271+0000 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3f4385fba6] 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.477 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.478 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: clone() 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: *** Caught signal (Aborted) ** 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: in thread 7f3f2de00700 thread_name:md_log_replay 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f423dcd20] 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: gsignal() 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: abort() 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3f4385fced] 2024-05-27T07:27:30.479 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.480 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 19: clone() 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.087+0000 7f3f2de00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: in thread 7f3f2de00700 thread_name:md_log_replay 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f423dcd20] 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: gsignal() 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: abort() 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3f4385fced] 2024-05-27T07:27:30.481 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.482 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 19: clone() 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: -1> 2024-05-27T07:27:30.084+0000 7f3f2de00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3f2de00700 time 2024-05-27T07:27:30.082271+0000 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3f4385fba6] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.484 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: clone() 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 0> 2024-05-27T07:27:30.087+0000 7f3f2de00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: in thread 7f3f2de00700 thread_name:md_log_replay 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f423dcd20] 2024-05-27T07:27:30.485 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: gsignal() 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: abort() 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3f4385fced] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.486 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.487 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.487 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.487 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 19: clone() 2024-05-27T07:27:30.487 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:30.487 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.487 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: -1> 2024-05-27T07:27:30.084+0000 7f3f2de00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3f2de00700 time 2024-05-27T07:27:30.082271+0000 2024-05-27T07:27:30.487 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3f4385fba6] 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.488 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: clone() 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 0> 2024-05-27T07:27:30.087+0000 7f3f2de00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: in thread 7f3f2de00700 thread_name:md_log_replay 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:30.489 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f423dcd20] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2: gsignal() 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 3: abort() 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3f4385fced] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3f4385fdd1] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563f1f20a23d] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563f1f1f5971] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 8: (SnapRealm::check_cache() const+0x13e) [0x563f1f1f3fa0] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563f1f1f4651] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563f1f189347] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563f1ee8587e] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563f1ee8ae94] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563f1ee95969] 2024-05-27T07:27:30.490 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 14: (MDLog::_replay_thread()+0x1f38) [0x563f1f22e48c] 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563f1f2379cb] 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 16: (Thread::entry_wrapper()+0x43) [0x7f3f4382e72d] 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3f4382e749] 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3f423d21ca] 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 19: clone() 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73181]: 2024-05-27T07:27:30.491 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 podman[73216]: 2024-05-27 07:27:30.210309699 +0000 UTC m=+0.036692791 container died 1d3cd80af82dd8593c34241af6d4167e002128c7a50943cc5b60da48512e4e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:27:30.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:30 smithi012 podman[110155]: 2024-05-27 07:27:30.268854076 +0000 UTC m=+0.039831628 container died 49c7c89e3875cc510dee7921b9234441b4950243b659c04df6f6ee925e87c671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-27T07:27:30.696 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_create_long_dirs 2024-05-27T07:27:30.696 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_create_short_dirs 2024-05-27T07:27:30.732 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_enospc 2024-05-27T07:27:30.774 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_writev_overlap 2024-05-27T07:27:31.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:30 smithi012 ceph-mon[31473]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:27:31.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:30 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/3542733052,v1:172.21.15.162:6833/3542733052] up:boot 2024-05-27T07:27:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:30 smithi012 ceph-mon[31473]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:30 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:31.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:30 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 podman[73216]: 2024-05-27 07:27:30.931233596 +0000 UTC m=+0.757616644 container remove 1d3cd80af82dd8593c34241af6d4167e002128c7a50943cc5b60da48512e4e83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:30 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:30 smithi162 ceph-mon[35676]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:30 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/3542733052,v1:172.21.15.162:6833/3542733052] up:boot 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:30 smithi162 ceph-mon[35676]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:30 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:31.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:30 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:31.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:30 smithi062 ceph-mon[35304]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:27:31.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:30 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/3542733052,v1:172.21.15.162:6833/3542733052] up:boot 2024-05-27T07:27:31.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:30 smithi062 ceph-mon[35304]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:31.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:30 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:31.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:31.236 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:30 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:31.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:31 smithi012 podman[110155]: 2024-05-27 07:27:31.151343173 +0000 UTC m=+0.922320727 container remove 49c7c89e3875cc510dee7921b9234441b4950243b659c04df6f6ee925e87c671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:27:31.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:31 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:31.651 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] checkpoint_journal 2024-05-27T07:27:31.691 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] mmap-rw-fault 2024-05-27T07:27:31.718 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] allocstale 2024-05-27T07:27:31.723 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:31 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:27:31.723 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:31 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.607s CPU time. 2024-05-27T07:27:31.765 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_cow_memory_failure 2024-05-27T07:27:32.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:31 smithi012 ceph-mon[31473]: pgmap v175: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 191 KiB/s wr, 71 op/s 2024-05-27T07:27:32.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:31 smithi162 ceph-mon[35676]: pgmap v175: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 191 KiB/s wr, 71 op/s 2024-05-27T07:27:32.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:31 smithi062 ceph-mon[35304]: pgmap v175: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 191 KiB/s wr, 71 op/s 2024-05-27T07:27:32.538 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:32 smithi012 podman[110866]: 2024-05-27 07:27:32.148329802 +0000 UTC m=+0.281951313 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:32.538 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:32 smithi012 podman[110866]: 2024-05-27 07:27:32.3788431 +0000 UTC m=+0.512464616 container create a084131c859561fa54d536dca8ab31de5c2442236bd521327c93a8c8925d3879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:27:32.658 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fake-dump-rootino 2024-05-27T07:27:32.742 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] dio-buf-fault 2024-05-27T07:27:32.772 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] rewinddir-test 2024-05-27T07:27:32.776 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] readdir-while-renames 2024-05-27T07:27:33.178 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:32 smithi012 podman[110866]: 2024-05-27 07:27:32.979456086 +0000 UTC m=+1.113077603 container init a084131c859561fa54d536dca8ab31de5c2442236bd521327c93a8c8925d3879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:27:33.178 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:32 smithi012 podman[110866]: 2024-05-27 07:27:32.987452337 +0000 UTC m=+1.121073849 container start a084131c859561fa54d536dca8ab31de5c2442236bd521327c93a8c8925d3879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:27:33.178 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:33 smithi012 ceph-mds[111370]: -- 172.21.15.12:0/1998987493 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c5395afa40 con 0x55c5395f7a80 2024-05-27T07:27:33.178 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:33 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: starting mds.a at 2024-05-27T07:27:33.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.371+0000 7fc3cee00700 1 -- 172.21.15.12:0/2258427921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c8105e20 msgr2=0x7fc3c810e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.371+0000 7fc3cee00700 1 --2- 172.21.15.12:0/2258427921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c8105e20 0x7fc3c810e370 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009990 tx=0x7fc3b4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.371+0000 7fc3cee00700 1 -- 172.21.15.12:0/2258427921 shutdown_connections 2024-05-27T07:27:33.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.371+0000 7fc3cee00700 1 --2- 172.21.15.12:0/2258427921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c81054d0 0x7fc3c81058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.371+0000 7fc3cee00700 1 --2- 172.21.15.12:0/2258427921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3c810e900 0x7fc3c8110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.371+0000 7fc3cee00700 1 --2- 172.21.15.12:0/2258427921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c8105e20 0x7fc3c810e370 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.371+0000 7fc3cee00700 1 -- 172.21.15.12:0/2258427921 >> 172.21.15.12:0/2258427921 conn(0x7fc3c80fedb0 msgr2=0x7fc3c81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:33.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.372+0000 7fc3cee00700 1 -- 172.21.15.12:0/2258427921 shutdown_connections 2024-05-27T07:27:33.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.373+0000 7fc3cee00700 1 -- 172.21.15.12:0/2258427921 wait complete. 2024-05-27T07:27:33.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.374+0000 7fc3cee00700 1 Processor -- start 2024-05-27T07:27:33.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.374+0000 7fc3cee00700 1 -- start start 2024-05-27T07:27:33.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c81054d0 0x7fc3c81030f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cee00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3c8105e20 0x7fc3c8103630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cee00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c810e900 0x7fc3c8103b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cee00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81142e0 con 0x7fc3c810e900 2024-05-27T07:27:33.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8114160 con 0x7fc3c81054d0 2024-05-27T07:27:33.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cee00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8114460 con 0x7fc3c8105e20 2024-05-27T07:27:33.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c810e900 0x7fc3c8103b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c810e900 0x7fc3c8103b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34336/0 (socket says 172.21.15.12:34336) 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3cce00700 1 -- 172.21.15.12:0/1314747027 learned_addr learned my addr 172.21.15.12:0/1314747027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.375+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c81054d0 0x7fc3c81030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.376+0000 7fc3c7400700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3c8105e20 0x7fc3c8103630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.376+0000 7fc3cce00700 1 -- 172.21.15.12:0/1314747027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3c8105e20 msgr2=0x7fc3c8103630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.376+0000 7fc3cce00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3c8105e20 0x7fc3c8103630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.376+0000 7fc3cce00700 1 -- 172.21.15.12:0/1314747027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c81054d0 msgr2=0x7fc3c81030f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.376+0000 7fc3cce00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c81054d0 0x7fc3c81030f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.376+0000 7fc3cce00700 1 -- 172.21.15.12:0/1314747027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c80746c0 con 0x7fc3c810e900 2024-05-27T07:27:33.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.377+0000 7fc3cce00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c810e900 0x7fc3c8103b70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000ef30 tx=0x7fc3c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:33.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.377+0000 7fc3c4c00700 1 -- 172.21.15.12:0/1314747027 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c00092e0 con 0x7fc3c810e900 2024-05-27T07:27:33.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.378+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c8074950 con 0x7fc3c810e900 2024-05-27T07:27:33.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.378+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c8074e90 con 0x7fc3c810e900 2024-05-27T07:27:33.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.378+0000 7fc3c4c00700 1 -- 172.21.15.12:0/1314747027 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3c0004bb0 con 0x7fc3c810e900 2024-05-27T07:27:33.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.378+0000 7fc3c4c00700 1 -- 172.21.15.12:0/1314747027 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0004020 con 0x7fc3c810e900 2024-05-27T07:27:33.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.383+0000 7fc3c4c00700 1 -- 172.21.15.12:0/1314747027 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc3c001b030 con 0x7fc3c810e900 2024-05-27T07:27:33.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.385+0000 7fc3dd9a2b80 1 Processor -- start 2024-05-27T07:27:33.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.386+0000 7fc3c4c00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc3ac0b2a70 0x7fc3ac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.386+0000 7fc3c4c00700 1 -- 172.21.15.12:0/1314747027 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9466+0+0 (secure 0 0 0) 0x7fc3c00e2420 con 0x7fc3c810e900 2024-05-27T07:27:33.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.386+0000 7fc3c7400700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc3ac0b2a70 0x7fc3ac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.387+0000 7fc3dd9a2b80 1 -- start start 2024-05-27T07:27:33.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.388+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b351de9f0 0x556b35271330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.388+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35271870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.388+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b35271e70 0x556b3527e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.388+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556b3519fe40 con 0x556b350f2e40 2024-05-27T07:27:33.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.388+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556b3518f9e0 con 0x556b351de9f0 2024-05-27T07:27:33.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.388+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b35193a20 con 0x556b35271e70 2024-05-27T07:27:33.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.388+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b351de9f0 0x556b35271330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.389+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b351de9f0 0x556b35271330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:51084/0 (socket says 172.21.15.12:51084) 2024-05-27T07:27:33.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.389+0000 7fc3c7e00700 1 -- 172.21.15.12:0/2160935864 learned_addr learned my addr 172.21.15.12:0/2160935864 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:33.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.389+0000 7fc3cce00700 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b35271e70 0x556b3527e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.389+0000 7fc3c7400700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc3ac0b2a70 0x7fc3ac0b4f80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400b2e0 tx=0x7fc3b4005fb0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:33.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.389+0000 7fc3c7e00700 1 -- 172.21.15.12:0/2160935864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b35271e70 msgr2=0x556b3527e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:33.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.390+0000 7fc3c7400700 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35271870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.390+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b35271e70 0x556b3527e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.390+0000 7fc3c7e00700 1 -- 172.21.15.12:0/2160935864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 msgr2=0x556b35271870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:33.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.390+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35271870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.390+0000 7fc3c7e00700 1 -- 172.21.15.12:0/2160935864 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b35158420 con 0x556b351de9f0 2024-05-27T07:27:33.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.390+0000 7fc3cce00700 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b35271e70 0x556b3527e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:33.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.390+0000 7fc3c7400700 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35271870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:33.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.391+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b351de9f0 0x556b35271330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800bd70 tx=0x7fc3b80106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae5baa054e87123 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:33.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.392+0000 7fc3bd600700 1 -- 172.21.15.12:0/2160935864 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8010e10 con 0x556b351de9f0 2024-05-27T07:27:33.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.392+0000 7fc3bd600700 1 -- 172.21.15.12:0/2160935864 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3b801c910 con 0x556b351de9f0 2024-05-27T07:27:33.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.392+0000 7fc3bd600700 1 -- 172.21.15.12:0/2160935864 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8011870 con 0x556b351de9f0 2024-05-27T07:27:33.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.393+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2160935864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b351de9f0 msgr2=0x556b35271330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.393+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b351de9f0 0x556b35271330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800bd70 tx=0x7fc3b80106b0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.394+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2160935864 shutdown_connections 2024-05-27T07:27:33.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.394+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b351de9f0 0x556b35271330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.394+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b35271e70 0x556b3527e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.394+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2160935864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35271870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.394+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2160935864 >> 172.21.15.12:0/2160935864 conn(0x556b3518c390 msgr2=0x556b3526d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:33.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.394+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2160935864 shutdown_connections 2024-05-27T07:27:33.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.394+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2160935864 wait complete. 2024-05-27T07:27:33.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.395+0000 7fc3dd9a2b80 1 -- start start 2024-05-27T07:27:33.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.395+0000 7fc3dd9a2b80 1 Processor -- start 2024-05-27T07:27:33.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.396+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b350f2e40 0x556b3527b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.396+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b351de9f0 0x556b3527bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.396+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b3527c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.396+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556b35095fa0 con 0x556b351de9f0 2024-05-27T07:27:33.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.396+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556b35130760 con 0x556b35271e70 2024-05-27T07:27:33.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.396+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b3527eb80 con 0x556b350f2e40 2024-05-27T07:27:33.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.396+0000 7fc3dd9a2b80 10 client.0 _ll_register_callbacks cb 0x556b35139c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:33.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.397+0000 7fc3c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b351de9f0 0x556b3527bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.397+0000 7fc3c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b351de9f0 0x556b3527bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34354/0 (socket says 172.21.15.12:34354) 2024-05-27T07:27:33.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.397+0000 7fc3c7400700 1 -- 172.21.15.12:0/2634727446 learned_addr learned my addr 172.21.15.12:0/2634727446 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:33.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.398+0000 7fc3c7400700 1 -- 172.21.15.12:0/2634727446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b350f2e40 msgr2=0x556b3527b5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:27:33.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.398+0000 7fc3c7400700 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b350f2e40 0x556b3527b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.398+0000 7fc3c7400700 1 -- 172.21.15.12:0/2634727446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 msgr2=0x556b3527c040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:27:33.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.398+0000 7fc3c7400700 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b3527c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.398+0000 7fc3c7400700 1 -- 172.21.15.12:0/2634727446 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b350df8d0 con 0x556b351de9f0 2024-05-27T07:27:33.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.399+0000 7fc3c7400700 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b351de9f0 0x556b3527bb00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009990 tx=0x7fc3b4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53b4fe1bd098b03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:33.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.399+0000 7fc3b2000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:33.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.400+0000 7fc3b2000700 1 -- 172.21.15.12:0/2634727446 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400b9c0 con 0x556b351de9f0 2024-05-27T07:27:33.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.400+0000 7fc3b2000700 1 -- 172.21.15.12:0/2634727446 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9466+0+0 (secure 0 0 0) 0x7fc3b400bb60 con 0x556b351de9f0 2024-05-27T07:27:33.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.401+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b801c910 con 0x556b351de9f0 2024-05-27T07:27:33.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.401+0000 7fc3b2000700 1 -- 172.21.15.12:0/2634727446 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fc3b400bd60 con 0x556b351de9f0 2024-05-27T07:27:33.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.401+0000 7fc3dd9a2b80 10 client.16032 fetch_fsmap learned FSMap version 62 2024-05-27T07:27:33.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.401+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x556b35160060 con 0x556b351de9f0 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.402+0000 7fc3b2000700 1 -- 172.21.15.12:0/2634727446 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3b801c910 con 0x556b351de9f0 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3b2000700 1 -- 172.21.15.12:0/2634727446 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 62) ==== 5942+0+0 (secure 0 0 0) 0x7fc3b4030bd0 con 0x556b351de9f0 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3dd9a2b80 10 client.16032 fetch_fsmap finished waiting for FSMap version 62 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3dd9a2b80 10 client.16032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3dd9a2b80 20 client.16032 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b3528bd50 0x556b3528e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3dd9a2b80 4 client.16032 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:33.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b351a1d90 con 0x556b3528bd50 2024-05-27T07:27:33.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.403+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b3528bd50 0x556b3528e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.404+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b3528bd50 0x556b3528e220 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec375a619b43c91e server_cookie=fbe977dac9d58ad7 in_seq=0 out_seq=0 2024-05-27T07:27:33.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.404+0000 7fc3b2000700 10 client.16032 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:27:33.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3b2000700 1 -- 172.21.15.12:0/2634727446 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556b351a1d90 con 0x556b3528bd50 2024-05-27T07:27:33.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3b2000700 10 client.16032 handle_command_reply: tid=0 2024-05-27T07:27:33.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 client.16032 shutdown 2024-05-27T07:27:33.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b351de9f0 msgr2=0x556b3527bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b351de9f0 0x556b3527bb00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009990 tx=0x7fc3b4009310 comp rx=0 tx=0).stop 2024-05-27T07:27:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 shutdown_connections 2024-05-27T07:27:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b3528bd50 0x556b3528e220 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b3527c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b350f2e40 0x556b3527b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/2634727446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b351de9f0 0x556b3527bb00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3b2000700 0 client.16032 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:27:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.407+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 >> 172.21.15.12:0/2634727446 conn(0x556b3518c390 msgr2=0x556b35271310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.408+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 shutdown_connections 2024-05-27T07:27:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.408+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/2634727446 wait complete. 2024-05-27T07:27:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.408+0000 7fc3dd9a2b80 20 client.16032 trim_cache size 0 max 16384 2024-05-27T07:27:33.428 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:33 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:27:33.428 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:33 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.936s CPU time. 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 Processor -- start 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 -- start start 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35276e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b3527a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35287c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800e4b0 con 0x556b350f2e40 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a80100b0 con 0x556b35271e70 2024-05-27T07:27:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.436+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800eb80 con 0x556b352717c0 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.437+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35276e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.437+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35276e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34366/0 (socket says 172.21.15.12:34366) 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.437+0000 7fc3cce00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35287c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.437+0000 7fc3c7e00700 1 -- 172.21.15.12:0/884264807 learned_addr learned my addr 172.21.15.12:0/884264807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.437+0000 7fc3c7400700 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b3527a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.438+0000 7fc3c7e00700 1 -- 172.21.15.12:0/884264807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 msgr2=0x556b35287c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.438+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35287c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.438+0000 7fc3c7e00700 1 -- 172.21.15.12:0/884264807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 msgr2=0x556b3527a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.438+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b3527a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.438+0000 7fc3c7e00700 1 -- 172.21.15.12:0/884264807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b3527c3b0 con 0x556b350f2e40 2024-05-27T07:27:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.438+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35276e30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8005d90 tx=0x7fc3b8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45d1890667e2388 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:33.431 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:33 smithi012 bash[110866]: a084131c859561fa54d536dca8ab31de5c2442236bd521327c93a8c8925d3879 2024-05-27T07:27:33.431 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:33 smithi012 systemd[1]: Started Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.439+0000 7fc3bcc00700 1 -- 172.21.15.12:0/884264807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80052c0 con 0x556b350f2e40 2024-05-27T07:27:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.439+0000 7fc3bcc00700 1 -- 172.21.15.12:0/884264807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3b8005460 con 0x556b350f2e40 2024-05-27T07:27:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.439+0000 7fc3bcc00700 1 -- 172.21.15.12:0/884264807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b801c480 con 0x556b350f2e40 2024-05-27T07:27:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.440+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/884264807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 msgr2=0x556b35276e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.440+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35276e30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8005d90 tx=0x7fc3b8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.440+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/884264807 shutdown_connections 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.440+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35287c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.440+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b350f2e40 0x556b35276e30 secure :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8005d90 tx=0x7fc3b8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.440+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/884264807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b3527a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.440+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/884264807 >> 172.21.15.12:0/884264807 conn(0x556b3518c390 msgr2=0x556b352710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.441+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/884264807 shutdown_connections 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.441+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/884264807 wait complete. 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.441+0000 7fc3dd9a2b80 1 -- start start 2024-05-27T07:27:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.442+0000 7fc3dd9a2b80 1 Processor -- start 2024-05-27T07:27:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.442+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35275a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.442+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b35275f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.442+0000 7fc3dd9a2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b35282e70 0x556b35283280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.442+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800f980 con 0x556b35282e70 2024-05-27T07:27:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.442+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800f250 con 0x556b35271e70 2024-05-27T07:27:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.442+0000 7fc3dd9a2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800cc40 con 0x556b352717c0 2024-05-27T07:27:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.443+0000 7fc3dd9a2b80 10 client.0 _ll_register_callbacks cb 0x556b35139c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.443+0000 7fc3c7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b35275f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.443+0000 7fc3cce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b35282e70 0x556b35283280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.443+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35275a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.444+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35275a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:49282/0 (socket says 172.21.15.12:49282) 2024-05-27T07:27:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.444+0000 7fc3c7e00700 1 -- 172.21.15.12:0/1858286200 learned_addr learned my addr 172.21.15.12:0/1858286200 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.445+0000 7fc3cce00700 1 -- 172.21.15.12:0/1858286200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 msgr2=0x556b35275a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.445+0000 7fc3cce00700 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35275a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.445+0000 7fc3cce00700 1 -- 172.21.15.12:0/1858286200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 msgr2=0x556b35275f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.445+0000 7fc3cce00700 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b35275f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.445+0000 7fc3cce00700 1 -- 172.21.15.12:0/1858286200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b352837c0 con 0x556b35282e70 2024-05-27T07:27:33.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.446+0000 7fc3cce00700 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b35282e70 0x556b35283280 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00df760 tx=0x7fc3c00dfcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0571b32ec4d68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:33.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.446+0000 7fc3b2000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:33.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.446+0000 7fc3b2000700 1 -- 172.21.15.12:0/1858286200 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c00c2070 con 0x556b35282e70 2024-05-27T07:27:33.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.446+0000 7fc3b2000700 1 -- 172.21.15.12:0/1858286200 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9466+0+0 (secure 0 0 0) 0x7fc3c00b1ed0 con 0x556b35282e70 2024-05-27T07:27:33.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.446+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b8005460 con 0x556b35282e70 2024-05-27T07:27:33.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.446+0000 7fc3b2000700 1 -- 172.21.15.12:0/1858286200 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fc3c00a9380 con 0x556b35282e70 2024-05-27T07:27:33.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.448+0000 7fc3dd9a2b80 10 client.16038 fetch_fsmap learned FSMap version 62 2024-05-27T07:27:33.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.448+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x556b35284990 con 0x556b35282e70 2024-05-27T07:27:33.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.449+0000 7fc3b2000700 1 -- 172.21.15.12:0/1858286200 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3b8005460 con 0x556b35282e70 2024-05-27T07:27:33.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.449+0000 7fc3b2000700 1 -- 172.21.15.12:0/1858286200 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 62) ==== 5942+0+0 (secure 0 0 0) 0x7fc3c00a4cf0 con 0x556b35282e70 2024-05-27T07:27:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.449+0000 7fc3dd9a2b80 10 client.16038 fetch_fsmap finished waiting for FSMap version 62 2024-05-27T07:27:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.449+0000 7fc3dd9a2b80 10 client.16038 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:27:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.449+0000 7fc3dd9a2b80 20 client.16038 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.450+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b35307d50 0x556b3530a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.450+0000 7fc3dd9a2b80 4 client.16038 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.450+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b3519b570 con 0x556b35307d50 2024-05-27T07:27:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.451+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b35307d50 0x556b3530a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:33.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.453+0000 7fc3c7e00700 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b35307d50 0x556b3530a170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ad28b3a1e1ca295 server_cookie=c54589261b4b71d3 in_seq=0 out_seq=0 2024-05-27T07:27:33.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.453+0000 7fc3b2000700 10 client.16038 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:27:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.454+0000 7fc3b2000700 1 -- 172.21.15.12:0/1858286200 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b3519b570 con 0x556b35307d50 2024-05-27T07:27:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.454+0000 7fc3b2000700 10 client.16038 handle_command_reply: tid=0 2024-05-27T07:27:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.454+0000 7fc3dd9a2b80 1 client.16038 shutdown 2024-05-27T07:27:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.454+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b35282e70 msgr2=0x556b35283280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.454+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b35282e70 0x556b35283280 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00df760 tx=0x7fc3c00dfcc0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 shutdown_connections 2024-05-27T07:27:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556b35282e70 0x556b35283280 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b352717c0 0x556b35275a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556b35307d50 0x556b3530a170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 --2- 172.21.15.12:0/1858286200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b35271e70 0x556b35275f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3b2000700 0 client.16038 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 >> 172.21.15.12:0/1858286200 conn(0x556b3518c390 msgr2=0x556b352710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 shutdown_connections 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 1 -- 172.21.15.12:0/1858286200 wait complete. 2024-05-27T07:27:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.455+0000 7fc3dd9a2b80 20 client.16038 trim_cache size 0 max 16384 2024-05-27T07:27:33.447 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:27:33.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.457+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc3ac0b2a70 msgr2=0x7fc3ac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.457+0000 7fc3cee00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc3ac0b2a70 0x7fc3ac0b4f80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400b2e0 tx=0x7fc3b4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.457+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c810e900 msgr2=0x7fc3c8103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:33.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.457+0000 7fc3cee00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c810e900 0x7fc3c8103b70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000ef30 tx=0x7fc3c000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 shutdown_connections 2024-05-27T07:27:33.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc3ac0b2a70 0x7fc3ac0b4f80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c81054d0 0x7fc3c81030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3c810e900 0x7fc3c8103b70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 --2- 172.21.15.12:0/1314747027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3c8105e20 0x7fc3c8103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 >> 172.21.15.12:0/1314747027 conn(0x7fc3c80fedb0 msgr2=0x7fc3c80ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 shutdown_connections 2024-05-27T07:27:33.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:33.458+0000 7fc3cee00700 1 -- 172.21.15.12:0/1314747027 wait complete. 2024-05-27T07:27:33.526 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] xfsctl 2024-05-27T07:27:33.551 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] bstat 2024-05-27T07:27:33.638 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:27:33.639 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 systemd[1]: Stopped Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:33.639 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.087s CPU time. 2024-05-27T07:27:33.639 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 systemd[1]: Starting Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:33.666 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mtab 2024-05-27T07:27:33.669 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] getdevicesize 2024-05-27T07:27:33.917 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 podman[73307]: 2024-05-27 07:27:33.55195843 +0000 UTC m=+0.035184708 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:33.917 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 podman[73307]: 2024-05-27 07:27:33.657262425 +0000 UTC m=+0.140488709 container create a80899b79dfe90858db38c1bd0506e191caaa829a233c9f77e62d8b93d56ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:27:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:33 smithi162 ceph-mon[35676]: pgmap v176: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 182 KiB/s wr, 68 op/s 2024-05-27T07:27:34.223 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 podman[73307]: 2024-05-27 07:27:33.927735739 +0000 UTC m=+0.410962108 container init a80899b79dfe90858db38c1bd0506e191caaa829a233c9f77e62d8b93d56ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-05-27T07:27:34.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:33 smithi162 podman[73307]: 2024-05-27 07:27:33.939877671 +0000 UTC m=+0.423103943 container start a80899b79dfe90858db38c1bd0506e191caaa829a233c9f77e62d8b93d56ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:27:34.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:34 smithi162 ceph-mds[73325]: -- 172.21.15.162:0/3682488491 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d264afa40 con 0x558d264f7180 2024-05-27T07:27:34.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:34 smithi162 bash[73307]: a80899b79dfe90858db38c1bd0506e191caaa829a233c9f77e62d8b93d56ecf7 2024-05-27T07:27:34.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: starting mds.i at 2024-05-27T07:27:34.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:34 smithi162 systemd[1]: Started Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:34.235 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:33 smithi062 ceph-mon[35304]: pgmap v176: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 182 KiB/s wr, 68 op/s 2024-05-27T07:27:34.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:33 smithi012 ceph-mon[31473]: pgmap v176: 129 pgs: 129 active+clean; 83 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 182 KiB/s wr, 68 op/s 2024-05-27T07:27:34.380 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] preallo_rw_pattern_reader 2024-05-27T07:27:34.416 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] preallo_rw_pattern_writer 2024-05-27T07:27:34.465 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] ftrunc 2024-05-27T07:27:34.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:27:34.502 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] trunc 2024-05-27T07:27:34.964 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:35.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:35 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/2017379615,v1:172.21.15.12:6839/2017379615] up:boot 2024-05-27T07:27:35.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:35 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:35.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:35 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:27:35.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:35 smithi012 ceph-mon[31473]: daemon mds.i restarted 2024-05-27T07:27:35.369 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] testx 2024-05-27T07:27:35.369 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fs_perms 2024-05-27T07:27:35.400 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] looptest 2024-05-27T07:27:35.413 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] locktest 2024-05-27T07:27:35.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:35 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/2017379615,v1:172.21.15.12:6839/2017379615] up:boot 2024-05-27T07:27:35.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:35 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:35.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:35 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:27:35.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:35 smithi162 ceph-mon[35676]: daemon mds.i restarted 2024-05-27T07:27:35.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:35 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/2017379615,v1:172.21.15.12:6839/2017379615] up:boot 2024-05-27T07:27:35.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:35 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:35.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:35 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:27:35.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:35 smithi062 ceph-mon[35304]: daemon mds.i restarted 2024-05-27T07:27:35.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:36 smithi012 ceph-mon[31473]: pgmap v177: 129 pgs: 129 active+clean; 83 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 182 KiB/s wr, 71 op/s 2024-05-27T07:27:35.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:36 smithi012 ceph-mon[31473]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:27:35.997 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:36 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/880249625,v1:172.21.15.162:6833/880249625] up:boot 2024-05-27T07:27:35.998 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:36 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:35.998 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:27:35.998 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:36 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:36.155 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] unwritten_mmap 2024-05-27T07:27:36.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:36 smithi162 ceph-mon[35676]: pgmap v177: 129 pgs: 129 active+clean; 83 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 182 KiB/s wr, 71 op/s 2024-05-27T07:27:36.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:36 smithi162 ceph-mon[35676]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:27:36.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:36 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/880249625,v1:172.21.15.162:6833/880249625] up:boot 2024-05-27T07:27:36.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:36 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:36.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:27:36.224 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:36 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:36.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cc9c00700 time 2024-05-27T07:27:35.950632+0000 2024-05-27T07:27:36.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:36.224 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cdf6feba6] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.225 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: clone() 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: *** Caught signal (Aborted) ** 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: in thread 7f0cc9c00700 thread_name:md_log_replay 2024-05-27T07:27:36.226 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:35.952+0000 7f0cc9c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cc9c00700 time 2024-05-27T07:27:35.950632+0000 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cdf6feba6] 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.227 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: clone() 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.228 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cde27bd20] 2024-05-27T07:27:36.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: gsignal() 2024-05-27T07:27:36.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: abort() 2024-05-27T07:27:36.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cdf6feced] 2024-05-27T07:27:36.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.229 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.230 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 19: clone() 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:35.954+0000 7f0cc9c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: in thread 7f0cc9c00700 thread_name:md_log_replay 2024-05-27T07:27:36.231 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cde27bd20] 2024-05-27T07:27:36.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: gsignal() 2024-05-27T07:27:36.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: abort() 2024-05-27T07:27:36.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cdf6feced] 2024-05-27T07:27:36.232 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.233 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 19: clone() 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: -1> 2024-05-27T07:27:35.952+0000 7f0cc9c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cc9c00700 time 2024-05-27T07:27:35.950632+0000 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.234 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cdf6feba6] 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.235 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.236 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: clone() 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 0> 2024-05-27T07:27:35.954+0000 7f0cc9c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: in thread 7f0cc9c00700 thread_name:md_log_replay 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cde27bd20] 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: gsignal() 2024-05-27T07:27:36.237 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: abort() 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cdf6feced] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.238 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 19: clone() 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: -1> 2024-05-27T07:27:35.952+0000 7f0cc9c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cc9c00700 time 2024-05-27T07:27:35.950632+0000 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.239 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cdf6feba6] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.240 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: clone() 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 0> 2024-05-27T07:27:35.954+0000 7f0cc9c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: in thread 7f0cc9c00700 thread_name:md_log_replay 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cde27bd20] 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2: gsignal() 2024-05-27T07:27:36.241 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 3: abort() 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cdf6feced] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cdf6fedd1] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558d0ddf423d] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558d0dddf971] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 8: (SnapRealm::check_cache() const+0x13e) [0x558d0ddddfa0] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558d0ddde651] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558d0dd73347] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558d0da6f87e] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558d0da74e94] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558d0da7f969] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 14: (MDLog::_replay_thread()+0x1f38) [0x558d0de1848c] 2024-05-27T07:27:36.242 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558d0de219cb] 2024-05-27T07:27:36.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cdf6cd72d] 2024-05-27T07:27:36.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cdf6cd749] 2024-05-27T07:27:36.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cde2711ca] 2024-05-27T07:27:36.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 19: clone() 2024-05-27T07:27:36.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:36.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i[73321]: 2024-05-27T07:27:36.243 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:36 smithi162 podman[73356]: 2024-05-27 07:27:36.079656449 +0000 UTC m=+0.033950232 container died a80899b79dfe90858db38c1bd0506e191caaa829a233c9f77e62d8b93d56ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:27:36.243 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] bulkstat_unlink_test 2024-05-27T07:27:36.335 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] deduperace 2024-05-27T07:27:36.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:36 smithi062 ceph-mon[35304]: pgmap v177: 129 pgs: 129 active+clean; 83 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 182 KiB/s wr, 71 op/s 2024-05-27T07:27:36.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:36 smithi062 ceph-mon[35304]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:27:36.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:36 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/880249625,v1:172.21.15.162:6833/880249625] up:boot 2024-05-27T07:27:36.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:36 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:36.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:27:36.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:36 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:36.511 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] bulkstat_unlink_test_modified 2024-05-27T07:27:37.003 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_dir_offset 2024-05-27T07:27:37.082 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_futimens 2024-05-27T07:27:37.158 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_immutable 2024-05-27T07:27:37.218 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:36 smithi162 podman[73356]: 2024-05-27 07:27:36.791472371 +0000 UTC m=+0.745766174 container remove a80899b79dfe90858db38c1bd0506e191caaa829a233c9f77e62d8b93d56ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-i, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:27:37.218 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:36 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:37.301 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] stale_handle 2024-05-27T07:27:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.638+0000 7f671dc00700 1 -- 172.21.15.12:0/868101367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6718106eb0 msgr2=0x7f6718113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.638+0000 7f671dc00700 1 --2- 172.21.15.12:0/868101367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6718106eb0 0x7f6718113340 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f671000b0a0 tx=0x7f671000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.638+0000 7f671dc00700 1 -- 172.21.15.12:0/868101367 shutdown_connections 2024-05-27T07:27:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.638+0000 7f671dc00700 1 --2- 172.21.15.12:0/868101367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6718106eb0 0x7f6718113340 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.638+0000 7f671dc00700 1 --2- 172.21.15.12:0/868101367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f671810c5a0 0x7f671810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.638+0000 7f671dc00700 1 --2- 172.21.15.12:0/868101367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6718106560 0x7f6718106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.638+0000 7f671dc00700 1 -- 172.21.15.12:0/868101367 >> 172.21.15.12:0/868101367 conn(0x7f6718102040 msgr2=0x7f6718104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.639+0000 7f671dc00700 1 -- 172.21.15.12:0/868101367 shutdown_connections 2024-05-27T07:27:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.639+0000 7f671dc00700 1 -- 172.21.15.12:0/868101367 wait complete. 2024-05-27T07:27:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 Processor -- start 2024-05-27T07:27:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 -- start start 2024-05-27T07:27:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6718106560 0x7f671819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6718106eb0 0x7f671819d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f671810c5a0 0x7f67181a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f67181165d0 con 0x7f671810c5a0 2024-05-27T07:27:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6718116450 con 0x7f6718106eb0 2024-05-27T07:27:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.641+0000 7f671dc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6718116750 con 0x7f6718106560 2024-05-27T07:27:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.642+0000 7f6717400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f671810c5a0 0x7f67181a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.642+0000 7f6717400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f671810c5a0 0x7f67181a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34390/0 (socket says 172.21.15.12:34390) 2024-05-27T07:27:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.642+0000 7f6717400700 1 -- 172.21.15.12:0/1189556833 learned_addr learned my addr 172.21.15.12:0/1189556833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.643+0000 7f6716a00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6718106560 0x7f671819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.643+0000 7f6716000700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6718106eb0 0x7f671819d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.643+0000 7f6717400700 1 -- 172.21.15.12:0/1189556833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6718106560 msgr2=0x7f671819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.643+0000 7f6717400700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6718106560 0x7f671819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.643+0000 7f6717400700 1 -- 172.21.15.12:0/1189556833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6718106eb0 msgr2=0x7f671819d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.643+0000 7f6717400700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6718106eb0 0x7f671819d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.643+0000 7f6717400700 1 -- 172.21.15.12:0/1189556833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67181a1e10 con 0x7f671810c5a0 2024-05-27T07:27:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.644+0000 7f6717400700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f671810c5a0 0x7f67181a1770 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f671000b070 tx=0x7f6710004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.645+0000 7f670f400700 1 -- 172.21.15.12:0/1189556833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710021070 con 0x7f671810c5a0 2024-05-27T07:27:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.645+0000 7f670f400700 1 -- 172.21.15.12:0/1189556833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6710007b80 con 0x7f671810c5a0 2024-05-27T07:27:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.645+0000 7f670f400700 1 -- 172.21.15.12:0/1189556833 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100045f0 con 0x7f671810c5a0 2024-05-27T07:27:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.645+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67181a20a0 con 0x7f671810c5a0 2024-05-27T07:27:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.645+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67181a2530 con 0x7f671810c5a0 2024-05-27T07:27:37.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.650+0000 7f670f400700 1 -- 172.21.15.12:0/1189556833 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f671001d450 con 0x7f671810c5a0 2024-05-27T07:27:37.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.652+0000 7f670f400700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f66fc0b2b40 0x7f66fc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.652+0000 7f670f400700 1 -- 172.21.15.12:0/1189556833 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f671000f020 con 0x7f671810c5a0 2024-05-27T07:27:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.653+0000 7f6716a00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f66fc0b2b40 0x7f66fc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.653+0000 7f672c722b80 1 Processor -- start 2024-05-27T07:27:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.653+0000 7f672c722b80 1 -- start start 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.654+0000 7f6716a00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f66fc0b2b40 0x7f66fc0b5050 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67181079d0 tx=0x7f6708009260 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.654+0000 7f672c722b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc627f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.654+0000 7f672c722b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc62d30 0x56190dc5c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.654+0000 7f672c722b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc5cdb0 0x56190dc5d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.654+0000 7f672c722b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56190db8ae40 con 0x56190dadde40 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.654+0000 7f672c722b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56190db7a9e0 con 0x56190dc62d30 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.654+0000 7f672c722b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56190db7ea20 con 0x56190dc5cdb0 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.655+0000 7f6717400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc5cdb0 0x56190dc5d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.655+0000 7f6717400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc5cdb0 0x56190dc5d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:49312/0 (socket says 172.21.15.12:49312) 2024-05-27T07:27:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.655+0000 7f6717400700 1 -- 172.21.15.12:0/2857630570 learned_addr learned my addr 172.21.15.12:0/2857630570 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.655+0000 7f6716a00700 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.655+0000 7f6717400700 1 -- 172.21.15.12:0/2857630570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc62d30 msgr2=0x56190dc5c870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.655+0000 7f6717400700 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc62d30 0x56190dc5c870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.655+0000 7f6717400700 1 -- 172.21.15.12:0/2857630570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 msgr2=0x56190dc627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.656+0000 7f6717400700 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc627f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.656+0000 7f6717400700 1 -- 172.21.15.12:0/2857630570 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56190db43420 con 0x56190dc5cdb0 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.656+0000 7f6716a00700 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.656+0000 7f6717400700 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc5cdb0 0x56190dc5d1c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6710004fd0 tx=0x7f67100aa5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab48c6dbc0f5c71 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.657+0000 7f6703e00700 1 -- 172.21.15.12:0/2857630570 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100b33a0 con 0x56190dc5cdb0 2024-05-27T07:27:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.657+0000 7f6703e00700 1 -- 172.21.15.12:0/2857630570 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f67100aa790 con 0x56190dc5cdb0 2024-05-27T07:27:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.657+0000 7f6703e00700 1 -- 172.21.15.12:0/2857630570 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100a6fc0 con 0x56190dc5cdb0 2024-05-27T07:27:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 -- 172.21.15.12:0/2857630570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc5cdb0 msgr2=0x56190dc5d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc5cdb0 0x56190dc5d1c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6710004fd0 tx=0x7f67100aa5d0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 -- 172.21.15.12:0/2857630570 shutdown_connections 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc5cdb0 0x56190dc5d1c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 --2- 172.21.15.12:0/2857630570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc62d30 0x56190dc5c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 -- 172.21.15.12:0/2857630570 >> 172.21.15.12:0/2857630570 conn(0x56190db3dfc0 msgr2=0x56190dc578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 -- 172.21.15.12:0/2857630570 shutdown_connections 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.659+0000 7f672c722b80 1 -- 172.21.15.12:0/2857630570 wait complete. 2024-05-27T07:27:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.660+0000 7f672c722b80 1 -- start start 2024-05-27T07:27:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.660+0000 7f672c722b80 1 Processor -- start 2024-05-27T07:27:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f672c722b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc665c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f672c722b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc66b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f672c722b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc62d30 0x56190dc67040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f672c722b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56190da80fa0 con 0x56190dadde40 2024-05-27T07:27:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f672c722b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56190db1b760 con 0x56190dc5cdb0 2024-05-27T07:27:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f672c722b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56190dc71dd0 con 0x56190dc62d30 2024-05-27T07:27:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f672c722b80 10 client.0 _ll_register_callbacks cb 0x56190db24c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f6716a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f6716a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc665c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34408/0 (socket says 172.21.15.12:34408) 2024-05-27T07:27:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.661+0000 7f6716a00700 1 -- 172.21.15.12:0/3203965262 learned_addr learned my addr 172.21.15.12:0/3203965262 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.662+0000 7f6716000700 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc66b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.662+0000 7f6717400700 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc62d30 0x56190dc67040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.663+0000 7f6716a00700 1 -- 172.21.15.12:0/3203965262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc62d30 msgr2=0x56190dc67040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.663+0000 7f6716a00700 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc62d30 0x56190dc67040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.663+0000 7f6716a00700 1 -- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 msgr2=0x56190dc66b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.663+0000 7f6716a00700 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc66b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.663+0000 7f6716a00700 1 -- 172.21.15.12:0/3203965262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56190daca8d0 con 0x56190dadde40 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.663+0000 7f6716a00700 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc665c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f670801d920 tx=0x7f67080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f775fb64ea85a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.663+0000 7f6700c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.664+0000 7f6700c00700 1 -- 172.21.15.12:0/3203965262 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708004230 con 0x56190dadde40 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.664+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67100aa790 con 0x56190dadde40 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.664+0000 7f6700c00700 1 -- 172.21.15.12:0/3203965262 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f67080043d0 con 0x56190dadde40 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.664+0000 7f6700c00700 1 -- 172.21.15.12:0/3203965262 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f670800e6c0 con 0x56190dadde40 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.664+0000 7f672c722b80 10 client.16053 fetch_fsmap learned FSMap version 65 2024-05-27T07:27:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.664+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x56190db4b060 con 0x56190dadde40 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f6700c00700 1 -- 172.21.15.12:0/3203965262 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f67100aa790 con 0x56190dadde40 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f6700c00700 1 -- 172.21.15.12:0/3203965262 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 65) ==== 6547+0+0 (secure 0 0 0) 0x7f6708003b20 con 0x56190dadde40 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f672c722b80 10 client.16053 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f672c722b80 10 client.16053 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f672c722b80 20 client.16053 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f672c722b80 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dc7f010 0x56190dc814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f672c722b80 4 client.16053 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.665+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56190db8cd90 con 0x56190dc7f010 2024-05-27T07:27:37.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.666+0000 7f6716000700 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dc7f010 0x56190dc814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.667+0000 7f6716000700 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dc7f010 0x56190dc814c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d504541c4a4e50bb server_cookie=876d29be81ece306 in_seq=0 out_seq=0 2024-05-27T07:27:37.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.667+0000 7f6700c00700 10 client.16053 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:27:37.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.669+0000 7f6700c00700 1 -- 172.21.15.12:0/3203965262 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56190db8cd90 con 0x56190dc7f010 2024-05-27T07:27:37.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.669+0000 7f6700c00700 10 client.16053 handle_command_reply: tid=0 2024-05-27T07:27:37.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.669+0000 7f672c722b80 1 client.16053 shutdown 2024-05-27T07:27:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.669+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 msgr2=0x56190dc665c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.669+0000 7f672c722b80 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc665c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f670801d920 tx=0x7f67080047c0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 shutdown_connections 2024-05-27T07:27:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f672c722b80 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dc7f010 0x56190dc814c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f672c722b80 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dc62d30 0x56190dc67040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f672c722b80 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dadde40 0x56190dc665c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f672c722b80 1 --2- 172.21.15.12:0/3203965262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc66b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f6700c00700 0 client.16053 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:27:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 >> 172.21.15.12:0/3203965262 conn(0x56190db3dfc0 msgr2=0x56190dc57580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.670+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 shutdown_connections 2024-05-27T07:27:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.671+0000 7f672c722b80 1 -- 172.21.15.12:0/3203965262 wait complete. 2024-05-27T07:27:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.671+0000 7f672c722b80 20 client.16053 trim_cache size 0 max 16384 2024-05-27T07:27:37.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.694+0000 7f672c722b80 1 Processor -- start 2024-05-27T07:27:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.695+0000 7f672c722b80 1 -- start start 2024-05-27T07:27:37.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.695+0000 7f672c722b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 0x56190dc5c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.695+0000 7f672c722b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc61d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.695+0000 7f672c722b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc60c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.695+0000 7f672c722b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f67181afcb0 con 0x56190dc62d30 2024-05-27T07:27:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.695+0000 7f672c722b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f67181ae780 con 0x56190dc5cdb0 2024-05-27T07:27:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.695+0000 7f672c722b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f67181aee50 con 0x56190dadde40 2024-05-27T07:27:37.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.696+0000 7f6717400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc60c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.696+0000 7f6717400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc60c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34422/0 (socket says 172.21.15.12:34422) 2024-05-27T07:27:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.696+0000 7f6717400700 1 -- 172.21.15.12:0/3878284164 learned_addr learned my addr 172.21.15.12:0/3878284164 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.696+0000 7f6716a00700 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 0x56190dc5c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.696+0000 7f6716000700 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc61d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.697+0000 7f6717400700 1 -- 172.21.15.12:0/3878284164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 msgr2=0x56190dc5c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.697+0000 7f6717400700 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 0x56190dc5c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.697+0000 7f6717400700 1 -- 172.21.15.12:0/3878284164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 msgr2=0x56190dc61d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.697+0000 7f6717400700 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc61d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.697+0000 7f6717400700 1 -- 172.21.15.12:0/3878284164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56190dc673b0 con 0x56190dc62d30 2024-05-27T07:27:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.698+0000 7f6717400700 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc60c40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f67100aac10 tx=0x7f67100099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bbb8542b9a562d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:37.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.698+0000 7f6703400700 1 -- 172.21.15.12:0/3878284164 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100aac80 con 0x56190dc62d30 2024-05-27T07:27:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.698+0000 7f6703400700 1 -- 172.21.15.12:0/3878284164 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f67100af030 con 0x56190dc62d30 2024-05-27T07:27:37.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.698+0000 7f6703400700 1 -- 172.21.15.12:0/3878284164 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100c5270 con 0x56190dc62d30 2024-05-27T07:27:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 -- 172.21.15.12:0/3878284164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 msgr2=0x56190dc60c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc60c40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f67100aac10 tx=0x7f67100099e0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 -- 172.21.15.12:0/3878284164 shutdown_connections 2024-05-27T07:27:37.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc60c40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 0x56190dc5c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 --2- 172.21.15.12:0/3878284164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc61d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 -- 172.21.15.12:0/3878284164 >> 172.21.15.12:0/3878284164 conn(0x56190db3dfc0 msgr2=0x56190dc64d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 -- 172.21.15.12:0/3878284164 shutdown_connections 2024-05-27T07:27:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.700+0000 7f672c722b80 1 -- 172.21.15.12:0/3878284164 wait complete. 2024-05-27T07:27:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.701+0000 7f672c722b80 1 -- start start 2024-05-27T07:27:37.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.701+0000 7f672c722b80 1 Processor -- start 2024-05-27T07:27:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f672c722b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 0x56190dc76e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f672c722b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc77350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f672c722b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc85c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f672c722b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f67181ace70 con 0x56190dc62d30 2024-05-27T07:27:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f672c722b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f67181af580 con 0x56190dc5cdb0 2024-05-27T07:27:37.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f672c722b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f67181ac130 con 0x56190dadde40 2024-05-27T07:27:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f672c722b80 10 client.0 _ll_register_callbacks cb 0x56190db24c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f6717400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc85c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f6717400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc85c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34432/0 (socket says 172.21.15.12:34432) 2024-05-27T07:27:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f6717400700 1 -- 172.21.15.12:0/2702119634 learned_addr learned my addr 172.21.15.12:0/2702119634 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:37.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.702+0000 7f6716000700 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc77350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.703+0000 7f6717400700 1 -- 172.21.15.12:0/2702119634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 msgr2=0x56190dc76e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.703+0000 7f6717400700 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 0x56190dc76e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.703+0000 7f6717400700 1 -- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 msgr2=0x56190dc77350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.703+0000 7f6717400700 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc77350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.703+0000 7f6717400700 1 -- 172.21.15.12:0/2702119634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56190dc876e0 con 0x56190dc62d30 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.703+0000 7f6717400700 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc85c70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f67100af1f0 tx=0x7f67100e1110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26b7c5113b32d36 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.703+0000 7f6700c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.704+0000 7f6700c00700 1 -- 172.21.15.12:0/2702119634 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100aac80 con 0x56190dc62d30 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.704+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56190db86570 con 0x56190dc62d30 2024-05-27T07:27:37.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.704+0000 7f6700c00700 1 -- 172.21.15.12:0/2702119634 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f67100a7270 con 0x56190dc62d30 2024-05-27T07:27:37.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.704+0000 7f6700c00700 1 -- 172.21.15.12:0/2702119634 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f67100b3a50 con 0x56190dc62d30 2024-05-27T07:27:37.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.704+0000 7f672c722b80 10 client.16059 fetch_fsmap learned FSMap version 65 2024-05-27T07:27:37.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.704+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x56190dce8f10 con 0x56190dc62d30 2024-05-27T07:27:37.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f6700c00700 1 -- 172.21.15.12:0/2702119634 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56190db86570 con 0x56190dc62d30 2024-05-27T07:27:37.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f6700c00700 1 -- 172.21.15.12:0/2702119634 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 65) ==== 6547+0+0 (secure 0 0 0) 0x7f6710011070 con 0x56190dc62d30 2024-05-27T07:27:37.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f672c722b80 10 client.16059 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:27:37.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f672c722b80 10 client.16059 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:37.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f672c722b80 20 client.16059 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:37.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f672c722b80 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dce90b0 0x56190dceb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:37.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f672c722b80 4 client.16059 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:37.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.705+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56190d9fd250 con 0x56190dce90b0 2024-05-27T07:27:37.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.706+0000 7f6716a00700 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dce90b0 0x56190dceb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:37.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.707+0000 7f6716a00700 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dce90b0 0x56190dceb540 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce847e32fee3f811 server_cookie=6398f1b30de62526 in_seq=0 out_seq=0 2024-05-27T07:27:37.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.707+0000 7f6700c00700 10 client.16059 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:27:37.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.708+0000 7f6700c00700 1 -- 172.21.15.12:0/2702119634 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56190d9fd250 con 0x56190dce90b0 2024-05-27T07:27:37.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.708+0000 7f6700c00700 10 client.16059 handle_command_reply: tid=0 2024-05-27T07:27:37.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.708+0000 7f672c722b80 1 client.16059 shutdown 2024-05-27T07:27:37.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.708+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 msgr2=0x56190dc85c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.708+0000 7f672c722b80 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc85c70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f67100af1f0 tx=0x7f67100e1110 comp rx=0 tx=0).stop 2024-05-27T07:27:37.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 shutdown_connections 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56190dce90b0 0x56190dceb540 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56190dc62d30 0x56190dc85c70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190dadde40 0x56190dc76e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 --2- 172.21.15.12:0/2702119634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56190dc5cdb0 0x56190dc77350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f6700c00700 0 client.16059 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 >> 172.21.15.12:0/2702119634 conn(0x56190db3dfc0 msgr2=0x56190dc65510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 shutdown_connections 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 1 -- 172.21.15.12:0/2702119634 wait complete. 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.709+0000 7f672c722b80 20 client.16059 trim_cache size 0 max 16384 2024-05-27T07:27:37.700 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:27:37.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.710+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f66fc0b2b40 msgr2=0x7f66fc0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.710+0000 7f671dc00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f66fc0b2b40 0x7f66fc0b5050 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67181079d0 tx=0x7f6708009260 comp rx=0 tx=0).stop 2024-05-27T07:27:37.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.710+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f671810c5a0 msgr2=0x7f67181a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:37.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.710+0000 7f671dc00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f671810c5a0 0x7f67181a1770 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f671000b070 tx=0x7f6710004c60 comp rx=0 tx=0).stop 2024-05-27T07:27:37.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 shutdown_connections 2024-05-27T07:27:37.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f66fc0b2b40 0x7f66fc0b5050 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f671810c5a0 0x7f67181a1770 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6718106560 0x7f671819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 --2- 172.21.15.12:0/1189556833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6718106eb0 0x7f671819d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:37.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 >> 172.21.15.12:0/1189556833 conn(0x7f6718102040 msgr2=0x7f6718103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:37.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 shutdown_connections 2024-05-27T07:27:37.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:37.711+0000 7f671dc00700 1 -- 172.21.15.12:0/1189556833 wait complete. 2024-05-27T07:27:37.723 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:37 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-27T07:27:37.723 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:37 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.183s CPU time. 2024-05-27T07:27:37.790 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] pwrite_mmap_blocked 2024-05-27T07:27:37.828 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_dir_offset2 2024-05-27T07:27:38.122 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] seek_sanity_test 2024-05-27T07:27:38.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:38 smithi012 ceph-mon[31473]: pgmap v179: 129 pgs: 129 active+clean; 83 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 92 KiB/s wr, 38 op/s 2024-05-27T07:27:38.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:38 smithi162 ceph-mon[35676]: pgmap v179: 129 pgs: 129 active+clean; 83 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 92 KiB/s wr, 38 op/s 2024-05-27T07:27:38.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:38 smithi062 ceph-mon[35304]: pgmap v179: 129 pgs: 129 active+clean; 83 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 92 KiB/s wr, 38 op/s 2024-05-27T07:27:38.551 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] seek_copy_test 2024-05-27T07:27:38.613 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_readdir_1 2024-05-27T07:27:38.661 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_readdir_2 2024-05-27T07:27:38.727 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:27:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:39.289 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fsync-tester 2024-05-27T07:27:39.324 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] nsexec 2024-05-27T07:27:39.388 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] cloner 2024-05-27T07:27:39.431 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] renameat2 2024-05-27T07:27:40.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:40 smithi012 ceph-mon[31473]: pgmap v180: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 200 KiB/s wr, 81 op/s 2024-05-27T07:27:40.174 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_getcwd 2024-05-27T07:27:40.202 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] e4compact 2024-05-27T07:27:40.256 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] test-nextquota 2024-05-27T07:27:40.280 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] punch-alternating 2024-05-27T07:27:40.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:40 smithi162 ceph-mon[35676]: pgmap v180: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 200 KiB/s wr, 81 op/s 2024-05-27T07:27:40.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:40 smithi062 ceph-mon[35304]: pgmap v180: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 200 KiB/s wr, 81 op/s 2024-05-27T07:27:41.249 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] attr-list-by-handle-cursor-test 2024-05-27T07:27:41.270 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] listxattr 2024-05-27T07:27:41.285 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] dio-interleaved 2024-05-27T07:27:41.327 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_dir_type 2024-05-27T07:27:41.873 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:41 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:27:41.874 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:41 smithi162 systemd[1]: Stopped Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:41.874 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:41 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.607s CPU time. 2024-05-27T07:27:41.874 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:41 smithi162 systemd[1]: Starting Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.888+0000 7f7237a00700 1 -- 172.21.15.12:0/3576298581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 msgr2=0x7f7230075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 --2- 172.21.15.12:0/3576298581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f7230075ee0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7224009990 tx=0x7f7224009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 -- 172.21.15.12:0/3576298581 shutdown_connections 2024-05-27T07:27:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 --2- 172.21.15.12:0/3576298581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7230110c10 0x7f7230113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 --2- 172.21.15.12:0/3576298581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f7230075ee0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 --2- 172.21.15.12:0/3576298581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7230076420 0x7f72301106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 -- 172.21.15.12:0/3576298581 >> 172.21.15.12:0/3576298581 conn(0x7f7230100030 msgr2=0x7f7230102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 -- 172.21.15.12:0/3576298581 shutdown_connections 2024-05-27T07:27:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.889+0000 7f7237a00700 1 -- 172.21.15.12:0/3576298581 wait complete. 2024-05-27T07:27:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.890+0000 7f7237a00700 1 Processor -- start 2024-05-27T07:27:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.891+0000 7f7237a00700 1 -- start start 2024-05-27T07:27:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.891+0000 7f7237a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f723019d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.891+0000 7f7237a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7230076420 0x7f723019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.891+0000 7f7237a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7230110c10 0x7f72301a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.891+0000 7f7237a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7230116590 con 0x7f7230075ad0 2024-05-27T07:27:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.891+0000 7f7237a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7230116410 con 0x7f7230110c10 2024-05-27T07:27:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.891+0000 7f7237a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7230116710 con 0x7f7230076420 2024-05-27T07:27:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f723019d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f723019d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53950/0 (socket says 172.21.15.12:53950) 2024-05-27T07:27:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 -- 172.21.15.12:0/2781698222 learned_addr learned my addr 172.21.15.12:0/2781698222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:41.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7230110c10 0x7f72301a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 -- 172.21.15.12:0/2781698222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7230076420 msgr2=0x7f723019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f722fe00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7230076420 0x7f723019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7230076420 0x7f723019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 -- 172.21.15.12:0/2781698222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7230110c10 msgr2=0x7f72301a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7230110c10 0x7f72301a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235000700 1 -- 172.21.15.12:0/2781698222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72301a1ee0 con 0x7f7230075ad0 2024-05-27T07:27:41.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.892+0000 7f7235a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7230110c10 0x7f72301a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:41.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.893+0000 7f7235000700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f723019d060 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7224005910 tx=0x7f72240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.893+0000 7f722d600700 1 -- 172.21.15.12:0/2781698222 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224027070 con 0x7f7230075ad0 2024-05-27T07:27:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.893+0000 7f722d600700 1 -- 172.21.15.12:0/2781698222 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f722400fc00 con 0x7f7230075ad0 2024-05-27T07:27:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.893+0000 7f722d600700 1 -- 172.21.15.12:0/2781698222 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224022d80 con 0x7f7230075ad0 2024-05-27T07:27:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.894+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72301a2170 con 0x7f7230075ad0 2024-05-27T07:27:41.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.895+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72301a25d0 con 0x7f7230075ad0 2024-05-27T07:27:41.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.900+0000 7f722d600700 1 -- 172.21.15.12:0/2781698222 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7224022470 con 0x7f7230075ad0 2024-05-27T07:27:41.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.900+0000 7f724661fb80 1 Processor -- start 2024-05-27T07:27:41.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.900+0000 7f724661fb80 1 -- start start 2024-05-27T07:27:41.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.901+0000 7f724661fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.901+0000 7f724661fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.901+0000 7f724661fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8df110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.901+0000 7f724661fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56363e800e40 con 0x56363e753e40 2024-05-27T07:27:41.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.901+0000 7f724661fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56363e7f09e0 con 0x56363e83f9f0 2024-05-27T07:27:41.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.901+0000 7f724661fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56363e7f4a20 con 0x56363e8d2e70 2024-05-27T07:27:41.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8d2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53960/0 (socket says 172.21.15.12:53960) 2024-05-27T07:27:41.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235000700 1 -- 172.21.15.12:0/3818730796 learned_addr learned my addr 172.21.15.12:0/3818730796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:41.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f722d600700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72180b29f0 0x7f72180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235a00700 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f722fe00700 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8df110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235000700 1 -- 172.21.15.12:0/3818730796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 msgr2=0x56363e8df110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235000700 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8df110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235000700 1 -- 172.21.15.12:0/3818730796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 msgr2=0x56363e8d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.902+0000 7f7235000700 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.903+0000 7f7235000700 1 -- 172.21.15.12:0/3818730796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56363e7b9420 con 0x56363e753e40 2024-05-27T07:27:41.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.903+0000 7f722d600700 1 -- 172.21.15.12:0/2781698222 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f72240e3670 con 0x7f7230075ad0 2024-05-27T07:27:41.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.903+0000 7f7235a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72180b29f0 0x7f72180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.903+0000 7f7235000700 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8d2870 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f72240a78a0 tx=0x7f7224009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d315c891a9f1d791 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.903+0000 7f7222000700 1 -- 172.21.15.12:0/3818730796 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224065590 con 0x56363e753e40 2024-05-27T07:27:41.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.904+0000 7f7222000700 1 -- 172.21.15.12:0/3818730796 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72240aad30 con 0x56363e753e40 2024-05-27T07:27:41.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.904+0000 7f7222000700 1 -- 172.21.15.12:0/3818730796 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224009590 con 0x56363e753e40 2024-05-27T07:27:41.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 -- 172.21.15.12:0/3818730796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 msgr2=0x56363e8d2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f7235a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72180b29f0 0x7f72180b4f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x56363e8d4ff0 tx=0x7f7228004b40 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8d2870 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f72240a78a0 tx=0x7f7224009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 -- 172.21.15.12:0/3818730796 shutdown_connections 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8df110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 --2- 172.21.15.12:0/3818730796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8d2870 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 -- 172.21.15.12:0/3818730796 >> 172.21.15.12:0/3818730796 conn(0x56363e7ed390 msgr2=0x56363e8ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 -- 172.21.15.12:0/3818730796 shutdown_connections 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.905+0000 7f724661fb80 1 -- 172.21.15.12:0/3818730796 wait complete. 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.906+0000 7f724661fb80 1 -- start start 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.906+0000 7f724661fb80 1 Processor -- start 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.907+0000 7f724661fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e753e40 0x56363e8dc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.907+0000 7f724661fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8dcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.907+0000 7f724661fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8dd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.907+0000 7f724661fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56363e6f6fa0 con 0x56363e8d2e70 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.907+0000 7f724661fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56363e791760 con 0x56363e83f9f0 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.907+0000 7f724661fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56363e800e40 con 0x56363e753e40 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.907+0000 7f724661fb80 10 client.0 _ll_register_callbacks cb 0x56363e79ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.908+0000 7f7235000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e753e40 0x56363e8dc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.908+0000 7f7235000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e753e40 0x56363e8dc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35926/0 (socket says 172.21.15.12:35926) 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.908+0000 7f7235000700 1 -- 172.21.15.12:0/4148127706 learned_addr learned my addr 172.21.15.12:0/4148127706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.908+0000 7f7235a00700 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8dd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.908+0000 7f722fe00700 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8dcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.909+0000 7f7235a00700 1 -- 172.21.15.12:0/4148127706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e753e40 msgr2=0x56363e8dc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.909+0000 7f7235a00700 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e753e40 0x56363e8dc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.909+0000 7f7235a00700 1 -- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 msgr2=0x56363e8dcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.909+0000 7f7235a00700 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8dcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.909+0000 7f7235a00700 1 -- 172.21.15.12:0/4148127706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56363e7408d0 con 0x56363e8d2e70 2024-05-27T07:27:41.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.909+0000 7f7235a00700 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8dd1d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7228007ce0 tx=0x7f7228004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680cb892962204d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.909+0000 7f7216a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.910+0000 7f7216a00700 1 -- 172.21.15.12:0/4148127706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7228012040 con 0x56363e8d2e70 2024-05-27T07:27:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.910+0000 7f7216a00700 1 -- 172.21.15.12:0/4148127706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f7228004bb0 con 0x56363e8d2e70 2024-05-27T07:27:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.910+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72240aad30 con 0x56363e8d2e70 2024-05-27T07:27:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.910+0000 7f7216a00700 1 -- 172.21.15.12:0/4148127706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f722802e050 con 0x56363e8d2e70 2024-05-27T07:27:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.910+0000 7f724661fb80 10 client.16071 fetch_fsmap learned FSMap version 66 2024-05-27T07:27:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.910+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x56363e7c1060 con 0x56363e8d2e70 2024-05-27T07:27:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f7216a00700 1 -- 172.21.15.12:0/4148127706 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72240aad30 con 0x56363e8d2e70 2024-05-27T07:27:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f7216a00700 1 -- 172.21.15.12:0/4148127706 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 66) ==== 6551+0+0 (secure 0 0 0) 0x7f72280051e0 con 0x56363e8d2e70 2024-05-27T07:27:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f724661fb80 10 client.16071 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:27:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f724661fb80 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f724661fb80 20 client.16071 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f724661fb80 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e8ec8e0 0x56363e8eed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f724661fb80 4 client.16071 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.911+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56363e802d90 con 0x56363e8ec8e0 2024-05-27T07:27:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.912+0000 7f7235000700 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e8ec8e0 0x56363e8eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.913+0000 7f7235000700 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e8ec8e0 0x56363e8eed50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d93f04320b7128a server_cookie=82defe9570be27da in_seq=0 out_seq=0 2024-05-27T07:27:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.913+0000 7f7216a00700 10 client.16071 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f7216a00700 1 -- 172.21.15.12:0/4148127706 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56363e802d90 con 0x56363e8ec8e0 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f7216a00700 10 client.16071 handle_command_reply: tid=0 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 client.16071 shutdown 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 msgr2=0x56363e8dd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8dd1d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7228007ce0 tx=0x7f7228004ad0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 shutdown_connections 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8dd1d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e753e40 0x56363e8dc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e8ec8e0 0x56363e8eed50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 --2- 172.21.15.12:0/4148127706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8dcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f7216a00700 0 client.16071 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.916+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 >> 172.21.15.12:0/4148127706 conn(0x56363e7ed390 msgr2=0x56363e8d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.917+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 shutdown_connections 2024-05-27T07:27:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.917+0000 7f724661fb80 1 -- 172.21.15.12:0/4148127706 wait complete. 2024-05-27T07:27:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.917+0000 7f724661fb80 20 client.16071 trim_cache size 0 max 16384 2024-05-27T07:27:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.945+0000 7f724661fb80 1 Processor -- start 2024-05-27T07:27:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.945+0000 7f724661fb80 1 -- start start 2024-05-27T07:27:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.946+0000 7f724661fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e753e40 0x56363e8d7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.946+0000 7f724661fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e83f9f0 0x56363e8db6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.946+0000 7f724661fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8e8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.946+0000 7f724661fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72301b42b0 con 0x56363e8d2e70 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.946+0000 7f724661fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f72301b2d50 con 0x56363e753e40 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.946+0000 7f724661fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f72301b3450 con 0x56363e83f9f0 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.946+0000 7f7235a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f7235000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e753e40 0x56363e8d7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f722fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e83f9f0 0x56363e8db6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f722fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e83f9f0 0x56363e8db6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35930/0 (socket says 172.21.15.12:35930) 2024-05-27T07:27:41.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f7235000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e753e40 0x56363e8d7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:53066/0 (socket says 172.21.15.12:53066) 2024-05-27T07:27:41.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f722fe00700 1 -- 172.21.15.12:0/1791480452 learned_addr learned my addr 172.21.15.12:0/1791480452 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:41.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f7235a00700 1 -- 172.21.15.12:0/1791480452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e83f9f0 msgr2=0x56363e8db6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f7235a00700 1 --2- 172.21.15.12:0/1791480452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e83f9f0 0x56363e8db6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f7235a00700 1 -- 172.21.15.12:0/1791480452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e753e40 msgr2=0x56363e8d7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.947+0000 7f7235a00700 1 --2- 172.21.15.12:0/1791480452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e753e40 0x56363e8d7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.948+0000 7f7235a00700 1 -- 172.21.15.12:0/1791480452 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56363e8cf9c0 con 0x56363e8d2e70 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.948+0000 7f7235a00700 1 --2- 172.21.15.12:0/1791480452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8e8860 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f722800b010 tx=0x7f7228005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5565713718433d17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.948+0000 7f7221600700 1 -- 172.21.15.12:0/1791480452 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7228037070 con 0x56363e8d2e70 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.948+0000 7f7221600700 1 -- 172.21.15.12:0/1791480452 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7228017e00 con 0x56363e8d2e70 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.948+0000 7f7221600700 1 -- 172.21.15.12:0/1791480452 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7228005110 con 0x56363e8d2e70 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.950+0000 7f724661fb80 1 -- 172.21.15.12:0/1791480452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 msgr2=0x56363e8e8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.950+0000 7f724661fb80 1 --2- 172.21.15.12:0/1791480452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8e8860 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f722800b010 tx=0x7f7228005a30 comp rx=0 tx=0).stop 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.950+0000 7f724661fb80 1 -- 172.21.15.12:0/1791480452 shutdown_connections 2024-05-27T07:27:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.950+0000 7f724661fb80 1 --2- 172.21.15.12:0/1791480452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e753e40 0x56363e8d7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.950+0000 7f724661fb80 1 --2- 172.21.15.12:0/1791480452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e8d2e70 0x56363e8e8860 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.950+0000 7f724661fb80 1 --2- 172.21.15.12:0/1791480452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e83f9f0 0x56363e8db6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.950+0000 7f724661fb80 1 -- 172.21.15.12:0/1791480452 >> 172.21.15.12:0/1791480452 conn(0x56363e7ed390 msgr2=0x56363e8dc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.952+0000 7f724661fb80 1 -- 172.21.15.12:0/1791480452 shutdown_connections 2024-05-27T07:27:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.952+0000 7f724661fb80 1 -- 172.21.15.12:0/1791480452 wait complete. 2024-05-27T07:27:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.952+0000 7f724661fb80 1 -- start start 2024-05-27T07:27:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.953+0000 7f724661fb80 1 Processor -- start 2024-05-27T07:27:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.953+0000 7f724661fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8e5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.953+0000 7f724661fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.953+0000 7f724661fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8d7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.953+0000 7f724661fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72301b1440 con 0x56363e753e40 2024-05-27T07:27:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.953+0000 7f724661fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f72301b3b80 con 0x56363e83f9f0 2024-05-27T07:27:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.953+0000 7f724661fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f72301b0700 con 0x56363e8d2e70 2024-05-27T07:27:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.954+0000 7f724661fb80 10 client.0 _ll_register_callbacks cb 0x56363e79ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.954+0000 7f722fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.954+0000 7f722fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:53068/0 (socket says 172.21.15.12:53068) 2024-05-27T07:27:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.954+0000 7f7235000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8e5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.954+0000 7f722fe00700 1 -- 172.21.15.12:0/1729763564 learned_addr learned my addr 172.21.15.12:0/1729763564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.955+0000 7f7235a00700 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8d7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f7235000700 1 -- 172.21.15.12:0/1729763564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 msgr2=0x56363e8d7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f7235000700 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8d7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f7235000700 1 -- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 msgr2=0x56363e8d7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f7235000700 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f7235000700 1 -- 172.21.15.12:0/1729763564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56363e8ca240 con 0x56363e753e40 2024-05-27T07:27:41.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f722fe00700 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:41.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f7235000700 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8e5280 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f72240aa330 tx=0x7f72240b1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2df4d4879539fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.956+0000 7f7216a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.957+0000 7f7216a00700 1 -- 172.21.15.12:0/1729763564 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72240aad30 con 0x56363e753e40 2024-05-27T07:27:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.957+0000 7f7216a00700 1 -- 172.21.15.12:0/1729763564 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9466+0+0 (secure 0 0 0) 0x7f72240e3260 con 0x56363e753e40 2024-05-27T07:27:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.957+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7228017e00 con 0x56363e753e40 2024-05-27T07:27:41.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.958+0000 7f7216a00700 1 -- 172.21.15.12:0/1729763564 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f72240aa630 con 0x56363e753e40 2024-05-27T07:27:41.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.958+0000 7f724661fb80 10 client.16077 fetch_fsmap learned FSMap version 66 2024-05-27T07:27:41.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.958+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x56363e8e3ac0 con 0x56363e753e40 2024-05-27T07:27:41.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.959+0000 7f7216a00700 1 -- 172.21.15.12:0/1729763564 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7228017e00 con 0x56363e753e40 2024-05-27T07:27:41.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.960+0000 7f7216a00700 1 -- 172.21.15.12:0/1729763564 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 66) ==== 6551+0+0 (secure 0 0 0) 0x7f72240b2f70 con 0x56363e753e40 2024-05-27T07:27:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.960+0000 7f724661fb80 10 client.16077 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:27:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.960+0000 7f724661fb80 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.960+0000 7f724661fb80 20 client.16077 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.960+0000 7f724661fb80 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e9516d0 0x56363e953af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.960+0000 7f724661fb80 4 client.16077 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.960+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56363e7fc570 con 0x56363e9516d0 2024-05-27T07:27:41.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.961+0000 7f722fe00700 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e9516d0 0x56363e953af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.962+0000 7f722fe00700 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e9516d0 0x56363e953af0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f15d9732aaad4c server_cookie=93facfbc9ecfe552 in_seq=0 out_seq=0 2024-05-27T07:27:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.962+0000 7f7216a00700 10 client.16077 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.963+0000 7f7216a00700 1 -- 172.21.15.12:0/1729763564 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56363e7fc570 con 0x56363e9516d0 2024-05-27T07:27:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.963+0000 7f7216a00700 10 client.16077 handle_command_reply: tid=0 2024-05-27T07:27:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.963+0000 7f724661fb80 1 client.16077 shutdown 2024-05-27T07:27:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 msgr2=0x56363e8e5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8e5280 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f72240aa330 tx=0x7f72240b1040 comp rx=0 tx=0).stop 2024-05-27T07:27:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 shutdown_connections 2024-05-27T07:27:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56363e8d2e70 0x56363e8d7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56363e753e40 0x56363e8e5280 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56363e9516d0 0x56363e953af0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 --2- 172.21.15.12:0/1729763564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56363e83f9f0 0x56363e8d7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.964+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 >> 172.21.15.12:0/1729763564 conn(0x56363e7ed390 msgr2=0x56363e8dc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.965+0000 7f7216a00700 0 client.16077 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.965+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 shutdown_connections 2024-05-27T07:27:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.965+0000 7f724661fb80 1 -- 172.21.15.12:0/1729763564 wait complete. 2024-05-27T07:27:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.966+0000 7f724661fb80 20 client.16077 trim_cache size 0 max 16384 2024-05-27T07:27:41.956 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:27:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.967+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72180b29f0 msgr2=0x7f72180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.967+0000 7f7237a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72180b29f0 0x7f72180b4f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x56363e8d4ff0 tx=0x7f7228004b40 comp rx=0 tx=0).stop 2024-05-27T07:27:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.967+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 msgr2=0x7f723019d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.967+0000 7f7237a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f723019d060 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7224005910 tx=0x7f72240047b0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 shutdown_connections 2024-05-27T07:27:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72180b29f0 0x7f72180b4f00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7230110c10 0x7f72301a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7230075ad0 0x7f723019d060 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 --2- 172.21.15.12:0/2781698222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7230076420 0x7f723019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 >> 172.21.15.12:0/2781698222 conn(0x7f7230100030 msgr2=0x7f7230101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 shutdown_connections 2024-05-27T07:27:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:41.968+0000 7f7237a00700 1 -- 172.21.15.12:0/2781698222 wait complete. 2024-05-27T07:27:42.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:42 smithi162 ceph-mon[35676]: pgmap v181: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 171 KiB/s wr, 69 op/s 2024-05-27T07:27:42.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:42 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:42.144 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:41 smithi162 podman[73447]: 2024-05-27 07:27:41.787803561 +0000 UTC m=+0.033178412 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:42.144 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:41 smithi162 podman[73447]: 2024-05-27 07:27:41.891996377 +0000 UTC m=+0.137371294 container create f663c50b6eea84f4ab7d0fd0ddf43ca020b8ddcd5daf2f767a370280f2ae75cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:27:42.253 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] dio-invalidate-cache 2024-05-27T07:27:42.291 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] stat_test 2024-05-27T07:27:42.297 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_encrypted_d_revalidate 2024-05-27T07:27:42.337 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] attr_replace_test 2024-05-27T07:27:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:42 smithi012 ceph-mon[31473]: pgmap v181: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 171 KiB/s wr, 69 op/s 2024-05-27T07:27:42.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:42 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:42.473 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:42 smithi162 podman[73447]: 2024-05-27 07:27:42.154408506 +0000 UTC m=+0.399783459 container init f663c50b6eea84f4ab7d0fd0ddf43ca020b8ddcd5daf2f767a370280f2ae75cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:27:42.474 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:42 smithi162 podman[73447]: 2024-05-27 07:27:42.164039037 +0000 UTC m=+0.409413931 container start f663c50b6eea84f4ab7d0fd0ddf43ca020b8ddcd5daf2f767a370280f2ae75cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:27:42.474 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:42 smithi162 bash[73447]: f663c50b6eea84f4ab7d0fd0ddf43ca020b8ddcd5daf2f767a370280f2ae75cc 2024-05-27T07:27:42.474 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:42 smithi162 systemd[1]: Started Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:42.474 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:42 smithi162 ceph-mds[73465]: -- 172.21.15.162:0/1381179248 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a42cf6ba40 con 0x55a42cfb3a80 2024-05-27T07:27:42.474 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: starting mds.l at 2024-05-27T07:27:42.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:42 smithi062 ceph-mon[35304]: pgmap v181: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 171 KiB/s wr, 69 op/s 2024-05-27T07:27:42.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:42 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:43.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:43 smithi162 ceph-mon[35676]: daemon mds.l restarted 2024-05-27T07:27:43.311 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ef8a00700 time 2024-05-27T07:27:43.164998+0000 2024-05-27T07:27:43.311 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:43.311 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f0e5d8ba6] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.312 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.313 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.313 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.313 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.313 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: clone() 2024-05-27T07:27:43.313 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.166+0000 7f9ef8a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ef8a00700 time 2024-05-27T07:27:43.164998+0000 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f0e5d8ba6] 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.314 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.315 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: clone() 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: *** Caught signal (Aborted) ** 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: in thread 7f9ef8a00700 thread_name:md_log_replay 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f0d155d20] 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: gsignal() 2024-05-27T07:27:43.316 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: abort() 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f0e5d8ced] 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.317 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.318 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.318 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.318 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.318 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.318 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.318 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.318 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.319 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 19: clone() 2024-05-27T07:27:43.319 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.169+0000 7f9ef8a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:43.319 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: in thread 7f9ef8a00700 thread_name:md_log_replay 2024-05-27T07:27:43.319 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.319 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.319 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f0d155d20] 2024-05-27T07:27:43.320 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: gsignal() 2024-05-27T07:27:43.320 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: abort() 2024-05-27T07:27:43.320 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f0e5d8ced] 2024-05-27T07:27:43.320 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.320 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.320 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.320 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.321 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.321 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.322 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.322 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 19: clone() 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.323 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: -1> 2024-05-27T07:27:43.166+0000 7f9ef8a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ef8a00700 time 2024-05-27T07:27:43.164998+0000 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f0e5d8ba6] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.324 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: clone() 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 0> 2024-05-27T07:27:43.169+0000 7f9ef8a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: in thread 7f9ef8a00700 thread_name:md_log_replay 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f0d155d20] 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: gsignal() 2024-05-27T07:27:43.325 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: abort() 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f0e5d8ced] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.326 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 19: clone() 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: -1> 2024-05-27T07:27:43.166+0000 7f9ef8a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ef8a00700 time 2024-05-27T07:27:43.164998+0000 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9f0e5d8ba6] 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.327 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: clone() 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 0> 2024-05-27T07:27:43.169+0000 7f9ef8a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:43.328 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: in thread 7f9ef8a00700 thread_name:md_log_replay 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9f0d155d20] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2: gsignal() 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 3: abort() 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9f0e5d8ced] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9f0e5d8dd1] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a41dae323d] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a41dace971] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a41daccfa0] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a41dacd651] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a41da62347] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a41d75e87e] 2024-05-27T07:27:43.329 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a41d763e94] 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a41d76e969] 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a41db0748c] 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a41db109cb] 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 16: (Thread::entry_wrapper()+0x43) [0x7f9f0e5a772d] 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9f0e5a7749] 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9f0d14b1ca] 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 19: clone() 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73461]: 2024-05-27T07:27:43.330 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:43 smithi162 podman[73496]: 2024-05-27 07:27:43.321731204 +0000 UTC m=+0.035403621 container died f663c50b6eea84f4ab7d0fd0ddf43ca020b8ddcd5daf2f767a370280f2ae75cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:27:43.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:43 smithi012 ceph-mon[31473]: daemon mds.l restarted 2024-05-27T07:27:43.365 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_attr_corruption 2024-05-27T07:27:43.365 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] mkswap 2024-05-27T07:27:43.365 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] swapon 2024-05-27T07:27:43.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:43 smithi062 ceph-mon[35304]: daemon mds.l restarted 2024-05-27T07:27:43.562 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_open_tmpfiles 2024-05-27T07:27:43.782 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:43 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:27:43.782 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:43 smithi012 systemd[1]: Stopped Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:43.782 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:43 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.936s CPU time. 2024-05-27T07:27:43.783 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:43 smithi012 systemd[1]: Starting Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:44.027 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:27:44.027 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:27:44.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:44 smithi012 ceph-mon[31473]: pgmap v182: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 171 KiB/s wr, 69 op/s 2024-05-27T07:27:44.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:44 smithi012 ceph-mon[31473]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:27:44.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:44 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/296525582,v1:172.21.15.162:6833/296525582] up:boot 2024-05-27T07:27:44.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:44 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:44.035 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:43 smithi012 podman[116042]: 2024-05-27 07:27:43.701712384 +0000 UTC m=+0.044506231 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:44.036 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:43 smithi012 podman[116042]: 2024-05-27 07:27:43.944164785 +0000 UTC m=+0.286958714 container create a371ccda58649f030e4b28b49ec04dc90ce871e1ee55ef25c2db827c95aa2790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:27:44.341 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:44 smithi012 podman[116042]: 2024-05-27 07:27:44.28171775 +0000 UTC m=+0.624511604 container init a371ccda58649f030e4b28b49ec04dc90ce871e1ee55ef25c2db827c95aa2790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:27:44.341 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:44 smithi012 podman[116042]: 2024-05-27 07:27:44.293793624 +0000 UTC m=+0.636587461 container start a371ccda58649f030e4b28b49ec04dc90ce871e1ee55ef25c2db827c95aa2790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:27:44.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:44 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:44.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:44 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:44.355 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] bulkstat_null_ocount 2024-05-27T07:27:44.356 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] splice-test 2024-05-27T07:27:44.356 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fscrypt-crypt-util 2024-05-27T07:27:44.387 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:44 smithi062 ceph-mon[35304]: pgmap v182: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 171 KiB/s wr, 69 op/s 2024-05-27T07:27:44.387 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:44 smithi062 ceph-mon[35304]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:27:44.387 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:44 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/296525582,v1:172.21.15.162:6833/296525582] up:boot 2024-05-27T07:27:44.387 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:44 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:44.388 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:44 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:44.388 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:44 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:44.473 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:44 smithi162 podman[73496]: 2024-05-27 07:27:44.026383329 +0000 UTC m=+0.740055763 container remove f663c50b6eea84f4ab7d0fd0ddf43ca020b8ddcd5daf2f767a370280f2ae75cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:27:44.474 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:44 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:44 smithi162 ceph-mon[35676]: pgmap v182: 129 pgs: 129 active+clean; 84 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 171 KiB/s wr, 69 op/s 2024-05-27T07:27:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:44 smithi162 ceph-mon[35676]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:27:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:44 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/296525582,v1:172.21.15.162:6833/296525582] up:boot 2024-05-27T07:27:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:44 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:44 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:44.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:44 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:44.581 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] chprojid_fail 2024-05-27T07:27:44.612 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:44.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:44 smithi012 bash[116042]: a371ccda58649f030e4b28b49ec04dc90ce871e1ee55ef25c2db827c95aa2790 2024-05-27T07:27:44.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:44 smithi012 systemd[1]: Started Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:44.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: starting mds.g at 2024-05-27T07:27:44.973 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:44 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:27:44.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:44 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.667s CPU time. 2024-05-27T07:27:45.298 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:45 smithi012 ceph-mon[31473]: daemon mds.g restarted 2024-05-27T07:27:45.298 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2655400700 time 2024-05-27T07:27:45.259280+0000 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f266b006ba6] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.299 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: clone() 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.260+0000 7f2655400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2655400700 time 2024-05-27T07:27:45.259280+0000 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f266b006ba6] 2024-05-27T07:27:45.300 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: clone() 2024-05-27T07:27:45.301 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: *** Caught signal (Aborted) ** 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: in thread 7f2655400700 thread_name:md_log_replay 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2669b83d20] 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: gsignal() 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: abort() 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f266b006ced] 2024-05-27T07:27:45.302 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 17: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.303 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 19: clone() 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.264+0000 7f2655400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: in thread 7f2655400700 thread_name:md_log_replay 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2669b83d20] 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: gsignal() 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: abort() 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f266b006ced] 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.304 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 17: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 19: clone() 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: -1> 2024-05-27T07:27:45.260+0000 7f2655400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2655400700 time 2024-05-27T07:27:45.259280+0000 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:45.305 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.306 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.306 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f266b006ba6] 2024-05-27T07:27:45.306 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.306 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.432 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] detached_mounts_propagation 2024-05-27T07:27:45.432 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] ext4_resize 2024-05-27T07:27:45.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:45 smithi162 ceph-mon[35676]: daemon mds.g restarted 2024-05-27T07:27:45.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:45 smithi062 ceph-mon[35304]: daemon mds.g restarted 2024-05-27T07:27:45.511 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_readdir_3 2024-05-27T07:27:45.587 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.587 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.588 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: clone() 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 0> 2024-05-27T07:27:45.264+0000 7f2655400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: in thread 7f2655400700 thread_name:md_log_replay 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2669b83d20] 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: gsignal() 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: abort() 2024-05-27T07:27:45.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f266b006ced] 2024-05-27T07:27:45.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.591 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 17: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 19: clone() 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: -1> 2024-05-27T07:27:45.260+0000 7f2655400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f2655400700 time 2024-05-27T07:27:45.259280+0000 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f266b006ba6] 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.592 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: clone() 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 0> 2024-05-27T07:27:45.264+0000 7f2655400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:45.593 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: in thread 7f2655400700 thread_name:md_log_replay 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2669b83d20] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2: gsignal() 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 3: abort() 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f266b006ced] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f266b006dd1] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555a64b9d23d] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555a64b88971] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 8: (SnapRealm::check_cache() const+0x13e) [0x555a64b86fa0] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555a64b87651] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555a64b1c347] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555a6481887e] 2024-05-27T07:27:45.594 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555a6481de94] 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555a64828969] 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 14: (MDLog::_replay_thread()+0x1f38) [0x555a64bc148c] 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555a64bca9cb] 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 16: (Thread::entry_wrapper()+0x43) [0x7f266afd572d] 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 17: (Thread::_entry_func(void*)+0xd) [0x7f266afd5749] 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2669b791ca] 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 19: clone() 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:45.595 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[116192]: 2024-05-27T07:27:46.022 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] splice2pipe 2024-05-27T07:27:46.246 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:46 smithi012 ceph-mon[31473]: pgmap v184: 129 pgs: 129 active+clean; 85 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 227 KiB/s wr, 78 op/s 2024-05-27T07:27:46.246 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:46 smithi012 ceph-mon[31473]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:27:46.247 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:46 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6840/827104269,v1:172.21.15.12:6841/827104269] up:boot 2024-05-27T07:27:46.247 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:46 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:46.247 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:46 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:27:46.247 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:46 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:46.247 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:45 smithi012 podman[116683]: 2024-05-27 07:27:45.870703464 +0000 UTC m=+0.489347296 container died a371ccda58649f030e4b28b49ec04dc90ce871e1ee55ef25c2db827c95aa2790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:27:46.283 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] uuid_ioctl 2024-05-27T07:27:46.431 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_snapshot_deleted_subvolume 2024-05-27T07:27:46.472 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fiemap-fault 2024-05-27T07:27:46.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:46 smithi162 ceph-mon[35676]: pgmap v184: 129 pgs: 129 active+clean; 85 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 227 KiB/s wr, 78 op/s 2024-05-27T07:27:46.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:46 smithi162 ceph-mon[35676]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:27:46.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:46 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6840/827104269,v1:172.21.15.12:6841/827104269] up:boot 2024-05-27T07:27:46.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:46 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:46.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:46 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:27:46.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:46 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:46.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:46 smithi062 ceph-mon[35304]: pgmap v184: 129 pgs: 129 active+clean; 85 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 227 KiB/s wr, 78 op/s 2024-05-27T07:27:46.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:46 smithi062 ceph-mon[35304]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:27:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:46 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6840/827104269,v1:172.21.15.12:6841/827104269] up:boot 2024-05-27T07:27:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:46 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 4 up:standby 2024-05-27T07:27:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:46 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:27:46.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:46 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:47.042 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fiemap-tester 2024-05-27T07:27:47.268 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_stripealign 2024-05-27T07:27:47.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:47 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:47.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:46 smithi012 podman[116683]: 2024-05-27 07:27:46.857382095 +0000 UTC m=+1.476025894 container remove a371ccda58649f030e4b28b49ec04dc90ce871e1ee55ef25c2db827c95aa2790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-27T07:27:47.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:46 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:47.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:47 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:47.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:47 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 3 up:standby 2024-05-27T07:27:47.973 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:47 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:27:47.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:47 smithi162 systemd[1]: Stopped Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:47.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:47 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Consumed 3.183s CPU time. 2024-05-27T07:27:47.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:47 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Start request repeated too quickly. 2024-05-27T07:27:47.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:47 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-27T07:27:47.974 INFO:journalctl@ceph.mds.i.smithi162.stdout:May 27 07:27:47 smithi162 systemd[1]: Failed to start Ceph mds.i for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:48.258 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:48 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:27:48.258 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:48 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.711s CPU time. 2024-05-27T07:27:48.259 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:48 smithi012 ceph-mon[31473]: pgmap v186: 129 pgs: 129 active+clean; 85 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 147 KiB/s wr, 45 op/s 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 -- 172.21.15.12:0/848880957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2270108f00 msgr2=0x7f2270109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 --2- 172.21.15.12:0/848880957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2270108f00 0x7f2270109310 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f225c0099c0 tx=0x7f225c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 -- 172.21.15.12:0/848880957 shutdown_connections 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 --2- 172.21.15.12:0/848880957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2270109850 0x7f22701111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 --2- 172.21.15.12:0/848880957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22701045d0 0x7f22701049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 --2- 172.21.15.12:0/848880957 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2270108f00 0x7f2270109310 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 -- 172.21.15.12:0/848880957 >> 172.21.15.12:0/848880957 conn(0x7f22700ffe60 msgr2=0x7f22701022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 -- 172.21.15.12:0/848880957 shutdown_connections 2024-05-27T07:27:48.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.439+0000 7f2275400700 1 -- 172.21.15.12:0/848880957 wait complete. 2024-05-27T07:27:48.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.441+0000 7f2275400700 1 Processor -- start 2024-05-27T07:27:48.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.441+0000 7f2275400700 1 -- start start 2024-05-27T07:27:48.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f2275400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22701045d0 0x7f2270075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f2275400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2270108f00 0x7f2270075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f2275400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2270109850 0x7f2270076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f2275400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2270114360 con 0x7f22701045d0 2024-05-27T07:27:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f2275400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f22701141e0 con 0x7f2270108f00 2024-05-27T07:27:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f2275400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22701144e0 con 0x7f2270109850 2024-05-27T07:27:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f226e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22701045d0 0x7f2270075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f226e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22701045d0 0x7f2270075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54036/0 (socket says 172.21.15.12:54036) 2024-05-27T07:27:48.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.442+0000 7f226e800700 1 -- 172.21.15.12:0/994628378 learned_addr learned my addr 172.21.15.12:0/994628378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:48.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226de00700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2270108f00 0x7f2270075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226f200700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2270109850 0x7f2270076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226e800700 1 -- 172.21.15.12:0/994628378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2270109850 msgr2=0x7f2270076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226e800700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2270109850 0x7f2270076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226e800700 1 -- 172.21.15.12:0/994628378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2270108f00 msgr2=0x7f2270075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226e800700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2270108f00 0x7f2270075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226e800700 1 -- 172.21.15.12:0/994628378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22701a9110 con 0x7f22701045d0 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.443+0000 7f226f200700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2270109850 0x7f2270076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.444+0000 7f226e800700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22701045d0 0x7f2270075ab0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f226000d980 tx=0x7f226000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.444+0000 7f2267400700 1 -- 172.21.15.12:0/994628378 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260016070 con 0x7f22701045d0 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.444+0000 7f2267400700 1 -- 172.21.15.12:0/994628378 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2260004830 con 0x7f22701045d0 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.444+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22701a9400 con 0x7f22701045d0 2024-05-27T07:27:48.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.444+0000 7f2267400700 1 -- 172.21.15.12:0/994628378 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260005440 con 0x7f22701045d0 2024-05-27T07:27:48.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.444+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22701a9960 con 0x7f22701045d0 2024-05-27T07:27:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.449+0000 7f228402bb80 1 Processor -- start 2024-05-27T07:27:48.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.449+0000 7f228402bb80 1 -- start start 2024-05-27T07:27:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f228402bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f228402bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfccb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f228402bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfccc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f228402bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a3cfbf9ec0 con 0x55a3cfcd1db0 2024-05-27T07:27:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f228402bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a3cfbe9a60 con 0x55a3cfb4cec0 2024-05-27T07:27:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f228402bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3cfbedaa0 con 0x55a3cfccbe30 2024-05-27T07:27:48.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f2267400700 1 -- 172.21.15.12:0/994628378 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f22600055e0 con 0x7f22701045d0 2024-05-27T07:27:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f226e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfccb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.450+0000 7f226f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfccb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54052/0 (socket says 172.21.15.12:54052) 2024-05-27T07:27:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226de00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfccc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226e800700 1 -- 172.21.15.12:0/543707893 learned_addr learned my addr 172.21.15.12:0/543707893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:53114/0 (socket says 172.21.15.12:53114) 2024-05-27T07:27:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226e800700 1 -- 172.21.15.12:0/543707893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 msgr2=0x55a3cfccc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226e800700 1 --2- 172.21.15.12:0/543707893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfccc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226e800700 1 -- 172.21.15.12:0/543707893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 msgr2=0x55a3cfcd1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226e800700 1 --2- 172.21.15.12:0/543707893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.451+0000 7f226e800700 1 -- 172.21.15.12:0/543707893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3cfbb24a0 con 0x55a3cfcd1db0 2024-05-27T07:27:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.452+0000 7f226e800700 1 --2- 172.21.15.12:0/543707893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfccb8f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f22600dfa20 tx=0x7f22600e8ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f968f9aed029854 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.452+0000 7f225be00700 1 -- 172.21.15.12:0/543707893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600f1070 con 0x55a3cfcd1db0 2024-05-27T07:27:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.452+0000 7f2267400700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f22540b2a70 0x7f22540b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.452+0000 7f225be00700 1 -- 172.21.15.12:0/543707893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22600a5c60 con 0x55a3cfcd1db0 2024-05-27T07:27:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.452+0000 7f225be00700 1 -- 172.21.15.12:0/543707893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600a8de0 con 0x55a3cfcd1db0 2024-05-27T07:27:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.452+0000 7f2267400700 1 -- 172.21.15.12:0/994628378 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9466+0+0 (secure 0 0 0) 0x7f22600e27a0 con 0x7f22701045d0 2024-05-27T07:27:48.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.452+0000 7f226f200700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f22540b2a70 0x7f22540b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 -- 172.21.15.12:0/543707893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 msgr2=0x55a3cfccb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f226f200700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f22540b2a70 0x7f22540b4f80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55a3cfcccbb0 tx=0x7f2268009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 --2- 172.21.15.12:0/543707893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfccb8f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f22600dfa20 tx=0x7f22600e8ed0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 -- 172.21.15.12:0/543707893 shutdown_connections 2024-05-27T07:27:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 --2- 172.21.15.12:0/543707893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 --2- 172.21.15.12:0/543707893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfccc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 --2- 172.21.15.12:0/543707893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfccb8f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 -- 172.21.15.12:0/543707893 >> 172.21.15.12:0/543707893 conn(0x55a3cfbad040 msgr2=0x55a3cfcc6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 -- 172.21.15.12:0/543707893 shutdown_connections 2024-05-27T07:27:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.454+0000 7f228402bb80 1 -- 172.21.15.12:0/543707893 wait complete. 2024-05-27T07:27:48.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.455+0000 7f228402bb80 1 -- start start 2024-05-27T07:27:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.455+0000 7f228402bb80 1 Processor -- start 2024-05-27T07:27:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.456+0000 7f228402bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.456+0000 7f228402bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.456+0000 7f228402bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.456+0000 7f228402bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a3cfaf0020 con 0x55a3cfcd1db0 2024-05-27T07:27:48.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.456+0000 7f228402bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a3cfb8a7e0 con 0x55a3cfb4cec0 2024-05-27T07:27:48.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.456+0000 7f228402bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3cfcd8880 con 0x55a3cfccbe30 2024-05-27T07:27:48.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.456+0000 7f228402bb80 10 client.0 _ll_register_callbacks cb 0x55a3cfb93cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54060/0 (socket says 172.21.15.12:54060) 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 -- 172.21.15.12:0/293380033 learned_addr learned my addr 172.21.15.12:0/293380033 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226de00700 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 -- 172.21.15.12:0/293380033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 msgr2=0x55a3cfcd5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 -- 172.21.15.12:0/293380033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 msgr2=0x55a3cfcd5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.457+0000 7f226f200700 1 -- 172.21.15.12:0/293380033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3cfb39950 con 0x55a3cfcd1db0 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.458+0000 7f226f200700 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd5fe0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2268018e70 tx=0x7f2268004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61a18ba7e355d64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.458+0000 7f2258c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:48.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.458+0000 7f2258c00700 1 -- 172.21.15.12:0/293380033 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226801e050 con 0x55a3cfcd1db0 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.458+0000 7f2258c00700 1 -- 172.21.15.12:0/293380033 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9466+0+0 (secure 0 0 0) 0x7f2268004bb0 con 0x55a3cfcd1db0 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.458+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22600a5c60 con 0x55a3cfcd1db0 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f2258c00700 1 -- 172.21.15.12:0/293380033 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f2268030050 con 0x55a3cfcd1db0 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 10 client.16098 fetch_fsmap learned FSMap version 71 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a3cfbba0e0 con 0x55a3cfcd1db0 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f2258c00700 1 -- 172.21.15.12:0/293380033 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22600a5c60 con 0x55a3cfcd1db0 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f2258c00700 1 -- 172.21.15.12:0/293380033 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 71) ==== 6555+0+0 (secure 0 0 0) 0x7f22680237b0 con 0x55a3cfcd1db0 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 10 client.16098 fetch_fsmap finished waiting for FSMap version 71 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 20 client.16098 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfce5870 0x55a3cfce7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 4 client.16098 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:48.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.459+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3cfbfbe10 con 0x55a3cfce5870 2024-05-27T07:27:48.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.460+0000 7f226e800700 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfce5870 0x55a3cfce7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.462+0000 7f226e800700 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfce5870 0x55a3cfce7d10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac25e993a97ebce8 server_cookie=a53f8df4ca05dcde in_seq=0 out_seq=0 2024-05-27T07:27:48.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.462+0000 7f2258c00700 10 client.16098 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:48.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.463+0000 7f2258c00700 1 -- 172.21.15.12:0/293380033 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a3cfbfbe10 con 0x55a3cfce5870 2024-05-27T07:27:48.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.463+0000 7f2258c00700 10 client.16098 handle_command_reply: tid=0 2024-05-27T07:27:48.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 client.16098 shutdown 2024-05-27T07:27:48.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 msgr2=0x55a3cfcd5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd5fe0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2268018e70 tx=0x7f2268004ad0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 shutdown_connections 2024-05-27T07:27:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfce5870 0x55a3cfce7d10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcd5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd5fe0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 --2- 172.21.15.12:0/293380033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 >> 172.21.15.12:0/293380033 conn(0x55a3cfbad040 msgr2=0x55a3cfcc6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f2258c00700 0 client.16098 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:48.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 shutdown_connections 2024-05-27T07:27:48.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.464+0000 7f228402bb80 1 -- 172.21.15.12:0/293380033 wait complete. 2024-05-27T07:27:48.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.465+0000 7f228402bb80 20 client.16098 trim_cache size 0 max 16384 2024-05-27T07:27:48.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:48 smithi162 ceph-mon[35676]: pgmap v186: 129 pgs: 129 active+clean; 85 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 147 KiB/s wr, 45 op/s 2024-05-27T07:27:48.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.487+0000 7f228402bb80 1 Processor -- start 2024-05-27T07:27:48.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f228402bb80 1 -- start start 2024-05-27T07:27:48.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f228402bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfccb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f228402bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f228402bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f228402bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f22701b42e0 con 0x55a3cfcd1db0 2024-05-27T07:27:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f228402bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f22701b2db0 con 0x55a3cfb4cec0 2024-05-27T07:27:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f228402bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22701b34e0 con 0x55a3cfccbe30 2024-05-27T07:27:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.488+0000 7f226f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54062/0 (socket says 172.21.15.12:54062) 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226f200700 1 -- 172.21.15.12:0/1154670373 learned_addr learned my addr 172.21.15.12:0/1154670373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226e800700 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfccb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226de00700 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226f200700 1 -- 172.21.15.12:0/1154670373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 msgr2=0x55a3cfcd5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226f200700 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226f200700 1 -- 172.21.15.12:0/1154670373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 msgr2=0x55a3cfccb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226f200700 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfccb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.489+0000 7f226f200700 1 -- 172.21.15.12:0/1154670373 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3cfcc7750 con 0x55a3cfcd1db0 2024-05-27T07:27:48.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.490+0000 7f226f200700 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd7a00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2268009450 tx=0x7f2268007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcc03b576e04b6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.490+0000 7f225b400700 1 -- 172.21.15.12:0/1154670373 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2268039070 con 0x55a3cfcd1db0 2024-05-27T07:27:48.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.490+0000 7f225b400700 1 -- 172.21.15.12:0/1154670373 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f226801f040 con 0x55a3cfcd1db0 2024-05-27T07:27:48.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.490+0000 7f225b400700 1 -- 172.21.15.12:0/1154670373 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226801e040 con 0x55a3cfcd1db0 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.491+0000 7f228402bb80 1 -- 172.21.15.12:0/1154670373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 msgr2=0x55a3cfcd7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.491+0000 7f228402bb80 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd7a00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2268009450 tx=0x7f2268007f90 comp rx=0 tx=0).stop 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.491+0000 7f228402bb80 1 -- 172.21.15.12:0/1154670373 shutdown_connections 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.491+0000 7f228402bb80 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfb4cec0 0x55a3cfccb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.491+0000 7f228402bb80 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcd7a00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.491+0000 7f228402bb80 1 --2- 172.21.15.12:0/1154670373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcd5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.491+0000 7f228402bb80 1 -- 172.21.15.12:0/1154670373 >> 172.21.15.12:0/1154670373 conn(0x55a3cfbad040 msgr2=0x55a3cfcd44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.492+0000 7f228402bb80 1 -- 172.21.15.12:0/1154670373 shutdown_connections 2024-05-27T07:27:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.492+0000 7f228402bb80 1 -- 172.21.15.12:0/1154670373 wait complete. 2024-05-27T07:27:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.492+0000 7f228402bb80 1 -- start start 2024-05-27T07:27:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 1 Processor -- start 2024-05-27T07:27:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcdab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcdb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcdb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f22701b14a0 con 0x55a3cfb4cec0 2024-05-27T07:27:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f22701b3c10 con 0x55a3cfcd1db0 2024-05-27T07:27:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22701b0760 con 0x55a3cfccbe30 2024-05-27T07:27:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.493+0000 7f228402bb80 10 client.0 _ll_register_callbacks cb 0x55a3cfb93cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcdab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcdab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54076/0 (socket says 172.21.15.12:54076) 2024-05-27T07:27:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 -- 172.21.15.12:0/3144171351 learned_addr learned my addr 172.21.15.12:0/3144171351 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226de00700 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcdb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226f200700 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcdb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 -- 172.21.15.12:0/3144171351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 msgr2=0x55a3cfcdb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcdb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 -- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfcd1db0 msgr2=0x55a3cfcdb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcdb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.494+0000 7f226e800700 1 -- 172.21.15.12:0/3144171351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3cfd503f0 con 0x55a3cfb4cec0 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f226e800700 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcdab10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2260000c00 tx=0x7f2260007640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbe8d1b7b4b32cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f2258c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:48.487 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:48 smithi062 ceph-mon[35304]: pgmap v186: 129 pgs: 129 active+clean; 85 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 147 KiB/s wr, 45 op/s 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f2258c00700 1 -- 172.21.15.12:0/3144171351 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600a7820 con 0x55a3cfb4cec0 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f2258c00700 1 -- 172.21.15.12:0/3144171351 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9466+0+0 (secure 0 0 0) 0x7f22600a5c60 con 0x55a3cfb4cec0 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f226801f040 con 0x55a3cfb4cec0 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f2258c00700 1 -- 172.21.15.12:0/3144171351 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f22600a7a40 con 0x55a3cfb4cec0 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f228402bb80 10 client.16104 fetch_fsmap learned FSMap version 71 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.495+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a3cfd51500 con 0x55a3cfb4cec0 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f2258c00700 1 -- 172.21.15.12:0/3144171351 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f226801f040 con 0x55a3cfb4cec0 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f2258c00700 1 -- 172.21.15.12:0/3144171351 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 71) ==== 6555+0+0 (secure 0 0 0) 0x7f22600910b0 con 0x55a3cfb4cec0 2024-05-27T07:27:48.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f228402bb80 10 client.16104 fetch_fsmap finished waiting for FSMap version 71 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f228402bb80 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f228402bb80 20 client.16104 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f228402bb80 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfd51960 0x55a3cfd53d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f228402bb80 4 client.16104 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.496+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a3cfbf55f0 con 0x55a3cfd51960 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.497+0000 7f226de00700 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfd51960 0x55a3cfd53d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.498+0000 7f226de00700 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfd51960 0x55a3cfd53d80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e182d74124021b3 server_cookie=2fc29820e21990d4 in_seq=0 out_seq=0 2024-05-27T07:27:48.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.498+0000 7f2258c00700 10 client.16104 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:48.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.501+0000 7f2258c00700 1 -- 172.21.15.12:0/3144171351 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a3cfbf55f0 con 0x55a3cfd51960 2024-05-27T07:27:48.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.501+0000 7f2258c00700 10 client.16104 handle_command_reply: tid=0 2024-05-27T07:27:48.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.501+0000 7f228402bb80 1 client.16104 shutdown 2024-05-27T07:27:48.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.501+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfb4cec0 msgr2=0x55a3cfcdab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.501+0000 7f228402bb80 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcdab10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2260000c00 tx=0x7f2260007640 comp rx=0 tx=0).stop 2024-05-27T07:27:48.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 shutdown_connections 2024-05-27T07:27:48.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a3cfd51960 0x55a3cfd53d80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a3cfcd1db0 0x55a3cfcdb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3cfb4cec0 0x55a3cfcdab10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 --2- 172.21.15.12:0/3144171351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3cfccbe30 0x55a3cfcdb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f2258c00700 0 client.16104 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 >> 172.21.15.12:0/3144171351 conn(0x55a3cfbad040 msgr2=0x55a3cfcd24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 shutdown_connections 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 1 -- 172.21.15.12:0/3144171351 wait complete. 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.502+0000 7f228402bb80 20 client.16104 trim_cache size 0 max 16384 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:27:48.493 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:27:48.494 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "3f8a777e-ea2e-48af-a1bd-93afff007720", 2024-05-27T07:27:48.494 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:27:48.494 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:27:48.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f22540b2a70 msgr2=0x7f22540b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f22540b2a70 0x7f22540b4f80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55a3cfcccbb0 tx=0x7f2268009040 comp rx=0 tx=0).stop 2024-05-27T07:27:48.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22701045d0 msgr2=0x7f2270075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22701045d0 0x7f2270075ab0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f226000d980 tx=0x7f226000dc90 comp rx=0 tx=0).stop 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 shutdown_connections 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f22540b2a70 0x7f22540b4f80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2270109850 0x7f2270076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22701045d0 0x7f2270075ab0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 --2- 172.21.15.12:0/994628378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2270108f00 0x7f2270075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 >> 172.21.15.12:0/994628378 conn(0x7f22700ffe60 msgr2=0x7f22701071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.503+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 shutdown_connections 2024-05-27T07:27:48.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:48.504+0000 7f2275400700 1 -- 172.21.15.12:0/994628378 wait complete. 2024-05-27T07:27:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:49 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:27:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:49 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:49 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:27:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:49 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:27:49.304 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:49 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:49.356 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:27:49.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:49 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:27:49.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:49 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:49.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:49 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:27:49.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:49 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:27:49.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:49 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:49.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:49 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:27:49.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:49 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:49.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:49 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:27:49.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:49 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:27:49.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:49 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:49.794 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:50.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:50 smithi012 ceph-mon[31473]: pgmap v187: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 416 KiB/s wr, 93 op/s 2024-05-27T07:27:50.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:50 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:27:50.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:50 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:50.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:50 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:50.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:50 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:50.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:50 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:50.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:50 smithi162 ceph-mon[35676]: pgmap v187: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 416 KiB/s wr, 93 op/s 2024-05-27T07:27:50.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:50 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:27:50.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:50 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:50.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:50 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:50.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:50 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:50.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:50 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:50.474 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] open_by_handle 2024-05-27T07:27:50.479 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_mmap_dio 2024-05-27T07:27:50.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:50 smithi062 ceph-mon[35304]: pgmap v187: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 416 KiB/s wr, 93 op/s 2024-05-27T07:27:50.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:50 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:27:50.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:50 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:50.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:50 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:50.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:50 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:50.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:50 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:51.091 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] xfsfind 2024-05-27T07:27:51.093 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] t_reflink_read_race 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:51.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:51 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:51.271 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] fssum 2024-05-27T07:27:51.412 INFO:tasks.workunit.client.0.smithi012.stdout:Building log-writes 2024-05-27T07:27:51.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:27:51.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:51.474 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:51 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:51.481 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] replay-log 2024-05-27T07:27:51.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:27:51.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:51.486 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:51 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:27:51.973 INFO:tasks.workunit.client.0.smithi012.stdout:Building perf 2024-05-27T07:27:52.021 INFO:tasks.workunit.client.0.smithi012.stdout:Building aio-dio-regress 2024-05-27T07:27:52.022 INFO:tasks.workunit.client.0.smithi012.stdout:Building vfs 2024-05-27T07:27:52.112 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] vfstest 2024-05-27T07:27:52.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:52 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:52.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:52 smithi012 ceph-mon[31473]: pgmap v188: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 416 KiB/s wr, 93 op/s 2024-05-27T07:27:52.125 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aiocp 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 -- 172.21.15.12:0/717550945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68075b50 msgr2=0x7f7b68075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 --2- 172.21.15.12:0/717550945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68075b50 0x7f7b68075f60 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009990 tx=0x7f7b5c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 -- 172.21.15.12:0/717550945 shutdown_connections 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 --2- 172.21.15.12:0/717550945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b68115360 0x7f7b68117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 --2- 172.21.15.12:0/717550945 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68075b50 0x7f7b68075f60 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 --2- 172.21.15.12:0/717550945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b680764a0 0x7f7b68114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 -- 172.21.15.12:0/717550945 >> 172.21.15.12:0/717550945 conn(0x7f7b68100030 msgr2=0x7f7b68102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 -- 172.21.15.12:0/717550945 shutdown_connections 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.351+0000 7f7b70600700 1 -- 172.21.15.12:0/717550945 wait complete. 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.352+0000 7f7b70600700 1 Processor -- start 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.352+0000 7f7b70600700 1 -- start start 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b70600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b680764a0 0x7f7b6806f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b70600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68115360 0x7f7b6806fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b70600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b68075670 0x7f7b681b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b70600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6806d930 con 0x7f7b68115360 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b70600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6806d7b0 con 0x7f7b680764a0 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b70600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6806dab0 con 0x7f7b68075670 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b6d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68115360 0x7f7b6806fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b6d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68115360 0x7f7b6806fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40408/0 (socket says 172.21.15.12:40408) 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b6d200700 1 -- 172.21.15.12:0/1298383454 learned_addr learned my addr 172.21.15.12:0/1298383454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.353+0000 7f7b6e600700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b68075670 0x7f7b681b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.354+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b680764a0 0x7f7b6806f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.354+0000 7f7b6dc00700 1 -- 172.21.15.12:0/1298383454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b68075670 msgr2=0x7f7b681b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.354+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b68075670 0x7f7b681b5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.354+0000 7f7b6dc00700 1 -- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68115360 msgr2=0x7f7b6806fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.355+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68115360 0x7f7b6806fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.355+0000 7f7b6dc00700 1 -- 172.21.15.12:0/1298383454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b681b64f0 con 0x7f7b680764a0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.356+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b680764a0 0x7f7b6806f5f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c008000 tx=0x7f7b5c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.356+0000 7f7b66000700 1 -- 172.21.15.12:0/1298383454 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c026070 con 0x7f7b680764a0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.357+0000 7f7b66000700 1 -- 172.21.15.12:0/1298383454 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b5c00fb00 con 0x7f7b680764a0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.357+0000 7f7b66000700 1 -- 172.21.15.12:0/1298383454 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c021970 con 0x7f7b680764a0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.357+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b681b66c0 con 0x7f7b680764a0 2024-05-27T07:27:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.357+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b681b6a60 con 0x7f7b680764a0 2024-05-27T07:27:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.364+0000 7f7b66000700 1 -- 172.21.15.12:0/1298383454 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7b5c00f670 con 0x7f7b680764a0 2024-05-27T07:27:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.365+0000 7f7b66000700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7b500b2b20 0x7f7b500b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.366+0000 7f7b66000700 1 -- 172.21.15.12:0/1298383454 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9466+0+0 (secure 0 0 0) 0x7f7b5c0e3170 con 0x7f7b680764a0 2024-05-27T07:27:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.366+0000 7f7b7f0c6b80 1 Processor -- start 2024-05-27T07:27:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.366+0000 7f7b6d200700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7b500b2b20 0x7f7b500b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.367+0000 7f7b7f0c6b80 1 -- start start 2024-05-27T07:27:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5611d33dbe40 con 0x5611d34b3d30 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5611d33cb9e0 con 0x5611d34addb0 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5611d33cfa20 con 0x5611d332ee40 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b6d200700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7b500b2b20 0x7f7b500b5030 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7b681037c0 tx=0x7f7b5400b500 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b6dc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b6dc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:55028/0 (socket says 172.21.15.12:55028) 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b6dc00700 1 -- 172.21.15.12:0/2343109819 learned_addr learned my addr 172.21.15.12:0/2343109819 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.368+0000 7f7b6d200700 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.369+0000 7f7b6dc00700 1 -- 172.21.15.12:0/2343109819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 msgr2=0x5611d34ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.369+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.369+0000 7f7b6dc00700 1 -- 172.21.15.12:0/2343109819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 msgr2=0x5611d34ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.369+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.369+0000 7f7b6dc00700 1 -- 172.21.15.12:0/2343109819 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611d3394420 con 0x5611d332ee40 2024-05-27T07:27:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.369+0000 7f7b6d200700 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.370+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34b37f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0a9dd0 tx=0x7f7b5c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8324ea10cc746c55 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.371+0000 7f7b5aa00700 1 -- 172.21.15.12:0/2343109819 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c0aa7d0 con 0x5611d332ee40 2024-05-27T07:27:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.371+0000 7f7b5aa00700 1 -- 172.21.15.12:0/2343109819 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b5c0aa0d0 con 0x5611d332ee40 2024-05-27T07:27:52.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.371+0000 7f7b5aa00700 1 -- 172.21.15.12:0/2343109819 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c0b2a10 con 0x5611d332ee40 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.373+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/2343109819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 msgr2=0x5611d34b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.373+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34b37f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0a9dd0 tx=0x7f7b5c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.374+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/2343109819 shutdown_connections 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.374+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.374+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34b37f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.374+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/2343109819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.374+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/2343109819 >> 172.21.15.12:0/2343109819 conn(0x5611d338efc0 msgr2=0x5611d34a88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.374+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/2343109819 shutdown_connections 2024-05-27T07:27:52.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.374+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/2343109819 wait complete. 2024-05-27T07:27:52.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.375+0000 7f7b7f0c6b80 1 -- start start 2024-05-27T07:27:52.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.375+0000 7f7b7f0c6b80 1 Processor -- start 2024-05-27T07:27:52.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.376+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d332ee40 0x5611d34b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.376+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34b7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.376+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34b3d30 0x5611d34b8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.376+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5611d32d1fa0 con 0x5611d332ee40 2024-05-27T07:27:52.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.376+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5611d336c760 con 0x5611d34addb0 2024-05-27T07:27:52.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.376+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5611d34cb5d0 con 0x5611d34b3d30 2024-05-27T07:27:52.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.376+0000 7f7b7f0c6b80 10 client.0 _ll_register_callbacks cb 0x5611d3375c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:52.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.377+0000 7f7b6dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d332ee40 0x5611d34b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.377+0000 7f7b6dc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d332ee40 0x5611d34b75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40434/0 (socket says 172.21.15.12:40434) 2024-05-27T07:27:52.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.377+0000 7f7b6dc00700 1 -- 172.21.15.12:0/3080788063 learned_addr learned my addr 172.21.15.12:0/3080788063 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:52.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.377+0000 7f7b6e600700 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34b3d30 0x5611d34b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.377+0000 7f7b6d200700 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34b7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b6dc00700 1 -- 172.21.15.12:0/3080788063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34b3d30 msgr2=0x5611d34b8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34b3d30 0x5611d34b8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b6dc00700 1 -- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 msgr2=0x5611d34b7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34b7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b6dc00700 1 -- 172.21.15.12:0/3080788063 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611d331b8d0 con 0x5611d332ee40 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d332ee40 0x5611d34b75c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0aa730 tx=0x7f7b5c0ae3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5b00b22dac0876 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b4f400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:27:52.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.378+0000 7f7b4f400700 1 -- 172.21.15.12:0/3080788063 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c0c4060 con 0x5611d332ee40 2024-05-27T07:27:52.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.379+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b5c0aa0d0 con 0x5611d332ee40 2024-05-27T07:27:52.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.379+0000 7f7b4f400700 1 -- 172.21.15.12:0/3080788063 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9466+0+0 (secure 0 0 0) 0x7f7b5c0b2c20 con 0x5611d332ee40 2024-05-27T07:27:52.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.379+0000 7f7b4f400700 1 -- 172.21.15.12:0/3080788063 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f7b5c0b2ea0 con 0x5611d332ee40 2024-05-27T07:27:52.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.379+0000 7f7b7f0c6b80 10 client.16116 fetch_fsmap learned FSMap version 72 2024-05-27T07:27:52.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.379+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x5611d339c060 con 0x5611d332ee40 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b4f400700 1 -- 172.21.15.12:0/3080788063 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b5c0aa7d0 con 0x5611d332ee40 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b4f400700 1 -- 172.21.15.12:0/3080788063 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 72) ==== 5950+0+0 (secure 0 0 0) 0x7f7b5c0cc030 con 0x5611d332ee40 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b7f0c6b80 10 client.16116 fetch_fsmap finished waiting for FSMap version 72 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b7f0c6b80 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b7f0c6b80 20 client.16116 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d34d8310 0x5611d34da730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b7f0c6b80 4 client.16116 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.381+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7b5c0aa0d0 con 0x5611d34d8310 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.382+0000 7f7b6d200700 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d34d8310 0x5611d34da730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.383+0000 7f7b6d200700 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d34d8310 0x5611d34da730 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e1bc70beb21e1e server_cookie=9e8e479aa3c771fd in_seq=0 out_seq=0 2024-05-27T07:27:52.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.384+0000 7f7b4f400700 10 client.16116 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:52.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b4f400700 1 -- 172.21.15.12:0/3080788063 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f7b5c0aa0d0 con 0x5611d34d8310 2024-05-27T07:27:52.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b4f400700 10 client.16116 handle_command_reply: tid=0 2024-05-27T07:27:52.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 client.16116 shutdown 2024-05-27T07:27:52.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d332ee40 msgr2=0x5611d34b75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d332ee40 0x5611d34b75c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0aa730 tx=0x7f7b5c0ae3a0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 shutdown_connections 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34b3d30 0x5611d34b8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d332ee40 0x5611d34b75c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d34d8310 0x5611d34da730 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3080788063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34b7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 >> 172.21.15.12:0/3080788063 conn(0x5611d338efc0 msgr2=0x5611d34a8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.388+0000 7f7b4f400700 0 client.16116 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.389+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 shutdown_connections 2024-05-27T07:27:52.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.389+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3080788063 wait complete. 2024-05-27T07:27:52.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.390+0000 7f7b7f0c6b80 20 client.16116 trim_cache size 0 max 16384 2024-05-27T07:27:52.387 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-append-write-fallocate-race 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.430+0000 7f7b7f0c6b80 1 Processor -- start 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.430+0000 7f7b7f0c6b80 1 -- start start 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d332ee40 0x5611d34ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34addb0 0x5611d34b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34b9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7b681108a0 con 0x5611d34b3d30 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b681b8390 con 0x5611d332ee40 2024-05-27T07:27:52.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7b681b6e60 con 0x5611d34addb0 2024-05-27T07:27:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b6e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34b9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.431+0000 7f7b6d200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34addb0 0x5611d34b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34b9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40444/0 (socket says 172.21.15.12:40444) 2024-05-27T07:27:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6e600700 1 -- 172.21.15.12:0/4107888724 learned_addr learned my addr 172.21.15.12:0/4107888724 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:52.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d332ee40 0x5611d34ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6e600700 1 -- 172.21.15.12:0/4107888724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34addb0 msgr2=0x5611d34b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6e600700 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34addb0 0x5611d34b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6e600700 1 -- 172.21.15.12:0/4107888724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d332ee40 msgr2=0x5611d34ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6e600700 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d332ee40 0x5611d34ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.432+0000 7f7b6e600700 1 -- 172.21.15.12:0/4107888724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611d34b5b30 con 0x5611d34b3d30 2024-05-27T07:27:52.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.433+0000 7f7b6e600700 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34b9920 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000a760 tx=0x7f7b6000aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372150cafbf9f787 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.434+0000 7f7b5a000700 1 -- 172.21.15.12:0/4107888724 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6001a030 con 0x5611d34b3d30 2024-05-27T07:27:52.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.434+0000 7f7b5a000700 1 -- 172.21.15.12:0/4107888724 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b60004020 con 0x5611d34b3d30 2024-05-27T07:27:52.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.434+0000 7f7b5a000700 1 -- 172.21.15.12:0/4107888724 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60016690 con 0x5611d34b3d30 2024-05-27T07:27:52.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.435+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/4107888724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 msgr2=0x5611d34b9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.435+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34b9920 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000a760 tx=0x7f7b6000aa70 comp rx=0 tx=0).stop 2024-05-27T07:27:52.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.436+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/4107888724 shutdown_connections 2024-05-27T07:27:52.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.436+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d332ee40 0x5611d34ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.436+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34b9920 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.436+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/4107888724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d34addb0 0x5611d34b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.436+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/4107888724 >> 172.21.15.12:0/4107888724 conn(0x5611d338efc0 msgr2=0x5611d34b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.436+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/4107888724 shutdown_connections 2024-05-27T07:27:52.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.437+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/4107888724 wait complete. 2024-05-27T07:27:52.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.437+0000 7f7b7f0c6b80 1 -- start start 2024-05-27T07:27:52.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.437+0000 7f7b7f0c6b80 1 Processor -- start 2024-05-27T07:27:52.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.438+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.438+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.438+0000 7f7b7f0c6b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34cd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.438+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7b681b7cc0 con 0x5611d34b3d30 2024-05-27T07:27:52.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.438+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b681b7590 con 0x5611d34addb0 2024-05-27T07:27:52.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.438+0000 7f7b7f0c6b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6810f030 con 0x5611d332ee40 2024-05-27T07:27:52.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.438+0000 7f7b7f0c6b80 10 client.0 _ll_register_callbacks cb 0x5611d3375c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:52.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.439+0000 7f7b6e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.439+0000 7f7b6d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.439+0000 7f7b6dc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.439+0000 7f7b6d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34d0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:55480/0 (socket says 172.21.15.12:55480) 2024-05-27T07:27:52.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.439+0000 7f7b6d200700 1 -- 172.21.15.12:0/3626829379 learned_addr learned my addr 172.21.15.12:0/3626829379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:27:52.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.440+0000 7f7b6d200700 1 -- 172.21.15.12:0/3626829379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 msgr2=0x5611d34d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.440+0000 7f7b6d200700 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.440+0000 7f7b6d200700 1 -- 172.21.15.12:0/3626829379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 msgr2=0x5611d34cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:52.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.440+0000 7f7b6d200700 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.440+0000 7f7b6d200700 1 -- 172.21.15.12:0/3626829379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611d34d1290 con 0x5611d34addb0 2024-05-27T07:27:52.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.441+0000 7f7b6d200700 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34d0bb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400c010 tx=0x7f7b540047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e671c92c56d9693b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:52.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.441+0000 7f7b4f400700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:27:52.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.441+0000 7f7b4f400700 1 -- 172.21.15.12:0/3626829379 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b54020bb0 con 0x5611d34addb0 2024-05-27T07:27:52.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.442+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b60004020 con 0x5611d34addb0 2024-05-27T07:27:52.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.442+0000 7f7b4f400700 1 -- 172.21.15.12:0/3626829379 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9466+0+0 (secure 0 0 0) 0x7f7b54020d50 con 0x5611d34addb0 2024-05-27T07:27:52.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.442+0000 7f7b4f400700 1 -- 172.21.15.12:0/3626829379 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f7b54037050 con 0x5611d34addb0 2024-05-27T07:27:52.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.442+0000 7f7b7f0c6b80 10 client.35708 fetch_fsmap learned FSMap version 72 2024-05-27T07:27:52.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.443+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x5611d3543150 con 0x5611d34addb0 2024-05-27T07:27:52.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.443+0000 7f7b4f400700 1 -- 172.21.15.12:0/3626829379 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b60004020 con 0x5611d34addb0 2024-05-27T07:27:52.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.444+0000 7f7b4f400700 1 -- 172.21.15.12:0/3626829379 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 72) ==== 5950+0+0 (secure 0 0 0) 0x7f7b54032bd0 con 0x5611d34addb0 2024-05-27T07:27:52.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.444+0000 7f7b7f0c6b80 10 client.35708 fetch_fsmap finished waiting for FSMap version 72 2024-05-27T07:27:52.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.444+0000 7f7b7f0c6b80 10 client.35708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:52.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.444+0000 7f7b7f0c6b80 20 client.35708 populate_metadata read hostname 'smithi012' 2024-05-27T07:27:52.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.444+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d35438f0 0x5611d3545d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:52.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.444+0000 7f7b7f0c6b80 4 client.35708 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:27:52.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.444+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5611d33ddd90 con 0x5611d35438f0 2024-05-27T07:27:52.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.445+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d35438f0 0x5611d3545d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.446+0000 7f7b6dc00700 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d35438f0 0x5611d3545d10 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da7718db9829342d server_cookie=98d4ee93a3a51b18 in_seq=0 out_seq=0 2024-05-27T07:27:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.446+0000 7f7b4f400700 10 client.35708 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b4f400700 1 -- 172.21.15.12:0/3626829379 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5611d33ddd90 con 0x5611d35438f0 2024-05-27T07:27:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b4f400700 10 client.35708 handle_command_reply: tid=0 2024-05-27T07:27:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 client.35708 shutdown 2024-05-27T07:27:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 msgr2=0x5611d34d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34d0bb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400c010 tx=0x7f7b540047c0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 shutdown_connections 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611d34b3d30 0x5611d34cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5611d332ee40 0x5611d34d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5611d35438f0 0x5611d3545d10 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 --2- 172.21.15.12:0/3626829379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611d34addb0 0x5611d34d0bb0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400c010 tx=0x7f7b540047c0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 >> 172.21.15.12:0/3626829379 conn(0x5611d338efc0 msgr2=0x5611d34b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b4f400700 0 client.35708 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:27:52.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.447+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 shutdown_connections 2024-05-27T07:27:52.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.448+0000 7f7b7f0c6b80 1 -- 172.21.15.12:0/3626829379 wait complete. 2024-05-27T07:27:52.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.448+0000 7f7b7f0c6b80 20 client.35708 trim_cache size 0 max 16384 2024-05-27T07:27:52.439 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:27:52.439 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:27:52.439 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:27:52.439 INFO:teuthology.orchestra.run.smithi012.stdout: "3f8a777e-ea2e-48af-a1bd-93afff007720": { 2024-05-27T07:27:52.440 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:27:52.440 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "3f8a777e-ea2e-48af-a1bd-93afff007720", 2024-05-27T07:27:52.440 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:27:52.440 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:27:52.440 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:27:52.440 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:27:52.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.450+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7b500b2b20 msgr2=0x7f7b500b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.450+0000 7f7b70600700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7b500b2b20 0x7f7b500b5030 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7b681037c0 tx=0x7f7b5400b500 comp rx=0 tx=0).stop 2024-05-27T07:27:52.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.450+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b680764a0 msgr2=0x7f7b6806f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:52.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.450+0000 7f7b70600700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b680764a0 0x7f7b6806f5f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c008000 tx=0x7f7b5c0057c0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 shutdown_connections 2024-05-27T07:27:52.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7b500b2b20 0x7f7b500b5030 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b680764a0 0x7f7b6806f5f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b68075670 0x7f7b681b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 --2- 172.21.15.12:0/1298383454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b68115360 0x7f7b6806fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:52.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 >> 172.21.15.12:0/1298383454 conn(0x7f7b68100030 msgr2=0x7f7b68100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:52.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 shutdown_connections 2024-05-27T07:27:52.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:27:52.451+0000 7f7b70600700 1 -- 172.21.15.12:0/1298383454 wait complete. 2024-05-27T07:27:52.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:52 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:52.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:52 smithi162 ceph-mon[35676]: pgmap v188: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 416 KiB/s wr, 93 op/s 2024-05-27T07:27:52.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:52 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:52.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:52 smithi062 ceph-mon[35304]: pgmap v188: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 416 KiB/s wr, 93 op/s 2024-05-27T07:27:52.558 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] mount-idmapped 2024-05-27T07:27:53.056 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-append-write-read-race 2024-05-27T07:27:53.124 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-mon[31473]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:53.124 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:53.148 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-cow-race 2024-05-27T07:27:53.387 INFO:tasks.cephfs.filesystem:scrub status for tag:3f8a777e-ea2e-48af-a1bd-93afff007720 - {'path': '/', 'tag': '3f8a777e-ea2e-48af-a1bd-93afff007720', 'options': 'recursive,force'} 2024-05-27T07:27:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:53 smithi162 ceph-mon[35676]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:53.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:53 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:53.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:53 smithi062 ceph-mon[35304]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-05-27T07:27:53.485 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:53 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:53.716 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb07400700 time 2024-05-27T07:27:53.565755+0000 2024-05-27T07:27:53.716 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:53.716 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.716 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffb1d006ba6] 2024-05-27T07:27:53.716 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.716 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.716 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.717 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.718 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: clone() 2024-05-27T07:27:53.718 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.567+0000 7ffb07400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb07400700 time 2024-05-27T07:27:53.565755+0000 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffb1d006ba6] 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.719 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.720 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: clone() 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: *** Caught signal (Aborted) ** 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: in thread 7ffb07400700 thread_name:md_log_replay 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffb1bb83d20] 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: gsignal() 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: abort() 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffb1d006ced] 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.721 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.722 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 19: clone() 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.570+0000 7ffb07400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: in thread 7ffb07400700 thread_name:md_log_replay 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffb1bb83d20] 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: gsignal() 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: abort() 2024-05-27T07:27:53.723 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffb1d006ced] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.724 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 19: clone() 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: -1> 2024-05-27T07:27:53.567+0000 7ffb07400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb07400700 time 2024-05-27T07:27:53.565755+0000 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.725 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffb1d006ba6] 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.726 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.727 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.727 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.727 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.727 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.727 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: clone() 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 0> 2024-05-27T07:27:53.570+0000 7ffb07400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: in thread 7ffb07400700 thread_name:md_log_replay 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.728 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffb1bb83d20] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: gsignal() 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: abort() 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffb1d006ced] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.729 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 19: clone() 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: -1> 2024-05-27T07:27:53.567+0000 7ffb07400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffb07400700 time 2024-05-27T07:27:53.565755+0000 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.730 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffb1d006ba6] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.731 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: clone() 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 0> 2024-05-27T07:27:53.570+0000 7ffb07400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: in thread 7ffb07400700 thread_name:md_log_replay 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffb1bb83d20] 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2: gsignal() 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 3: abort() 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffb1d006ced] 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffb1d006dd1] 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c51872c23d] 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c518717971] 2024-05-27T07:27:53.732 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c518715fa0] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c518716651] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5186ab347] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5183a787e] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c5183ace94] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5183b7969] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c51875048c] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5187599cb] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 16: (Thread::entry_wrapper()+0x43) [0x7ffb1cfd572d] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffb1cfd5749] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffb1bb791ca] 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 19: clone() 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[111103]: 2024-05-27T07:27:53.733 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:53 smithi012 podman[119140]: 2024-05-27 07:27:53.724452904 +0000 UTC m=+0.045877138 container died a084131c859561fa54d536dca8ab31de5c2442236bd521327c93a8c8925d3879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:27:53.796 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-cycle-write 2024-05-27T07:27:53.817 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-eof-race 2024-05-27T07:27:53.821 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-extend-stat 2024-05-27T07:27:54.433 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-hole-filling-race 2024-05-27T07:27:54.433 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-fcntl-race 2024-05-27T07:27:54.437 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-invalidate-failure 2024-05-27T07:27:54.523 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:54 smithi012 ceph-mon[31473]: pgmap v189: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 352 KiB/s wr, 79 op/s 2024-05-27T07:27:54.524 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:54 smithi012 podman[119140]: 2024-05-27 07:27:54.533585187 +0000 UTC m=+0.855009401 container remove a084131c859561fa54d536dca8ab31de5c2442236bd521327c93a8c8925d3879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:27:54.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:54 smithi162 ceph-mon[35676]: pgmap v189: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 352 KiB/s wr, 79 op/s 2024-05-27T07:27:54.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:54 smithi062 ceph-mon[35304]: pgmap v189: 129 pgs: 129 active+clean; 87 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 352 KiB/s wr, 79 op/s 2024-05-27T07:27:54.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:54 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:55.057 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-invalidate-readahead 2024-05-27T07:27:55.064 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aiodio_sparse2 2024-05-27T07:27:55.076 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-subblock-eof-read 2024-05-27T07:27:55.127 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:54 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:27:55.127 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:54 smithi162 systemd[1]: Stopped Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:55.127 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:54 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.667s CPU time. 2024-05-27T07:27:55.127 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:54 smithi162 systemd[1]: Starting Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:55.339 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:54 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:27:55.339 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:27:54 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.860s CPU time. 2024-05-27T07:27:55.396 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 podman[73715]: 2024-05-27 07:27:55.042024362 +0000 UTC m=+0.033419038 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:55.396 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 podman[73715]: 2024-05-27 07:27:55.243915387 +0000 UTC m=+0.235310100 container create 3c85e299d40704530479096cb190ae7c2568b4cffa9ffc60e7885b2fd09a4e39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:27:55.635 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-dio-write-verify 2024-05-27T07:27:55.646 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-free-ring-with-bogus-nr-pages 2024-05-27T07:27:55.732 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-io-setup-with-nonwritable-context-pointer 2024-05-27T07:27:56.823 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:56 smithi062 ceph-mon[35304]: pgmap v190: 129 pgs: 129 active+clean; 88 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 294 KiB/s wr, 67 op/s 2024-05-27T07:27:56.823 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:56 smithi062 ceph-mon[35304]: daemon mds.l restarted 2024-05-27T07:27:56.823 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 podman[73715]: 2024-05-27 07:27:55.522717614 +0000 UTC m=+0.514112334 container init 3c85e299d40704530479096cb190ae7c2568b4cffa9ffc60e7885b2fd09a4e39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T07:27:56.823 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 podman[73715]: 2024-05-27 07:27:55.53246291 +0000 UTC m=+0.523857594 container start 3c85e299d40704530479096cb190ae7c2568b4cffa9ffc60e7885b2fd09a4e39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T07:27:56.823 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 bash[73715]: 3c85e299d40704530479096cb190ae7c2568b4cffa9ffc60e7885b2fd09a4e39 2024-05-27T07:27:56.823 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 systemd[1]: Started Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:56.823 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 ceph-mds[73733]: -- 172.21.15.162:0/1873398895 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b82d31a40 con 0x556b82d79a80 2024-05-27T07:27:56.824 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: starting mds.l at 2024-05-27T07:27:56.824 INFO:tasks.workunit.client.0.smithi012.stdout: [CC] aio-last-ref-held-by-io 2024-05-27T07:27:56.824 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:56 smithi162 ceph-mon[35676]: pgmap v190: 129 pgs: 129 active+clean; 88 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 294 KiB/s wr, 67 op/s 2024-05-27T07:27:56.824 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:56 smithi162 ceph-mon[35676]: daemon mds.l restarted 2024-05-27T07:27:56.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:56 smithi012 ceph-mon[31473]: pgmap v190: 129 pgs: 129 active+clean; 88 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 294 KiB/s wr, 67 op/s 2024-05-27T07:27:56.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:56 smithi012 ceph-mon[31473]: daemon mds.l restarted 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f79cc400700 time 2024-05-27T07:27:56.410618+0000 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79e1e3eba6] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.826 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: clone() 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: *** Caught signal (Aborted) ** 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: in thread 7f79cc400700 thread_name:md_log_replay 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.411+0000 7f79cc400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f79cc400700 time 2024-05-27T07:27:56.410618+0000 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:56.827 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79e1e3eba6] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.828 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: clone() 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79e09bbd20] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: gsignal() 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: abort() 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79e1e3eced] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.829 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 19: clone() 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.414+0000 7f79cc400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: in thread 7f79cc400700 thread_name:md_log_replay 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.830 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79e09bbd20] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: gsignal() 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: abort() 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79e1e3eced] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.831 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 19: clone() 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: -1> 2024-05-27T07:27:56.411+0000 7f79cc400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f79cc400700 time 2024-05-27T07:27:56.410618+0000 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79e1e3eba6] 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.832 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: clone() 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.833 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 0> 2024-05-27T07:27:56.414+0000 7f79cc400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: in thread 7f79cc400700 thread_name:md_log_replay 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79e09bbd20] 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: gsignal() 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: abort() 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79e1e3eced] 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.834 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 19: clone() 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: -1> 2024-05-27T07:27:56.411+0000 7f79cc400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f79cc400700 time 2024-05-27T07:27:56.410618+0000 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.835 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79e1e3eba6] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.836 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: clone() 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 0> 2024-05-27T07:27:56.414+0000 7f79cc400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: in thread 7f79cc400700 thread_name:md_log_replay 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79e09bbd20] 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2: gsignal() 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 3: abort() 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79e1e3eced] 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79e1e3edd1] 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x556b815e423d] 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556b815cf971] 2024-05-27T07:27:56.837 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 8: (SnapRealm::check_cache() const+0x13e) [0x556b815cdfa0] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556b815ce651] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556b81563347] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556b8125f87e] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556b81264e94] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556b8126f969] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 14: (MDLog::_replay_thread()+0x1f38) [0x556b8160848c] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 15: (MDLog::ReplayThread::entry()+0x11) [0x556b816119cb] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 16: (Thread::entry_wrapper()+0x43) [0x7f79e1e0d72d] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79e1e0d749] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79e09b11ca] 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 19: clone() 2024-05-27T07:27:56.838 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:27:56.839 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73729]: 2024-05-27T07:27:56.839 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:56 smithi162 podman[73764]: 2024-05-27 07:27:56.539029826 +0000 UTC m=+0.035921567 container died 3c85e299d40704530479096cb190ae7c2568b4cffa9ffc60e7885b2fd09a4e39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:27:57.082 INFO:tasks.workunit.client.0.smithi012.stderr:+ cd .. 2024-05-27T07:27:57.096 INFO:tasks.workunit.client.0.smithi012.stderr:+ cp xfstests-dev/ltp/fsx . 2024-05-27T07:27:57.119 INFO:tasks.workunit.client.0.smithi012.stderr:+ OPTIONS=-z 2024-05-27T07:27:57.132 INFO:tasks.workunit.client.0.smithi012.stderr:+ ./fsx -z 1MB -N 50000 -p 10000 -l 1048576 2024-05-27T07:27:57.132 INFO:tasks.workunit.client.0.smithi012.stdout:Seed set to 1 2024-05-27T07:27:57.228 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-05-27T07:27:57.228 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-05-27T07:27:57.229 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support clone range, disabling! 2024-05-27T07:27:57.239 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support dedupe range, disabling! 2024-05-27T07:27:57.246 INFO:tasks.workunit.client.0.smithi012.stdout:skipping zero size read 2024-05-27T07:27:57.254 INFO:tasks.workunit.client.0.smithi012.stdout:skipping zero size read 2024-05-27T07:27:57.256 INFO:tasks.workunit.client.0.smithi012.stdout:fallocating to largest ever: 0x9abd6 2024-05-27T07:27:57.457 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0xaeaec 2024-05-27T07:27:57.585 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0xf3194 2024-05-27T07:27:57.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:57 smithi012 ceph-mon[31473]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:27:57.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:57 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/2410898914,v1:172.21.15.162:6833/2410898914] up:boot 2024-05-27T07:27:57.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:57 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:27:57.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:57 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:57.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:57 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:57.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:57 smithi162 ceph-mon[35676]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:27:57.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:57 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/2410898914,v1:172.21.15.162:6833/2410898914] up:boot 2024-05-27T07:27:57.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:57 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:27:57.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:57 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:57.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:57 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:57.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:57 smithi162 podman[73764]: 2024-05-27 07:27:57.294423382 +0000 UTC m=+0.791315161 container remove 3c85e299d40704530479096cb190ae7c2568b4cffa9ffc60e7885b2fd09a4e39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T07:27:57.724 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:57 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:27:57.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:57 smithi062 ceph-mon[35304]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:27:57.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:57 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/2410898914,v1:172.21.15.162:6833/2410898914] up:boot 2024-05-27T07:27:57.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:57 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:27:57.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:57 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:27:57.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:57 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:27:57.765 INFO:tasks.workunit.client.0.smithi012.stdout:copying to largest ever: 0xf3ceb 2024-05-27T07:27:58.224 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:57 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:27:58.224 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:27:57 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.652s CPU time. 2024-05-27T07:27:58.433 INFO:tasks.workunit.client.0.smithi012.stdout:copying to largest ever: 0xfaccc 2024-05-27T07:27:58.457 INFO:tasks.workunit.client.0.smithi012.stdout:fallocating to largest ever: 0xfbb6d 2024-05-27T07:27:58.531 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:58 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:27:58.531 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:58 smithi012 systemd[1]: Stopped Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:58.531 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:58 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.711s CPU time. 2024-05-27T07:27:58.531 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:58 smithi012 systemd[1]: Starting Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:27:58.532 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:27:58 smithi012 ceph-mon[31473]: pgmap v192: 129 pgs: 129 active+clean; 88 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 294 KiB/s wr, 67 op/s 2024-05-27T07:27:58.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:27:58 smithi162 ceph-mon[35676]: pgmap v192: 129 pgs: 129 active+clean; 88 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 294 KiB/s wr, 67 op/s 2024-05-27T07:27:58.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:27:58 smithi062 ceph-mon[35304]: pgmap v192: 129 pgs: 129 active+clean; 88 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 294 KiB/s wr, 67 op/s 2024-05-27T07:27:58.839 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:58 smithi012 podman[120441]: 2024-05-27 07:27:58.447015693 +0000 UTC m=+0.039978938 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:27:59.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:59 smithi012 podman[120441]: 2024-05-27 07:27:59.093466453 +0000 UTC m=+0.686429693 container create 07e699ca63cfea125c7cbd701c4b9d66476718f3cad00b50663503f3d9b7d62d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:59 smithi012 podman[120441]: 2024-05-27 07:27:59.519767544 +0000 UTC m=+1.112730931 container init 07e699ca63cfea125c7cbd701c4b9d66476718f3cad00b50663503f3d9b7d62d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:59 smithi012 podman[120441]: 2024-05-27 07:27:59.530410631 +0000 UTC m=+1.123373845 container start 07e699ca63cfea125c7cbd701c4b9d66476718f3cad00b50663503f3d9b7d62d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:59 smithi012 bash[120441]: 07e699ca63cfea125c7cbd701c4b9d66476718f3cad00b50663503f3d9b7d62d 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:59 smithi012 systemd[1]: Started Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:59 smithi012 ceph-mds[120459]: -- 172.21.15.12:0/1120264917 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55668c91da40 con 0x55668c965a80 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:27:59 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: starting mds.g at 2024-05-27T07:27:59.977 INFO:tasks.workunit.client.0.smithi012.stdout:fallocating to largest ever: 0xffee7 2024-05-27T07:28:00.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:00 smithi012 ceph-mon[31473]: pgmap v193: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 307 KiB/s wr, 72 op/s 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:00 smithi012 ceph-mon[31473]: daemon mds.g restarted 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1440600700 time 2024-05-27T07:28:00.527897+0000 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1456152ba6] 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.685 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.686 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: clone() 2024-05-27T07:28:00.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: *** Caught signal (Aborted) ** 2024-05-27T07:28:00.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: in thread 7f1440600700 thread_name:md_log_replay 2024-05-27T07:28:00.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.529+0000 7f1440600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1440600700 time 2024-05-27T07:28:00.527897+0000 2024-05-27T07:28:00.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:00.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1456152ba6] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: clone() 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1454ccfd20] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: gsignal() 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: abort() 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1456152ced] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 19: clone() 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.531+0000 7f1440600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: in thread 7f1440600700 thread_name:md_log_replay 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1454ccfd20] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: gsignal() 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: abort() 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1456152ced] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 19: clone() 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: -1> 2024-05-27T07:28:00.529+0000 7f1440600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1440600700 time 2024-05-27T07:28:00.527897+0000 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1456152ba6] 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: clone() 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 0> 2024-05-27T07:28:00.531+0000 7f1440600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: in thread 7f1440600700 thread_name:md_log_replay 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1454ccfd20] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: gsignal() 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: abort() 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1456152ced] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 19: clone() 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: -1> 2024-05-27T07:28:00.529+0000 7f1440600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1440600700 time 2024-05-27T07:28:00.527897+0000 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1456152ba6] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: clone() 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 0> 2024-05-27T07:28:00.531+0000 7f1440600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: in thread 7f1440600700 thread_name:md_log_replay 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1454ccfd20] 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2: gsignal() 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 3: abort() 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1456152ced] 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1456152dd1] 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55667911a23d] 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 7: (SnapRealm::build_snap_set() const+0x323) [0x556679105971] 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 8: (SnapRealm::check_cache() const+0x13e) [0x556679103fa0] 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x556679104651] 2024-05-27T07:28:00.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556679099347] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x556678d9587e] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556678d9ae94] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x556678da5969] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 14: (MDLog::_replay_thread()+0x1f38) [0x55667913e48c] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5566791479cb] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 16: (Thread::entry_wrapper()+0x43) [0x7f145612172d] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1456121749] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1454cc51ca] 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 19: clone() 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120454]: 2024-05-27T07:28:00.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:00 smithi012 podman[120491]: 2024-05-27 07:28:00.69446912 +0000 UTC m=+0.037462572 container died 07e699ca63cfea125c7cbd701c4b9d66476718f3cad00b50663503f3d9b7d62d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:28:00.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:00 smithi162 ceph-mon[35676]: pgmap v193: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 307 KiB/s wr, 72 op/s 2024-05-27T07:28:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:00 smithi162 ceph-mon[35676]: daemon mds.g restarted 2024-05-27T07:28:00.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:00 smithi062 ceph-mon[35304]: pgmap v193: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 307 KiB/s wr, 72 op/s 2024-05-27T07:28:00.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:00 smithi062 ceph-mon[35304]: daemon mds.g restarted 2024-05-27T07:28:01.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:01 smithi162 ceph-mon[35676]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:28:01.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:01 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/615814966,v1:172.21.15.12:6839/615814966] up:boot 2024-05-27T07:28:01.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:01 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:01.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:01 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:01.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:01 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:01.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:01 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:28:01.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:01 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:28:01.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:01 smithi062 ceph-mon[35304]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:28:01.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:01 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/615814966,v1:172.21.15.12:6839/615814966] up:boot 2024-05-27T07:28:01.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:01 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:01.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:01 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:01.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:01 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:01.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:01 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:28:01.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:01 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:28:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:01 smithi012 ceph-mon[31473]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:28:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:01 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/615814966,v1:172.21.15.12:6839/615814966] up:boot 2024-05-27T07:28:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:01 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:01 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:01 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:01.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:01 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:28:01.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:01 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:28:02.279 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:01 smithi012 podman[120491]: 2024-05-27 07:28:01.872328477 +0000 UTC m=+1.215321983 container remove 07e699ca63cfea125c7cbd701c4b9d66476718f3cad00b50663503f3d9b7d62d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:28:02.280 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:01 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:02.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:02 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:02.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:02 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.729s CPU time. 2024-05-27T07:28:02.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:02 smithi012 ceph-mon[31473]: pgmap v195: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 384 KiB/s wr, 90 op/s 2024-05-27T07:28:02.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:02 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:28:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:02 smithi162 ceph-mon[35676]: pgmap v195: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 384 KiB/s wr, 90 op/s 2024-05-27T07:28:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:02 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:28:02.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:02 smithi062 ceph-mon[35304]: pgmap v195: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 384 KiB/s wr, 90 op/s 2024-05-27T07:28:02.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:02 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:28:04.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:04 smithi162 ceph-mon[35676]: pgmap v196: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 286 KiB/s wr, 55 op/s 2024-05-27T07:28:04.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:04 smithi062 ceph-mon[35304]: pgmap v196: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 286 KiB/s wr, 55 op/s 2024-05-27T07:28:04.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:04 smithi012 ceph-mon[31473]: pgmap v196: 129 pgs: 129 active+clean; 89 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 286 KiB/s wr, 55 op/s 2024-05-27T07:28:05.504 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:28:05.504 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 systemd[1]: Stopped Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:05.504 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.860s CPU time. 2024-05-27T07:28:05.504 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 systemd[1]: Starting Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:05.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 podman[120582]: 2024-05-27 07:28:05.421147529 +0000 UTC m=+0.038460758 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:05.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 podman[120582]: 2024-05-27 07:28:05.538590264 +0000 UTC m=+0.155903544 container create 9b8e3aeb8130f6474e2569baf05b8daad531e3e463383e44d503fc6bf9b118ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:28:06.283 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 podman[120582]: 2024-05-27 07:28:05.866719263 +0000 UTC m=+0.484032530 container init 9b8e3aeb8130f6474e2569baf05b8daad531e3e463383e44d503fc6bf9b118ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:06.283 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:05 smithi012 podman[120582]: 2024-05-27 07:28:05.878667634 +0000 UTC m=+0.495980870 container start 9b8e3aeb8130f6474e2569baf05b8daad531e3e463383e44d503fc6bf9b118ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:28:06.283 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:06 smithi012 ceph-mds[120601]: -- 172.21.15.12:0/494411664 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5558e80b3a40 con 0x5558e80fb600 2024-05-27T07:28:06.283 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: starting mds.a at 2024-05-27T07:28:06.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:06 smithi012 ceph-mon[31473]: pgmap v197: 129 pgs: 129 active+clean; 91 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 184 op/s 2024-05-27T07:28:06.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:06 smithi012 ceph-mon[31473]: daemon mds.a restarted 2024-05-27T07:28:06.590 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:06 smithi012 bash[120582]: 9b8e3aeb8130f6474e2569baf05b8daad531e3e463383e44d503fc6bf9b118ae 2024-05-27T07:28:06.590 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:06 smithi012 systemd[1]: Started Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:06.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:06 smithi162 ceph-mon[35676]: pgmap v197: 129 pgs: 129 active+clean; 91 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 184 op/s 2024-05-27T07:28:06.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:06 smithi162 ceph-mon[35676]: daemon mds.a restarted 2024-05-27T07:28:06.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:06 smithi062 ceph-mon[35304]: pgmap v197: 129 pgs: 129 active+clean; 91 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 184 op/s 2024-05-27T07:28:06.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:06 smithi062 ceph-mon[35304]: daemon mds.a restarted 2024-05-27T07:28:07.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-mon[31473]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:28:07.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/2924967159,v1:172.21.15.12:6839/2924967159] up:boot 2024-05-27T07:28:07.696 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:07.696 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:07.696 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:07.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbaa0200700 time 2024-05-27T07:28:07.560978+0000 2024-05-27T07:28:07.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbab5d6dba6] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: clone() 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: *** Caught signal (Aborted) ** 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: in thread 7fbaa0200700 thread_name:md_log_replay 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.562+0000 7fbaa0200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbaa0200700 time 2024-05-27T07:28:07.560978+0000 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbab5d6dba6] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: clone() 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbab48ead20] 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: gsignal() 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: abort() 2024-05-27T07:28:07.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbab5d6dced] 2024-05-27T07:28:07.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 19: clone() 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.565+0000 7fbaa0200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: in thread 7fbaa0200700 thread_name:md_log_replay 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbab48ead20] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: gsignal() 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: abort() 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbab5d6dced] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 19: clone() 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: -1> 2024-05-27T07:28:07.562+0000 7fbaa0200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbaa0200700 time 2024-05-27T07:28:07.560978+0000 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbab5d6dba6] 2024-05-27T07:28:07.704 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.705 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: clone() 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 0> 2024-05-27T07:28:07.565+0000 7fbaa0200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: in thread 7fbaa0200700 thread_name:md_log_replay 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbab48ead20] 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: gsignal() 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: abort() 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbab5d6dced] 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.706 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 19: clone() 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: -1> 2024-05-27T07:28:07.562+0000 7fbaa0200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fbaa0200700 time 2024-05-27T07:28:07.560978+0000 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbab5d6dba6] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.708 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: clone() 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 0> 2024-05-27T07:28:07.565+0000 7fbaa0200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: in thread 7fbaa0200700 thread_name:md_log_replay 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbab48ead20] 2024-05-27T07:28:07.709 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2: gsignal() 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 3: abort() 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbab5d6dced] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbab5d6ddd1] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5558b226123d] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5558b224c971] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 8: (SnapRealm::check_cache() const+0x13e) [0x5558b224afa0] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5558b224b651] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5558b21e0347] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5558b1edc87e] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5558b1ee1e94] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5558b1eec969] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 14: (MDLog::_replay_thread()+0x1f38) [0x5558b228548c] 2024-05-27T07:28:07.710 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5558b228e9cb] 2024-05-27T07:28:07.711 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 16: (Thread::entry_wrapper()+0x43) [0x7fbab5d3c72d] 2024-05-27T07:28:07.711 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 17: (Thread::_entry_func(void*)+0xd) [0x7fbab5d3c749] 2024-05-27T07:28:07.711 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fbab48e01ca] 2024-05-27T07:28:07.711 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 19: clone() 2024-05-27T07:28:07.711 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:07.711 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120596]: 2024-05-27T07:28:07.711 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:07 smithi012 podman[120632]: 2024-05-27 07:28:07.706110111 +0000 UTC m=+0.035156642 container died 9b8e3aeb8130f6474e2569baf05b8daad531e3e463383e44d503fc6bf9b118ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:28:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:07 smithi162 ceph-mon[35676]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:28:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:07 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/2924967159,v1:172.21.15.12:6839/2924967159] up:boot 2024-05-27T07:28:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:07 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:07 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:07 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:07 smithi062 ceph-mon[35304]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:28:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:07 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/2924967159,v1:172.21.15.12:6839/2924967159] up:boot 2024-05-27T07:28:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:07 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:07 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:07.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:07 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:08.383 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:07 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:28:08.383 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:07 smithi162 systemd[1]: Stopped Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:08.383 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:07 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.652s CPU time. 2024-05-27T07:28:08.383 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 systemd[1]: Starting Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:08.655 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:08 smithi162 ceph-mon[35676]: pgmap v199: 129 pgs: 129 active+clean; 91 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 964 KiB/s wr, 134 op/s 2024-05-27T07:28:08.656 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 podman[73865]: 2024-05-27 07:28:08.305802743 +0000 UTC m=+0.035392590 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:08.656 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 podman[73865]: 2024-05-27 07:28:08.411186667 +0000 UTC m=+0.140776559 container create 8e93f819e7358cacbe562a60f17ab6a8763ee9f7066dc458c2a489d6082533fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:28:08.656 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 podman[73865]: 2024-05-27 07:28:08.665739722 +0000 UTC m=+0.395329691 container init 8e93f819e7358cacbe562a60f17ab6a8763ee9f7066dc458c2a489d6082533fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T07:28:08.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:08 smithi062 ceph-mon[35304]: pgmap v199: 129 pgs: 129 active+clean; 91 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 964 KiB/s wr, 134 op/s 2024-05-27T07:28:08.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:08 smithi012 ceph-mon[31473]: pgmap v199: 129 pgs: 129 active+clean; 91 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 964 KiB/s wr, 134 op/s 2024-05-27T07:28:08.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:08 smithi012 podman[120632]: 2024-05-27 07:28:08.457813864 +0000 UTC m=+0.786860419 container remove 9b8e3aeb8130f6474e2569baf05b8daad531e3e463383e44d503fc6bf9b118ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:28:08.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:08 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:08.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 podman[73865]: 2024-05-27 07:28:08.677420978 +0000 UTC m=+0.407010830 container start 8e93f819e7358cacbe562a60f17ab6a8763ee9f7066dc458c2a489d6082533fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:28:08.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 bash[73865]: 8e93f819e7358cacbe562a60f17ab6a8763ee9f7066dc458c2a489d6082533fe 2024-05-27T07:28:08.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 systemd[1]: Started Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:08.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 ceph-mds[73884]: -- 172.21.15.162:0/1697640290 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5591d1537a40 con 0x5591d157f180 2024-05-27T07:28:08.974 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: starting mds.l at 2024-05-27T07:28:09.340 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:08 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:28:09.340 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:08 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.639s CPU time. 2024-05-27T07:28:09.701 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:09 smithi162 ceph-mon[35676]: daemon mds.l restarted 2024-05-27T07:28:09.701 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa9a7800700 time 2024-05-27T07:28:09.584176+0000 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9bd25aba6] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: clone() 2024-05-27T07:28:09.703 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.586+0000 7fa9a7800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa9a7800700 time 2024-05-27T07:28:09.584176+0000 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9bd25aba6] 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.704 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: clone() 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.705 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: *** Caught signal (Aborted) ** 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: in thread 7fa9a7800700 thread_name:md_log_replay 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa9bbdd7d20] 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: gsignal() 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: abort() 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9bd25aced] 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.706 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 19: clone() 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.590+0000 7fa9a7800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: in thread 7fa9a7800700 thread_name:md_log_replay 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa9bbdd7d20] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: gsignal() 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: abort() 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9bd25aced] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.708 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.709 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.709 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.709 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.709 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.709 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 19: clone() 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: -1> 2024-05-27T07:28:09.586+0000 7fa9a7800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa9a7800700 time 2024-05-27T07:28:09.584176+0000 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9bd25aba6] 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.710 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.711 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.711 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.711 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.711 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.711 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.711 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: clone() 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 0> 2024-05-27T07:28:09.590+0000 7fa9a7800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: in thread 7fa9a7800700 thread_name:md_log_replay 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.712 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa9bbdd7d20] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: gsignal() 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: abort() 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9bd25aced] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.713 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 19: clone() 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: -1> 2024-05-27T07:28:09.586+0000 7fa9a7800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa9a7800700 time 2024-05-27T07:28:09.584176+0000 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9bd25aba6] 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.714 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: clone() 2024-05-27T07:28:09.715 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 0> 2024-05-27T07:28:09.590+0000 7fa9a7800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: in thread 7fa9a7800700 thread_name:md_log_replay 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa9bbdd7d20] 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2: gsignal() 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 3: abort() 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9bd25aced] 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9bd25add1] 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5591c1e2423d] 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5591c1e0f971] 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 8: (SnapRealm::check_cache() const+0x13e) [0x5591c1e0dfa0] 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5591c1e0e651] 2024-05-27T07:28:09.716 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5591c1da3347] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5591c1a9f87e] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5591c1aa4e94] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5591c1aaf969] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 14: (MDLog::_replay_thread()+0x1f38) [0x5591c1e4848c] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5591c1e519cb] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 16: (Thread::entry_wrapper()+0x43) [0x7fa9bd22972d] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa9bd229749] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa9bbdcd1ca] 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 19: clone() 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[73880]: 2024-05-27T07:28:09.717 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:09 smithi162 podman[73915]: 2024-05-27 07:28:09.711795294 +0000 UTC m=+0.035641146 container died 8e93f819e7358cacbe562a60f17ab6a8763ee9f7066dc458c2a489d6082533fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:28:09.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:09 smithi062 ceph-mon[35304]: daemon mds.l restarted 2024-05-27T07:28:09.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:09 smithi012 ceph-mon[31473]: daemon mds.l restarted 2024-05-27T07:28:10.695 INFO:tasks.workunit.client.0.smithi012.stdout:fallocating to largest ever: 0x100000 2024-05-27T07:28:10.722 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:10 smithi162 podman[73915]: 2024-05-27 07:28:10.462553649 +0000 UTC m=+0.786399560 container remove 8e93f819e7358cacbe562a60f17ab6a8763ee9f7066dc458c2a489d6082533fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:28:10.723 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:10 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:10.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:10 smithi162 ceph-mon[35676]: pgmap v200: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-05-27T07:28:10.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:10 smithi162 ceph-mon[35676]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:28:10.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:10 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/3309517807,v1:172.21.15.162:6833/3309517807] up:boot 2024-05-27T07:28:10.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:10 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:10.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:28:10.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:10 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:10.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:10 smithi062 ceph-mon[35304]: pgmap v200: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-05-27T07:28:10.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:10 smithi062 ceph-mon[35304]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:28:10.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:10 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/3309517807,v1:172.21.15.162:6833/3309517807] up:boot 2024-05-27T07:28:10.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:10 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:10.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:28:10.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:10 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:10.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:10 smithi012 ceph-mon[31473]: pgmap v200: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-05-27T07:28:10.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:10 smithi012 ceph-mon[31473]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:28:10.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:10 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/3309517807,v1:172.21.15.162:6833/3309517807] up:boot 2024-05-27T07:28:10.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:10 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:10.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:28:10.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:10 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:11.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:10 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:28:11.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:10 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.651s CPU time. 2024-05-27T07:28:12.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:12 smithi162 ceph-mon[35676]: pgmap v202: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-05-27T07:28:12.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:12 smithi062 ceph-mon[35304]: pgmap v202: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-05-27T07:28:12.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:12 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:28:12.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:12 smithi012 systemd[1]: Stopped Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:12.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:12 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.729s CPU time. 2024-05-27T07:28:12.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:12 smithi012 systemd[1]: Starting Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:12.763 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:12 smithi012 ceph-mon[31473]: pgmap v202: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.9 MiB/s wr, 245 op/s 2024-05-27T07:28:13.090 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:12 smithi012 podman[120719]: 2024-05-27 07:28:12.679842811 +0000 UTC m=+0.038016716 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:13.090 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:12 smithi012 podman[120719]: 2024-05-27 07:28:12.816750397 +0000 UTC m=+0.174924324 container create 186546c862508fe5bcb982a1e196765117fcebef5c4b4f7122ea553f5f16f81b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:28:13.404 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 podman[120719]: 2024-05-27 07:28:13.112661744 +0000 UTC m=+0.470835642 container init 186546c862508fe5bcb982a1e196765117fcebef5c4b4f7122ea553f5f16f81b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:28:13.404 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 podman[120719]: 2024-05-27 07:28:13.125289564 +0000 UTC m=+0.483463467 container start 186546c862508fe5bcb982a1e196765117fcebef5c4b4f7122ea553f5f16f81b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:28:13.404 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 bash[120719]: 186546c862508fe5bcb982a1e196765117fcebef5c4b4f7122ea553f5f16f81b 2024-05-27T07:28:13.404 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 systemd[1]: Started Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:13.404 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-mds[120737]: -- 172.21.15.12:0/506419236 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624ee261a40 con 0x5624ee2a9600 2024-05-27T07:28:13.404 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: starting mds.g at 2024-05-27T07:28:13.664 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:13 smithi012 ceph-mon[31473]: daemon mds.g restarted 2024-05-27T07:28:13.666 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff8c7c00700 time 2024-05-27T07:28:13.617772+0000 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff8dd62eba6] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.667 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: clone() 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.619+0000 7ff8c7c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff8c7c00700 time 2024-05-27T07:28:13.617772+0000 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff8dd62eba6] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.668 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.669 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: clone() 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: *** Caught signal (Aborted) ** 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: in thread 7ff8c7c00700 thread_name:md_log_replay 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff8dc1abd20] 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: gsignal() 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: abort() 2024-05-27T07:28:13.670 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff8dd62eced] 2024-05-27T07:28:13.671 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.671 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.671 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.671 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.671 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.671 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.671 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 19: clone() 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.622+0000 7ff8c7c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: in thread 7ff8c7c00700 thread_name:md_log_replay 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff8dc1abd20] 2024-05-27T07:28:13.672 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: gsignal() 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: abort() 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff8dd62eced] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.673 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 19: clone() 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: -1> 2024-05-27T07:28:13.619+0000 7ff8c7c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff8c7c00700 time 2024-05-27T07:28:13.617772+0000 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff8dd62eba6] 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.674 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: clone() 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 0> 2024-05-27T07:28:13.622+0000 7ff8c7c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:13.675 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: in thread 7ff8c7c00700 thread_name:md_log_replay 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff8dc1abd20] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: gsignal() 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: abort() 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff8dd62eced] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.676 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 19: clone() 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:13.677 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: -1> 2024-05-27T07:28:13.619+0000 7ff8c7c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff8c7c00700 time 2024-05-27T07:28:13.617772+0000 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff8dd62eba6] 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.678 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: clone() 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 0> 2024-05-27T07:28:13.622+0000 7ff8c7c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: in thread 7ff8c7c00700 thread_name:md_log_replay 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff8dc1abd20] 2024-05-27T07:28:13.679 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2: gsignal() 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 3: abort() 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff8dd62eced] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff8dd62edd1] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5624b015323d] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5624b013e971] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 8: (SnapRealm::check_cache() const+0x13e) [0x5624b013cfa0] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5624b013d651] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5624b00d2347] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5624afdce87e] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5624afdd3e94] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5624afdde969] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 14: (MDLog::_replay_thread()+0x1f38) [0x5624b017748c] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5624b01809cb] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 16: (Thread::entry_wrapper()+0x43) [0x7ff8dd5fd72d] 2024-05-27T07:28:13.680 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff8dd5fd749] 2024-05-27T07:28:13.681 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff8dc1a11ca] 2024-05-27T07:28:13.681 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 19: clone() 2024-05-27T07:28:13.681 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:13.681 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[120733]: 2024-05-27T07:28:13.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:13 smithi162 ceph-mon[35676]: daemon mds.g restarted 2024-05-27T07:28:13.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:13 smithi062 ceph-mon[35304]: daemon mds.g restarted 2024-05-27T07:28:14.087 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:13 smithi012 podman[120768]: 2024-05-27 07:28:13.764732316 +0000 UTC m=+0.038927318 container died 186546c862508fe5bcb982a1e196765117fcebef5c4b4f7122ea553f5f16f81b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:28:14.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:14 smithi162 ceph-mon[35676]: pgmap v203: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 942 KiB/s wr, 111 op/s 2024-05-27T07:28:14.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:14 smithi162 ceph-mon[35676]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:28:14.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:14 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/1014325092,v1:172.21.15.12:6839/1014325092] up:boot 2024-05-27T07:28:14.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:14 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:14 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:14 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:14.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:14 smithi062 ceph-mon[35304]: pgmap v203: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 942 KiB/s wr, 111 op/s 2024-05-27T07:28:14.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:14 smithi062 ceph-mon[35304]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:28:14.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:14 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/1014325092,v1:172.21.15.12:6839/1014325092] up:boot 2024-05-27T07:28:14.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:14 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:14.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:14 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:14.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:14 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:14.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:14 smithi012 ceph-mon[31473]: pgmap v203: 129 pgs: 129 active+clean; 93 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 942 KiB/s wr, 111 op/s 2024-05-27T07:28:14.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:14 smithi012 ceph-mon[31473]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:28:14.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:14 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/1014325092,v1:172.21.15.12:6839/1014325092] up:boot 2024-05-27T07:28:14.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:14 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:14.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:14 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:14.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:14 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:14.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:14 smithi012 podman[120768]: 2024-05-27 07:28:14.468696083 +0000 UTC m=+0.742891136 container remove 186546c862508fe5bcb982a1e196765117fcebef5c4b4f7122ea553f5f16f81b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:28:14.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:14 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:15.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:14 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:15.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:14 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.657s CPU time. 2024-05-27T07:28:16.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:16 smithi162 ceph-mon[35676]: pgmap v205: 129 pgs: 129 active+clean; 93 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-05-27T07:28:16.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:16 smithi062 ceph-mon[35304]: pgmap v205: 129 pgs: 129 active+clean; 93 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-05-27T07:28:16.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:16 smithi012 ceph-mon[31473]: pgmap v205: 129 pgs: 129 active+clean; 93 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-05-27T07:28:17.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:17 smithi162 ceph-mon[35676]: pgmap v206: 129 pgs: 129 active+clean; 93 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1004 KiB/s wr, 103 op/s 2024-05-27T07:28:17.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:17 smithi062 ceph-mon[35304]: pgmap v206: 129 pgs: 129 active+clean; 93 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1004 KiB/s wr, 103 op/s 2024-05-27T07:28:17.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:17 smithi012 ceph-mon[31473]: pgmap v206: 129 pgs: 129 active+clean; 93 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1004 KiB/s wr, 103 op/s 2024-05-27T07:28:19.522 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:19 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:28:19.523 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:19 smithi012 systemd[1]: Stopped Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:19.523 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:19 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.639s CPU time. 2024-05-27T07:28:19.523 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:19 smithi012 systemd[1]: Starting Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:19.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:19 smithi012 podman[120866]: 2024-05-27 07:28:19.436515844 +0000 UTC m=+0.039760003 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:19.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:19 smithi012 podman[120866]: 2024-05-27 07:28:19.569235081 +0000 UTC m=+0.172479219 container create e9655eca477ac1a6d8e699ebc66c901e626681b637fa9db14d18eeb2b9f626b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:28:19.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:19 smithi012 ceph-mon[31473]: pgmap v207: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-27T07:28:19.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:19 smithi162 ceph-mon[35676]: pgmap v207: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-27T07:28:19.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:19 smithi062 ceph-mon[35304]: pgmap v207: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-27T07:28:20.560 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:20 smithi012 podman[120866]: 2024-05-27 07:28:20.116569677 +0000 UTC m=+0.719813921 container init e9655eca477ac1a6d8e699ebc66c901e626681b637fa9db14d18eeb2b9f626b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:28:20.561 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:20 smithi012 podman[120866]: 2024-05-27 07:28:20.130716269 +0000 UTC m=+0.733960445 container start e9655eca477ac1a6d8e699ebc66c901e626681b637fa9db14d18eeb2b9f626b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:28:20.561 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:20 smithi012 bash[120866]: e9655eca477ac1a6d8e699ebc66c901e626681b637fa9db14d18eeb2b9f626b1 2024-05-27T07:28:20.561 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:20 smithi012 ceph-mds[120886]: -- 172.21.15.12:0/1288588102 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557639f85a40 con 0x557639fcd180 2024-05-27T07:28:20.561 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: starting mds.a at 2024-05-27T07:28:20.561 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:20 smithi012 systemd[1]: Started Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:20.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:20 smithi012 ceph-mon[31473]: daemon mds.a restarted 2024-05-27T07:28:20.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:20 smithi162 ceph-mon[35676]: daemon mds.a restarted 2024-05-27T07:28:20.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:20 smithi062 ceph-mon[35304]: daemon mds.a restarted 2024-05-27T07:28:21.374 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:20 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:28:21.374 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:20 smithi162 systemd[1]: Stopped Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:21.374 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:20 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.651s CPU time. 2024-05-27T07:28:21.375 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 systemd[1]: Starting Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:21.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:21 smithi162 ceph-mon[35676]: pgmap v208: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:21 smithi162 ceph-mon[35676]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:21 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/1730848646,v1:172.21.15.12:6839/1730848646] up:boot 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:21 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:21 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:21 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 podman[74006]: 2024-05-27 07:28:21.29088838 +0000 UTC m=+0.035910757 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 podman[74006]: 2024-05-27 07:28:21.416059676 +0000 UTC m=+0.161082090 container create 9ce05056aa3a57caee22370d05442b918fd248ce6e2fcae2ad78c227877200b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:28:21.675 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 podman[74006]: 2024-05-27 07:28:21.68499797 +0000 UTC m=+0.430020424 container init 9ce05056aa3a57caee22370d05442b918fd248ce6e2fcae2ad78c227877200b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T07:28:21.838 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb779200700 time 2024-05-27T07:28:21.806029+0000 2024-05-27T07:28:21.838 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb78ec2bba6] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: clone() 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: *** Caught signal (Aborted) ** 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: in thread 7fb779200700 thread_name:md_log_replay 2024-05-27T07:28:21.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-mon[31473]: pgmap v208: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-mon[31473]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/1730848646,v1:172.21.15.12:6839/1730848646] up:boot 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.808+0000 7fb779200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb779200700 time 2024-05-27T07:28:21.806029+0000 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:21.841 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb78ec2bba6] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:21.842 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: clone() 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb78d7a8d20] 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: gsignal() 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: abort() 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb78ec2bced] 2024-05-27T07:28:21.843 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:21.844 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 19: clone() 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.811+0000 7fb779200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: in thread 7fb779200700 thread_name:md_log_replay 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb78d7a8d20] 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: gsignal() 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: abort() 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb78ec2bced] 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:21.845 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:21.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:21.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:21.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:21.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:21.846 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 19: clone() 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: -1> 2024-05-27T07:28:21.808+0000 7fb779200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb779200700 time 2024-05-27T07:28:21.806029+0000 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb78ec2bba6] 2024-05-27T07:28:21.847 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:21.848 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: clone() 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 0> 2024-05-27T07:28:21.811+0000 7fb779200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: in thread 7fb779200700 thread_name:md_log_replay 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb78d7a8d20] 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: gsignal() 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: abort() 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb78ec2bced] 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:21.849 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 19: clone() 2024-05-27T07:28:21.850 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:21.851 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:21.941 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 podman[74006]: 2024-05-27 07:28:21.697046462 +0000 UTC m=+0.442068880 container start 9ce05056aa3a57caee22370d05442b918fd248ce6e2fcae2ad78c227877200b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:28:21.941 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 ceph-mds[74025]: -- 172.21.15.162:0/830071750 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55721d751a40 con 0x55721d799180 2024-05-27T07:28:21.941 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: starting mds.l at 2024-05-27T07:28:21.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:21 smithi062 ceph-mon[35304]: pgmap v208: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-05-27T07:28:21.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:21 smithi062 ceph-mon[35304]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:28:21.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:21 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/1730848646,v1:172.21.15.12:6839/1730848646] up:boot 2024-05-27T07:28:21.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:21 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:21.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:21 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:21.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:21 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:22.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: -1> 2024-05-27T07:28:21.808+0000 7fb779200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb779200700 time 2024-05-27T07:28:21.806029+0000 2024-05-27T07:28:22.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:22.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:22.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb78ec2bba6] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:22.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: clone() 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 0> 2024-05-27T07:28:21.811+0000 7fb779200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: in thread 7fb779200700 thread_name:md_log_replay 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb78d7a8d20] 2024-05-27T07:28:22.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2: gsignal() 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 3: abort() 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb78ec2bced] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb78ec2bdd1] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55761751b23d] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557617506971] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 8: (SnapRealm::check_cache() const+0x13e) [0x557617504fa0] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557617505651] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55761749a347] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55761719687e] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55761719be94] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576171a6969] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 14: (MDLog::_replay_thread()+0x1f38) [0x55761753f48c] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576175489cb] 2024-05-27T07:28:22.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 16: (Thread::entry_wrapper()+0x43) [0x7fb78ebfa72d] 2024-05-27T07:28:22.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb78ebfa749] 2024-05-27T07:28:22.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb78d79e1ca] 2024-05-27T07:28:22.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 19: clone() 2024-05-27T07:28:22.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:22.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[120881]: 2024-05-27T07:28:22.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:21 smithi012 podman[120917]: 2024-05-27 07:28:21.944992145 +0000 UTC m=+0.037473358 container died e9655eca477ac1a6d8e699ebc66c901e626681b637fa9db14d18eeb2b9f626b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True) 2024-05-27T07:28:22.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 bash[74006]: 9ce05056aa3a57caee22370d05442b918fd248ce6e2fcae2ad78c227877200b0 2024-05-27T07:28:22.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:21 smithi162 systemd[1]: Started Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:22.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:22 smithi162 ceph-mon[35676]: daemon mds.l restarted 2024-05-27T07:28:22.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:22 smithi162 ceph-mon[35676]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:28:22.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:22 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.162:6832/3323642950,v1:172.21.15.162:6833/3323642950] up:boot 2024-05-27T07:28:22.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:22 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:22.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:22 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb25b400700 time 2024-05-27T07:28:22.419774+0000 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb270ec1ba6] 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.725 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: clone() 2024-05-27T07:28:22.726 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.421+0000 7fb25b400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb25b400700 time 2024-05-27T07:28:22.419774+0000 2024-05-27T07:28:22.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:22.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb270ec1ba6] 2024-05-27T07:28:22.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.727 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: clone() 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.728 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: *** Caught signal (Aborted) ** 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: in thread 7fb25b400700 thread_name:md_log_replay 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26fa3ed20] 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: gsignal() 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: abort() 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb270ec1ced] 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.730 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 19: clone() 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.424+0000 7fb25b400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: in thread 7fb25b400700 thread_name:md_log_replay 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26fa3ed20] 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: gsignal() 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: abort() 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb270ec1ced] 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.731 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.732 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 19: clone() 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: -1> 2024-05-27T07:28:22.421+0000 7fb25b400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb25b400700 time 2024-05-27T07:28:22.419774+0000 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb270ec1ba6] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.733 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: clone() 2024-05-27T07:28:22.734 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 0> 2024-05-27T07:28:22.424+0000 7fb25b400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: in thread 7fb25b400700 thread_name:md_log_replay 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26fa3ed20] 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: gsignal() 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: abort() 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb270ec1ced] 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.735 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.736 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 19: clone() 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: -1> 2024-05-27T07:28:22.421+0000 7fb25b400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb25b400700 time 2024-05-27T07:28:22.419774+0000 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb270ec1ba6] 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.737 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: clone() 2024-05-27T07:28:22.738 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 0> 2024-05-27T07:28:22.424+0000 7fb25b400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: in thread 7fb25b400700 thread_name:md_log_replay 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26fa3ed20] 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2: gsignal() 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 3: abort() 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb270ec1ced] 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb270ec1dd1] 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fb4c723d] 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fb4b2971] 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fb4b0fa0] 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fb4b1651] 2024-05-27T07:28:22.739 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fb446347] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fb14287e] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fb147e94] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fb152969] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fb4eb48c] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fb4f49cb] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 16: (Thread::entry_wrapper()+0x43) [0x7fb270e9072d] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb270e90749] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb26fa341ca] 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 19: clone() 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l[74021]: 2024-05-27T07:28:22.740 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:22 smithi162 podman[74056]: 2024-05-27 07:28:22.535084688 +0000 UTC m=+0.032816210 container died 9ce05056aa3a57caee22370d05442b918fd248ce6e2fcae2ad78c227877200b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:22 smithi012 ceph-mon[31473]: daemon mds.l restarted 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:22 smithi012 ceph-mon[31473]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:22 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.162:6832/3323642950,v1:172.21.15.162:6833/3323642950] up:boot 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:22 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:22 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:28:22.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:22 smithi062 ceph-mon[35304]: daemon mds.l restarted 2024-05-27T07:28:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:22 smithi062 ceph-mon[35304]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:28:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:22 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.162:6832/3323642950,v1:172.21.15.162:6833/3323642950] up:boot 2024-05-27T07:28:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:22 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:22 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:28:23.340 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:23 smithi012 podman[120917]: 2024-05-27 07:28:23.015072034 +0000 UTC m=+1.107553297 container remove e9655eca477ac1a6d8e699ebc66c901e626681b637fa9db14d18eeb2b9f626b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-27T07:28:23.340 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:23 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:23.388 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:28:23.723 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:23 smithi162 podman[74056]: 2024-05-27 07:28:23.272116265 +0000 UTC m=+0.769847788 container remove 9ce05056aa3a57caee22370d05442b918fd248ce6e2fcae2ad78c227877200b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:28:23.723 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:23 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:23.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:23 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:28:23.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:23 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.686s CPU time. 2024-05-27T07:28:23.884 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:24.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:23 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:28:24.223 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:23 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.623s CPU time. 2024-05-27T07:28:24.293 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:24 smithi012 ceph-mon[31473]: pgmap v211: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 831 KiB/s wr, 102 op/s 2024-05-27T07:28:24.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:24 smithi162 ceph-mon[35676]: pgmap v211: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 831 KiB/s wr, 102 op/s 2024-05-27T07:28:24.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:24 smithi062 ceph-mon[35304]: pgmap v211: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 831 KiB/s wr, 102 op/s 2024-05-27T07:28:25.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:25 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:28:25.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:25 smithi012 systemd[1]: Stopped Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:25.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:25 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.657s CPU time. 2024-05-27T07:28:25.589 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:25 smithi012 systemd[1]: Starting Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:26.372 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:26 smithi012 ceph-mon[31473]: pgmap v212: 129 pgs: 129 active+clean; 95 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-05-27T07:28:26.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:26 smithi162 ceph-mon[35676]: pgmap v212: 129 pgs: 129 active+clean; 95 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-05-27T07:28:26.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:26 smithi062 ceph-mon[35304]: pgmap v212: 129 pgs: 129 active+clean; 95 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-05-27T07:28:26.839 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:26 smithi012 podman[121098]: 2024-05-27 07:28:26.288906907 +0000 UTC m=+0.043490088 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:27.590 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:27 smithi012 podman[121098]: 2024-05-27 07:28:27.345406976 +0000 UTC m=+1.099990178 container create f6bfe6662b6a7540d494905ef1ef64a7b65410398107eaa7bd0797a429a9bc86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:28:27.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:27 smithi012 podman[121098]: 2024-05-27 07:28:27.854338269 +0000 UTC m=+1.608921470 container init f6bfe6662b6a7540d494905ef1ef64a7b65410398107eaa7bd0797a429a9bc86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T07:28:28.168 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:27 smithi012 podman[121098]: 2024-05-27 07:28:27.864516959 +0000 UTC m=+1.619100148 container start f6bfe6662b6a7540d494905ef1ef64a7b65410398107eaa7bd0797a429a9bc86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:28:28.168 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:27 smithi012 bash[121098]: f6bfe6662b6a7540d494905ef1ef64a7b65410398107eaa7bd0797a429a9bc86 2024-05-27T07:28:28.168 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:27 smithi012 systemd[1]: Started Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:28.168 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-mds[121139]: -- 172.21.15.12:0/3461973569 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ced60da40 con 0x562ced655180 2024-05-27T07:28:28.168 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: starting mds.g at 2024-05-27T07:28:28.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 -- 172.21.15.12:0/2079223267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dfc108f10 msgr2=0x7f0dfc109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 --2- 172.21.15.12:0/2079223267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dfc108f10 0x7f0dfc109320 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0de80099c0 tx=0x7f0de8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 -- 172.21.15.12:0/2079223267 shutdown_connections 2024-05-27T07:28:28.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 --2- 172.21.15.12:0/2079223267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dfc109860 0x7f0dfc1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 --2- 172.21.15.12:0/2079223267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 0x7f0dfc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 --2- 172.21.15.12:0/2079223267 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dfc108f10 0x7f0dfc109320 secure :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0de80099c0 tx=0x7f0de8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 -- 172.21.15.12:0/2079223267 >> 172.21.15.12:0/2079223267 conn(0x7f0dfc100030 msgr2=0x7f0dfc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 -- 172.21.15.12:0/2079223267 shutdown_connections 2024-05-27T07:28:28.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.361+0000 7f0e01200700 1 -- 172.21.15.12:0/2079223267 wait complete. 2024-05-27T07:28:28.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.363+0000 7f0e01200700 1 Processor -- start 2024-05-27T07:28:28.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.363+0000 7f0e01200700 1 -- start start 2024-05-27T07:28:28.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.364+0000 7f0e01200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 0x7f0dfc198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.364+0000 7f0e01200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dfc109860 0x7f0dfc199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.364+0000 7f0e01200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dfc1995b0 0x7f0dfc19d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.364+0000 7f0e01200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112210 con 0x7f0dfc109860 2024-05-27T07:28:28.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.364+0000 7f0e01200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112090 con 0x7f0dfc1995b0 2024-05-27T07:28:28.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.364+0000 7f0e01200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112390 con 0x7f0dfc1045e0 2024-05-27T07:28:28.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.364+0000 7f0df9e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 0x7f0dfc198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.365+0000 7f0df9e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 0x7f0dfc198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:53620/0 (socket says 172.21.15.12:53620) 2024-05-27T07:28:28.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.365+0000 7f0df9e00700 1 -- 172.21.15.12:0/521548114 learned_addr learned my addr 172.21.15.12:0/521548114 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:28.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.365+0000 7f0dfa800700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dfc1995b0 0x7f0dfc19d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.366+0000 7f0df9e00700 1 -- 172.21.15.12:0/521548114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dfc1995b0 msgr2=0x7f0dfc19d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.366+0000 7f0df9e00700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dfc1995b0 0x7f0dfc19d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.366+0000 7f0df9e00700 1 -- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dfc109860 msgr2=0x7f0dfc199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.366+0000 7f0df9e00700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dfc109860 0x7f0dfc199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.366+0000 7f0df9e00700 1 -- 172.21.15.12:0/521548114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc19da20 con 0x7f0dfc1045e0 2024-05-27T07:28:28.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.366+0000 7f0df9e00700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 0x7f0dfc198b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00b530 tx=0x7f0dec00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.367+0000 7f0df2a00700 1 -- 172.21.15.12:0/521548114 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec015020 con 0x7f0dfc1045e0 2024-05-27T07:28:28.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.367+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc19dd10 con 0x7f0dfc1045e0 2024-05-27T07:28:28.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.367+0000 7f0df2a00700 1 -- 172.21.15.12:0/521548114 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dec011cc0 con 0x7f0dfc1045e0 2024-05-27T07:28:28.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.367+0000 7f0df2a00700 1 -- 172.21.15.12:0/521548114 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec010dc0 con 0x7f0dfc1045e0 2024-05-27T07:28:28.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.367+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc19e250 con 0x7f0dfc1045e0 2024-05-27T07:28:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.373+0000 7f0e0fd71b80 1 Processor -- start 2024-05-27T07:28:28.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.374+0000 7f0e0fd71b80 1 -- start start 2024-05-27T07:28:28.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.374+0000 7f0df2a00700 1 -- 172.21.15.12:0/521548114 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0dec018020 con 0x7f0dfc1045e0 2024-05-27T07:28:28.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b17797f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1779d30 0x5590b1773870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b17741c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5590b16a1e40 con 0x5590b1779d30 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5590b16919e0 con 0x5590b1773db0 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5590b1695a20 con 0x5590b15f4e40 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0df9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1779d30 0x5590b1773870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0df9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1779d30 0x5590b1773870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35840/0 (socket says 172.21.15.12:35840) 2024-05-27T07:28:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0dfa800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b17797f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.375+0000 7f0df9e00700 1 -- 172.21.15.12:0/4270102111 learned_addr learned my addr 172.21.15.12:0/4270102111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.376+0000 7f0df9e00700 1 -- 172.21.15.12:0/4270102111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 msgr2=0x5590b17797f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.376+0000 7f0df9400700 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b17741c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.376+0000 7f0df9e00700 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b17797f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.376+0000 7f0df9e00700 1 -- 172.21.15.12:0/4270102111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 msgr2=0x5590b17741c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.376+0000 7f0df9e00700 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b17741c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.376+0000 7f0df9e00700 1 -- 172.21.15.12:0/4270102111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590b165a420 con 0x5590b1779d30 2024-05-27T07:28:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.376+0000 7f0df9e00700 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1779d30 0x5590b1773870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0df470 tx=0x7f0dec0ea5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948f8cd66692dd6f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.377+0000 7f0df2a00700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0de00b29d0 0x7f0de00b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.377+0000 7f0de7400700 1 -- 172.21.15.12:0/4270102111 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0f5070 con 0x5590b1779d30 2024-05-27T07:28:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.377+0000 7f0de7400700 1 -- 172.21.15.12:0/4270102111 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dec0a8ba0 con 0x5590b1779d30 2024-05-27T07:28:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.377+0000 7f0de7400700 1 -- 172.21.15.12:0/4270102111 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0a89e0 con 0x5590b1779d30 2024-05-27T07:28:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.378+0000 7f0df2a00700 1 -- 172.21.15.12:0/521548114 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9466+0+0 (secure 0 0 0) 0x7f0dec0e2050 con 0x7f0dfc1045e0 2024-05-27T07:28:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.378+0000 7f0dfa800700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0de00b29d0 0x7f0de00b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.380+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/4270102111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1779d30 msgr2=0x5590b1773870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.380+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1779d30 0x5590b1773870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0df470 tx=0x7f0dec0ea5d0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.380+0000 7f0dfa800700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0de00b29d0 0x7f0de00b4ee0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5590b1774b30 tx=0x7f0df4009100 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.381+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/4270102111 shutdown_connections 2024-05-27T07:28:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.381+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b17741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.381+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b17797f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.381+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/4270102111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1779d30 0x5590b1773870 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.381+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/4270102111 >> 172.21.15.12:0/4270102111 conn(0x5590b1654fc0 msgr2=0x5590b176e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.381+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/4270102111 shutdown_connections 2024-05-27T07:28:28.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.381+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/4270102111 wait complete. 2024-05-27T07:28:28.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.382+0000 7f0e0fd71b80 1 -- start start 2024-05-27T07:28:28.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.382+0000 7f0e0fd71b80 1 Processor -- start 2024-05-27T07:28:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.384+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b177d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.384+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b177d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.384+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1779d30 0x5590b177df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.384+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5590b1597fa0 con 0x5590b1773db0 2024-05-27T07:28:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.384+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5590b1632760 con 0x5590b1779d30 2024-05-27T07:28:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.384+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5590b1780680 con 0x5590b15f4e40 2024-05-27T07:28:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.384+0000 7f0e0fd71b80 10 client.0 _ll_register_callbacks cb 0x5590b163bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.385+0000 7f0df9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b177d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.385+0000 7f0df9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b177d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35850/0 (socket says 172.21.15.12:35850) 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.385+0000 7f0df9400700 1 -- 172.21.15.12:0/163612483 learned_addr learned my addr 172.21.15.12:0/163612483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.385+0000 7f0dfa800700 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1779d30 0x5590b177df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.385+0000 7f0df9e00700 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b177d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.386+0000 7f0df9400700 1 -- 172.21.15.12:0/163612483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 msgr2=0x5590b177d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.386+0000 7f0df9400700 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b177d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.386+0000 7f0df9400700 1 -- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1779d30 msgr2=0x5590b177df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.386+0000 7f0df9400700 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1779d30 0x5590b177df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.386+0000 7f0df9400700 1 -- 172.21.15.12:0/163612483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590b15e18d0 con 0x5590b1773db0 2024-05-27T07:28:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.386+0000 7f0df9400700 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b177d9f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009990 tx=0x7f0de800bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d39b9cbdcc46ec8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.386+0000 7f0ddbe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.387+0000 7f0ddbe00700 1 -- 172.21.15.12:0/163612483 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de8025060 con 0x5590b1773db0 2024-05-27T07:28:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.387+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dec0a8ba0 con 0x5590b1773db0 2024-05-27T07:28:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.387+0000 7f0ddbe00700 1 -- 172.21.15.12:0/163612483 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9466+0+0 (secure 0 0 0) 0x7f0de8005070 con 0x5590b1773db0 2024-05-27T07:28:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.388+0000 7f0ddbe00700 1 -- 172.21.15.12:0/163612483 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f0de802c050 con 0x5590b1773db0 2024-05-27T07:28:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.388+0000 7f0e0fd71b80 10 client.16173 fetch_fsmap learned FSMap version 87 2024-05-27T07:28:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.388+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x5590b1662060 con 0x5590b1773db0 2024-05-27T07:28:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.390+0000 7f0ddbe00700 1 -- 172.21.15.12:0/163612483 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dec0a8ba0 con 0x5590b1773db0 2024-05-27T07:28:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.391+0000 7f0ddbe00700 1 -- 172.21.15.12:0/163612483 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 87) ==== 5950+0+0 (secure 0 0 0) 0x7f0de800f460 con 0x5590b1773db0 2024-05-27T07:28:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.391+0000 7f0e0fd71b80 10 client.16173 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:28:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.391+0000 7f0e0fd71b80 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.391+0000 7f0e0fd71b80 20 client.16173 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.391+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b178d8e0 0x5590b178fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.391+0000 7f0e0fd71b80 4 client.16173 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.391+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590b16a3d90 con 0x5590b178d8e0 2024-05-27T07:28:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.392+0000 7f0df9e00700 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b178d8e0 0x5590b178fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.393+0000 7f0df9e00700 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b178d8e0 0x5590b178fd30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=923db155579664c4 server_cookie=ae938a26f65aa489 in_seq=0 out_seq=0 2024-05-27T07:28:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.393+0000 7f0ddbe00700 10 client.16173 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.396+0000 7f0ddbe00700 1 -- 172.21.15.12:0/163612483 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5590b16a3d90 con 0x5590b178d8e0 2024-05-27T07:28:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.396+0000 7f0ddbe00700 10 client.16173 handle_command_reply: tid=0 2024-05-27T07:28:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.396+0000 7f0e0fd71b80 1 client.16173 shutdown 2024-05-27T07:28:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.397+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 msgr2=0x5590b177d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.397+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b177d9f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009990 tx=0x7f0de800bee0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.398+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 shutdown_connections 2024-05-27T07:28:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.398+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b178d8e0 0x5590b178fd30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.398+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1779d30 0x5590b177df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.398+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b15f4e40 0x5590b177d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.398+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/163612483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b177d9f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.398+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 >> 172.21.15.12:0/163612483 conn(0x5590b1654fc0 msgr2=0x5590b176e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.398+0000 7f0ddbe00700 0 client.16173 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.399+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 shutdown_connections 2024-05-27T07:28:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.400+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/163612483 wait complete. 2024-05-27T07:28:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.401+0000 7f0e0fd71b80 20 client.16173 trim_cache size 0 max 16384 2024-05-27T07:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.446+0000 7f0e0fd71b80 1 Processor -- start 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.446+0000 7f0e0fd71b80 1 -- start start 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.447+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b15f4e40 0x5590b1773870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.447+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b177c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.447+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1777b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.447+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc00e4e0 con 0x5590b15f4e40 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.447+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc0100e0 con 0x5590b1773db0 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.447+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc00ebb0 con 0x5590b1779d30 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.447+0000 7f0df9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b15f4e40 0x5590b1773870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.448+0000 7f0df9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b15f4e40 0x5590b1773870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35862/0 (socket says 172.21.15.12:35862) 2024-05-27T07:28:28.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.448+0000 7f0df9e00700 1 -- 172.21.15.12:0/2907413443 learned_addr learned my addr 172.21.15.12:0/2907413443 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:28 smithi012 ceph-mon[31473]: pgmap v213: 129 pgs: 129 active+clean; 95 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-05-27T07:28:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:28 smithi012 ceph-mon[31473]: daemon mds.g restarted 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.448+0000 7f0df9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b177c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.448+0000 7f0dfa800700 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1777b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.449+0000 7f0df9e00700 1 -- 172.21.15.12:0/2907413443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 msgr2=0x5590b1777b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.449+0000 7f0df9e00700 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1777b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.449+0000 7f0df9e00700 1 -- 172.21.15.12:0/2907413443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 msgr2=0x5590b177c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.449+0000 7f0df9e00700 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b177c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.449+0000 7f0df9e00700 1 -- 172.21.15.12:0/2907413443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590b177ba20 con 0x5590b15f4e40 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.449+0000 7f0df9400700 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b177c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.449+0000 7f0df9e00700 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b15f4e40 0x5590b1773870 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00bc40 tx=0x7f0dec010f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53ab2d2ac7d154 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.450+0000 7f0de6a00700 1 -- 172.21.15.12:0/2907413443 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec04f680 con 0x5590b15f4e40 2024-05-27T07:28:28.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.450+0000 7f0de6a00700 1 -- 172.21.15.12:0/2907413443 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dec0eae70 con 0x5590b15f4e40 2024-05-27T07:28:28.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.450+0000 7f0de6a00700 1 -- 172.21.15.12:0/2907413443 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0a8800 con 0x5590b15f4e40 2024-05-27T07:28:28.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.453+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2907413443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b15f4e40 msgr2=0x5590b1773870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.453+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b15f4e40 0x5590b1773870 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00bc40 tx=0x7f0dec010f60 comp rx=0 tx=0).stop 2024-05-27T07:28:28.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.457+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2907413443 shutdown_connections 2024-05-27T07:28:28.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.457+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1777b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.457+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b15f4e40 0x5590b1773870 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.457+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2907413443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b1773db0 0x5590b177c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.457+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2907413443 >> 172.21.15.12:0/2907413443 conn(0x5590b1654fc0 msgr2=0x5590b177d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.457+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2907413443 shutdown_connections 2024-05-27T07:28:28.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.457+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2907413443 wait complete. 2024-05-27T07:28:28.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.458+0000 7f0e0fd71b80 1 -- start start 2024-05-27T07:28:28.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.458+0000 7f0e0fd71b80 1 Processor -- start 2024-05-27T07:28:28.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.462+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b15f4e40 0x5590b1785b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.462+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b17860d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.462+0000 7f0e0fd71b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1794900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc00fa10 con 0x5590b1773db0 2024-05-27T07:28:28.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc00f2e0 con 0x5590b15f4e40 2024-05-27T07:28:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0e0fd71b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc00cc70 con 0x5590b1779d30 2024-05-27T07:28:28.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0e0fd71b80 10 client.0 _ll_register_callbacks cb 0x5590b163bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:28.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0df9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b17860d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0df9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b15f4e40 0x5590b1785b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0df9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b17860d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35864/0 (socket says 172.21.15.12:35864) 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.463+0000 7f0df9400700 1 -- 172.21.15.12:0/2330814127 learned_addr learned my addr 172.21.15.12:0/2330814127 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.464+0000 7f0dfa800700 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1794900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.464+0000 7f0df9400700 1 -- 172.21.15.12:0/2330814127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 msgr2=0x5590b1794900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.464+0000 7f0df9400700 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1794900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.464+0000 7f0df9400700 1 -- 172.21.15.12:0/2330814127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b15f4e40 msgr2=0x5590b1785b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.464+0000 7f0df9400700 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b15f4e40 0x5590b1785b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.465+0000 7f0df9400700 1 -- 172.21.15.12:0/2330814127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590b17950b0 con 0x5590b1773db0 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.465+0000 7f0dfa800700 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1794900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.465+0000 7f0df9400700 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b17860d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0de800a010 tx=0x7f0de800fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10f5841889cd0d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:28.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.465+0000 7f0ddbe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.466+0000 7f0ddbe00700 1 -- 172.21.15.12:0/2330814127 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de8025070 con 0x5590b1773db0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.466+0000 7f0ddbe00700 1 -- 172.21.15.12:0/2330814127 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9466+0+0 (secure 0 0 0) 0x7f0de800f460 con 0x5590b1773db0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.466+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dec0eae70 con 0x5590b1773db0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.467+0000 7f0ddbe00700 1 -- 172.21.15.12:0/2330814127 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f0de800f660 con 0x5590b1773db0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.467+0000 7f0e0fd71b80 10 client.16179 fetch_fsmap learned FSMap version 87 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.467+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x5590b1808d30 con 0x5590b1773db0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0ddbe00700 1 -- 172.21.15.12:0/2330814127 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dec0eae70 con 0x5590b1773db0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0ddbe00700 1 -- 172.21.15.12:0/2330814127 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 87) ==== 5950+0+0 (secure 0 0 0) 0x7f0de8021bb0 con 0x5590b1773db0 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0e0fd71b80 10 client.16179 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0e0fd71b80 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0e0fd71b80 20 client.16179 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b18093c0 0x5590b180b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:28.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0e0fd71b80 4 client.16179 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:28:28.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.468+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590b17902e0 con 0x5590b18093c0 2024-05-27T07:28:28.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.469+0000 7f0df9e00700 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b18093c0 0x5590b180b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:28.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.470+0000 7f0df9e00700 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b18093c0 0x5590b180b7e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bc8005f60025ed9 server_cookie=448b2cbf88223ac3 in_seq=0 out_seq=0 2024-05-27T07:28:28.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.470+0000 7f0ddbe00700 10 client.16179 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:28.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.471+0000 7f0ddbe00700 1 -- 172.21.15.12:0/2330814127 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5590b17902e0 con 0x5590b18093c0 2024-05-27T07:28:28.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.471+0000 7f0ddbe00700 10 client.16179 handle_command_reply: tid=0 2024-05-27T07:28:28.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.471+0000 7f0e0fd71b80 1 client.16179 shutdown 2024-05-27T07:28:28.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.471+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 msgr2=0x5590b17860d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.471+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b17860d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0de800a010 tx=0x7f0de800fa50 comp rx=0 tx=0).stop 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.471+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 shutdown_connections 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.471+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5590b18093c0 0x5590b180b7e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5590b15f4e40 0x5590b1785b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b1779d30 0x5590b1794900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0e0fd71b80 1 --2- 172.21.15.12:0/2330814127 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5590b1773db0 0x5590b17860d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 >> 172.21.15.12:0/2330814127 conn(0x5590b1654fc0 msgr2=0x5590b17778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0ddbe00700 0 client.16179 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 shutdown_connections 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0e0fd71b80 1 -- 172.21.15.12:0/2330814127 wait complete. 2024-05-27T07:28:28.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.472+0000 7f0e0fd71b80 20 client.16179 trim_cache size 0 max 16384 2024-05-27T07:28:28.463 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:28:28.463 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:28:28.463 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:28:28.463 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:28:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0de00b29d0 msgr2=0x7f0de00b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0de00b29d0 0x7f0de00b4ee0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5590b1774b30 tx=0x7f0df4009100 comp rx=0 tx=0).stop 2024-05-27T07:28:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 msgr2=0x7f0dfc198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 0x7f0dfc198b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00b530 tx=0x7f0dec00b840 comp rx=0 tx=0).stop 2024-05-27T07:28:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 shutdown_connections 2024-05-27T07:28:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f0de00b29d0 0x7f0de00b4ee0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dfc1995b0 0x7f0dfc19d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dfc1045e0 0x7f0dfc198b30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 --2- 172.21.15.12:0/521548114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0dfc109860 0x7f0dfc199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:28.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 >> 172.21.15.12:0/521548114 conn(0x7f0dfc100030 msgr2=0x7f0dfc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:28.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.474+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 shutdown_connections 2024-05-27T07:28:28.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:28.475+0000 7f0e01200700 1 -- 172.21.15.12:0/521548114 wait complete. 2024-05-27T07:28:28.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:28 smithi162 ceph-mon[35676]: pgmap v213: 129 pgs: 129 active+clean; 95 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-05-27T07:28:28.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:28 smithi162 ceph-mon[35676]: daemon mds.g restarted 2024-05-27T07:28:28.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:28 smithi062 ceph-mon[35304]: pgmap v213: 129 pgs: 129 active+clean; 95 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-05-27T07:28:28.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:28 smithi062 ceph-mon[35304]: daemon mds.g restarted 2024-05-27T07:28:28.755 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff7aea00700 time 2024-05-27T07:28:28.523720+0000 2024-05-27T07:28:28.755 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:28.755 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.755 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7c4481ba6] 2024-05-27T07:28:28.755 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.756 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: clone() 2024-05-27T07:28:28.757 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.526+0000 7ff7aea00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff7aea00700 time 2024-05-27T07:28:28.523720+0000 2024-05-27T07:28:28.757 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:28.757 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.757 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7c4481ba6] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.758 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: clone() 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: *** Caught signal (Aborted) ** 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: in thread 7ff7aea00700 thread_name:md_log_replay 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff7c2ffed20] 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: gsignal() 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: abort() 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7c4481ced] 2024-05-27T07:28:28.759 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.760 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 19: clone() 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.529+0000 7ff7aea00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: in thread 7ff7aea00700 thread_name:md_log_replay 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff7c2ffed20] 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: gsignal() 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: abort() 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7c4481ced] 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.761 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.762 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 19: clone() 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: -1> 2024-05-27T07:28:28.526+0000 7ff7aea00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff7aea00700 time 2024-05-27T07:28:28.523720+0000 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7c4481ba6] 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.763 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.764 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.764 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.764 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.764 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: clone() 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 0> 2024-05-27T07:28:28.529+0000 7ff7aea00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: in thread 7ff7aea00700 thread_name:md_log_replay 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff7c2ffed20] 2024-05-27T07:28:28.765 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: gsignal() 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: abort() 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7c4481ced] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.766 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 19: clone() 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: -1> 2024-05-27T07:28:28.526+0000 7ff7aea00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff7aea00700 time 2024-05-27T07:28:28.523720+0000 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff7c4481ba6] 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.767 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: clone() 2024-05-27T07:28:28.768 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 0> 2024-05-27T07:28:28.529+0000 7ff7aea00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: in thread 7ff7aea00700 thread_name:md_log_replay 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff7c2ffed20] 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2: gsignal() 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 3: abort() 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff7c4481ced] 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7c4481dd1] 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ce343123d] 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ce341c971] 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ce341afa0] 2024-05-27T07:28:28.769 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ce341b651] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ce33b0347] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ce30ac87e] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ce30b1e94] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ce30bc969] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ce345548c] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ce345e9cb] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 16: (Thread::entry_wrapper()+0x43) [0x7ff7c445072d] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff7c4450749] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff7c2ff41ca] 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 19: clone() 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121127]: 2024-05-27T07:28:28.770 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:28 smithi012 podman[121227]: 2024-05-27 07:28:28.765425542 +0000 UTC m=+0.144083451 container died f6bfe6662b6a7540d494905ef1ef64a7b65410398107eaa7bd0797a429a9bc86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:28:29.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:29 smithi012 ceph-mon[31473]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:28:29.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:29 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/26871050,v1:172.21.15.12:6839/26871050] up:boot 2024-05-27T07:28:29.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:29 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:29.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:29 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:29.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:29 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:29.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:29 smithi162 ceph-mon[35676]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:28:29.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:29 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/26871050,v1:172.21.15.12:6839/26871050] up:boot 2024-05-27T07:28:29.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:29 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:29.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:29 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:29.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:29 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:29.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:29 smithi062 ceph-mon[35304]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:28:29.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:29 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/26871050,v1:172.21.15.12:6839/26871050] up:boot 2024-05-27T07:28:29.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:29 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:29.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:29 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:29.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:29 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:30.339 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:29 smithi012 podman[121227]: 2024-05-27 07:28:29.906060314 +0000 UTC m=+1.284718240 container remove f6bfe6662b6a7540d494905ef1ef64a7b65410398107eaa7bd0797a429a9bc86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:28:30.339 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:29 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:30.458 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:28:30.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:28:30.699 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:30 smithi012 ceph-mon[31473]: pgmap v215: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-05-27T07:28:30.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:30 smithi162 ceph-mon[35676]: pgmap v215: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-05-27T07:28:30.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:30 smithi062 ceph-mon[35304]: pgmap v215: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 2.0 MiB/s wr, 237 op/s 2024-05-27T07:28:30.980 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:31.089 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:30 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:31.089 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:30 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.756s CPU time. 2024-05-27T07:28:32.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:32 smithi012 ceph-mon[31473]: pgmap v216: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 1.9 MiB/s wr, 229 op/s 2024-05-27T07:28:32.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:32 smithi162 ceph-mon[35676]: pgmap v216: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 1.9 MiB/s wr, 229 op/s 2024-05-27T07:28:32.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:32 smithi062 ceph-mon[35304]: pgmap v216: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 1.9 MiB/s wr, 229 op/s 2024-05-27T07:28:33.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 -- 172.21.15.12:0/3313357843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 msgr2=0x7f9430113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:33.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 --2- 172.21.15.12:0/3313357843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f9430113340 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f942800b0a0 tx=0x7f942800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 -- 172.21.15.12:0/3313357843 shutdown_connections 2024-05-27T07:28:33.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 --2- 172.21.15.12:0/3313357843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f9430113340 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 --2- 172.21.15.12:0/3313357843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f943010c5a0 0x7f943010c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 --2- 172.21.15.12:0/3313357843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9430106560 0x7f9430106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 -- 172.21.15.12:0/3313357843 >> 172.21.15.12:0/3313357843 conn(0x7f9430102040 msgr2=0x7f9430104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:33.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 -- 172.21.15.12:0/3313357843 shutdown_connections 2024-05-27T07:28:33.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.715+0000 7f9437000700 1 -- 172.21.15.12:0/3313357843 wait complete. 2024-05-27T07:28:33.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.717+0000 7f9437000700 1 Processor -- start 2024-05-27T07:28:33.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.718+0000 7f9437000700 1 -- start start 2024-05-27T07:28:33.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.718+0000 7f9437000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9430106560 0x7f9430075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.718+0000 7f9437000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f943006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.718+0000 7f9437000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f943010c5a0 0x7f943006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.718+0000 7f9437000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f94301165d0 con 0x7f9430106eb0 2024-05-27T07:28:33.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.718+0000 7f9437000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9430116450 con 0x7f9430106560 2024-05-27T07:28:33.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.718+0000 7f9437000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9430116750 con 0x7f943010c5a0 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.719+0000 7f942f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f943006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.719+0000 7f942f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f943006e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60448/0 (socket says 172.21.15.12:60448) 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.719+0000 7f942f400700 1 -- 172.21.15.12:0/2664252727 learned_addr learned my addr 172.21.15.12:0/2664252727 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.719+0000 7f9435000700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f943010c5a0 0x7f943006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.719+0000 7f942fe00700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9430106560 0x7f9430075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.720+0000 7f942f400700 1 -- 172.21.15.12:0/2664252727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f943010c5a0 msgr2=0x7f943006ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.720+0000 7f942f400700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f943010c5a0 0x7f943006ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.720+0000 7f942f400700 1 -- 172.21.15.12:0/2664252727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9430106560 msgr2=0x7f9430075320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:33.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.720+0000 7f942f400700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9430106560 0x7f9430075320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.720+0000 7f942f400700 1 -- 172.21.15.12:0/2664252727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943006f2a0 con 0x7f9430106eb0 2024-05-27T07:28:33.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.720+0000 7f942fe00700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9430106560 0x7f9430075320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:33.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.720+0000 7f942f400700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f943006e4a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f941c00e940 tx=0x7f941c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:33.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.721+0000 7f942cc00700 1 -- 172.21.15.12:0/2664252727 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c00cbd0 con 0x7f9430106eb0 2024-05-27T07:28:33.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.721+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9430072f70 con 0x7f9430106eb0 2024-05-27T07:28:33.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.721+0000 7f942cc00700 1 -- 172.21.15.12:0/2664252727 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f941c00cd70 con 0x7f9430106eb0 2024-05-27T07:28:33.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.721+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9430073500 con 0x7f9430106eb0 2024-05-27T07:28:33.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.721+0000 7f942cc00700 1 -- 172.21.15.12:0/2664252727 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c018680 con 0x7f9430106eb0 2024-05-27T07:28:33.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.723+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9430107c80 con 0x7f9430106eb0 2024-05-27T07:28:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.727+0000 7f942cc00700 1 -- 172.21.15.12:0/2664252727 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f941c010430 con 0x7f9430106eb0 2024-05-27T07:28:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.728+0000 7f942cc00700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f94140b2bf0 0x7f94140b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.728+0000 7f942cc00700 1 -- 172.21.15.12:0/2664252727 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9466+0+0 (secure 0 0 0) 0x7f941c018420 con 0x7f9430106eb0 2024-05-27T07:28:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.728+0000 7f942fe00700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f94140b2bf0 0x7f94140b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.729+0000 7f942fe00700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f94140b2bf0 0x7f94140b5100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f943006fca0 tx=0x7f9420009260 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:33.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:33.737+0000 7f942cc00700 1 -- 172.21.15.12:0/2664252727 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f941c0aa080 con 0x7f9430106eb0 2024-05-27T07:28:33.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:33 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:28:33.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:33 smithi012 systemd[1]: Stopped Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:33.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:33 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.686s CPU time. 2024-05-27T07:28:33.839 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:33 smithi012 systemd[1]: Starting Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:34.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.127+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f943006f980 con 0x7f9430106eb0 2024-05-27T07:28:34.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.129+0000 7f942cc00700 1 -- 172.21.15.12:0/2664252727 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+8562 (secure 0 0 0) 0x7f941c0b2100 con 0x7f9430106eb0 2024-05-27T07:28:34.118 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:28:34.119 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2219705283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2219705283},{"type":"v1","addr":"172.21.15.12:6835","nonce":2219705283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/4064394372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":4064394372},{"type":"v1","addr":"172.21.15.12:6837","nonce":4064394372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":87,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:28:28.318875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16164":{"gid":16164,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6839/26871050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":26871050},{"type":"v1","addr":"172.21.15.12:6839","nonce":26871050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25828":{"gid":25828,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3323642950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3323642950},{"type":"v1","addr":"172.21.15.162:6833","nonce":3323642950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_35726":{"gid":35726,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6839/1730848646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":1730848646},{"type":"v1","addr":"172.21.15.12:6839","nonce":1730848646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:28:34.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f94140b2bf0 msgr2=0x7f94140b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:34.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f94140b2bf0 0x7f94140b5100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f943006fca0 tx=0x7f9420009260 comp rx=0 tx=0).stop 2024-05-27T07:28:34.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 msgr2=0x7f943006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:34.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f943006e4a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f941c00e940 tx=0x7f941c00ed00 comp rx=0 tx=0).stop 2024-05-27T07:28:34.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 shutdown_connections 2024-05-27T07:28:34.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f94140b2bf0 0x7f94140b5100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9430106560 0x7f9430075320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f943010c5a0 0x7f943006ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 --2- 172.21.15.12:0/2664252727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9430106eb0 0x7f943006e4a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:34.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.133+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 >> 172.21.15.12:0/2664252727 conn(0x7f9430102040 msgr2=0x7f94301035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:34.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.134+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 shutdown_connections 2024-05-27T07:28:34.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:34.134+0000 7f9437000700 1 -- 172.21.15.12:0/2664252727 wait complete. 2024-05-27T07:28:34.124 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 87 2024-05-27T07:28:34.339 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 podman[121450]: 2024-05-27 07:28:33.928591016 +0000 UTC m=+0.046397056 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:34.340 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 podman[121450]: 2024-05-27 07:28:34.094888849 +0000 UTC m=+0.212694910 container create 145483ce587486fee9d83ff301f01f00d0aa9e4b9d648fdec5a34326b40c7279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-27T07:28:34.355 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:33 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:28:34.355 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:33 smithi162 systemd[1]: Stopped Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:34.355 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:33 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.623s CPU time. 2024-05-27T07:28:34.355 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:33 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-05-27T07:28:34.356 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:33 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:28:34.356 INFO:journalctl@ceph.mds.l.smithi162.stdout:May 27 07:28:33 smithi162 systemd[1]: Failed to start Ceph mds.l for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:34.641 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:34 smithi012 ceph-mon[31473]: pgmap v217: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-05-27T07:28:34.641 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:34 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2664252727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:28:34.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 podman[121450]: 2024-05-27 07:28:34.536446681 +0000 UTC m=+0.654252751 container init 145483ce587486fee9d83ff301f01f00d0aa9e4b9d648fdec5a34326b40c7279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:28:34.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 podman[121450]: 2024-05-27 07:28:34.547255956 +0000 UTC m=+0.665062002 container start 145483ce587486fee9d83ff301f01f00d0aa9e4b9d648fdec5a34326b40c7279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:34.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:34 smithi162 ceph-mon[35676]: pgmap v217: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-05-27T07:28:34.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:34 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2664252727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:28:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:34 smithi062 ceph-mon[35304]: pgmap v217: 129 pgs: 129 active+clean; 96 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-05-27T07:28:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:34 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2664252727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:28:35.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 bash[121450]: 145483ce587486fee9d83ff301f01f00d0aa9e4b9d648fdec5a34326b40c7279 2024-05-27T07:28:35.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 systemd[1]: Started Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:35.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 ceph-mds[121480]: -- 172.21.15.12:0/3582345736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb0e92ba40 con 0x55eb0e973180 2024-05-27T07:28:35.090 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:34 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: starting mds.a at 2024-05-27T07:28:35.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:28:35.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:35 smithi012 ceph-mon[31473]: daemon mds.a restarted 2024-05-27T07:28:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:35 smithi162 ceph-mon[35676]: daemon mds.a restarted 2024-05-27T07:28:35.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:35 smithi062 ceph-mon[35304]: daemon mds.a restarted 2024-05-27T07:28:35.895 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:36.632 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-mon[31473]: pgmap v218: 129 pgs: 129 active+clean; 97 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 178 op/s 2024-05-27T07:28:36.632 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-mon[31473]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:28:36.632 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6838/3335198443,v1:172.21.15.12:6839/3335198443] up:boot 2024-05-27T07:28:36.632 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:36.632 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:36.632 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f296b800700 time 2024-05-27T07:28:36.600169+0000 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2981322ba6] 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:36.633 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: clone() 2024-05-27T07:28:36.634 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.602+0000 7f296b800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f296b800700 time 2024-05-27T07:28:36.600169+0000 2024-05-27T07:28:36.635 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:36.635 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.635 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:36.635 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2981322ba6] 2024-05-27T07:28:36.635 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:36.635 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:36.635 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:36.636 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: clone() 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: *** Caught signal (Aborted) ** 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: in thread 7f296b800700 thread_name:md_log_replay 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f297fe9fd20] 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: gsignal() 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: abort() 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2981322ced] 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:36.637 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 17: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:36.638 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 19: clone() 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.605+0000 7f296b800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: in thread 7f296b800700 thread_name:md_log_replay 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f297fe9fd20] 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: gsignal() 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: abort() 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2981322ced] 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:36.639 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:36.640 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:36.640 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:36.640 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:36.640 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:36.640 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:36.640 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 17: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 19: clone() 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: -1> 2024-05-27T07:28:36.602+0000 7f296b800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f296b800700 time 2024-05-27T07:28:36.600169+0000 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.641 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2981322ba6] 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:36.642 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: clone() 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 0> 2024-05-27T07:28:36.605+0000 7f296b800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: in thread 7f296b800700 thread_name:md_log_replay 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:36.643 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f297fe9fd20] 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: gsignal() 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: abort() 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2981322ced] 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:36.644 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 17: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 19: clone() 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:36.645 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:36 smithi162 ceph-mon[35676]: pgmap v218: 129 pgs: 129 active+clean; 97 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 178 op/s 2024-05-27T07:28:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:36 smithi162 ceph-mon[35676]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:28:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:36 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6838/3335198443,v1:172.21.15.12:6839/3335198443] up:boot 2024-05-27T07:28:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:36 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:36.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:36 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:36 smithi062 ceph-mon[35304]: pgmap v218: 129 pgs: 129 active+clean; 97 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 178 op/s 2024-05-27T07:28:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:36 smithi062 ceph-mon[35304]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:28:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:36 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6838/3335198443,v1:172.21.15.12:6839/3335198443] up:boot 2024-05-27T07:28:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:36 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 3 up:standby 2024-05-27T07:28:36.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:28:36.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:36 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2 up:standby 2024-05-27T07:28:37.091 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: -1> 2024-05-27T07:28:36.602+0000 7f296b800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f296b800700 time 2024-05-27T07:28:36.600169+0000 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2981322ba6] 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:37.092 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: clone() 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 0> 2024-05-27T07:28:36.605+0000 7f296b800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: in thread 7f296b800700 thread_name:md_log_replay 2024-05-27T07:28:37.093 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:37.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f297fe9fd20] 2024-05-27T07:28:37.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2: gsignal() 2024-05-27T07:28:37.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 3: abort() 2024-05-27T07:28:37.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2981322ced] 2024-05-27T07:28:37.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2981322dd1] 2024-05-27T07:28:37.094 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eacf34a23d] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eacf335971] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eacf333fa0] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eacf334651] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eacf2c9347] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eacefc587e] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eacefcae94] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eacefd5969] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eacf36e48c] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eacf3779cb] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 16: (Thread::entry_wrapper()+0x43) [0x7f29812f172d] 2024-05-27T07:28:37.095 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 17: (Thread::_entry_func(void*)+0xd) [0x7f29812f1749] 2024-05-27T07:28:37.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f297fe951ca] 2024-05-27T07:28:37.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 19: clone() 2024-05-27T07:28:37.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:37.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a[121474]: 2024-05-27T07:28:37.202 INFO:tasks.workunit.client.0.smithi012.stdout:10000 punch from 0x713e to 0x9907, (0x27c9 bytes) 2024-05-27T07:28:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:37 smithi162 ceph-mon[35676]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:37 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 1 up:standby 2024-05-27T07:28:37.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:37 smithi062 ceph-mon[35304]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:37.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:37 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 1 up:standby 2024-05-27T07:28:37.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:37 smithi012 ceph-mon[31473]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:37.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:37 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 1 up:standby 2024-05-27T07:28:37.840 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:37 smithi012 podman[121600]: 2024-05-27 07:28:37.473781712 +0000 UTC m=+0.767550713 container died 145483ce587486fee9d83ff301f01f00d0aa9e4b9d648fdec5a34326b40c7279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:37.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb27aa00700 time 2024-05-27T07:28:37.424813+0000 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb29048fba6] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: clone() 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.427+0000 7fb27aa00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb27aa00700 time 2024-05-27T07:28:37.424813+0000 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:37.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb29048fba6] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: clone() 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: *** Caught signal (Aborted) ** 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: in thread 7fb27aa00700 thread_name:md_log_replay 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb28f00cd20] 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: gsignal() 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: abort() 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb29048fced] 2024-05-27T07:28:37.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 19: clone() 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.430+0000 7fb27aa00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: in thread 7fb27aa00700 thread_name:md_log_replay 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb28f00cd20] 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: gsignal() 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: abort() 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb29048fced] 2024-05-27T07:28:37.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 19: clone() 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: -1> 2024-05-27T07:28:37.427+0000 7fb27aa00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb27aa00700 time 2024-05-27T07:28:37.424813+0000 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb29048fba6] 2024-05-27T07:28:37.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: clone() 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 0> 2024-05-27T07:28:37.430+0000 7fb27aa00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:37.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: in thread 7fb27aa00700 thread_name:md_log_replay 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb28f00cd20] 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: gsignal() 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: abort() 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb29048fced] 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 19: clone() 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: -1> 2024-05-27T07:28:37.427+0000 7fb27aa00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb27aa00700 time 2024-05-27T07:28:37.424813+0000 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:37.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb29048fba6] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: clone() 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 0> 2024-05-27T07:28:37.430+0000 7fb27aa00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: in thread 7fb27aa00700 thread_name:md_log_replay 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb28f00cd20] 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2: gsignal() 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 3: abort() 2024-05-27T07:28:37.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb29048fced] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb29048fdd1] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x560cb96a223d] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 7: (SnapRealm::build_snap_set() const+0x323) [0x560cb968d971] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 8: (SnapRealm::check_cache() const+0x13e) [0x560cb968bfa0] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x560cb968c651] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x560cb9621347] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560cb931d87e] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560cb9322e94] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560cb932d969] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 14: (MDLog::_replay_thread()+0x1f38) [0x560cb96c648c] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 15: (MDLog::ReplayThread::entry()+0x11) [0x560cb96cf9cb] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 16: (Thread::entry_wrapper()+0x43) [0x7fb29045e72d] 2024-05-27T07:28:37.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb29045e749] 2024-05-27T07:28:37.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb28f0021ca] 2024-05-27T07:28:37.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 19: clone() 2024-05-27T07:28:37.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:37.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[72344]: 2024-05-27T07:28:38.506 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:38 smithi012 podman[121623]: 2024-05-27 07:28:38.205176365 +0000 UTC m=+0.677289943 container died bdd79a7e3870a44300c72d72894968d4b4dd64f2e326d4ffa15510e1cece4146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:28:38.507 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:38 smithi012 ceph-mon[31473]: pgmap v220: 129 pgs: 129 active+clean; 97 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-05-27T07:28:38.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:38 smithi162 ceph-mon[35676]: pgmap v220: 129 pgs: 129 active+clean; 97 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-05-27T07:28:38.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:38 smithi062 ceph-mon[35304]: pgmap v220: 129 pgs: 129 active+clean; 97 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 217 op/s 2024-05-27T07:28:39.589 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:39 smithi012 podman[121623]: 2024-05-27 07:28:39.330587643 +0000 UTC m=+1.802701220 container remove bdd79a7e3870a44300c72d72894968d4b4dd64f2e326d4ffa15510e1cece4146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:28:39.590 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:39 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:40.000 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:39 smithi012 podman[121600]: 2024-05-27 07:28:39.667366329 +0000 UTC m=+2.961135351 container remove 145483ce587486fee9d83ff301f01f00d0aa9e4b9d648fdec5a34326b40c7279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:28:40.001 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:39 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.473+0000 7f6de9800700 1 -- 172.21.15.12:0/3794373756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de410c5a0 msgr2=0x7f6de410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.473+0000 7f6de9800700 1 --2- 172.21.15.12:0/3794373756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de410c5a0 0x7f6de410c9b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009990 tx=0x7f6dd4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.473+0000 7f6de9800700 1 -- 172.21.15.12:0/3794373756 shutdown_connections 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.473+0000 7f6de9800700 1 --2- 172.21.15.12:0/3794373756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106eb0 0x7f6de4113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.473+0000 7f6de9800700 1 --2- 172.21.15.12:0/3794373756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de410c5a0 0x7f6de410c9b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.473+0000 7f6de9800700 1 --2- 172.21.15.12:0/3794373756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6de4106560 0x7f6de4106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.473+0000 7f6de9800700 1 -- 172.21.15.12:0/3794373756 >> 172.21.15.12:0/3794373756 conn(0x7f6de4102040 msgr2=0x7f6de4104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.474+0000 7f6de9800700 1 -- 172.21.15.12:0/3794373756 shutdown_connections 2024-05-27T07:28:40.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.474+0000 7f6de9800700 1 -- 172.21.15.12:0/3794373756 wait complete. 2024-05-27T07:28:40.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.476+0000 7f6de9800700 1 Processor -- start 2024-05-27T07:28:40.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.476+0000 7f6de9800700 1 -- start start 2024-05-27T07:28:40.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106560 0x7f6de4111cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de9800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de4106eb0 0x7f6de41121f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de9800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6de410c5a0 0x7f6de41acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de9800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6de4078750 con 0x7f6de4106eb0 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de9800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6de40785d0 con 0x7f6de4106560 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de9800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6de40788d0 con 0x7f6de410c5a0 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106560 0x7f6de4111cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106560 0x7f6de4111cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:38548/0 (socket says 172.21.15.12:38548) 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de2a00700 1 -- 172.21.15.12:0/303914334 learned_addr learned my addr 172.21.15.12:0/303914334 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.477+0000 7f6de3400700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6de410c5a0 0x7f6de41acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.478+0000 7f6de2a00700 1 -- 172.21.15.12:0/303914334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6de410c5a0 msgr2=0x7f6de41acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.478+0000 7f6de2a00700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6de410c5a0 0x7f6de41acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.478+0000 7f6de2a00700 1 -- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de4106eb0 msgr2=0x7f6de41121f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.478+0000 7f6de2a00700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de4106eb0 0x7f6de41121f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.478+0000 7f6de2a00700 1 -- 172.21.15.12:0/303914334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de4115a50 con 0x7f6de4106560 2024-05-27T07:28:40.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.479+0000 7f6de2a00700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106560 0x7f6de4111cb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40047b0 tx=0x7f6dd400bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:40.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.481+0000 7f6ddb400700 1 -- 172.21.15.12:0/303914334 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4026070 con 0x7f6de4106560 2024-05-27T07:28:40.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.481+0000 7f6ddb400700 1 -- 172.21.15.12:0/303914334 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6dd4021410 con 0x7f6de4106560 2024-05-27T07:28:40.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.481+0000 7f6ddb400700 1 -- 172.21.15.12:0/303914334 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4017650 con 0x7f6de4106560 2024-05-27T07:28:40.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.481+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de4115ce0 con 0x7f6de4106560 2024-05-27T07:28:40.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.482+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de4116170 con 0x7f6de4106560 2024-05-27T07:28:40.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.489+0000 7f6df83b3b80 1 Processor -- start 2024-05-27T07:28:40.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.490+0000 7f6df83b3b80 1 -- start start 2024-05-27T07:28:40.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.490+0000 7f6ddb400700 1 -- 172.21.15.12:0/303914334 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6dd4029070 con 0x7f6de4106560 2024-05-27T07:28:40.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.491+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564729f39f0 0x556472a86330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.491+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a86870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.491+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556472a86e70 0x556472a93110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.491+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5564729b4e40 con 0x556472a86e70 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.491+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5564729a49e0 con 0x5564729f39f0 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.491+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5564729a8a20 con 0x556472907e40 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.491+0000 7f6de3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564729f39f0 0x556472a86330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.492+0000 7f6de3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564729f39f0 0x556472a86330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:38550/0 (socket says 172.21.15.12:38550) 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.492+0000 7f6de3400700 1 -- 172.21.15.12:0/2226427034 learned_addr learned my addr 172.21.15.12:0/2226427034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:40.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.492+0000 7f6de2a00700 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a86870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.492+0000 7f6de2000700 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556472a86e70 0x556472a93110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6de2a00700 1 -- 172.21.15.12:0/2226427034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564729f39f0 msgr2=0x556472a86330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6de2a00700 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564729f39f0 0x556472a86330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6de2a00700 1 -- 172.21.15.12:0/2226427034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556472a86e70 msgr2=0x556472a93110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6de2a00700 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556472a86e70 0x556472a93110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6de2a00700 1 -- 172.21.15.12:0/2226427034 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55647296d420 con 0x556472907e40 2024-05-27T07:28:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6de2000700 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556472a86e70 0x556472a93110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6ddb400700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dc80b2a70 0x7f6dc80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6de3400700 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564729f39f0 0x556472a86330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.493+0000 7f6ddb400700 1 -- 172.21.15.12:0/303914334 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7f6dd40e38f0 con 0x7f6de4106560 2024-05-27T07:28:40.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.494+0000 7f6de2a00700 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a86870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40e0c90 tx=0x7f6dd40e9ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7797884d1cb3c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:40.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.494+0000 7f6de3400700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dc80b2a70 0x7f6dc80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.494+0000 7f6dcfe00700 1 -- 172.21.15.12:0/2226427034 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40a6770 con 0x556472907e40 2024-05-27T07:28:40.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.495+0000 7f6dcfe00700 1 -- 172.21.15.12:0/2226427034 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6dd40aa3c0 con 0x556472907e40 2024-05-27T07:28:40.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.495+0000 7f6dcfe00700 1 -- 172.21.15.12:0/2226427034 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40aa020 con 0x556472907e40 2024-05-27T07:28:40.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.496+0000 7f6df83b3b80 1 -- 172.21.15.12:0/2226427034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 msgr2=0x556472a86870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.496+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a86870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40e0c90 tx=0x7f6dd40e9ef0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.496+0000 7f6de3400700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dc80b2a70 0x7f6dc80b4f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x556472a88ff0 tx=0x7f6ddc009130 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:40.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.497+0000 7f6df83b3b80 1 -- 172.21.15.12:0/2226427034 shutdown_connections 2024-05-27T07:28:40.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.497+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564729f39f0 0x556472a86330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.497+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556472a86e70 0x556472a93110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.497+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/2226427034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a86870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.497+0000 7f6df83b3b80 1 -- 172.21.15.12:0/2226427034 >> 172.21.15.12:0/2226427034 conn(0x5564729a1390 msgr2=0x556472a824c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:40.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.497+0000 7f6df83b3b80 1 -- 172.21.15.12:0/2226427034 shutdown_connections 2024-05-27T07:28:40.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.498+0000 7f6df83b3b80 1 -- 172.21.15.12:0/2226427034 wait complete. 2024-05-27T07:28:40.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.498+0000 7f6df83b3b80 1 -- start start 2024-05-27T07:28:40.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.499+0000 7f6df83b3b80 1 Processor -- start 2024-05-27T07:28:40.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a90680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a90bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472a86e70 0x556472a91100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5564728aafa0 con 0x5564729f39f0 2024-05-27T07:28:40.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556472945760 con 0x556472a86e70 2024-05-27T07:28:40.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556472a93940 con 0x556472907e40 2024-05-27T07:28:40.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6df83b3b80 10 client.0 _ll_register_callbacks cb 0x55647294ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:40.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.500+0000 7f6de2a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a90680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472a86e70 0x556472a91100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de2a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a90680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:54984/0 (socket says 172.21.15.12:54984) 2024-05-27T07:28:40.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de2a00700 1 -- 172.21.15.12:0/3893922538 learned_addr learned my addr 172.21.15.12:0/3893922538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:40.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de2000700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a90bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de2a00700 1 -- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472a86e70 msgr2=0x556472a91100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de2a00700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472a86e70 0x556472a91100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de2a00700 1 -- 172.21.15.12:0/3893922538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 msgr2=0x556472a90bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.501+0000 7f6de2a00700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a90bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.502+0000 7f6de2a00700 1 -- 172.21.15.12:0/3893922538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564728f48d0 con 0x556472907e40 2024-05-27T07:28:40.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.502+0000 7f6de3400700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472a86e70 0x556472a91100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:40.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.502+0000 7f6de2000700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a90bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:40.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.502+0000 7f6de2a00700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a90680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40e0d40 tx=0x7f6dd40b30c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=926673af3ed2a0f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:40.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.503+0000 7f6dccc00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:28:40.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.503+0000 7f6dccc00700 1 -- 172.21.15.12:0/3893922538 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40aa200 con 0x556472907e40 2024-05-27T07:28:40.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.503+0000 7f6dccc00700 1 -- 172.21.15.12:0/3893922538 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7f6dd40b10d0 con 0x556472907e40 2024-05-27T07:28:40.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.503+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dd40aa3c0 con 0x556472907e40 2024-05-27T07:28:40.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.503+0000 7f6dccc00700 1 -- 172.21.15.12:0/3893922538 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6dd40aa680 con 0x556472907e40 2024-05-27T07:28:40.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.504+0000 7f6df83b3b80 10 client.25870 fetch_fsmap learned FSMap version 90 2024-05-27T07:28:40.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.504+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=4+}) -- 0x556472975060 con 0x556472907e40 2024-05-27T07:28:40.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.504+0000 7f6dccc00700 1 -- 172.21.15.12:0/3893922538 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6dd40aa3c0 con 0x556472907e40 2024-05-27T07:28:40.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.505+0000 7f6dccc00700 1 -- 172.21.15.12:0/3893922538 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 90) ==== 5345+0+0 (secure 0 0 0) 0x7f6dd40218e0 con 0x556472907e40 2024-05-27T07:28:40.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.506+0000 7f6df83b3b80 10 client.25870 fetch_fsmap finished waiting for FSMap version 90 2024-05-27T07:28:40.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.506+0000 7f6df83b3b80 10 client.25870 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:28:40.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.506+0000 7f6df83b3b80 20 client.25870 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:40.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.506+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472aa06a0 0x556472aa2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.506+0000 7f6df83b3b80 4 client.25870 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:40.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.506+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564729b6d90 con 0x556472aa06a0 2024-05-27T07:28:40.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.507+0000 7f6de2000700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472aa06a0 0x556472aa2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.508+0000 7f6de2000700 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472aa06a0 0x556472aa2ac0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3dc98293fa3ac1f server_cookie=f2e6641b1bd68c18 in_seq=0 out_seq=0 2024-05-27T07:28:40.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.508+0000 7f6dccc00700 10 client.25870 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:28:40.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.510+0000 7f6dccc00700 1 -- 172.21.15.12:0/3893922538 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5564729b6d90 con 0x556472aa06a0 2024-05-27T07:28:40.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.510+0000 7f6dccc00700 10 client.25870 handle_command_reply: tid=0 2024-05-27T07:28:40.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 client.25870 shutdown 2024-05-27T07:28:40.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 msgr2=0x556472a90680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a90680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40e0d40 tx=0x7f6dd40b30c0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 shutdown_connections 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472aa06a0 0x556472aa2ac0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472a86e70 0x556472a91100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472907e40 0x556472a90680 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/3893922538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a90bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 >> 172.21.15.12:0/3893922538 conn(0x5564729a1390 msgr2=0x556472a86310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.511+0000 7f6dccc00700 0 client.25870 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.512+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 shutdown_connections 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.512+0000 7f6df83b3b80 1 -- 172.21.15.12:0/3893922538 wait complete. 2024-05-27T07:28:40.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.512+0000 7f6df83b3b80 20 client.25870 trim_cache size 0 max 16384 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.534+0000 7f6df83b3b80 1 Processor -- start 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.535+0000 7f6df83b3b80 1 -- start start 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.535+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.535+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a867c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.535+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472aa8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.535+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6de41c05d0 con 0x5564729f39f0 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.535+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6de411dfd0 con 0x556472907e40 2024-05-27T07:28:40.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.535+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6de41bff00 con 0x556472a86e70 2024-05-27T07:28:40.527 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:28:40.527 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.042s CPU time. 2024-05-27T07:28:40.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:40 smithi012 ceph-mon[31473]: pgmap v221: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-05-27T07:28:40.527 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:28:40.527 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.742s CPU time. 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a867c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a867c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60522/0 (socket says 172.21.15.12:60522) 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 -- 172.21.15.12:0/1275415626 learned_addr learned my addr 172.21.15.12:0/1275415626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2a00700 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de3400700 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472aa8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 -- 172.21.15.12:0/1275415626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 msgr2=0x556472aa8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472aa8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 -- 172.21.15.12:0/1275415626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 msgr2=0x556472a8be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.536+0000 7f6de2000700 1 -- 172.21.15.12:0/1275415626 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556472a7e240 con 0x5564729f39f0 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.537+0000 7f6de2a00700 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.537+0000 7f6de2000700 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a867c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6de41057c0 tx=0x7f6dd0009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edc3cd95eceadbe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:40.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.537+0000 7f6dcf400700 1 -- 172.21.15.12:0/1275415626 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00142d0 con 0x5564729f39f0 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.537+0000 7f6dcf400700 1 -- 172.21.15.12:0/1275415626 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6dd00129c0 con 0x5564729f39f0 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.537+0000 7f6dcf400700 1 -- 172.21.15.12:0/1275415626 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd001d870 con 0x5564729f39f0 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.538+0000 7f6df83b3b80 1 -- 172.21.15.12:0/1275415626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 msgr2=0x556472a867c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.538+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a867c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6de41057c0 tx=0x7f6dd0009670 comp rx=0 tx=0).stop 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 -- 172.21.15.12:0/1275415626 shutdown_connections 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472aa8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/1275415626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a867c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 -- 172.21.15.12:0/1275415626 >> 172.21.15.12:0/1275415626 conn(0x5564729a1390 msgr2=0x556472a860b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 -- 172.21.15.12:0/1275415626 shutdown_connections 2024-05-27T07:28:40.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 -- 172.21.15.12:0/1275415626 wait complete. 2024-05-27T07:28:40.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.539+0000 7f6df83b3b80 1 -- start start 2024-05-27T07:28:40.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.540+0000 7f6df83b3b80 1 Processor -- start 2024-05-27T07:28:40.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.540+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.540+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a8ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.540+0000 7f6df83b3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472a97b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.540+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6de41bf0a0 con 0x5564729f39f0 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6de41be970 con 0x556472907e40 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6df83b3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6de41bf7d0 con 0x556472a86e70 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6df83b3b80 10 client.0 _ll_register_callbacks cb 0x55647294ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6de2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a8ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6de2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a8ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60528/0 (socket says 172.21.15.12:60528) 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6de2000700 1 -- 172.21.15.12:0/559859041 learned_addr learned my addr 172.21.15.12:0/559859041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6de2a00700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.541+0000 7f6de3400700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472a97b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6de2000700 1 -- 172.21.15.12:0/559859041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 msgr2=0x556472a97b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6de2000700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472a97b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6de2000700 1 -- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 msgr2=0x556472a8a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:40.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6de2000700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6de2000700 1 -- 172.21.15.12:0/559859041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556472a91470 con 0x5564729f39f0 2024-05-27T07:28:40.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6de3400700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472a97b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:40.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6de2000700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a8ab90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0004790 tx=0x7f6dd0014160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9317161d56427a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:40.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.542+0000 7f6dccc00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:40.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.543+0000 7f6dccc00700 1 -- 172.21.15.12:0/559859041 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0012530 con 0x5564729f39f0 2024-05-27T07:28:40.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.543+0000 7f6dccc00700 1 -- 172.21.15.12:0/559859041 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9466+0+0 (secure 0 0 0) 0x7f6dd00126d0 con 0x5564729f39f0 2024-05-27T07:28:40.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.543+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dd00129c0 con 0x5564729f39f0 2024-05-27T07:28:40.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.545+0000 7f6dccc00700 1 -- 172.21.15.12:0/559859041 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6dd0004040 con 0x5564729f39f0 2024-05-27T07:28:40.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.545+0000 7f6df83b3b80 10 client.16206 fetch_fsmap learned FSMap version 90 2024-05-27T07:28:40.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.545+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=4+}) -- 0x556472a98f00 con 0x5564729f39f0 2024-05-27T07:28:40.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6dccc00700 1 -- 172.21.15.12:0/559859041 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6dd00129c0 con 0x5564729f39f0 2024-05-27T07:28:40.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6dccc00700 1 -- 172.21.15.12:0/559859041 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 90) ==== 5345+0+0 (secure 0 0 0) 0x7f6dd003a980 con 0x5564729f39f0 2024-05-27T07:28:40.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6df83b3b80 10 client.16206 fetch_fsmap finished waiting for FSMap version 90 2024-05-27T07:28:40.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6df83b3b80 10 client.16206 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:28:40.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6df83b3b80 20 client.16206 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:40.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472b0b0f0 0x556472b0d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:40.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6df83b3b80 4 client.16206 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:40.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.546+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564729b0570 con 0x556472b0b0f0 2024-05-27T07:28:40.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.549+0000 7f6de2a00700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472b0b0f0 0x556472b0d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:40.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.550+0000 7f6de2a00700 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472b0b0f0 0x556472b0d510 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=48d1122d88bddaf1 server_cookie=40fe5b2cf22f69a0 in_seq=0 out_seq=0 2024-05-27T07:28:40.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.550+0000 7f6dccc00700 10 client.16206 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:28:40.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.551+0000 7f6dccc00700 1 -- 172.21.15.12:0/559859041 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564729b0570 con 0x556472b0b0f0 2024-05-27T07:28:40.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.551+0000 7f6dccc00700 10 client.16206 handle_command_reply: tid=0 2024-05-27T07:28:40.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.551+0000 7f6df83b3b80 1 client.16206 shutdown 2024-05-27T07:28:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 msgr2=0x556472a8ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a8ab90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0004790 tx=0x7f6dd0014160 comp rx=0 tx=0).stop 2024-05-27T07:28:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 shutdown_connections 2024-05-27T07:28:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556472b0b0f0 0x556472b0d510 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556472907e40 0x556472a8a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556472a86e70 0x556472a97b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 --2- 172.21.15.12:0/559859041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564729f39f0 0x556472a8ab90 secure :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0004790 tx=0x7f6dd0014160 comp rx=0 tx=0).stop 2024-05-27T07:28:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 >> 172.21.15.12:0/559859041 conn(0x5564729a1390 msgr2=0x556472a87ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.552+0000 7f6dccc00700 0 client.16206 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:28:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.553+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 shutdown_connections 2024-05-27T07:28:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.553+0000 7f6df83b3b80 1 -- 172.21.15.12:0/559859041 wait complete. 2024-05-27T07:28:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.553+0000 7f6df83b3b80 20 client.16206 trim_cache size 0 max 16384 2024-05-27T07:28:40.544 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:28:40.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dc80b2a70 msgr2=0x7f6dc80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dc80b2a70 0x7f6dc80b4f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x556472a88ff0 tx=0x7f6ddc009130 comp rx=0 tx=0).stop 2024-05-27T07:28:40.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106560 msgr2=0x7f6de4111cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:40.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106560 0x7f6de4111cb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40047b0 tx=0x7f6dd400bfa0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 shutdown_connections 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6dc80b2a70 0x7f6dc80b4f80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6de4106560 0x7f6de4111cb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6de410c5a0 0x7f6de41acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 --2- 172.21.15.12:0/303914334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6de4106eb0 0x7f6de41121f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.554+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 >> 172.21.15.12:0/303914334 conn(0x7f6de4102040 msgr2=0x7f6de4103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.555+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 shutdown_connections 2024-05-27T07:28:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:40.555+0000 7f6de9800700 1 -- 172.21.15.12:0/303914334 wait complete. 2024-05-27T07:28:40.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:40 smithi162 ceph-mon[35676]: pgmap v221: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-05-27T07:28:40.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:40 smithi062 ceph-mon[35304]: pgmap v221: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-05-27T07:28:41.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:28:41.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: Stopped Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:41.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.756s CPU time. 2024-05-27T07:28:41.340 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:40 smithi012 systemd[1]: Starting Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:41.582 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:28:42.199 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:41 smithi012 podman[121849]: 2024-05-27 07:28:41.795796649 +0000 UTC m=+0.045712372 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:42.199 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:41 smithi012 podman[121849]: 2024-05-27 07:28:41.964148945 +0000 UTC m=+0.214064652 container create 88b46d3a208da7961a13a0e50b2e7ce2f8f52587208986a12191e06000a8eac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:42.407 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:42.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:42 smithi012 ceph-mon[31473]: pgmap v222: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-05-27T07:28:42.565 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:42 smithi012 ceph-mon[31473]: daemon mds.g restarted 2024-05-27T07:28:42.566 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 podman[121849]: 2024-05-27 07:28:42.20999655 +0000 UTC m=+0.459912294 container init 88b46d3a208da7961a13a0e50b2e7ce2f8f52587208986a12191e06000a8eac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-27T07:28:42.566 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 podman[121849]: 2024-05-27 07:28:42.219591375 +0000 UTC m=+0.469507082 container start 88b46d3a208da7961a13a0e50b2e7ce2f8f52587208986a12191e06000a8eac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:28:42.566 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 bash[121849]: 88b46d3a208da7961a13a0e50b2e7ce2f8f52587208986a12191e06000a8eac5 2024-05-27T07:28:42.566 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 systemd[1]: Started Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:42.566 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-mds[121895]: -- 172.21.15.12:0/3533873485 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56072a039a40 con 0x56072a081600 2024-05-27T07:28:42.566 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: starting mds.g at 2024-05-27T07:28:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:42 smithi162 ceph-mon[35676]: pgmap v222: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-05-27T07:28:42.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:42 smithi162 ceph-mon[35676]: daemon mds.g restarted 2024-05-27T07:28:42.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:42 smithi062 ceph-mon[35304]: pgmap v222: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.4 MiB/s wr, 185 op/s 2024-05-27T07:28:42.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:42 smithi062 ceph-mon[35304]: daemon mds.g restarted 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5991e00700 time 2024-05-27T07:28:42.601761+0000 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a7840ba6] 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.840 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: clone() 2024-05-27T07:28:42.841 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.603+0000 7f5991e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5991e00700 time 2024-05-27T07:28:42.601761+0000 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a7840ba6] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.842 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: clone() 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: *** Caught signal (Aborted) ** 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: in thread 7f5991e00700 thread_name:md_log_replay 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a63bdd20] 2024-05-27T07:28:42.843 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: gsignal() 2024-05-27T07:28:42.844 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: abort() 2024-05-27T07:28:42.844 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a7840ced] 2024-05-27T07:28:42.844 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.844 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.844 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.844 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 17: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 19: clone() 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.606+0000 7f5991e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: in thread 7f5991e00700 thread_name:md_log_replay 2024-05-27T07:28:42.845 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a63bdd20] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: gsignal() 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: abort() 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a7840ced] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.846 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 17: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 19: clone() 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: -1> 2024-05-27T07:28:42.603+0000 7f5991e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5991e00700 time 2024-05-27T07:28:42.601761+0000 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.847 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a7840ba6] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.848 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: clone() 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 0> 2024-05-27T07:28:42.606+0000 7f5991e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: in thread 7f5991e00700 thread_name:md_log_replay 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a63bdd20] 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: gsignal() 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: abort() 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a7840ced] 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.849 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.850 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 17: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 19: clone() 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: -1> 2024-05-27T07:28:42.603+0000 7f5991e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5991e00700 time 2024-05-27T07:28:42.601761+0000 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a7840ba6] 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.851 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: clone() 2024-05-27T07:28:42.852 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 0> 2024-05-27T07:28:42.606+0000 7f5991e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: in thread 7f5991e00700 thread_name:md_log_replay 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a63bdd20] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2: gsignal() 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 3: abort() 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a7840ced] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a7840dd1] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56072633123d] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56072631c971] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 8: (SnapRealm::check_cache() const+0x13e) [0x56072631afa0] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56072631b651] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5607262b0347] 2024-05-27T07:28:42.853 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x560725fac87e] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x560725fb1e94] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x560725fbc969] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 14: (MDLog::_replay_thread()+0x1f38) [0x56072635548c] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56072635e9cb] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 16: (Thread::entry_wrapper()+0x43) [0x7f59a780f72d] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 17: (Thread::_entry_func(void*)+0xd) [0x7f59a780f749] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f59a63b31ca] 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 19: clone() 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g[121891]: 2024-05-27T07:28:42.854 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:42 smithi012 podman[121949]: 2024-05-27 07:28:42.780619148 +0000 UTC m=+0.076059302 container died 88b46d3a208da7961a13a0e50b2e7ce2f8f52587208986a12191e06000a8eac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:28:43.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:43 smithi012 ceph-mon[31473]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:28:43.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:43 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6836/1971929829,v1:172.21.15.12:6837/1971929829] up:boot 2024-05-27T07:28:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:43 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2 up:standby 2024-05-27T07:28:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:43 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:43.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:43 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 1 up:standby 2024-05-27T07:28:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:43 smithi162 ceph-mon[35676]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:28:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:43 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6836/1971929829,v1:172.21.15.12:6837/1971929829] up:boot 2024-05-27T07:28:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:43 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2 up:standby 2024-05-27T07:28:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:43 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:43.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:43 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 1 up:standby 2024-05-27T07:28:43.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:43 smithi062 ceph-mon[35304]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:28:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:43 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6836/1971929829,v1:172.21.15.12:6837/1971929829] up:boot 2024-05-27T07:28:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:43 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2 up:standby 2024-05-27T07:28:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:43 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:43 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 1 up:standby 2024-05-27T07:28:44.054 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:43 smithi012 podman[121949]: 2024-05-27 07:28:43.930850759 +0000 UTC m=+1.226290928 container remove 88b46d3a208da7961a13a0e50b2e7ce2f8f52587208986a12191e06000a8eac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:28:44.054 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:43 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:44.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:44 smithi162 ceph-mon[35676]: pgmap v224: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 834 KiB/s wr, 121 op/s 2024-05-27T07:28:44.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:44 smithi062 ceph-mon[35304]: pgmap v224: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 834 KiB/s wr, 121 op/s 2024-05-27T07:28:44.736 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:44 smithi012 ceph-mon[31473]: pgmap v224: 129 pgs: 129 active+clean; 99 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 834 KiB/s wr, 121 op/s 2024-05-27T07:28:45.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:45 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:45.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:45 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.697s CPU time. 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 -- 172.21.15.12:0/392949741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac110c60 msgr2=0x7fdaac113080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 --2- 172.21.15.12:0/392949741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac110c60 0x7fdaac113080 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b0a0 tx=0x7fdaa400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 -- 172.21.15.12:0/392949741 shutdown_connections 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 --2- 172.21.15.12:0/392949741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac110c60 0x7fdaac113080 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 --2- 172.21.15.12:0/392949741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdaac1030b0 0x7fdaac1034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 --2- 172.21.15.12:0/392949741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdaac103a00 0x7fdaac110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 -- 172.21.15.12:0/392949741 >> 172.21.15.12:0/392949741 conn(0x7fdaac0fedb0 msgr2=0x7fdaac1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 -- 172.21.15.12:0/392949741 shutdown_connections 2024-05-27T07:28:45.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.699+0000 7fdab2e00700 1 -- 172.21.15.12:0/392949741 wait complete. 2024-05-27T07:28:45.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:45 smithi012 ceph-mon[31473]: pgmap v225: 129 pgs: 129 active+clean; 99 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-05-27T07:28:45.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.700+0000 7fdab2e00700 1 Processor -- start 2024-05-27T07:28:45.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.700+0000 7fdab2e00700 1 -- start start 2024-05-27T07:28:45.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdab2e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdaac1030b0 0x7fdaac10db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdab2e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac103a00 0x7fdaac10e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdab2e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdaac110c60 0x7fdaac10e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdab2e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac116570 con 0x7fdaac103a00 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdab2e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1163f0 con 0x7fdaac1030b0 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdab2e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1166f0 con 0x7fdaac110c60 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdaab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac103a00 0x7fdaac10e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdaab400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac103a00 0x7fdaac10e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40298/0 (socket says 172.21.15.12:40298) 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdaab400700 1 -- 172.21.15.12:0/2359454544 learned_addr learned my addr 172.21.15.12:0/2359454544 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.701+0000 7fdaabe00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdaac1030b0 0x7fdaac10db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.702+0000 7fdaab400700 1 -- 172.21.15.12:0/2359454544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdaac110c60 msgr2=0x7fdaac10e5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.702+0000 7fdaab400700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdaac110c60 0x7fdaac10e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.702+0000 7fdaab400700 1 -- 172.21.15.12:0/2359454544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdaac1030b0 msgr2=0x7fdaac10db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.702+0000 7fdaab400700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdaac1030b0 0x7fdaac10db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.702+0000 7fdaab400700 1 -- 172.21.15.12:0/2359454544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac1ad3e0 con 0x7fdaac103a00 2024-05-27T07:28:45.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.702+0000 7fdaab400700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac103a00 0x7fdaac10e0a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fda9800c9e0 tx=0x7fda9800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.703+0000 7fdaa8c00700 1 -- 172.21.15.12:0/2359454544 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98007aa0 con 0x7fdaac103a00 2024-05-27T07:28:45.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.703+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaac1ad6d0 con 0x7fdaac103a00 2024-05-27T07:28:45.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.703+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaac1adc60 con 0x7fdaac103a00 2024-05-27T07:28:45.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.703+0000 7fdaa8c00700 1 -- 172.21.15.12:0/2359454544 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda98007c40 con 0x7fdaac103a00 2024-05-27T07:28:45.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.703+0000 7fdaa8c00700 1 -- 172.21.15.12:0/2359454544 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98018660 con 0x7fdaac103a00 2024-05-27T07:28:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.708+0000 7fdac1947b80 1 Processor -- start 2024-05-27T07:28:45.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdac1947b80 1 -- start start 2024-05-27T07:28:45.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdaa8c00700 1 -- 172.21.15.12:0/2359454544 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fda98018800 con 0x7fdaac103a00 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377af48e40 0x56377b0cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0cdd30 0x56377b0c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdac1947b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56377aff5e40 con 0x56377b0cdd30 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdac1947b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56377afe59e0 con 0x56377af48e40 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.709+0000 7fdac1947b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56377afe9a20 con 0x56377b0c7db0 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.710+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0cdd30 0x56377b0c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.710+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0cdd30 0x56377b0c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40310/0 (socket says 172.21.15.12:40310) 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.710+0000 7fdaabe00700 1 -- 172.21.15.12:0/2951997552 learned_addr learned my addr 172.21.15.12:0/2951997552 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:45.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.710+0000 7fdab0e00700 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377af48e40 0x56377b0cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.711+0000 7fdab0e00700 1 -- 172.21.15.12:0/2951997552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 msgr2=0x56377b0c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.711+0000 7fdab0e00700 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.711+0000 7fdab0e00700 1 -- 172.21.15.12:0/2951997552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0cdd30 msgr2=0x56377b0c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.711+0000 7fdab0e00700 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0cdd30 0x56377b0c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.711+0000 7fdab0e00700 1 -- 172.21.15.12:0/2951997552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56377afae420 con 0x56377af48e40 2024-05-27T07:28:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.712+0000 7fdaa8c00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fda900b2a70 0x7fda900b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.712+0000 7fdab0e00700 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377af48e40 0x56377b0cd7f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b070 tx=0x7fdaa400cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb363add5dd03058 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.712+0000 7fdaa8c00700 1 -- 172.21.15.12:0/2359454544 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9466+0+0 (secure 0 0 0) 0x7fda980e2840 con 0x7fdaac103a00 2024-05-27T07:28:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.712+0000 7fdaabe00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fda900b2a70 0x7fda900b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.712+0000 7fdaa1600700 1 -- 172.21.15.12:0/2951997552 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40090d0 con 0x56377af48e40 2024-05-27T07:28:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.713+0000 7fdaa1600700 1 -- 172.21.15.12:0/2951997552 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdaa4009270 con 0x56377af48e40 2024-05-27T07:28:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.713+0000 7fdaa1600700 1 -- 172.21.15.12:0/2951997552 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4007c50 con 0x56377af48e40 2024-05-27T07:28:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.714+0000 7fdac1947b80 1 -- 172.21.15.12:0/2951997552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377af48e40 msgr2=0x56377b0cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.714+0000 7fdac1947b80 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377af48e40 0x56377b0cd7f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b070 tx=0x7fdaa400cf10 comp rx=0 tx=0).stop 2024-05-27T07:28:45.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.714+0000 7fdac1947b80 1 -- 172.21.15.12:0/2951997552 shutdown_connections 2024-05-27T07:28:45.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.714+0000 7fdac1947b80 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377af48e40 0x56377b0cd7f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.714+0000 7fdac1947b80 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.714+0000 7fdac1947b80 1 --2- 172.21.15.12:0/2951997552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0cdd30 0x56377b0c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.714+0000 7fdac1947b80 1 -- 172.21.15.12:0/2951997552 >> 172.21.15.12:0/2951997552 conn(0x56377afa8fc0 msgr2=0x56377b0c28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.716+0000 7fdaabe00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fda900b2a70 0x7fda900b4f80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00de10 tx=0x7fda9c00d560 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.716+0000 7fdac1947b80 1 -- 172.21.15.12:0/2951997552 shutdown_connections 2024-05-27T07:28:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.716+0000 7fdac1947b80 1 -- 172.21.15.12:0/2951997552 wait complete. 2024-05-27T07:28:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.717+0000 7fdac1947b80 1 -- start start 2024-05-27T07:28:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.717+0000 7fdac1947b80 1 Processor -- start 2024-05-27T07:28:45.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0d14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0d1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0cdd30 0x56377b0d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0d14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdac1947b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56377aeebfa0 con 0x56377af48e40 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdac1947b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56377af86760 con 0x56377b0cdd30 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdac1947b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56377b0d47a0 con 0x56377b0c7db0 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0d14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40326/0 (socket says 172.21.15.12:40326) 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.718+0000 7fdaabe00700 1 -- 172.21.15.12:0/1904170992 learned_addr learned my addr 172.21.15.12:0/1904170992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdac1947b80 10 client.0 _ll_register_callbacks cb 0x56377af8fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdaab400700 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0d1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdab0e00700 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0cdd30 0x56377b0d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdaab400700 1 -- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0cdd30 msgr2=0x56377b0d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdaab400700 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0cdd30 0x56377b0d1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdaab400700 1 -- 172.21.15.12:0/1904170992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 msgr2=0x56377b0d14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdaab400700 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0d14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.719+0000 7fdaab400700 1 -- 172.21.15.12:0/1904170992 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56377af358d0 con 0x56377b0c7db0 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.720+0000 7fdaab400700 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0d1a10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda980dfb80 tx=0x7fda980e00b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1555cbfd122fc5ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.720+0000 7fda96000700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.720+0000 7fda96000700 1 -- 172.21.15.12:0/1904170992 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980a9e10 con 0x56377b0c7db0 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.720+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaa4009270 con 0x56377b0c7db0 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.721+0000 7fda96000700 1 -- 172.21.15.12:0/1904170992 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9466+0+0 (secure 0 0 0) 0x7fda980e23d0 con 0x56377b0c7db0 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.721+0000 7fda96000700 1 -- 172.21.15.12:0/1904170992 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fda980a9710 con 0x56377b0c7db0 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.721+0000 7fdac1947b80 10 client.25888 fetch_fsmap learned FSMap version 92 2024-05-27T07:28:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.721+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=4+}) -- 0x56377afb6060 con 0x56377b0c7db0 2024-05-27T07:28:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fda96000700 1 -- 172.21.15.12:0/1904170992 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdaa4009270 con 0x56377b0c7db0 2024-05-27T07:28:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fda96000700 1 -- 172.21.15.12:0/1904170992 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 92) ==== 5345+0+0 (secure 0 0 0) 0x7fda98020030 con 0x56377b0c7db0 2024-05-27T07:28:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fdac1947b80 10 client.25888 fetch_fsmap finished waiting for FSMap version 92 2024-05-27T07:28:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fdac1947b80 10 client.25888 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:28:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fdac1947b80 20 client.25888 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fdac1947b80 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b0e1850 0x56377b0e3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fdac1947b80 4 client.25888 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.722+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56377aff7d90 con 0x56377b0e1850 2024-05-27T07:28:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.723+0000 7fdaabe00700 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b0e1850 0x56377b0e3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.724+0000 7fdaabe00700 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b0e1850 0x56377b0e3cc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5df22ae793f68607 server_cookie=930af5459cafd8f6 in_seq=0 out_seq=0 2024-05-27T07:28:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.724+0000 7fda96000700 10 client.25888 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:28:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.726+0000 7fda96000700 1 -- 172.21.15.12:0/1904170992 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56377aff7d90 con 0x56377b0e1850 2024-05-27T07:28:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.726+0000 7fda96000700 10 client.25888 handle_command_reply: tid=0 2024-05-27T07:28:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.726+0000 7fdac1947b80 1 client.25888 shutdown 2024-05-27T07:28:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 msgr2=0x56377b0d1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0d1a10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda980dfb80 tx=0x7fda980e00b0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 shutdown_connections 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b0e1850 0x56377b0e3cc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0cdd30 0x56377b0d1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0d14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 --2- 172.21.15.12:0/1904170992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0c7db0 0x56377b0d1a10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 >> 172.21.15.12:0/1904170992 conn(0x56377afa8fc0 msgr2=0x56377b0c2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fda96000700 0 client.25888 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 shutdown_connections 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 1 -- 172.21.15.12:0/1904170992 wait complete. 2024-05-27T07:28:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.727+0000 7fdac1947b80 20 client.25888 trim_cache size 0 max 16384 2024-05-27T07:28:45.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:45 smithi162 ceph-mon[35676]: pgmap v225: 129 pgs: 129 active+clean; 99 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-05-27T07:28:45.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:45 smithi062 ceph-mon[35304]: pgmap v225: 129 pgs: 129 active+clean; 99 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-05-27T07:28:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 Processor -- start 2024-05-27T07:28:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 -- start start 2024-05-27T07:28:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0c7db0 0x56377b0dd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c010080 con 0x56377af48e40 2024-05-27T07:28:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c00dd50 con 0x56377b0c7db0 2024-05-27T07:28:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.752+0000 7fdac1947b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c00f9b0 con 0x56377b0cdd30 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.753+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.753+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40336/0 (socket says 172.21.15.12:40336) 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.753+0000 7fdaabe00700 1 -- 172.21.15.12:0/3415607546 learned_addr learned my addr 172.21.15.12:0/3415607546 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.753+0000 7fdaab400700 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0c7db0 0x56377b0dd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.753+0000 7fdab0e00700 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.753+0000 7fdaabe00700 1 -- 172.21.15.12:0/3415607546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 msgr2=0x56377b0cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.753+0000 7fdaabe00700 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.754+0000 7fdaabe00700 1 -- 172.21.15.12:0/3415607546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0c7db0 msgr2=0x56377b0dd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.754+0000 7fdaabe00700 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0c7db0 0x56377b0dd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.754+0000 7fdaabe00700 1 -- 172.21.15.12:0/3415607546 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56377b0def80 con 0x56377af48e40 2024-05-27T07:28:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.754+0000 7fdaabe00700 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0c7870 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fda9c014010 tx=0x7fda9c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcacc1c2f13af07 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.755+0000 7fdaa0c00700 1 -- 172.21.15.12:0/3415607546 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c004020 con 0x56377af48e40 2024-05-27T07:28:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.755+0000 7fdaa0c00700 1 -- 172.21.15.12:0/3415607546 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda9c003680 con 0x56377af48e40 2024-05-27T07:28:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.755+0000 7fdaa0c00700 1 -- 172.21.15.12:0/3415607546 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c029680 con 0x56377af48e40 2024-05-27T07:28:45.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.756+0000 7fdac1947b80 1 -- 172.21.15.12:0/3415607546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 msgr2=0x56377b0c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.756+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0c7870 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fda9c014010 tx=0x7fda9c015000 comp rx=0 tx=0).stop 2024-05-27T07:28:45.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.756+0000 7fdac1947b80 1 -- 172.21.15.12:0/3415607546 shutdown_connections 2024-05-27T07:28:45.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.756+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.756+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377af48e40 0x56377b0c7870 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.756+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3415607546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0c7db0 0x56377b0dd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.756+0000 7fdac1947b80 1 -- 172.21.15.12:0/3415607546 >> 172.21.15.12:0/3415607546 conn(0x56377afa8fc0 msgr2=0x56377b0cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.757+0000 7fdac1947b80 1 -- 172.21.15.12:0/3415607546 shutdown_connections 2024-05-27T07:28:45.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.757+0000 7fdac1947b80 1 -- 172.21.15.12:0/3415607546 wait complete. 2024-05-27T07:28:45.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.757+0000 7fdac1947b80 1 -- start start 2024-05-27T07:28:45.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 1 Processor -- start 2024-05-27T07:28:45.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0c7db0 0x56377b0d9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0d9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0e84b0 0x56377b0e88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c00eb50 con 0x56377b0c7db0 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c00e420 con 0x56377b0e84b0 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c00f280 con 0x56377b0cdd30 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.758+0000 7fdac1947b80 10 client.0 _ll_register_callbacks cb 0x56377af8fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0c7db0 0x56377b0d9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0c7db0 0x56377b0d9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40352/0 (socket says 172.21.15.12:40352) 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 -- 172.21.15.12:0/3765580026 learned_addr learned my addr 172.21.15.12:0/3765580026 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdab0e00700 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0e84b0 0x56377b0e88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaab400700 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0d9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 -- 172.21.15.12:0/3765580026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 msgr2=0x56377b0d9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0d9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 -- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0e84b0 msgr2=0x56377b0e88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0e84b0 0x56377b0e88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.759+0000 7fdaabe00700 1 -- 172.21.15.12:0/3765580026 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56377b0c1580 con 0x56377b0c7db0 2024-05-27T07:28:45.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.760+0000 7fda96000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:45.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.760+0000 7fdaabe00700 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0c7db0 0x56377b0d9a90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00dbc0 tx=0x7fda9c004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e077879a8cf4fa8b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.761+0000 7fda96000700 1 -- 172.21.15.12:0/3765580026 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c003a30 con 0x56377b0c7db0 2024-05-27T07:28:45.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.761+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda9c003680 con 0x56377b0c7db0 2024-05-27T07:28:45.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.761+0000 7fda96000700 1 -- 172.21.15.12:0/3765580026 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9466+0+0 (secure 0 0 0) 0x7fda9c003bd0 con 0x56377b0c7db0 2024-05-27T07:28:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.761+0000 7fda96000700 1 -- 172.21.15.12:0/3765580026 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fda9c003dd0 con 0x56377b0c7db0 2024-05-27T07:28:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.761+0000 7fdac1947b80 10 client.16230 fetch_fsmap learned FSMap version 92 2024-05-27T07:28:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.761+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=4+}) -- 0x56377b14bbf0 con 0x56377b0c7db0 2024-05-27T07:28:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fda96000700 1 -- 172.21.15.12:0/3765580026 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda9c003680 con 0x56377b0c7db0 2024-05-27T07:28:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fda96000700 1 -- 172.21.15.12:0/3765580026 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 92) ==== 5345+0+0 (secure 0 0 0) 0x7fda9c030070 con 0x56377b0c7db0 2024-05-27T07:28:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fdac1947b80 10 client.16230 fetch_fsmap finished waiting for FSMap version 92 2024-05-27T07:28:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fdac1947b80 10 client.16230 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:28:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fdac1947b80 20 client.16230 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b14c1b0 0x56377b14e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fdac1947b80 4 client.16230 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.762+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56377aff1570 con 0x56377b14c1b0 2024-05-27T07:28:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.763+0000 7fdaab400700 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b14c1b0 0x56377b14e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.764+0000 7fdaab400700 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b14c1b0 0x56377b14e5d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af069ba37990687c server_cookie=67612f101d2e7ce9 in_seq=0 out_seq=0 2024-05-27T07:28:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.764+0000 7fda96000700 10 client.16230 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:28:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.764+0000 7fda96000700 1 -- 172.21.15.12:0/3765580026 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56377aff1570 con 0x56377b14c1b0 2024-05-27T07:28:45.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.764+0000 7fda96000700 10 client.16230 handle_command_reply: tid=0 2024-05-27T07:28:45.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.764+0000 7fdac1947b80 1 client.16230 shutdown 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0c7db0 msgr2=0x56377b0d9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0c7db0 0x56377b0d9a90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00dbc0 tx=0x7fda9c004160 comp rx=0 tx=0).stop 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 shutdown_connections 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56377b14c1b0 0x56377b14e5d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56377b0e84b0 0x56377b0e88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56377b0c7db0 0x56377b0d9a90 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 --2- 172.21.15.12:0/3765580026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56377b0cdd30 0x56377b0d9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 >> 172.21.15.12:0/3765580026 conn(0x56377afa8fc0 msgr2=0x56377b0d1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.765+0000 7fda96000700 0 client.16230 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.766+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 shutdown_connections 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.766+0000 7fdac1947b80 1 -- 172.21.15.12:0/3765580026 wait complete. 2024-05-27T07:28:45.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.766+0000 7fdac1947b80 20 client.16230 trim_cache size 0 max 16384 2024-05-27T07:28:45.757 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:28:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.766+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fda900b2a70 msgr2=0x7fda900b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fda900b2a70 0x7fda900b4f80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00de10 tx=0x7fda9c00d560 comp rx=0 tx=0).stop 2024-05-27T07:28:45.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac103a00 msgr2=0x7fdaac10e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac103a00 0x7fdaac10e0a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fda9800c9e0 tx=0x7fda9800cda0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 shutdown_connections 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fda900b2a70 0x7fda900b4f80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdaac1030b0 0x7fdaac10db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdaac110c60 0x7fdaac10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 --2- 172.21.15.12:0/2359454544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaac103a00 0x7fdaac10e0a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 >> 172.21.15.12:0/2359454544 conn(0x7fdaac0fedb0 msgr2=0x7fdaac1127f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 shutdown_connections 2024-05-27T07:28:45.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:45.767+0000 7fdab2e00700 1 -- 172.21.15.12:0/2359454544 wait complete. 2024-05-27T07:28:46.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:28:47.177 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:47.578 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:47 smithi012 ceph-mon[31473]: pgmap v226: 129 pgs: 129 active+clean; 99 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.5 MiB/s wr, 194 op/s 2024-05-27T07:28:47.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:47 smithi162 ceph-mon[35676]: pgmap v226: 129 pgs: 129 active+clean; 99 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.5 MiB/s wr, 194 op/s 2024-05-27T07:28:47.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:47 smithi062 ceph-mon[35304]: pgmap v226: 129 pgs: 129 active+clean; 99 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.5 MiB/s wr, 194 op/s 2024-05-27T07:28:49.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:49 smithi012 ceph-mon[31473]: pgmap v227: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:49.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:49 smithi162 ceph-mon[35676]: pgmap v227: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:49.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:49 smithi062 ceph-mon[35304]: pgmap v227: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:50.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 -- 172.21.15.12:0/394675883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d41030b0 msgr2=0x7f91d41034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 --2- 172.21.15.12:0/394675883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d41030b0 0x7f91d41034c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009990 tx=0x7f91c4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 -- 172.21.15.12:0/394675883 shutdown_connections 2024-05-27T07:28:50.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 --2- 172.21.15.12:0/394675883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91d4110c60 0x7f91d4113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 --2- 172.21.15.12:0/394675883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d41030b0 0x7f91d41034c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 --2- 172.21.15.12:0/394675883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91d4103a00 0x7f91d4110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 -- 172.21.15.12:0/394675883 >> 172.21.15.12:0/394675883 conn(0x7f91d40fedb0 msgr2=0x7f91d41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 -- 172.21.15.12:0/394675883 shutdown_connections 2024-05-27T07:28:50.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.083+0000 7f91d9000700 1 -- 172.21.15.12:0/394675883 wait complete. 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.085+0000 7f91d9000700 1 Processor -- start 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.085+0000 7f91d9000700 1 -- start start 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.085+0000 7f91d9000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91d41030b0 0x7f91d419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d9000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d4103a00 0x7f91d419d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d9000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91d4110c60 0x7f91d41a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d9000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91d4116550 con 0x7f91d4103a00 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d9000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41163d0 con 0x7f91d4110c60 2024-05-27T07:28:50.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d9000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41166d0 con 0x7f91d41030b0 2024-05-27T07:28:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d4103a00 0x7f91d419d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d4103a00 0x7f91d419d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40370/0 (socket says 172.21.15.12:40370) 2024-05-27T07:28:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d1400700 1 -- 172.21.15.12:0/2631985151 learned_addr learned my addr 172.21.15.12:0/2631985151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d2800700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91d4110c60 0x7f91d41a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.086+0000 7f91d1e00700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91d41030b0 0x7f91d419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.087+0000 7f91d1400700 1 -- 172.21.15.12:0/2631985151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91d41030b0 msgr2=0x7f91d419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.087+0000 7f91d1400700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91d41030b0 0x7f91d419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.087+0000 7f91d1400700 1 -- 172.21.15.12:0/2631985151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91d4110c60 msgr2=0x7f91d41a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.087+0000 7f91d1400700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91d4110c60 0x7f91d41a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.087+0000 7f91d1400700 1 -- 172.21.15.12:0/2631985151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d41a1e50 con 0x7f91d4103a00 2024-05-27T07:28:50.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.088+0000 7f91d1400700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d4103a00 0x7f91d419d520 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f91c000d980 tx=0x7f91c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.088+0000 7f91caa00700 1 -- 172.21.15.12:0/2631985151 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0017070 con 0x7f91d4103a00 2024-05-27T07:28:50.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.088+0000 7f91caa00700 1 -- 172.21.15.12:0/2631985151 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91c000fd20 con 0x7f91d4103a00 2024-05-27T07:28:50.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.088+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d41a2140 con 0x7f91d4103a00 2024-05-27T07:28:50.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.089+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d41a9a60 con 0x7f91d4103a00 2024-05-27T07:28:50.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.089+0000 7f91caa00700 1 -- 172.21.15.12:0/2631985151 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c000e3b0 con 0x7f91d4103a00 2024-05-27T07:28:50.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.095+0000 7f91e7ab4b80 1 Processor -- start 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.095+0000 7f91e7ab4b80 1 -- start start 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.095+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138129f0 0x5644138a5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 0x5644138b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5644137d3e40 con 0x564413726e40 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5644137c39e0 con 0x5644138129f0 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644137c7a20 con 0x5644138a5e70 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91d2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138129f0 0x5644138a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91d2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138129f0 0x5644138a5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:59488/0 (socket says 172.21.15.12:59488) 2024-05-27T07:28:50.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.096+0000 7f91d2800700 1 -- 172.21.15.12:0/3416326841 learned_addr learned my addr 172.21.15.12:0/3416326841 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:50.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.097+0000 7f91d1e00700 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.097+0000 7f91d2800700 1 -- 172.21.15.12:0/3416326841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 msgr2=0x5644138b2110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:28:50.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.097+0000 7f91d2800700 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 0x5644138b2110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.097+0000 7f91d2800700 1 -- 172.21.15.12:0/3416326841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 msgr2=0x5644138a5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:50.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.098+0000 7f91d2800700 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.098+0000 7f91d2800700 1 -- 172.21.15.12:0/3416326841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441378c420 con 0x5644138129f0 2024-05-27T07:28:50.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.099+0000 7f91d2800700 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138129f0 0x5644138a5330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003fa0 tx=0x7f91cc009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a009ff6f640eff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.100+0000 7f91bf400700 1 -- 172.21.15.12:0/3416326841 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc004bb0 con 0x5644138129f0 2024-05-27T07:28:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.100+0000 7f91bf400700 1 -- 172.21.15.12:0/3416326841 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91cc0048e0 con 0x5644138129f0 2024-05-27T07:28:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.100+0000 7f91bf400700 1 -- 172.21.15.12:0/3416326841 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc007d20 con 0x5644138129f0 2024-05-27T07:28:50.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.102+0000 7f91caa00700 1 -- 172.21.15.12:0/2631985151 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f91c000f890 con 0x7f91d4103a00 2024-05-27T07:28:50.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.102+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3416326841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138129f0 msgr2=0x5644138a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.102+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138129f0 0x5644138a5330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003fa0 tx=0x7f91cc009ba0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.103+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3416326841 shutdown_connections 2024-05-27T07:28:50.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.103+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138129f0 0x5644138a5330 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.103+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 0x5644138b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.103+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3416326841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.103+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3416326841 >> 172.21.15.12:0/3416326841 conn(0x5644137c0390 msgr2=0x5644138a14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.103+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3416326841 shutdown_connections 2024-05-27T07:28:50.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.103+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3416326841 wait complete. 2024-05-27T07:28:50.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.104+0000 7f91e7ab4b80 1 -- start start 2024-05-27T07:28:50.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.105+0000 7f91e7ab4b80 1 Processor -- start 2024-05-27T07:28:50.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564413726e40 0x5644138af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644138129f0 0x5644138afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 0x5644138b0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5644136c9fa0 con 0x5644138129f0 2024-05-27T07:28:50.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564413764760 con 0x564413726e40 2024-05-27T07:28:50.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644138c3b90 con 0x5644138a5e70 2024-05-27T07:28:50.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91d2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564413726e40 0x5644138af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91caa00700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91b80b29a0 0x7f91b80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.106+0000 7f91e7ab4b80 10 client.0 _ll_register_callbacks cb 0x56441376dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:50.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.107+0000 7f91d2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564413726e40 0x5644138af680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:59496/0 (socket says 172.21.15.12:59496) 2024-05-27T07:28:50.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.107+0000 7f91d2800700 1 -- 172.21.15.12:0/2287072550 learned_addr learned my addr 172.21.15.12:0/2287072550 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:50.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.107+0000 7f91caa00700 1 -- 172.21.15.12:0/2631985151 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9466+0+0 (secure 0 0 0) 0x7f91c00e2680 con 0x7f91d4103a00 2024-05-27T07:28:50.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.107+0000 7f91d1e00700 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644138129f0 0x5644138afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.107+0000 7f91d2800700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91b80b29a0 0x7f91b80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.108+0000 7f91d1400700 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 0x5644138b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.108+0000 7f91d1e00700 1 -- 172.21.15.12:0/2287072550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 msgr2=0x5644138b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:50.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.108+0000 7f91d1e00700 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 0x5644138b0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.108+0000 7f91d1e00700 1 -- 172.21.15.12:0/2287072550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564413726e40 msgr2=0x5644138af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:50.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.108+0000 7f91d1e00700 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564413726e40 0x5644138af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.108+0000 7f91d1e00700 1 -- 172.21.15.12:0/2287072550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644137138d0 con 0x5644138129f0 2024-05-27T07:28:50.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.109+0000 7f91d1e00700 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644138129f0 0x5644138afbc0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f91c4006010 tx=0x7f91c4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264bcac55aac1686 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.109+0000 7f91b3e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.110+0000 7f91b3e00700 1 -- 172.21.15.12:0/2287072550 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4003bd0 con 0x5644138129f0 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.110+0000 7f91d2800700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91b80b29a0 0x7f91b80b4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91cc000f80 tx=0x7f91cc009100 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.110+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91cc0048e0 con 0x5644138129f0 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.110+0000 7f91b3e00700 1 -- 172.21.15.12:0/2287072550 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9466+0+0 (secure 0 0 0) 0x7f91c4003d70 con 0x5644138129f0 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.110+0000 7f91b3e00700 1 -- 172.21.15.12:0/2287072550 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f91c4031050 con 0x5644138129f0 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.110+0000 7f91e7ab4b80 10 client.16242 fetch_fsmap learned FSMap version 92 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.110+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=4+}) -- 0x564413794060 con 0x5644138129f0 2024-05-27T07:28:50.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91b3e00700 1 -- 172.21.15.12:0/2287072550 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91cc0048e0 con 0x5644138129f0 2024-05-27T07:28:50.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91b3e00700 1 -- 172.21.15.12:0/2287072550 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 92) ==== 5345+0+0 (secure 0 0 0) 0x7f91c4023670 con 0x5644138129f0 2024-05-27T07:28:50.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91e7ab4b80 10 client.16242 fetch_fsmap finished waiting for FSMap version 92 2024-05-27T07:28:50.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91e7ab4b80 10 client.16242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:50.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91e7ab4b80 20 client.16242 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:50.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5644138d0fe0 0x5644138d3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91e7ab4b80 4 client.16242 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:50.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.111+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644137d5d90 con 0x5644138d0fe0 2024-05-27T07:28:50.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.112+0000 7f91d1400700 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5644138d0fe0 0x5644138d3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.113+0000 7f91d1400700 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5644138d0fe0 0x5644138d3430 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1de053506db6a8d server_cookie=9e282f1b47bb1e39 in_seq=0 out_seq=0 2024-05-27T07:28:50.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.113+0000 7f91b3e00700 10 client.16242 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:50.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91b3e00700 1 -- 172.21.15.12:0/2287072550 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5644137d5d90 con 0x5644138d0fe0 2024-05-27T07:28:50.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91b3e00700 10 client.16242 handle_command_reply: tid=0 2024-05-27T07:28:50.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 client.16242 shutdown 2024-05-27T07:28:50.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644138129f0 msgr2=0x5644138afbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644138129f0 0x5644138afbc0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f91c4006010 tx=0x7f91c4005800 comp rx=0 tx=0).stop 2024-05-27T07:28:50.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 shutdown_connections 2024-05-27T07:28:50.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5644138d0fe0 0x5644138d3430 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564413726e40 0x5644138af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138a5e70 0x5644138b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/2287072550 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5644138129f0 0x5644138afbc0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91b3e00700 0 client.16242 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:50.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.117+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 >> 172.21.15.12:0/2287072550 conn(0x5644137c0390 msgr2=0x5644138a5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.118+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 shutdown_connections 2024-05-27T07:28:50.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.118+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/2287072550 wait complete. 2024-05-27T07:28:50.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.118+0000 7f91e7ab4b80 20 client.16242 trim_cache size 0 max 16384 2024-05-27T07:28:50.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.140+0000 7f91e7ab4b80 1 Processor -- start 2024-05-27T07:28:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91e7ab4b80 1 -- start start 2024-05-27T07:28:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138aae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 0x5644138cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91b4010090 con 0x564413726e40 2024-05-27T07:28:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91b400dd60 con 0x5644138a5e70 2024-05-27T07:28:50.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91b400f9c0 con 0x5644138129f0 2024-05-27T07:28:50.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91d1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91d1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40404/0 (socket says 172.21.15.12:40404) 2024-05-27T07:28:50.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.141+0000 7f91d1e00700 1 -- 172.21.15.12:0/3778175958 learned_addr learned my addr 172.21.15.12:0/3778175958 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:50.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.142+0000 7f91d1400700 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138aae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.142+0000 7f91d1e00700 1 -- 172.21.15.12:0/3778175958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 msgr2=0x5644138aae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:50.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.142+0000 7f91d1e00700 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138aae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.142+0000 7f91d1e00700 1 -- 172.21.15.12:0/3778175958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 msgr2=0x5644138cccc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:28:50.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.142+0000 7f91d1e00700 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 0x5644138cccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.142+0000 7f91d1e00700 1 -- 172.21.15.12:0/3778175958 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644138a2850 con 0x564413726e40 2024-05-27T07:28:50.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.143+0000 7f91d1e00700 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a1660 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f91c4023bd0 tx=0x7f91c4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e9a808d089a45b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.143+0000 7f91bea00700 1 -- 172.21.15.12:0/3778175958 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4023670 con 0x564413726e40 2024-05-27T07:28:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.144+0000 7f91bea00700 1 -- 172.21.15.12:0/3778175958 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91c4028070 con 0x564413726e40 2024-05-27T07:28:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.144+0000 7f91bea00700 1 -- 172.21.15.12:0/3778175958 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c400f530 con 0x564413726e40 2024-05-27T07:28:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3778175958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 msgr2=0x5644138a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a1660 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f91c4023bd0 tx=0x7f91c4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3778175958 shutdown_connections 2024-05-27T07:28:50.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 0x5644138cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138a1660 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/3778175958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138aae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3778175958 >> 172.21.15.12:0/3778175958 conn(0x5644137c0390 msgr2=0x5644138a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.145+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3778175958 shutdown_connections 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.146+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/3778175958 wait complete. 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.146+0000 7f91e7ab4b80 1 -- start start 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.146+0000 7f91e7ab4b80 1 Processor -- start 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.147+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.147+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.147+0000 7f91e7ab4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 0x5644138c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91b400eb60 con 0x564413726e40 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91b400e430 con 0x5644138a5e70 2024-05-27T07:28:50.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91e7ab4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91b400f290 con 0x5644138129f0 2024-05-27T07:28:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91d1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91d1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138c3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40420/0 (socket says 172.21.15.12:40420) 2024-05-27T07:28:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91d1e00700 1 -- 172.21.15.12:0/4218588121 learned_addr learned my addr 172.21.15.12:0/4218588121 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91e7ab4b80 10 client.0 _ll_register_callbacks cb 0x56441376dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91d1400700 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.148+0000 7f91d2800700 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 0x5644138c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.149+0000 7f91d1e00700 1 -- 172.21.15.12:0/4218588121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 msgr2=0x5644138c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.149+0000 7f91d1e00700 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138c43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.149+0000 7f91d1e00700 1 -- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 msgr2=0x5644138c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.149+0000 7f91d1e00700 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 0x5644138c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.149+0000 7f91d1e00700 1 -- 172.21.15.12:0/4218588121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564413939af0 con 0x564413726e40 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.149+0000 7f91d1e00700 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138c3e70 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f91c400b9a0 tx=0x7f91c4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927cfbc0ccf6ea98 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.149+0000 7f91b3e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91b3e00700 1 -- 172.21.15.12:0/4218588121 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c400bcd0 con 0x564413726e40 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91c4028070 con 0x564413726e40 2024-05-27T07:28:50.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91b3e00700 1 -- 172.21.15.12:0/4218588121 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9466+0+0 (secure 0 0 0) 0x7f91c40039d0 con 0x564413726e40 2024-05-27T07:28:50.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91b3e00700 1 -- 172.21.15.12:0/4218588121 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f91c404d050 con 0x564413726e40 2024-05-27T07:28:50.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91e7ab4b80 10 client.16248 fetch_fsmap learned FSMap version 92 2024-05-27T07:28:50.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=4+}) -- 0x56441393ac70 con 0x564413726e40 2024-05-27T07:28:50.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91b3e00700 1 -- 172.21.15.12:0/4218588121 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91c4028070 con 0x564413726e40 2024-05-27T07:28:50.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.150+0000 7f91b3e00700 1 -- 172.21.15.12:0/4218588121 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 92) ==== 5345+0+0 (secure 0 0 0) 0x7f91c400fa20 con 0x564413726e40 2024-05-27T07:28:50.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.151+0000 7f91e7ab4b80 10 client.16248 fetch_fsmap finished waiting for FSMap version 92 2024-05-27T07:28:50.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.151+0000 7f91e7ab4b80 10 client.16248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:50.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.151+0000 7f91e7ab4b80 20 client.16248 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.151+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56441393b230 0x56441393d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.151+0000 7f91e7ab4b80 4 client.16248 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.151+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644137cf570 con 0x56441393b230 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.151+0000 7f91d1400700 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56441393b230 0x56441393d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.153+0000 7f91d1400700 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56441393b230 0x56441393d650 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=403c6ba77c8edbc2 server_cookie=d662db9e0ef8c211 in_seq=0 out_seq=0 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.153+0000 7f91b3e00700 10 client.16248 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.153+0000 7f91b3e00700 1 -- 172.21.15.12:0/4218588121 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644137cf570 con 0x56441393b230 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.153+0000 7f91b3e00700 10 client.16248 handle_command_reply: tid=0 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.153+0000 7f91e7ab4b80 1 client.16248 shutdown 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 msgr2=0x5644138c3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138c3e70 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f91c400b9a0 tx=0x7f91c4009680 comp rx=0 tx=0).stop 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 shutdown_connections 2024-05-27T07:28:50.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56441393b230 0x56441393d650 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644138a5e70 0x5644138c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564413726e40 0x5644138c3e70 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 --2- 172.21.15.12:0/4218588121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644138129f0 0x5644138c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 >> 172.21.15.12:0/4218588121 conn(0x5644137c0390 msgr2=0x5644138b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91b3e00700 0 client.16248 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 shutdown_connections 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 1 -- 172.21.15.12:0/4218588121 wait complete. 2024-05-27T07:28:50.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.154+0000 7f91e7ab4b80 20 client.16248 trim_cache size 0 max 16384 2024-05-27T07:28:50.151 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91b80b29a0 msgr2=0x7f91b80b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91b80b29a0 0x7f91b80b4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91cc000f80 tx=0x7f91cc009100 comp rx=0 tx=0).stop 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d4103a00 msgr2=0x7f91d419d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d4103a00 0x7f91d419d520 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f91c000d980 tx=0x7f91c000dc90 comp rx=0 tx=0).stop 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 shutdown_connections 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91b80b29a0 0x7f91b80b4eb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91d4110c60 0x7f91d41a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91d41030b0 0x7f91d419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 --2- 172.21.15.12:0/2631985151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91d4103a00 0x7f91d419d520 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 >> 172.21.15.12:0/2631985151 conn(0x7f91d40fedb0 msgr2=0x7f91d4112890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 shutdown_connections 2024-05-27T07:28:50.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:50.155+0000 7f91d9000700 1 -- 172.21.15.12:0/2631985151 wait complete. 2024-05-27T07:28:50.981 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:28:50.981 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: Stopped Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:50.981 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Consumed 3.742s CPU time. 2024-05-27T07:28:50.981 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-05-27T07:28:50.981 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:28:50.981 INFO:journalctl@ceph.mds.a.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: Failed to start Ceph mds.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:50.982 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:28:50.982 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: Stopped Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:50.982 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.042s CPU time. 2024-05-27T07:28:50.982 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:50 smithi012 systemd[1]: Starting Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:28:51.586 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 podman[122329]: 2024-05-27 07:28:51.250282746 +0000 UTC m=+0.038157368 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:51.586 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 podman[122329]: 2024-05-27 07:28:51.391991112 +0000 UTC m=+0.179865731 container create ebed5c8c03d3196d84ec94df00e46dd9c637f8708267aa391f7a44456cf97b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:51.586 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:51 smithi012 ceph-mon[31473]: pgmap v228: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:51.586 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:51 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:51.839 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 podman[122329]: 2024-05-27 07:28:51.680954291 +0000 UTC m=+0.468828962 container init ebed5c8c03d3196d84ec94df00e46dd9c637f8708267aa391f7a44456cf97b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:28:51.839 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 podman[122329]: 2024-05-27 07:28:51.692822157 +0000 UTC m=+0.480696786 container start ebed5c8c03d3196d84ec94df00e46dd9c637f8708267aa391f7a44456cf97b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-27T07:28:51.839 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 bash[122329]: ebed5c8c03d3196d84ec94df00e46dd9c637f8708267aa391f7a44456cf97b8c 2024-05-27T07:28:51.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 systemd[1]: Started Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:51.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 ceph-mds[122414]: -- 172.21.15.12:0/427076872 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b6155da40 con 0x559b615a5a80 2024-05-27T07:28:51.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:51 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: starting mds.d at 2024-05-27T07:28:51.972 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:51 smithi162 ceph-mon[35676]: pgmap v228: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:51.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:51 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:51.984 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:51 smithi062 ceph-mon[35304]: pgmap v228: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:51.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:51 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:52.044 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:28:52.045 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:28:52.840 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb6ce00700 time 2024-05-27T07:28:52.494928+0000 2024-05-27T07:28:52.840 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:52.840 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb828f8ba6] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: clone() 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: *** Caught signal (Aborted) ** 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: in thread 7fcb6ce00700 thread_name:md_log_replay 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: daemon mds.d restarted 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:28:52.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:52.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6836/446700025,v1:172.21.15.12:6837/446700025] up:boot 2024-05-27T07:28:52.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2024-05-27T07:28:52.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.496+0000 7fcb6ce00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb6ce00700 time 2024-05-27T07:28:52.494928+0000 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb828f8ba6] 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.844 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: clone() 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.845 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb81475d20] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: gsignal() 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: abort() 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb828f8ced] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.846 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 19: clone() 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.499+0000 7fcb6ce00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: in thread 7fcb6ce00700 thread_name:md_log_replay 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb81475d20] 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: gsignal() 2024-05-27T07:28:52.847 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: abort() 2024-05-27T07:28:52.848 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb828f8ced] 2024-05-27T07:28:52.848 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.848 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.848 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.848 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.848 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.848 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.849 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 19: clone() 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: -1> 2024-05-27T07:28:52.496+0000 7fcb6ce00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb6ce00700 time 2024-05-27T07:28:52.494928+0000 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb828f8ba6] 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.850 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.851 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: clone() 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 0> 2024-05-27T07:28:52.499+0000 7fcb6ce00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: in thread 7fcb6ce00700 thread_name:md_log_replay 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb81475d20] 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: gsignal() 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: abort() 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb828f8ced] 2024-05-27T07:28:52.852 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.853 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 19: clone() 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: -1> 2024-05-27T07:28:52.496+0000 7fcb6ce00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcb6ce00700 time 2024-05-27T07:28:52.494928+0000 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.854 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcb828f8ba6] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.855 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: clone() 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 0> 2024-05-27T07:28:52.499+0000 7fcb6ce00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: in thread 7fcb6ce00700 thread_name:md_log_replay 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcb81475d20] 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2: gsignal() 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 3: abort() 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcb828f8ced] 2024-05-27T07:28:52.856 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcb828f8dd1] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c05aacf23d] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c05aaba971] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c05aab8fa0] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c05aab9651] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c05aa4e347] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c05a74a87e] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c05a74fe94] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c05a75a969] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c05aaf348c] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c05aafc9cb] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 16: (Thread::entry_wrapper()+0x43) [0x7fcb828c772d] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcb828c7749] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcb8146b1ca] 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 19: clone() 2024-05-27T07:28:52.857 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:52.858 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[71934]: 2024-05-27T07:28:52.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: daemon mds.d restarted 2024-05-27T07:28:52.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:28:52.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6836/446700025,v1:172.21.15.12:6837/446700025] up:boot 2024-05-27T07:28:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2024-05-27T07:28:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:28:52.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: daemon mds.d restarted 2024-05-27T07:28:52.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:28:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6836/446700025,v1:172.21.15.12:6837/446700025] up:boot 2024-05-27T07:28:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 2024-05-27T07:28:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:28:53.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4cd5800700 time 2024-05-27T07:28:53.571906+0000 2024-05-27T07:28:53.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ceb3f3ba6] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: clone() 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.573+0000 7f4cd5800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4cd5800700 time 2024-05-27T07:28:53.571906+0000 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ceb3f3ba6] 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: clone() 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: *** Caught signal (Aborted) ** 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: in thread 7f4cd5800700 thread_name:md_log_replay 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4ce9f70d20] 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: gsignal() 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: abort() 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4ceb3f3ced] 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 19: clone() 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.577+0000 7f4cd5800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: in thread 7f4cd5800700 thread_name:md_log_replay 2024-05-27T07:28:53.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4ce9f70d20] 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: gsignal() 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: abort() 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4ceb3f3ced] 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 19: clone() 2024-05-27T07:28:53.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: -1> 2024-05-27T07:28:53.573+0000 7f4cd5800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4cd5800700 time 2024-05-27T07:28:53.571906+0000 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ceb3f3ba6] 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: clone() 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 0> 2024-05-27T07:28:53.577+0000 7f4cd5800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: in thread 7f4cd5800700 thread_name:md_log_replay 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4ce9f70d20] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: gsignal() 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: abort() 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4ceb3f3ced] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 19: clone() 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: -1> 2024-05-27T07:28:53.573+0000 7f4cd5800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4cd5800700 time 2024-05-27T07:28:53.571906+0000 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ceb3f3ba6] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: clone() 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 0> 2024-05-27T07:28:53.577+0000 7f4cd5800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: in thread 7f4cd5800700 thread_name:md_log_replay 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4ce9f70d20] 2024-05-27T07:28:53.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2: gsignal() 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 3: abort() 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4ceb3f3ced] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4ceb3f3dd1] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559b5184223d] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559b5182d971] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 8: (SnapRealm::check_cache() const+0x13e) [0x559b5182bfa0] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559b5182c651] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559b517c1347] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559b514bd87e] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559b514c2e94] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559b514cd969] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 14: (MDLog::_replay_thread()+0x1f38) [0x559b5186648c] 2024-05-27T07:28:53.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559b5186f9cb] 2024-05-27T07:28:53.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 16: (Thread::entry_wrapper()+0x43) [0x7f4ceb3c272d] 2024-05-27T07:28:53.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4ceb3c2749] 2024-05-27T07:28:53.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4ce9f661ca] 2024-05-27T07:28:53.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 19: clone() 2024-05-27T07:28:53.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:53.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[122410]: 2024-05-27T07:28:54.302 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:54.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:54 smithi012 ceph-mon[31473]: pgmap v230: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:54.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:54 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:54.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:54 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:54.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:54 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:54.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:54 smithi162 ceph-mon[35676]: pgmap v230: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:54.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:54 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:54.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:54 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:54.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:54 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:54.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:54 smithi062 ceph-mon[35304]: pgmap v230: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-05-27T07:28:54.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:54 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:54.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:54 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:54.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:54 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:28:55.089 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:54 smithi012 podman[122538]: 2024-05-27 07:28:54.807661623 +0000 UTC m=+2.179645606 container remove f20e208d4b35181eec0e98b46cea3908edcd2325685a809881a08ebe75fac05f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-27T07:28:55.090 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:54 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:55.711 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:55 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:28:55.711 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:55 smithi012 systemd[1]: Stopped Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:55.711 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:55 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.697s CPU time. 2024-05-27T07:28:55.711 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:55 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-05-27T07:28:55.712 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:55 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:55.712 INFO:journalctl@ceph.mds.g.smithi012.stdout:May 27 07:28:55 smithi012 systemd[1]: Failed to start Ceph mds.g for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:28:56.558 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:56 smithi012 podman[122582]: 2024-05-27 07:28:56.393536272 +0000 UTC m=+2.717181966 container remove ebed5c8c03d3196d84ec94df00e46dd9c637f8708267aa391f7a44456cf97b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:28:56.558 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:56 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:56.558 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:56 smithi012 ceph-mon[31473]: pgmap v231: 129 pgs: 129 active+clean; 101 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:28:56.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:56 smithi162 ceph-mon[35676]: pgmap v231: 129 pgs: 129 active+clean; 101 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:28:56.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:56 smithi062 ceph-mon[35304]: pgmap v231: 129 pgs: 129 active+clean; 101 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:28:57.143 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:56 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:28:57.144 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:28:56 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 2.140s CPU time. 2024-05-27T07:28:57.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:57 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:28:57.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:28:57 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 3.746s CPU time. 2024-05-27T07:28:57.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 -- 172.21.15.12:0/2591435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f14105e40 msgr2=0x7f6f1410e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:57.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2591435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f14105e40 0x7f6f1410e390 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009990 tx=0x7f6f00009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 -- 172.21.15.12:0/2591435766 shutdown_connections 2024-05-27T07:28:57.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2591435766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1410e8d0 0x7f6f14110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2591435766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f141054f0 0x7f6f14105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2591435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f14105e40 0x7f6f1410e390 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 -- 172.21.15.12:0/2591435766 >> 172.21.15.12:0/2591435766 conn(0x7f6f140fedb0 msgr2=0x7f6f141011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:57.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.920+0000 7f6f19e00700 1 -- 172.21.15.12:0/2591435766 shutdown_connections 2024-05-27T07:28:57.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.922+0000 7f6f19e00700 1 -- 172.21.15.12:0/2591435766 wait complete. 2024-05-27T07:28:57.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.923+0000 7f6f19e00700 1 Processor -- start 2024-05-27T07:28:57.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.924+0000 7f6f19e00700 1 -- start start 2024-05-27T07:28:57.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.925+0000 7f6f19e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f141054f0 0x7f6f1419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.925+0000 7f6f19e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f14105e40 0x7f6f1419d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.925+0000 7f6f19e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f1410e8d0 0x7f6f141a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.925+0000 7f6f19e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f14114330 con 0x7f6f1410e8d0 2024-05-27T07:28:57.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.925+0000 7f6f19e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141141b0 con 0x7f6f14105e40 2024-05-27T07:28:57.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.925+0000 7f6f19e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141144b0 con 0x7f6f141054f0 2024-05-27T07:28:57.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f1410e8d0 0x7f6f141a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f1410e8d0 0x7f6f141a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54732/0 (socket says 172.21.15.12:54732) 2024-05-27T07:28:57.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 -- 172.21.15.12:0/2872663588 learned_addr learned my addr 172.21.15.12:0/2872663588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:57.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f12a00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f141054f0 0x7f6f1419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f12000700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f14105e40 0x7f6f1419d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 -- 172.21.15.12:0/2872663588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f141054f0 msgr2=0x7f6f1419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:57.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f141054f0 0x7f6f1419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 -- 172.21.15.12:0/2872663588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f14105e40 msgr2=0x7f6f1419d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:57.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f14105e40 0x7f6f1419d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.926+0000 7f6f13400700 1 -- 172.21.15.12:0/2872663588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f141a1e30 con 0x7f6f1410e8d0 2024-05-27T07:28:57.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.927+0000 7f6f13400700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f1410e8d0 0x7f6f141a1790 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00ef30 tx=0x7f6f0c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:57.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.927+0000 7f6f18a00700 1 -- 172.21.15.12:0/2872663588 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c0092e0 con 0x7f6f1410e8d0 2024-05-27T07:28:57.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.927+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f141a20c0 con 0x7f6f1410e8d0 2024-05-27T07:28:57.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.927+0000 7f6f18a00700 1 -- 172.21.15.12:0/2872663588 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f0c004bb0 con 0x7f6f1410e8d0 2024-05-27T07:28:57.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.927+0000 7f6f18a00700 1 -- 172.21.15.12:0/2872663588 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c004020 con 0x7f6f1410e8d0 2024-05-27T07:28:57.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.927+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f141a9a40 con 0x7f6f1410e8d0 2024-05-27T07:28:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.933+0000 7f6f18a00700 1 -- 172.21.15.12:0/2872663588 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6f0c01b030 con 0x7f6f1410e8d0 2024-05-27T07:28:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.934+0000 7f6f289f3b80 1 Processor -- start 2024-05-27T07:28:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.934+0000 7f6f289f3b80 1 -- start start 2024-05-27T07:28:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595edba70 0x55c595f6e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f6e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595f6eef0 0x55c595f7b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c595e9cec0 con 0x55c595edba70 2024-05-27T07:28:57.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c595e8ca60 con 0x55c595defec0 2024-05-27T07:28:57.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c595e90aa0 con 0x55c595f6eef0 2024-05-27T07:28:57.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595edba70 0x55c595f6e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595edba70 0x55c595f6e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54746/0 (socket says 172.21.15.12:54746) 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.935+0000 7f6f12000700 1 -- 172.21.15.12:0/432437584 learned_addr learned my addr 172.21.15.12:0/432437584 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.936+0000 7f6f12000700 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595f6eef0 0x55c595f7b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.936+0000 7f6f18a00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6ef80b29f0 0x7f6ef80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.936+0000 7f6f18a00700 1 -- 172.21.15.12:0/2872663588 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9466+0+0 (secure 0 0 0) 0x7f6f0c0e2500 con 0x7f6f1410e8d0 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.937+0000 7f6f12a00700 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f6e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.937+0000 7f6f12000700 1 -- 172.21.15.12:0/432437584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595f6eef0 msgr2=0x55c595f7b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.937+0000 7f6f12000700 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595f6eef0 0x55c595f7b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.937+0000 7f6f12000700 1 -- 172.21.15.12:0/432437584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 msgr2=0x55c595f6e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:57.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.937+0000 7f6f12000700 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f6e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.937+0000 7f6f13400700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6ef80b29f0 0x7f6ef80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.937+0000 7f6f12000700 1 -- 172.21.15.12:0/432437584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c595e554a0 con 0x55c595edba70 2024-05-27T07:28:57.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.938+0000 7f6f12000700 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595edba70 0x55c595f6e3b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6f00000c00 tx=0x7f6f000058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bb882fe6e75084 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:57.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.938+0000 7f6f08c00700 1 -- 172.21.15.12:0/432437584 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00021040 con 0x55c595edba70 2024-05-27T07:28:57.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.938+0000 7f6f08c00700 1 -- 172.21.15.12:0/432437584 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f00003df0 con 0x55c595edba70 2024-05-27T07:28:57.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.938+0000 7f6f08c00700 1 -- 172.21.15.12:0/432437584 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000f460 con 0x55c595edba70 2024-05-27T07:28:57.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.939+0000 7f6f13400700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6ef80b29f0 0x7f6ef80b4f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0a9450 tx=0x7f6f0c009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:57.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 -- 172.21.15.12:0/432437584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595edba70 msgr2=0x55c595f6e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:57.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595edba70 0x55c595f6e3b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6f00000c00 tx=0x7f6f000058e0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 -- 172.21.15.12:0/432437584 shutdown_connections 2024-05-27T07:28:57.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595f6eef0 0x55c595f7b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595edba70 0x55c595f6e3b0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/432437584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f6e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 -- 172.21.15.12:0/432437584 >> 172.21.15.12:0/432437584 conn(0x55c595e89410 msgr2=0x55c595f6a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:57.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 -- 172.21.15.12:0/432437584 shutdown_connections 2024-05-27T07:28:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.940+0000 7f6f289f3b80 1 -- 172.21.15.12:0/432437584 wait complete. 2024-05-27T07:28:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.941+0000 7f6f289f3b80 1 -- start start 2024-05-27T07:28:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 1 Processor -- start 2024-05-27T07:28:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f78d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6eef0 0x55c595f79260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c595d93020 con 0x55c595f6eef0 2024-05-27T07:28:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c595e2d7e0 con 0x55c595defec0 2024-05-27T07:28:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c595f843b0 con 0x55c595edba70 2024-05-27T07:28:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.942+0000 7f6f289f3b80 10 client.0 _ll_register_callbacks cb 0x55c595e36cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.943+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6eef0 0x55c595f79260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.943+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6eef0 0x55c595f79260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54758/0 (socket says 172.21.15.12:54758) 2024-05-27T07:28:57.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.943+0000 7f6f12000700 1 -- 172.21.15.12:0/353674575 learned_addr learned my addr 172.21.15.12:0/353674575 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:57.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.943+0000 7f6f12000700 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.943+0000 7f6f12a00700 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f78d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.944+0000 7f6f12000700 1 -- 172.21.15.12:0/353674575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 msgr2=0x55c595f78d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.944+0000 7f6f12000700 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f78d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.944+0000 7f6f12000700 1 -- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 msgr2=0x55c595f787e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.944+0000 7f6f12000700 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f787e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.944+0000 7f6f12000700 1 -- 172.21.15.12:0/353674575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c595ddc950 con 0x55c595f6eef0 2024-05-27T07:28:57.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.945+0000 7f6f12000700 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6eef0 0x55c595f79260 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x55c595f78060 tx=0x7f6f0000bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3adc8b61df143001 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:57.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.945+0000 7f6efd600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:57.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.945+0000 7f6efd600700 1 -- 172.21.15.12:0/353674575 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00031040 con 0x55c595f6eef0 2024-05-27T07:28:57.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.945+0000 7f6efd600700 1 -- 172.21.15.12:0/353674575 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9466+0+0 (secure 0 0 0) 0x7f6f00003b60 con 0x55c595f6eef0 2024-05-27T07:28:57.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.945+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f00003df0 con 0x55c595f6eef0 2024-05-27T07:28:57.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.946+0000 7f6efd600700 1 -- 172.21.15.12:0/353674575 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f6f0000b6b0 con 0x55c595f6eef0 2024-05-27T07:28:57.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.946+0000 7f6f289f3b80 10 client.16266 fetch_fsmap learned FSMap version 93 2024-05-27T07:28:57.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.946+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x55c595e5d0e0 con 0x55c595f6eef0 2024-05-27T07:28:57.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6efd600700 1 -- 172.21.15.12:0/353674575 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f00003df0 con 0x55c595f6eef0 2024-05-27T07:28:57.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6efd600700 1 -- 172.21.15.12:0/353674575 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 93) ==== 4740+0+0 (secure 0 0 0) 0x7f6f00028c10 con 0x55c595f6eef0 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6f289f3b80 10 client.16266 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6f289f3b80 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6f289f3b80 20 client.16266 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595f911b0 0x55c595f935d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6f289f3b80 4 client.16266 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.947+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c595e9ee10 con 0x55c595f911b0 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.949+0000 7f6f12a00700 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595f911b0 0x55c595f935d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.950+0000 7f6f12a00700 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595f911b0 0x55c595f935d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f6ea65dd51febb8 server_cookie=a8e078b7aeab85f5 in_seq=0 out_seq=0 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.951+0000 7f6efd600700 10 client.16266 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6efd600700 1 -- 172.21.15.12:0/353674575 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c595e9ee10 con 0x55c595f911b0 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6efd600700 10 client.16266 handle_command_reply: tid=0 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 client.16266 shutdown 2024-05-27T07:28:57.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6eef0 msgr2=0x55c595f79260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6eef0 0x55c595f79260 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x55c595f78060 tx=0x7f6f0000bb80 comp rx=0 tx=0).stop 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 shutdown_connections 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595f911b0 0x55c595f935d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6eef0 0x55c595f79260 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/353674575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f78d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.953+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 >> 172.21.15.12:0/353674575 conn(0x55c595e89410 msgr2=0x55c595f6e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.954+0000 7f6efd600700 0 client.16266 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.954+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 shutdown_connections 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.954+0000 7f6f289f3b80 1 -- 172.21.15.12:0/353674575 wait complete. 2024-05-27T07:28:57.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.954+0000 7f6f289f3b80 20 client.16266 trim_cache size 0 max 16384 2024-05-27T07:28:57.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.980+0000 7f6f289f3b80 1 Processor -- start 2024-05-27T07:28:58.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.980+0000 7f6f289f3b80 1 -- start start 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f72990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f8d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6e980 0x55c595f7a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141aef30 con 0x55c595f6e980 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141acc00 con 0x55c595defec0 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141ae860 con 0x55c595edba70 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6e980 0x55c595f7a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6e980 0x55c595f7a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54774/0 (socket says 172.21.15.12:54774) 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.981+0000 7f6f12000700 1 -- 172.21.15.12:0/3616052801 learned_addr learned my addr 172.21.15.12:0/3616052801 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.982+0000 7f6f12a00700 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f8d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.982+0000 7f6f12000700 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f72990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.982+0000 7f6f12000700 1 -- 172.21.15.12:0/3616052801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 msgr2=0x55c595f8d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.982+0000 7f6f12000700 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f8d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.982+0000 7f6f12000700 1 -- 172.21.15.12:0/3616052801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 msgr2=0x55c595f72990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.982+0000 7f6f12000700 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f72990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.982+0000 7f6f12000700 1 -- 172.21.15.12:0/3616052801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c595f662c0 con 0x55c595f6e980 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.983+0000 7f6f12000700 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6e980 0x55c595f7a5a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000f530 tx=0x7f6f00031040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e44a9fb62d535e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.983+0000 7f6effe00700 1 -- 172.21.15.12:0/3616052801 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000b850 con 0x55c595f6e980 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.983+0000 7f6effe00700 1 -- 172.21.15.12:0/3616052801 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f00005640 con 0x55c595f6e980 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.983+0000 7f6effe00700 1 -- 172.21.15.12:0/3616052801 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000e7c0 con 0x55c595f6e980 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 -- 172.21.15.12:0/3616052801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6e980 msgr2=0x55c595f7a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6e980 0x55c595f7a5a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000f530 tx=0x7f6f00031040 comp rx=0 tx=0).stop 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 -- 172.21.15.12:0/3616052801 shutdown_connections 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595defec0 0x55c595f72990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595f6e980 0x55c595f7a5a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/3616052801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f8d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 -- 172.21.15.12:0/3616052801 >> 172.21.15.12:0/3616052801 conn(0x55c595e89410 msgr2=0x55c595f79e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 -- 172.21.15.12:0/3616052801 shutdown_connections 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.985+0000 7f6f289f3b80 1 -- 172.21.15.12:0/3616052801 wait complete. 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.986+0000 7f6f289f3b80 1 -- start start 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.986+0000 7f6f289f3b80 1 Processor -- start 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.987+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595defec0 0x55c595f862f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.987+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f86830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.987+0000 7f6f289f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595f6e980 0x55c595f86d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.987+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141ada00 con 0x55c595defec0 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.987+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141ad2d0 con 0x55c595f6e980 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.987+0000 7f6f289f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141ae130 con 0x55c595edba70 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.987+0000 7f6f289f3b80 10 client.0 _ll_register_callbacks cb 0x55c595e36cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595defec0 0x55c595f862f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595defec0 0x55c595f862f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54776/0 (socket says 172.21.15.12:54776) 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 -- 172.21.15.12:0/1565547358 learned_addr learned my addr 172.21.15.12:0/1565547358 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12a00700 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f86830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 -- 172.21.15.12:0/1565547358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 msgr2=0x55c595f86830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f86830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 -- 172.21.15.12:0/1565547358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595f6e980 msgr2=0x55c595f86d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595f6e980 0x55c595f86d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.988+0000 7f6f12000700 1 -- 172.21.15.12:0/1565547358 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c595f795d0 con 0x55c595defec0 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6f12000700 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595defec0 0x55c595f862f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000f690 tx=0x7f6f0000e3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd12cf6cc7978ba4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6efd600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6efd600700 1 -- 172.21.15.12:0/1565547358 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000b330 con 0x55c595defec0 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6efd600700 1 -- 172.21.15.12:0/1565547358 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9466+0+0 (secure 0 0 0) 0x7f6f0000ebb0 con 0x55c595defec0 2024-05-27T07:28:58.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f00005640 con 0x55c595defec0 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6efd600700 1 -- 172.21.15.12:0/1565547358 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f6f00055050 con 0x55c595defec0 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6f289f3b80 10 client.16272 fetch_fsmap learned FSMap version 93 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.989+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x55c595ffcba0 con 0x55c595defec0 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6efd600700 1 -- 172.21.15.12:0/1565547358 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f00005640 con 0x55c595defec0 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6efd600700 1 -- 172.21.15.12:0/1565547358 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 93) ==== 4740+0+0 (secure 0 0 0) 0x7f6f00004c50 con 0x55c595defec0 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6f289f3b80 10 client.16272 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6f289f3b80 10 client.16272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6f289f3b80 20 client.16272 populate_metadata read hostname 'smithi012' 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595ffd0b0 0x55c595fff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6f289f3b80 4 client.16272 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:28:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.990+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c595e985f0 con 0x55c595ffd0b0 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.991+0000 7f6f12a00700 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595ffd0b0 0x55c595fff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.992+0000 7f6f12a00700 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595ffd0b0 0x55c595fff4d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c031cd710a9afb7 server_cookie=55078072eae7580f in_seq=0 out_seq=0 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.992+0000 7f6efd600700 10 client.16272 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6efd600700 1 -- 172.21.15.12:0/1565547358 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c595e985f0 con 0x55c595ffd0b0 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6efd600700 10 client.16272 handle_command_reply: tid=0 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 client.16272 shutdown 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595defec0 msgr2=0x55c595f862f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595defec0 0x55c595f862f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000f690 tx=0x7f6f0000e3e0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 shutdown_connections 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c595ffd0b0 0x55c595fff4d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c595f6e980 0x55c595f86d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c595defec0 0x55c595f862f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 --2- 172.21.15.12:0/1565547358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c595edba70 0x55c595f86830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 >> 172.21.15.12:0/1565547358 conn(0x55c595e89410 msgr2=0x55c595f79db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6efd600700 0 client.16272 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 shutdown_connections 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.996+0000 7f6f289f3b80 1 -- 172.21.15.12:0/1565547358 wait complete. 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.997+0000 7f6f289f3b80 20 client.16272 trim_cache size 0 max 16384 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "28a08984-c671-4e65-9faa-a8134d286ee1", 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:28:58.159 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:28:58.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.997+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6ef80b29f0 msgr2=0x7f6ef80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:58.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.997+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6ef80b29f0 0x7f6ef80b4f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0a9450 tx=0x7f6f0c009040 comp rx=0 tx=0).stop 2024-05-27T07:28:58.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.997+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f1410e8d0 msgr2=0x7f6f141a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:58.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.997+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f1410e8d0 0x7f6f141a1790 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00ef30 tx=0x7f6f0c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 shutdown_connections 2024-05-27T07:28:58.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6ef80b29f0 0x7f6ef80b4f00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f1410e8d0 0x7f6f141a1790 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f141054f0 0x7f6f1419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 --2- 172.21.15.12:0/2872663588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f14105e40 0x7f6f1419d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:58.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 >> 172.21.15.12:0/2872663588 conn(0x7f6f140fedb0 msgr2=0x7f6f140ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:58.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 shutdown_connections 2024-05-27T07:28:58.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:28:57.998+0000 7f6f19e00700 1 -- 172.21.15.12:0/2872663588 wait complete. 2024-05-27T07:28:58.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:58 smithi012 ceph-mon[31473]: pgmap v232: 129 pgs: 129 active+clean; 101 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:28:58.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:58 smithi162 ceph-mon[35676]: pgmap v232: 129 pgs: 129 active+clean; 101 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:28:58.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:58 smithi062 ceph-mon[35304]: pgmap v232: 129 pgs: 129 active+clean; 101 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:28:59.337 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:28:59.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:59 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:28:59.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:59 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:59 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:28:59.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:59 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:28:59.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:59 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:28:59.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:59 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:28:59 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:59 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:28:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:59 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:59 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:28:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:59 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:28:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:59 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:28:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:59 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:28:59 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:59 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:28:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:59 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:59 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:28:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:59 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:28:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:59 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:28:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:59 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:28:59 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:00.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:00 smithi012 ceph-mon[31473]: pgmap v233: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:00.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:00 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:00.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:00 smithi162 ceph-mon[35676]: pgmap v233: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:00.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:00 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:00.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:00 smithi062 ceph-mon[35304]: pgmap v233: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:00.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:00 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 -- 172.21.15.12:0/3058492166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 msgr2=0x7fd83c105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 --2- 172.21.15.12:0/3058492166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c105880 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009990 tx=0x7fd82c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 -- 172.21.15.12:0/3058492166 shutdown_connections 2024-05-27T07:29:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 --2- 172.21.15.12:0/3058492166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83c10e8e0 0x7fd83c110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 --2- 172.21.15.12:0/3058492166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c105880 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 --2- 172.21.15.12:0/3058492166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd83c105dc0 0x7fd83c10e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 -- 172.21.15.12:0/3058492166 >> 172.21.15.12:0/3058492166 conn(0x7fd83c0fef20 msgr2=0x7fd83c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 -- 172.21.15.12:0/3058492166 shutdown_connections 2024-05-27T07:29:02.430 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:02 smithi012 ceph-mon[31473]: pgmap v234: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:02.430 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:02 smithi012 ceph-mon[31473]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:29:02.430 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:02 smithi012 ceph-mon[31473]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:02.430 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:02 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.438+0000 7fd842800700 1 -- 172.21.15.12:0/3058492166 wait complete. 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.439+0000 7fd842800700 1 Processor -- start 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd842800700 1 -- start start 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd842800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd842800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd83c105dc0 0x7fd83c19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd842800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83c10e8e0 0x7fd83c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd842800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c1142e0 con 0x7fd83c105470 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd842800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c114160 con 0x7fd83c10e8e0 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd842800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c114460 con 0x7fd83c105dc0 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd83b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd83b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52262/0 (socket says 172.21.15.12:52262) 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.440+0000 7fd83b400700 1 -- 172.21.15.12:0/2073098745 learned_addr learned my addr 172.21.15.12:0/2073098745 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.441+0000 7fd83be00700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83c10e8e0 0x7fd83c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.441+0000 7fd83b400700 1 -- 172.21.15.12:0/2073098745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd83c105dc0 msgr2=0x7fd83c19d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.441+0000 7fd83b400700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd83c105dc0 0x7fd83c19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.441+0000 7fd83b400700 1 -- 172.21.15.12:0/2073098745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83c10e8e0 msgr2=0x7fd83c1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.441+0000 7fd83b400700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83c10e8e0 0x7fd83c1a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.441+0000 7fd83b400700 1 -- 172.21.15.12:0/2073098745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83c1a1dc0 con 0x7fd83c105470 2024-05-27T07:29:02.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.441+0000 7fd83b400700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c19cef0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0057c0 tx=0x7fd82c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:02.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.442+0000 7fd833e00700 1 -- 172.21.15.12:0/2073098745 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c027070 con 0x7fd83c105470 2024-05-27T07:29:02.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.442+0000 7fd833e00700 1 -- 172.21.15.12:0/2073098745 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd82c00fda0 con 0x7fd83c105470 2024-05-27T07:29:02.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.442+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd83c1a2050 con 0x7fd83c105470 2024-05-27T07:29:02.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.442+0000 7fd833e00700 1 -- 172.21.15.12:0/2073098745 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c022ca0 con 0x7fd83c105470 2024-05-27T07:29:02.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.442+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd83c1a2530 con 0x7fd83c105470 2024-05-27T07:29:02.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.447+0000 7fd833e00700 1 -- 172.21.15.12:0/2073098745 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd82c02a070 con 0x7fd83c105470 2024-05-27T07:29:02.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.448+0000 7fd8512d4b80 1 Processor -- start 2024-05-27T07:29:02.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.448+0000 7fd8512d4b80 1 -- start start 2024-05-27T07:29:02.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a785330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd833e00700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8200b2a70 0x7fd8200b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a785870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a792110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56498a6b3e40 con 0x56498a6f29f0 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56498a6a39e0 con 0x56498a785e70 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56498a6a7a20 con 0x56498a606e40 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd833e00700 1 -- 172.21.15.12:0/2073098745 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9642+0+0 (secure 0 0 0) 0x7fd82c0e3ad0 con 0x7fd83c105470 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd83be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a785330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd83be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a785330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52270/0 (socket says 172.21.15.12:52270) 2024-05-27T07:29:02.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.449+0000 7fd83be00700 1 -- 172.21.15.12:0/2487798223 learned_addr learned my addr 172.21.15.12:0/2487798223 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83aa00700 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a785870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83be00700 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a792110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83b400700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8200b2a70 0x7fd8200b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83be00700 1 -- 172.21.15.12:0/2487798223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 msgr2=0x56498a785870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83be00700 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a785870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83be00700 1 -- 172.21.15.12:0/2487798223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 msgr2=0x56498a792110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83be00700 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a792110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.450+0000 7fd83be00700 1 -- 172.21.15.12:0/2487798223 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56498a66c420 con 0x56498a6f29f0 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.451+0000 7fd83be00700 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a785330 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd8340076a0 tx=0x7fd8340093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d275bdb516ad6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.451+0000 7fd83b400700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8200b2a70 0x7fd8200b4f80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00ffb0 tx=0x7fd82c00b540 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:02.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.451+0000 7fd830c00700 1 -- 172.21.15.12:0/2487798223 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83400f050 con 0x56498a6f29f0 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.451+0000 7fd830c00700 1 -- 172.21.15.12:0/2487798223 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd834013040 con 0x56498a6f29f0 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.451+0000 7fd830c00700 1 -- 172.21.15.12:0/2487798223 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd834012040 con 0x56498a6f29f0 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.452+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2487798223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 msgr2=0x56498a785330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.452+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a785330 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd8340076a0 tx=0x7fd8340093f0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.452+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2487798223 shutdown_connections 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.452+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a792110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.452+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a785330 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.452+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2487798223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a785870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.453+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2487798223 >> 172.21.15.12:0/2487798223 conn(0x56498a6a0390 msgr2=0x56498a7814c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:02.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.453+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2487798223 shutdown_connections 2024-05-27T07:29:02.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.454+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2487798223 wait complete. 2024-05-27T07:29:02.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.454+0000 7fd8512d4b80 1 -- start start 2024-05-27T07:29:02.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.455+0000 7fd8512d4b80 1 Processor -- start 2024-05-27T07:29:02.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.455+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a606e40 0x56498a78f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.455+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.455+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a790190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.455+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56498a5a9fa0 con 0x56498a606e40 2024-05-27T07:29:02.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.455+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56498a644760 con 0x56498a785e70 2024-05-27T07:29:02.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.455+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56498a792b80 con 0x56498a6f29f0 2024-05-27T07:29:02.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.456+0000 7fd8512d4b80 10 client.0 _ll_register_callbacks cb 0x56498a64dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:02.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.456+0000 7fd83be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a606e40 0x56498a78f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.456+0000 7fd83be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a606e40 0x56498a78f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52286/0 (socket says 172.21.15.12:52286) 2024-05-27T07:29:02.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.456+0000 7fd83be00700 1 -- 172.21.15.12:0/3343436345 learned_addr learned my addr 172.21.15.12:0/3343436345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:02.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.456+0000 7fd83aa00700 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.457+0000 7fd83be00700 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a790190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.457+0000 7fd83be00700 1 -- 172.21.15.12:0/3343436345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 msgr2=0x56498a78fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.457+0000 7fd83be00700 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.457+0000 7fd83be00700 1 -- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 msgr2=0x56498a790190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.457+0000 7fd83be00700 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a790190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.457+0000 7fd83be00700 1 -- 172.21.15.12:0/3343436345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56498a5f38d0 con 0x56498a606e40 2024-05-27T07:29:02.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.458+0000 7fd83be00700 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a606e40 0x56498a78f710 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd8340093c0 tx=0x7fd834009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5df617e6178b1d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:02.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.458+0000 7fd825600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.458+0000 7fd825600700 1 -- 172.21.15.12:0/3343436345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83400f030 con 0x56498a606e40 2024-05-27T07:29:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.458+0000 7fd825600700 1 -- 172.21.15.12:0/3343436345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9642+0+0 (secure 0 0 0) 0x7fd834010ae0 con 0x56498a606e40 2024-05-27T07:29:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.458+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd834013040 con 0x56498a606e40 2024-05-27T07:29:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.459+0000 7fd825600700 1 -- 172.21.15.12:0/3343436345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fd834012070 con 0x56498a606e40 2024-05-27T07:29:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.459+0000 7fd8512d4b80 10 client.16284 fetch_fsmap learned FSMap version 94 2024-05-27T07:29:02.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.459+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=4+}) -- 0x56498a674060 con 0x56498a606e40 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd825600700 1 -- 172.21.15.12:0/3343436345 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd834013040 con 0x56498a606e40 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd825600700 1 -- 172.21.15.12:0/3343436345 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 94) ==== 4147+0+0 (secure 0 0 0) 0x7fd834027020 con 0x56498a606e40 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd8512d4b80 10 client.16284 fetch_fsmap finished waiting for FSMap version 94 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd8512d4b80 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd8512d4b80 20 client.16284 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a79ff70 0x56498a7a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd8512d4b80 4 client.16284 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:02.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.460+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56498a6b5d90 con 0x56498a79ff70 2024-05-27T07:29:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.461+0000 7fd83aa00700 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a79ff70 0x56498a7a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.462+0000 7fd83aa00700 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a79ff70 0x56498a7a23c0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f12af522b1e5749 server_cookie=2254e16ddbe4d8aa in_seq=0 out_seq=0 2024-05-27T07:29:02.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.462+0000 7fd825600700 10 client.16284 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:02.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd825600700 1 -- 172.21.15.12:0/3343436345 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56498a6b5d90 con 0x56498a79ff70 2024-05-27T07:29:02.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd825600700 10 client.16284 handle_command_reply: tid=0 2024-05-27T07:29:02.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 client.16284 shutdown 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a606e40 msgr2=0x56498a78f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a606e40 0x56498a78f710 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd8340093c0 tx=0x7fd834009350 comp rx=0 tx=0).stop 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 shutdown_connections 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a79ff70 0x56498a7a23c0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a790190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a606e40 0x56498a78f710 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/3343436345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd825600700 0 client.16284 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:02.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.464+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 >> 172.21.15.12:0/3343436345 conn(0x56498a6a0390 msgr2=0x56498a7853b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:02.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.465+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 shutdown_connections 2024-05-27T07:29:02.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.465+0000 7fd8512d4b80 1 -- 172.21.15.12:0/3343436345 wait complete. 2024-05-27T07:29:02.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.465+0000 7fd8512d4b80 20 client.16284 trim_cache size 0 max 16384 2024-05-27T07:29:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 Processor -- start 2024-05-27T07:29:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 -- start start 2024-05-27T07:29:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a606e40 0x56498a789910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a785e70 0x56498a7a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56498a7817e0 con 0x56498a785e70 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c1b2e10 con 0x56498a606e40 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.495+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c1b11b0 con 0x56498a6f29f0 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a785e70 0x56498a7a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a785e70 0x56498a7a7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52290/0 (socket says 172.21.15.12:52290) 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 -- 172.21.15.12:0/2413416772 learned_addr learned my addr 172.21.15.12:0/2413416772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a606e40 0x56498a789910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83aa00700 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 -- 172.21.15.12:0/2413416772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 msgr2=0x56498a78aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 -- 172.21.15.12:0/2413416772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a606e40 msgr2=0x56498a789910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a606e40 0x56498a789910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.496+0000 7fd83be00700 1 -- 172.21.15.12:0/2413416772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56498a790500 con 0x56498a785e70 2024-05-27T07:29:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.497+0000 7fd83be00700 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a785e70 0x56498a7a7b80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd834018600 tx=0x7fd8340093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a445b52127fc57c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.497+0000 7fd827e00700 1 -- 172.21.15.12:0/2413416772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83400f040 con 0x56498a785e70 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.497+0000 7fd827e00700 1 -- 172.21.15.12:0/2413416772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd834024040 con 0x56498a785e70 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.497+0000 7fd827e00700 1 -- 172.21.15.12:0/2413416772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83402ad60 con 0x56498a785e70 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2413416772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a785e70 msgr2=0x56498a7a7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a785e70 0x56498a7a7b80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd834018600 tx=0x7fd8340093f0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2413416772 shutdown_connections 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a606e40 0x56498a789910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a785e70 0x56498a7a7b80 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/2413416772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a6f29f0 0x56498a78aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2413416772 >> 172.21.15.12:0/2413416772 conn(0x56498a6a0390 msgr2=0x56498a78e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2413416772 shutdown_connections 2024-05-27T07:29:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.498+0000 7fd8512d4b80 1 -- 172.21.15.12:0/2413416772 wait complete. 2024-05-27T07:29:02.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.499+0000 7fd8512d4b80 1 -- start start 2024-05-27T07:29:02.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.499+0000 7fd8512d4b80 1 Processor -- start 2024-05-27T07:29:02.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a7935b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a793af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd8512d4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a802a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c1b1880 con 0x56498a6f29f0 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c1b26e0 con 0x56498a785e70 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd8512d4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c1b1fb0 con 0x56498a606e40 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd8512d4b80 10 client.0 _ll_register_callbacks cb 0x56498a64dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd83aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a793af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd83aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a793af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52304/0 (socket says 172.21.15.12:52304) 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd83aa00700 1 -- 172.21.15.12:0/1235953636 learned_addr learned my addr 172.21.15.12:0/1235953636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd83be00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a7935b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.500+0000 7fd83be00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a802a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd83aa00700 1 -- 172.21.15.12:0/1235953636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 msgr2=0x56498a7935b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd83aa00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a7935b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd83aa00700 1 -- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 msgr2=0x56498a802a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd83aa00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a802a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd83aa00700 1 -- 172.21.15.12:0/1235953636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56498a802fa0 con 0x56498a6f29f0 2024-05-27T07:29:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd83be00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a7935b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd83aa00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a793af0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd828005d90 tx=0x7fd828004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c587b8758ca2cd92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd825600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.501+0000 7fd825600700 1 -- 172.21.15.12:0/1235953636 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82801f070 con 0x56498a6f29f0 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd825600700 1 -- 172.21.15.12:0/1235953636 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9642+0+0 (secure 0 0 0) 0x7fd828003cc0 con 0x56498a6f29f0 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd834024040 con 0x56498a6f29f0 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd825600700 1 -- 172.21.15.12:0/1235953636 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fd82802b050 con 0x56498a6f29f0 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd8512d4b80 10 client.16290 fetch_fsmap learned FSMap version 94 2024-05-27T07:29:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=4+}) -- 0x56498a80a360 con 0x56498a6f29f0 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd825600700 1 -- 172.21.15.12:0/1235953636 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd834024040 con 0x56498a6f29f0 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd825600700 1 -- 172.21.15.12:0/1235953636 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 94) ==== 4147+0+0 (secure 0 0 0) 0x7fd82800f040 con 0x56498a6f29f0 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd8512d4b80 10 client.16290 fetch_fsmap finished waiting for FSMap version 94 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd8512d4b80 10 client.16290 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.502+0000 7fd8512d4b80 20 client.16290 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.503+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a80aa60 0x56498a80ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.503+0000 7fd8512d4b80 4 client.16290 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.503+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56498a6af570 con 0x56498a80aa60 2024-05-27T07:29:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.503+0000 7fd83be00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a80aa60 0x56498a80ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.504+0000 7fd83be00700 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a80aa60 0x56498a80ce80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a543e43a836c9f1f server_cookie=d157c16d6b5c5960 in_seq=0 out_seq=0 2024-05-27T07:29:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.504+0000 7fd825600700 10 client.16290 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd825600700 1 -- 172.21.15.12:0/1235953636 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56498a6af570 con 0x56498a80aa60 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd825600700 10 client.16290 handle_command_reply: tid=0 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 client.16290 shutdown 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 msgr2=0x56498a793af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a793af0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd828005d90 tx=0x7fd828004320 comp rx=0 tx=0).stop 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 shutdown_connections 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56498a80aa60 0x56498a80ce80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56498a785e70 0x56498a802a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56498a606e40 0x56498a7935b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd825600700 0 client.16290 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 --2- 172.21.15.12:0/1235953636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56498a6f29f0 0x56498a793af0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.505+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 >> 172.21.15.12:0/1235953636 conn(0x56498a6a0390 msgr2=0x56498a78f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.506+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 shutdown_connections 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.506+0000 7fd8512d4b80 1 -- 172.21.15.12:0/1235953636 wait complete. 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.506+0000 7fd8512d4b80 20 client.16290 trim_cache size 0 max 16384 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:29:02.497 INFO:teuthology.orchestra.run.smithi012.stdout: "28a08984-c671-4e65-9faa-a8134d286ee1": { 2024-05-27T07:29:02.498 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:29:02.498 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "28a08984-c671-4e65-9faa-a8134d286ee1", 2024-05-27T07:29:02.498 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:29:02.498 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:29:02.498 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:29:02.498 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8200b2a70 msgr2=0x7fd8200b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8200b2a70 0x7fd8200b4f80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00ffb0 tx=0x7fd82c00b540 comp rx=0 tx=0).stop 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 msgr2=0x7fd83c19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c19cef0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0057c0 tx=0x7fd82c009f90 comp rx=0 tx=0).stop 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 shutdown_connections 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8200b2a70 0x7fd8200b4f80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83c10e8e0 0x7fd83c1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd83c105470 0x7fd83c19cef0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 --2- 172.21.15.12:0/2073098745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd83c105dc0 0x7fd83c19d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.507+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 >> 172.21.15.12:0/2073098745 conn(0x7fd83c0fef20 msgr2=0x7fd83c103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.508+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 shutdown_connections 2024-05-27T07:29:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:02.508+0000 7fd842800700 1 -- 172.21.15.12:0/2073098745 wait complete. 2024-05-27T07:29:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:02 smithi162 ceph-mon[35676]: pgmap v234: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:02 smithi162 ceph-mon[35676]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:29:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:02 smithi162 ceph-mon[35676]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:02.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:02 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:02.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:02 smithi062 ceph-mon[35304]: pgmap v234: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:02.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:02 smithi062 ceph-mon[35304]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:29:02.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:02 smithi062 ceph-mon[35304]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:02 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:03.847 INFO:tasks.cephfs.filesystem:scrub status for tag:28a08984-c671-4e65-9faa-a8134d286ee1 - {'path': '/', 'tag': '28a08984-c671-4e65-9faa-a8134d286ee1', 'options': 'recursive,force'} 2024-05-27T07:29:04.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:04 smithi162 ceph-mon[35676]: pgmap v236: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:04 smithi062 ceph-mon[35304]: pgmap v236: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:04.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:04 smithi012 ceph-mon[31473]: pgmap v236: 129 pgs: 129 active+clean; 102 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:06.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:06 smithi162 ceph-mon[35676]: pgmap v237: 129 pgs: 129 active+clean; 104 MiB data, 916 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:29:06.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:06 smithi062 ceph-mon[35304]: pgmap v237: 129 pgs: 129 active+clean; 104 MiB data, 916 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:29:06.837 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:06 smithi012 ceph-mon[31473]: pgmap v237: 129 pgs: 129 active+clean; 104 MiB data, 916 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:29:07.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:29:07.342 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:29:07.342 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: Stopped Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:07.342 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 2.140s CPU time. 2024-05-27T07:29:07.342 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: Starting Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 podman[122999]: 2024-05-27 07:29:07.44742827 +0000 UTC m=+0.045045155 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: Stopped Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 3.746s CPU time. 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:07 smithi012 systemd[1]: Starting Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:07 smithi012 ceph-mon[31473]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:07 smithi012 ceph-mon[31473]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:07 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:29:07.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:07 smithi012 ceph-mon[31473]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:29:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:07 smithi162 ceph-mon[35676]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:07 smithi162 ceph-mon[35676]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:07 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:29:07.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:07 smithi162 ceph-mon[35676]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:29:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:07 smithi062 ceph-mon[35304]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:07 smithi062 ceph-mon[35304]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:07.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:07 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:29:07.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:07 smithi062 ceph-mon[35304]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:29:07.878 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 podman[122999]: 2024-05-27 07:29:07.609005006 +0000 UTC m=+0.206621913 container create 7c47cce290dd80940323997dbca7143eb2607301e887a7f34f20bdfebb481452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:07.878 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 podman[122999]: 2024-05-27 07:29:07.888043511 +0000 UTC m=+0.485660423 container init 7c47cce290dd80940323997dbca7143eb2607301e887a7f34f20bdfebb481452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:29:08.233 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:07 smithi012 podman[122999]: 2024-05-27 07:29:07.899672657 +0000 UTC m=+0.497289544 container start 7c47cce290dd80940323997dbca7143eb2607301e887a7f34f20bdfebb481452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:29:08.234 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:08 smithi012 bash[122999]: 7c47cce290dd80940323997dbca7143eb2607301e887a7f34f20bdfebb481452 2024-05-27T07:29:08.234 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:08 smithi012 systemd[1]: Started Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:08.234 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:08 smithi012 ceph-mds[123048]: -- 172.21.15.12:0/2246191974 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ff87907a40 con 0x55ff8794f600 2024-05-27T07:29:08.234 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: starting mds.j at 2024-05-27T07:29:08.557 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 podman[123078]: 2024-05-27 07:29:08.149128663 +0000 UTC m=+0.038777925 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:08.557 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 podman[123078]: 2024-05-27 07:29:08.289329665 +0000 UTC m=+0.178978947 container create bf19f4bb893638cc2734f8412358e81477fb34adf24d39f14e553db0bba2a2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:29:08.558 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:08 smithi012 ceph-mon[31473]: pgmap v238: 129 pgs: 129 active+clean; 104 MiB data, 916 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:29:08.657 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:08 smithi162 ceph-mon[35676]: pgmap v238: 129 pgs: 129 active+clean; 104 MiB data, 916 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:29:08.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:08 smithi062 ceph-mon[35304]: pgmap v238: 129 pgs: 129 active+clean; 104 MiB data, 916 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:29:08.839 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 podman[123078]: 2024-05-27 07:29:08.56793086 +0000 UTC m=+0.457580163 container init bf19f4bb893638cc2734f8412358e81477fb34adf24d39f14e553db0bba2a2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:08.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 podman[123078]: 2024-05-27 07:29:08.575542177 +0000 UTC m=+0.465191430 container start bf19f4bb893638cc2734f8412358e81477fb34adf24d39f14e553db0bba2a2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:08.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 bash[123078]: bf19f4bb893638cc2734f8412358e81477fb34adf24d39f14e553db0bba2a2cd 2024-05-27T07:29:08.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 systemd[1]: Started Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:08.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 ceph-mds[123108]: -- 172.21.15.12:0/2820567643 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fda639a40 con 0x557fda681600 2024-05-27T07:29:08.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: starting mds.d at 2024-05-27T07:29:09.659 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:09 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/1487209239,v1:172.21.15.12:6835/1487209239] up:boot 2024-05-27T07:29:09.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:09 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby 2024-05-27T07:29:09.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:09 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:09.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:09 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:29:09.661 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe317a00700 time 2024-05-27T07:29:09.584495+0000 2024-05-27T07:29:09.661 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.661 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.661 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe32d5f8ba6] 2024-05-27T07:29:09.661 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.662 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.663 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: clone() 2024-05-27T07:29:09.663 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.586+0000 7fe317a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe317a00700 time 2024-05-27T07:29:09.584495+0000 2024-05-27T07:29:09.663 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.663 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.663 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.663 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe32d5f8ba6] 2024-05-27T07:29:09.663 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.664 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: clone() 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: *** Caught signal (Aborted) ** 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: in thread 7fe317a00700 thread_name:md_log_replay 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe32c175d20] 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: gsignal() 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: abort() 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe32d5f8ced] 2024-05-27T07:29:09.665 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff1c2200700 time 2024-05-27T07:29:09.590817+0000 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff1d7ce4ba6] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.666 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: clone() 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.592+0000 7ff1c2200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff1c2200700 time 2024-05-27T07:29:09.590817+0000 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.667 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 19: clone() 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.590+0000 7fe317a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:09.668 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: in thread 7fe317a00700 thread_name:md_log_replay 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe32c175d20] 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: gsignal() 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: abort() 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe32d5f8ced] 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.669 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff1d7ce4ba6] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.670 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: clone() 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: *** Caught signal (Aborted) ** 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: in thread 7ff1c2200700 thread_name:md_log_replay 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff1d6861d20] 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: gsignal() 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: abort() 2024-05-27T07:29:09.671 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff1d7ce4ced] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 19: clone() 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.672 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: -1> 2024-05-27T07:29:09.586+0000 7fe317a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe317a00700 time 2024-05-27T07:29:09.584495+0000 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe32d5f8ba6] 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.673 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.674 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 19: clone() 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.596+0000 7ff1c2200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: in thread 7ff1c2200700 thread_name:md_log_replay 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff1d6861d20] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: gsignal() 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: abort() 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff1d7ce4ced] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.675 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 19: clone() 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:09.676 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: -1> 2024-05-27T07:29:09.592+0000 7ff1c2200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff1c2200700 time 2024-05-27T07:29:09.590817+0000 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff1d7ce4ba6] 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.677 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.678 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: clone() 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 0> 2024-05-27T07:29:09.590+0000 7fe317a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: in thread 7fe317a00700 thread_name:md_log_replay 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe32c175d20] 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: gsignal() 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: abort() 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe32d5f8ced] 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.679 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 19: clone() 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: -1> 2024-05-27T07:29:09.586+0000 7fe317a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe317a00700 time 2024-05-27T07:29:09.584495+0000 2024-05-27T07:29:09.680 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe32d5f8ba6] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.681 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: clone() 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 0> 2024-05-27T07:29:09.590+0000 7fe317a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: in thread 7fe317a00700 thread_name:md_log_replay 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe32c175d20] 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2: gsignal() 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 3: abort() 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe32d5f8ced] 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe32d5f8dd1] 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557fad8cd23d] 2024-05-27T07:29:09.682 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557fad8b8971] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 8: (SnapRealm::check_cache() const+0x13e) [0x557fad8b6fa0] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557fad8b7651] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557fad84c347] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557fad54887e] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557fad54de94] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557fad558969] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 14: (MDLog::_replay_thread()+0x1f38) [0x557fad8f148c] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557fad8fa9cb] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 16: (Thread::entry_wrapper()+0x43) [0x7fe32d5c772d] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe32d5c7749] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe32c16b1ca] 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 19: clone() 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:09.683 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123093]: 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: clone() 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 0> 2024-05-27T07:29:09.596+0000 7ff1c2200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: in thread 7ff1c2200700 thread_name:md_log_replay 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff1d6861d20] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: gsignal() 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: abort() 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff1d7ce4ced] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 19: clone() 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: -1> 2024-05-27T07:29:09.592+0000 7ff1c2200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff1c2200700 time 2024-05-27T07:29:09.590817+0000 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff1d7ce4ba6] 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: clone() 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 0> 2024-05-27T07:29:09.596+0000 7ff1c2200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:09.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: in thread 7ff1c2200700 thread_name:md_log_replay 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff1d6861d20] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2: gsignal() 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 3: abort() 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff1d7ce4ced] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff1d7ce4dd1] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ff58c2d23d] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ff58c18971] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ff58c16fa0] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ff58c17651] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ff58bac347] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ff588a887e] 2024-05-27T07:29:09.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ff588ade94] 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ff588b8969] 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ff58c5148c] 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ff58c5a9cb] 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 16: (Thread::entry_wrapper()+0x43) [0x7ff1d7cb372d] 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff1d7cb3749] 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff1d68571ca] 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 19: clone() 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:09.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123035]: 2024-05-27T07:29:09.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:09 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/1487209239,v1:172.21.15.12:6835/1487209239] up:boot 2024-05-27T07:29:09.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:09 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby 2024-05-27T07:29:09.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:09 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:09.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:09 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:29:09.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:09 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/1487209239,v1:172.21.15.12:6835/1487209239] up:boot 2024-05-27T07:29:09.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:09 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby 2024-05-27T07:29:09.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:09 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:09.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:09 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:29:10.090 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:09 smithi012 podman[123139]: 2024-05-27 07:29:09.75629563 +0000 UTC m=+0.045255439 container died bf19f4bb893638cc2734f8412358e81477fb34adf24d39f14e553db0bba2a2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:10.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:10 smithi162 ceph-mon[35676]: pgmap v240: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 214 op/s 2024-05-27T07:29:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:10 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6836/752138788,v1:172.21.15.12:6837/752138788] up:boot 2024-05-27T07:29:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:10 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:10.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:10 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:10.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:10 smithi062 ceph-mon[35304]: pgmap v240: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 214 op/s 2024-05-27T07:29:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:10 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6836/752138788,v1:172.21.15.12:6837/752138788] up:boot 2024-05-27T07:29:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:10 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:10.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:10 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:10.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:10 smithi012 podman[123139]: 2024-05-27 07:29:10.481996655 +0000 UTC m=+0.770956497 container remove bf19f4bb893638cc2734f8412358e81477fb34adf24d39f14e553db0bba2a2cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T07:29:10.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:10 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:10.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:10 smithi012 podman[123140]: 2024-05-27 07:29:10.491326192 +0000 UTC m=+0.767379139 container died 7c47cce290dd80940323997dbca7143eb2607301e887a7f34f20bdfebb481452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:29:10.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:10 smithi012 ceph-mon[31473]: pgmap v240: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 214 op/s 2024-05-27T07:29:10.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:10 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6836/752138788,v1:172.21.15.12:6837/752138788] up:boot 2024-05-27T07:29:10.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:10 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:10.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:10.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:10 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:11.559 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:11 smithi012 podman[123140]: 2024-05-27 07:29:11.212916145 +0000 UTC m=+1.488969065 container remove 7c47cce290dd80940323997dbca7143eb2607301e887a7f34f20bdfebb481452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5) 2024-05-27T07:29:11.559 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:11 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:11.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:11 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:29:11.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:11 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:29:11.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:11 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:29:11.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:11 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:29:11.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:11 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:29:11.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:11 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:29:11.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:11 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:29:11.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:11 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:29:11.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:11 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:29:11.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:11 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:11.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:11 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.635s CPU time. 2024-05-27T07:29:11.840 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:11 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:29:11.840 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:11 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.678s CPU time. 2024-05-27T07:29:12.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:12 smithi162 ceph-mon[35676]: pgmap v241: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 187 op/s 2024-05-27T07:29:12.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:12 smithi062 ceph-mon[35304]: pgmap v241: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 187 op/s 2024-05-27T07:29:12.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:12 smithi012 ceph-mon[31473]: pgmap v241: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 187 op/s 2024-05-27T07:29:14.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:14 smithi162 ceph-mon[35676]: pgmap v242: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-05-27T07:29:14.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:14 smithi062 ceph-mon[35304]: pgmap v242: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-05-27T07:29:14.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:14 smithi012 ceph-mon[31473]: pgmap v242: 129 pgs: 129 active+clean; 105 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-05-27T07:29:15.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:15 smithi062 ceph-mon[35304]: pgmap v243: 129 pgs: 129 active+clean; 106 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:29:15.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:15 smithi012 ceph-mon[31473]: pgmap v243: 129 pgs: 129 active+clean; 106 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:29:15.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:15 smithi162 ceph-mon[35676]: pgmap v243: 129 pgs: 129 active+clean; 106 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:29:17.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:17 smithi162 ceph-mon[35676]: pgmap v244: 129 pgs: 129 active+clean; 106 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:29:17.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:17 smithi062 ceph-mon[35304]: pgmap v244: 129 pgs: 129 active+clean; 106 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:29:18.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:17 smithi012 ceph-mon[31473]: pgmap v244: 129 pgs: 129 active+clean; 106 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:29:19.320 INFO:tasks.workunit.client.0.smithi012.stdout:20000 punch from 0x9116f to 0x978af, (0x6740 bytes) 2024-05-27T07:29:19.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:19 smithi012 ceph-mon[31473]: pgmap v245: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:19.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:19 smithi162 ceph-mon[35676]: pgmap v245: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:19.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:19 smithi062 ceph-mon[35304]: pgmap v245: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-05-27T07:29:21.859 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:21 smithi012 ceph-mon[31473]: pgmap v246: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-27T07:29:21.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:29:21.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: Stopped Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:21.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.635s CPU time. 2024-05-27T07:29:21.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: Starting Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:21.860 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:29:21.860 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: Stopped Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:21.860 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.678s CPU time. 2024-05-27T07:29:21.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:21 smithi162 ceph-mon[35676]: pgmap v246: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-27T07:29:21.985 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:21 smithi062 ceph-mon[35304]: pgmap v246: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-27T07:29:22.232 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 podman[123281]: 2024-05-27 07:29:21.948989955 +0000 UTC m=+0.046609459 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:22.232 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 podman[123281]: 2024-05-27 07:29:22.079676007 +0000 UTC m=+0.177295509 container create 87245be1db72d6eef81e7a8922593a1fdcf52febe68a185b79ceebbce4070ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-27T07:29:22.232 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:21 smithi012 systemd[1]: Starting Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:22.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:22 smithi012 ceph-mon[31473]: daemon mds.d restarted 2024-05-27T07:29:22.655 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 podman[123281]: 2024-05-27 07:29:22.383941103 +0000 UTC m=+0.481560673 container init 87245be1db72d6eef81e7a8922593a1fdcf52febe68a185b79ceebbce4070ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-27T07:29:22.655 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 podman[123281]: 2024-05-27 07:29:22.395706527 +0000 UTC m=+0.493326032 container start 87245be1db72d6eef81e7a8922593a1fdcf52febe68a185b79ceebbce4070ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:22.655 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 bash[123281]: 87245be1db72d6eef81e7a8922593a1fdcf52febe68a185b79ceebbce4070ac4 2024-05-27T07:29:22.655 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 systemd[1]: Started Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:22.655 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-mds[123329]: -- 172.21.15.12:0/3296504512 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55624bc184e0 con 0x55624bc63600 2024-05-27T07:29:22.655 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-mds[123329]: -- 172.21.15.12:0/3296504512 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55624bc1ba40 con 0x55624bc63600 2024-05-27T07:29:22.655 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: starting mds.d at 2024-05-27T07:29:22.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:22 smithi162 ceph-mon[35676]: daemon mds.d restarted 2024-05-27T07:29:22.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:22 smithi062 ceph-mon[35304]: daemon mds.d restarted 2024-05-27T07:29:22.996 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:22 smithi012 podman[123340]: 2024-05-27 07:29:22.571714563 +0000 UTC m=+0.043253869 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:22.996 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:22 smithi012 podman[123340]: 2024-05-27 07:29:22.709622718 +0000 UTC m=+0.181162010 container create ce03eb90ce4b58653ddd4f345e711f0b16cb185dd0a7e7d41828232e673ad192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True) 2024-05-27T07:29:22.996 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:23 smithi012 podman[123340]: 2024-05-27 07:29:23.005625462 +0000 UTC m=+0.477164777 container init ce03eb90ce4b58653ddd4f345e711f0b16cb185dd0a7e7d41828232e673ad192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe5b3e00700 time 2024-05-27T07:29:22.782024+0000 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5c9946ba6] 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:22.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: clone() 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: *** Caught signal (Aborted) ** 2024-05-27T07:29:22.998 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: in thread 7fe5b3e00700 thread_name:md_log_replay 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:22.783+0000 7fe5b3e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe5b3e00700 time 2024-05-27T07:29:22.782024+0000 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5c9946ba6] 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:22.999 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: clone() 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:23.000 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5c84c3d20] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: gsignal() 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: abort() 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5c9946ced] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:23.001 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 19: clone() 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:22.786+0000 7fe5b3e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: in thread 7fe5b3e00700 thread_name:md_log_replay 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5c84c3d20] 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: gsignal() 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: abort() 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5c9946ced] 2024-05-27T07:29:23.002 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:23.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:23.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:23.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:23.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:23.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 19: clone() 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: -1> 2024-05-27T07:29:22.783+0000 7fe5b3e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe5b3e00700 time 2024-05-27T07:29:22.782024+0000 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:23.004 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5c9946ba6] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:23.005 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: clone() 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 0> 2024-05-27T07:29:22.786+0000 7fe5b3e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: in thread 7fe5b3e00700 thread_name:md_log_replay 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5c84c3d20] 2024-05-27T07:29:23.006 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: gsignal() 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: abort() 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5c9946ced] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:23.007 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 19: clone() 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: -1> 2024-05-27T07:29:22.783+0000 7fe5b3e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe5b3e00700 time 2024-05-27T07:29:22.782024+0000 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:23.008 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5c9946ba6] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:23.009 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: clone() 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 0> 2024-05-27T07:29:22.786+0000 7fe5b3e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: in thread 7fe5b3e00700 thread_name:md_log_replay 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5c84c3d20] 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2: gsignal() 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 3: abort() 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5c9946ced] 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5c9946dd1] 2024-05-27T07:29:23.010 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55624981223d] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5562497fd971] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 8: (SnapRealm::check_cache() const+0x13e) [0x5562497fbfa0] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5562497fc651] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556249791347] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55624948d87e] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x556249492e94] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55624949d969] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 14: (MDLog::_replay_thread()+0x1f38) [0x55624983648c] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55624983f9cb] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5c991572d] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5c9915749] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5c84b91ca] 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 19: clone() 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:23.011 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123316]: 2024-05-27T07:29:23.246 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:23 smithi012 podman[123340]: 2024-05-27 07:29:23.018969001 +0000 UTC m=+0.490508294 container start ce03eb90ce4b58653ddd4f345e711f0b16cb185dd0a7e7d41828232e673ad192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:23.247 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:23 smithi012 bash[123340]: ce03eb90ce4b58653ddd4f345e711f0b16cb185dd0a7e7d41828232e673ad192 2024-05-27T07:29:23.247 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:23 smithi012 systemd[1]: Started Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:23.247 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mds[123398]: -- 172.21.15.12:0/1357517490 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d3251e9a40 con 0x55d325231a80 2024-05-27T07:29:23.247 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: starting mds.j at 2024-05-27T07:29:23.579 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:23 smithi012 podman[123386]: 2024-05-27 07:29:23.25639762 +0000 UTC m=+0.371480676 container died 87245be1db72d6eef81e7a8922593a1fdcf52febe68a185b79ceebbce4070ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mon[31473]: pgmap v247: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mon[31473]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/1375087071,v1:172.21.15.12:6835/1375087071] up:boot 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:23 smithi012 ceph-mon[31473]: daemon mds.j restarted 2024-05-27T07:29:23.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:23 smithi162 ceph-mon[35676]: pgmap v247: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-27T07:29:23.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:23 smithi162 ceph-mon[35676]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:29:23.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:23 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/1375087071,v1:172.21.15.12:6835/1375087071] up:boot 2024-05-27T07:29:23.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:23 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:23.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:23.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:23 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:23.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:23 smithi162 ceph-mon[35676]: daemon mds.j restarted 2024-05-27T07:29:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:23 smithi062 ceph-mon[35304]: pgmap v247: 129 pgs: 129 active+clean; 107 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-27T07:29:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:23 smithi062 ceph-mon[35304]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:29:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:23 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/1375087071,v1:172.21.15.12:6835/1375087071] up:boot 2024-05-27T07:29:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:23 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:23 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:23.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:23 smithi062 ceph-mon[35304]: daemon mds.j restarted 2024-05-27T07:29:24.339 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:24 smithi012 podman[123386]: 2024-05-27 07:29:24.07131171 +0000 UTC m=+1.186394737 container remove 87245be1db72d6eef81e7a8922593a1fdcf52febe68a185b79ceebbce4070ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T07:29:24.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:24 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:24.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:24 smithi012 ceph-mon[31473]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:29:24.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:24 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/274030823,v1:172.21.15.12:6835/274030823] up:boot 2024-05-27T07:29:24.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:24 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:24.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:24 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:24.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:24 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:24.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:24 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:24.841 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:24 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.581s CPU time. 2024-05-27T07:29:24.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:24 smithi162 ceph-mon[35676]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:29:24.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:24 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/274030823,v1:172.21.15.12:6835/274030823] up:boot 2024-05-27T07:29:24.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:24 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:24.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:24 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:24.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:24 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:24.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:24 smithi062 ceph-mon[35304]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:29:24.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:24 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/274030823,v1:172.21.15.12:6835/274030823] up:boot 2024-05-27T07:29:24.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:24 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:24.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:24 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:24.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:24 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82fda00700 time 2024-05-27T07:29:24.853206+0000 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f831354dba6] 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.296 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: clone() 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:24.854+0000 7f82fda00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82fda00700 time 2024-05-27T07:29:24.853206+0000 2024-05-27T07:29:25.297 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f831354dba6] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.298 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: clone() 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: *** Caught signal (Aborted) ** 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: in thread 7f82fda00700 thread_name:md_log_replay 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83120cad20] 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: gsignal() 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: abort() 2024-05-27T07:29:25.299 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f831354dced] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.300 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 17: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 19: clone() 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:24.857+0000 7f82fda00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: in thread 7f82fda00700 thread_name:md_log_replay 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83120cad20] 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: gsignal() 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: abort() 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f831354dced] 2024-05-27T07:29:25.301 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.302 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.302 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.302 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.302 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.302 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 17: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 19: clone() 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: -1> 2024-05-27T07:29:24.854+0000 7f82fda00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82fda00700 time 2024-05-27T07:29:24.853206+0000 2024-05-27T07:29:25.303 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f831354dba6] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.304 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: clone() 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 0> 2024-05-27T07:29:24.857+0000 7f82fda00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: in thread 7f82fda00700 thread_name:md_log_replay 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83120cad20] 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: gsignal() 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: abort() 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f831354dced] 2024-05-27T07:29:25.305 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 17: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.306 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 19: clone() 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: -1> 2024-05-27T07:29:24.854+0000 7f82fda00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f82fda00700 time 2024-05-27T07:29:24.853206+0000 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f831354dba6] 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.307 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: clone() 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 0> 2024-05-27T07:29:24.857+0000 7f82fda00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: in thread 7f82fda00700 thread_name:md_log_replay 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.308 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83120cad20] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2: gsignal() 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 3: abort() 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f831354dced] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f831354ddd1] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d2fbf7c23d] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d2fbf67971] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d2fbf65fa0] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d2fbf66651] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d2fbefb347] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d2fbbf787e] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d2fbbfce94] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d2fbc07969] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d2fbfa048c] 2024-05-27T07:29:25.309 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d2fbfa99cb] 2024-05-27T07:29:25.310 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 16: (Thread::entry_wrapper()+0x43) [0x7f831351c72d] 2024-05-27T07:29:25.310 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 17: (Thread::_entry_func(void*)+0xd) [0x7f831351c749] 2024-05-27T07:29:25.310 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f83120c01ca] 2024-05-27T07:29:25.310 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 19: clone() 2024-05-27T07:29:25.310 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:25.310 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123384]: 2024-05-27T07:29:25.310 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:24 smithi012 podman[123459]: 2024-05-27 07:29:24.985829728 +0000 UTC m=+0.036382381 container died ce03eb90ce4b58653ddd4f345e711f0b16cb185dd0a7e7d41828232e673ad192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T07:29:25.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:25 smithi162 ceph-mon[35676]: pgmap v250: 129 pgs: 129 active+clean; 108 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-05-27T07:29:25.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:25 smithi062 ceph-mon[35304]: pgmap v250: 129 pgs: 129 active+clean; 108 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-05-27T07:29:26.090 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:25 smithi012 podman[123459]: 2024-05-27 07:29:25.811285537 +0000 UTC m=+0.861838215 container remove ce03eb90ce4b58653ddd4f345e711f0b16cb185dd0a7e7d41828232e673ad192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:29:26.090 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:25 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:26.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:25 smithi012 ceph-mon[31473]: pgmap v250: 129 pgs: 129 active+clean; 108 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-05-27T07:29:26.590 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:26 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:29:26.590 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:26 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.617s CPU time. 2024-05-27T07:29:28.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:28 smithi012 ceph-mon[31473]: pgmap v251: 129 pgs: 129 active+clean; 108 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 860 KiB/s wr, 102 op/s 2024-05-27T07:29:28.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:28 smithi162 ceph-mon[35676]: pgmap v251: 129 pgs: 129 active+clean; 108 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 860 KiB/s wr, 102 op/s 2024-05-27T07:29:28.735 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:28 smithi062 ceph-mon[35304]: pgmap v251: 129 pgs: 129 active+clean; 108 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 860 KiB/s wr, 102 op/s 2024-05-27T07:29:30.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:30 smithi012 ceph-mon[31473]: pgmap v252: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-05-27T07:29:30.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:30 smithi162 ceph-mon[35676]: pgmap v252: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-05-27T07:29:30.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:30 smithi062 ceph-mon[35304]: pgmap v252: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-05-27T07:29:32.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:32 smithi012 ceph-mon[31473]: pgmap v253: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-05-27T07:29:32.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:32 smithi162 ceph-mon[35676]: pgmap v253: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-05-27T07:29:32.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:32 smithi062 ceph-mon[35304]: pgmap v253: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-05-27T07:29:33.848 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:29:34.381 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:34.610 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:34 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:29:34.610 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:34 smithi012 systemd[1]: Stopped Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:34.610 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:34 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.581s CPU time. 2024-05-27T07:29:34.610 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:34 smithi012 ceph-mon[31473]: pgmap v254: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-05-27T07:29:34.611 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:34 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:34.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:34 smithi162 ceph-mon[35676]: pgmap v254: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-05-27T07:29:34.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:34 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:34 smithi062 ceph-mon[35304]: pgmap v254: 129 pgs: 129 active+clean; 109 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-05-27T07:29:34.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:34 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:34.954 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:34 smithi012 systemd[1]: Starting Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:36.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:36 smithi012 podman[123631]: 2024-05-27 07:29:36.120074251 +0000 UTC m=+0.469146358 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:36.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:36 smithi012 podman[123631]: 2024-05-27 07:29:36.247530816 +0000 UTC m=+0.596602917 container create 3cc8a919febb803012a4a85898482225827943b877697bf5b38fd1bef1302322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:36.340 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:36 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:29:36.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:36 smithi012 systemd[1]: Stopped Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:36.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:36 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.617s CPU time. 2024-05-27T07:29:36.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:36 smithi012 systemd[1]: Starting Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:36.603 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:36 smithi012 ceph-mon[31473]: pgmap v255: 129 pgs: 129 active+clean; 110 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-05-27T07:29:36.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:36 smithi162 ceph-mon[35676]: pgmap v255: 129 pgs: 129 active+clean; 110 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-05-27T07:29:36.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:36 smithi062 ceph-mon[35304]: pgmap v255: 129 pgs: 129 active+clean; 110 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-05-27T07:29:36.879 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:36 smithi012 podman[123631]: 2024-05-27 07:29:36.689241251 +0000 UTC m=+1.038313411 container init 3cc8a919febb803012a4a85898482225827943b877697bf5b38fd1bef1302322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:29:36.879 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:36 smithi012 podman[123631]: 2024-05-27 07:29:36.698330953 +0000 UTC m=+1.047403051 container start 3cc8a919febb803012a4a85898482225827943b877697bf5b38fd1bef1302322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:29:36.879 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:36 smithi012 ceph-mds[123701]: -- 172.21.15.12:0/254697000 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e4f094ba40 con 0x55e4f0993a80 2024-05-27T07:29:36.879 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: starting mds.d at 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 -- 172.21.15.12:0/412354990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 msgr2=0x7fef90113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 --2- 172.21.15.12:0/412354990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 0x7fef90113030 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fef8800b0a0 tx=0x7fef8800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 -- 172.21.15.12:0/412354990 shutdown_connections 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 --2- 172.21.15.12:0/412354990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef90075ad0 0x7fef90075ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 --2- 172.21.15.12:0/412354990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 0x7fef90113030 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 --2- 172.21.15.12:0/412354990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 0x7fef901106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 -- 172.21.15.12:0/412354990 >> 172.21.15.12:0/412354990 conn(0x7fef90100030 msgr2=0x7fef90102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:37.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 -- 172.21.15.12:0/412354990 shutdown_connections 2024-05-27T07:29:37.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.072+0000 7fef97200700 1 -- 172.21.15.12:0/412354990 wait complete. 2024-05-27T07:29:37.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.073+0000 7fef97200700 1 Processor -- start 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.074+0000 7fef97200700 1 -- start start 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.074+0000 7fef97200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef90075ad0 0x7fef9010db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.074+0000 7fef97200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 0x7fef9010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.074+0000 7fef97200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 0x7fef9010e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.074+0000 7fef97200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef90116550 con 0x7fef90110c10 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.074+0000 7fef97200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef901163d0 con 0x7fef90075ad0 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.074+0000 7fef97200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef901166d0 con 0x7fef90076420 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.075+0000 7fef95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 0x7fef9010e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.076+0000 7fef95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 0x7fef9010e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54992/0 (socket says 172.21.15.12:54992) 2024-05-27T07:29:37.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.076+0000 7fef8fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef90075ad0 0x7fef9010db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.076+0000 7fef8f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 0x7fef9010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.076+0000 7fef8f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 0x7fef9010e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:50922/0 (socket says 172.21.15.12:50922) 2024-05-27T07:29:37.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.076+0000 7fef8f400700 1 -- 172.21.15.12:0/411428830 learned_addr learned my addr 172.21.15.12:0/411428830 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:37.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.077+0000 7fef8f400700 1 -- 172.21.15.12:0/411428830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef90075ad0 msgr2=0x7fef9010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.077+0000 7fef8f400700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef90075ad0 0x7fef9010db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.077+0000 7fef8f400700 1 -- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 msgr2=0x7fef9010e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.077+0000 7fef8f400700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 0x7fef9010e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.077+0000 7fef8f400700 1 -- 172.21.15.12:0/411428830 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9006d300 con 0x7fef90076420 2024-05-27T07:29:37.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.078+0000 7fef8f400700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 0x7fef9010e060 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e970 tx=0x7fef7c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:37.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.078+0000 7fef8cc00700 1 -- 172.21.15.12:0/411428830 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c004a30 con 0x7fef90076420 2024-05-27T07:29:37.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.078+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9006d590 con 0x7fef90076420 2024-05-27T07:29:37.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.079+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9006db20 con 0x7fef90076420 2024-05-27T07:29:37.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.079+0000 7fef8cc00700 1 -- 172.21.15.12:0/411428830 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef7c00cc70 con 0x7fef90076420 2024-05-27T07:29:37.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.079+0000 7fef8cc00700 1 -- 172.21.15.12:0/411428830 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c005520 con 0x7fef90076420 2024-05-27T07:29:37.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.087+0000 7fefa5d55b80 1 Processor -- start 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.087+0000 7fefa5d55b80 1 -- start start 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.087+0000 7fef8cc00700 1 -- 172.21.15.12:0/411428830 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fef7c00c7e0 con 0x7fef90076420 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.088+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x55581577c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.088+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5558155fde40 0x55581577c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.088+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55581577ce70 0x555815789110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.088+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5558156aae40 con 0x5558155fde40 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.088+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55581569a9e0 con 0x55581577ce70 2024-05-27T07:29:37.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.088+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55581569ea20 con 0x5558156e99f0 2024-05-27T07:29:37.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.089+0000 7fef8f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55581577ce70 0x555815789110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.089+0000 7fef8f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55581577ce70 0x555815789110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:43562/0 (socket says 172.21.15.12:43562) 2024-05-27T07:29:37.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.089+0000 7fef8f400700 1 -- 172.21.15.12:0/2655257933 learned_addr learned my addr 172.21.15.12:0/2655257933 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:37.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.089+0000 7fef95200700 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x55581577c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.089+0000 7fef8fe00700 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5558155fde40 0x55581577c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.089+0000 7fef8f400700 1 -- 172.21.15.12:0/2655257933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 msgr2=0x55581577c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.089+0000 7fef8f400700 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x55581577c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.090+0000 7fef8f400700 1 -- 172.21.15.12:0/2655257933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5558155fde40 msgr2=0x55581577c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.090+0000 7fef8f400700 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5558155fde40 0x55581577c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.090+0000 7fef8f400700 1 -- 172.21.15.12:0/2655257933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555815663420 con 0x55581577ce70 2024-05-27T07:29:37.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.090+0000 7fef8fe00700 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5558155fde40 0x55581577c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:37.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.090+0000 7fef8f400700 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55581577ce70 0x555815789110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0dfd70 tx=0x7fef7c0e6ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e76194ba5153eb3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:37.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.091+0000 7fef85600700 1 -- 172.21.15.12:0/2655257933 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0eb5e0 con 0x55581577ce70 2024-05-27T07:29:37.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.091+0000 7fef85600700 1 -- 172.21.15.12:0/2655257933 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef7c0f4070 con 0x55581577ce70 2024-05-27T07:29:37.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.091+0000 7fef85600700 1 -- 172.21.15.12:0/2655257933 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0ef9e0 con 0x55581577ce70 2024-05-27T07:29:37.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.092+0000 7fef8cc00700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef740b2a70 0x7fef740b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.092+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2655257933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55581577ce70 msgr2=0x555815789110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.092+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55581577ce70 0x555815789110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0dfd70 tx=0x7fef7c0e6ae0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.093+0000 7fef8cc00700 1 -- 172.21.15.12:0/411428830 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9466+0+0 (secure 0 0 0) 0x7fef7c0e2a60 con 0x7fef90076420 2024-05-27T07:29:37.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.093+0000 7fef95200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef740b2a70 0x7fef740b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2655257933 shutdown_connections 2024-05-27T07:29:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55581577ce70 0x555815789110 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x55581577c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2655257933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5558155fde40 0x55581577c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2655257933 >> 172.21.15.12:0/2655257933 conn(0x555815697390 msgr2=0x5558157784c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2655257933 shutdown_connections 2024-05-27T07:29:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2655257933 wait complete. 2024-05-27T07:29:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.094+0000 7fef95200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef740b2a70 0x7fef740b4f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x55581577eff0 tx=0x7fef8800b720 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.095+0000 7fefa5d55b80 1 -- start start 2024-05-27T07:29:37.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.095+0000 7fefa5d55b80 1 Processor -- start 2024-05-27T07:29:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.096+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558155fde40 0x5558157866a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.096+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x555815786be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.096+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815787120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.096+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5558155a0fa0 con 0x55581577ce70 2024-05-27T07:29:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.096+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55581563b760 con 0x5558155fde40 2024-05-27T07:29:37.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.096+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555815789b20 con 0x5558156e99f0 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.096+0000 7fefa5d55b80 10 client.0 _ll_register_callbacks cb 0x555815644c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef8fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558155fde40 0x5558157866a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef8fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558155fde40 0x5558157866a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:43572/0 (socket says 172.21.15.12:43572) 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef8fe00700 1 -- 172.21.15.12:0/3572820969 learned_addr learned my addr 172.21.15.12:0/3572820969 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef95200700 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815787120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef8f400700 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x555815786be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef95200700 1 -- 172.21.15.12:0/3572820969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 msgr2=0x555815786be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef95200700 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x555815786be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef95200700 1 -- 172.21.15.12:0/3572820969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558155fde40 msgr2=0x5558157866a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef95200700 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558155fde40 0x5558157866a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.097+0000 7fef95200700 1 -- 172.21.15.12:0/3572820969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558155ea8d0 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.098+0000 7fef95200700 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815787120 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fef88009900 tx=0x7fef880099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72182585562fe65e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.098+0000 7fef7a000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.098+0000 7fef7a000700 1 -- 172.21.15.12:0/3572820969 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88007c30 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.098+0000 7fef7a000700 1 -- 172.21.15.12:0/3572820969 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9466+0+0 (secure 0 0 0) 0x7fef8802a450 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.098+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef7c0f4070 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.099+0000 7fef7a000700 1 -- 172.21.15.12:0/3572820969 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fef88039050 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.099+0000 7fefa5d55b80 10 client.16329 fetch_fsmap learned FSMap version 104 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.099+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55581566b060 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.099+0000 7fef7a000700 1 -- 172.21.15.12:0/3572820969 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef7c0f4070 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.100+0000 7fef7a000700 1 -- 172.21.15.12:0/3572820969 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 104) ==== 4143+0+0 (secure 0 0 0) 0x7fef88011070 con 0x55581577ce70 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.100+0000 7fefa5d55b80 10 client.16329 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.100+0000 7fefa5d55b80 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:37.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.100+0000 7fefa5d55b80 20 client.16329 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.100+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815796e60 0x555815799310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.100+0000 7fefa5d55b80 4 client.16329 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.100+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558156acd90 con 0x555815796e60 2024-05-27T07:29:37.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.101+0000 7fef8fe00700 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815796e60 0x555815799310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.102+0000 7fef8fe00700 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815796e60 0x555815799310 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b15c798d96f14e85 server_cookie=c0639f46975073b6 in_seq=0 out_seq=0 2024-05-27T07:29:37.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.102+0000 7fef7a000700 10 client.16329 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:37.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.104+0000 7fef7a000700 1 -- 172.21.15.12:0/3572820969 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5558156acd90 con 0x555815796e60 2024-05-27T07:29:37.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.104+0000 7fef7a000700 10 client.16329 handle_command_reply: tid=0 2024-05-27T07:29:37.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.104+0000 7fefa5d55b80 1 client.16329 shutdown 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 msgr2=0x555815787120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815787120 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fef88009900 tx=0x7fef880099e0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 shutdown_connections 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815796e60 0x555815799310 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558155fde40 0x5558157866a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815787120 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/3572820969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558156e99f0 0x555815786be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fef7a000700 0 client.16329 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 >> 172.21.15.12:0/3572820969 conn(0x555815697390 msgr2=0x55581577c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:37.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 shutdown_connections 2024-05-27T07:29:37.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.105+0000 7fefa5d55b80 1 -- 172.21.15.12:0/3572820969 wait complete. 2024-05-27T07:29:37.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.106+0000 7fefa5d55b80 20 client.16329 trim_cache size 0 max 16384 2024-05-27T07:29:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.140+0000 7fefa5d55b80 1 Processor -- start 2024-05-27T07:29:37.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.140+0000 7fefa5d55b80 1 -- start start 2024-05-27T07:29:37.132 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 bash[123631]: 3cc8a919febb803012a4a85898482225827943b877697bf5b38fd1bef1302322 2024-05-27T07:29:37.132 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 systemd[1]: Started Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:37.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.141+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815785600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.141+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x555815781e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.141+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815786d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.142+0000 7fef95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815786d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.142+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef901bf7c0 con 0x55581577ce70 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.142+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef90075230 con 0x5558156e99f0 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.142+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef901bf090 con 0x5558155fde40 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.142+0000 7fef95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815786d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55026/0 (socket says 172.21.15.12:55026) 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.142+0000 7fef95200700 1 -- 172.21.15.12:0/2345773676 learned_addr learned my addr 172.21.15.12:0/2345773676 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.143+0000 7fef8fe00700 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815785600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.143+0000 7fef8f400700 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x555815781e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.143+0000 7fef95200700 1 -- 172.21.15.12:0/2345773676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 msgr2=0x555815785600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.143+0000 7fef95200700 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815785600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.143+0000 7fef95200700 1 -- 172.21.15.12:0/2345773676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 msgr2=0x555815781e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.143+0000 7fef95200700 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x555815781e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.143+0000 7fef95200700 1 -- 172.21.15.12:0/2345773676 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555815779000 con 0x55581577ce70 2024-05-27T07:29:37.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.144+0000 7fef95200700 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815786d70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fef880053a0 tx=0x7fef88006160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d63663e5c0ff68d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:37.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.145+0000 7fef84c00700 1 -- 172.21.15.12:0/2345773676 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88007c30 con 0x55581577ce70 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.145+0000 7fef84c00700 1 -- 172.21.15.12:0/2345773676 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef88042070 con 0x55581577ce70 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.145+0000 7fef84c00700 1 -- 172.21.15.12:0/2345773676 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef880123f0 con 0x55581577ce70 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.146+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2345773676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 msgr2=0x555815786d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.146+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815786d70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fef880053a0 tx=0x7fef88006160 comp rx=0 tx=0).stop 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.146+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2345773676 shutdown_connections 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.146+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815786d70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.146+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815785600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.146+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/2345773676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x555815781e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.146+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2345773676 >> 172.21.15.12:0/2345773676 conn(0x555815697390 msgr2=0x555815786660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.147+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2345773676 shutdown_connections 2024-05-27T07:29:37.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.147+0000 7fefa5d55b80 1 -- 172.21.15.12:0/2345773676 wait complete. 2024-05-27T07:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.147+0000 7fefa5d55b80 1 -- start start 2024-05-27T07:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.148+0000 7fefa5d55b80 1 Processor -- start 2024-05-27T07:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.149+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815781080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.149+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x5558157815c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.149+0000 7fefa5d55b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815781b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.149+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef901be960 con 0x55581577ce70 2024-05-27T07:29:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.149+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef901be330 con 0x5558156e99f0 2024-05-27T07:29:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.149+0000 7fefa5d55b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5558156acd90 con 0x5558155fde40 2024-05-27T07:29:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.149+0000 7fefa5d55b80 10 client.0 _ll_register_callbacks cb 0x555815644c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.150+0000 7fef95200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815781b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.150+0000 7fef8f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x5558157815c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.150+0000 7fef8f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x5558157815c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:43582/0 (socket says 172.21.15.12:43582) 2024-05-27T07:29:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.150+0000 7fef8f400700 1 -- 172.21.15.12:0/4260159855 learned_addr learned my addr 172.21.15.12:0/4260159855 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.150+0000 7fef8fe00700 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815781080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.151+0000 7fef95200700 1 -- 172.21.15.12:0/4260159855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 msgr2=0x555815781080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.151+0000 7fef95200700 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815781080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.151+0000 7fef95200700 1 -- 172.21.15.12:0/4260159855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 msgr2=0x5558157815c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.151+0000 7fef95200700 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x5558157815c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.151+0000 7fef95200700 1 -- 172.21.15.12:0/4260159855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55581578d210 con 0x55581577ce70 2024-05-27T07:29:37.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.152+0000 7fef95200700 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815781b00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fef8800b3f0 tx=0x7fef88007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabe43939797b5a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:37.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.152+0000 7fef7a000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:37.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.152+0000 7fef7a000700 1 -- 172.21.15.12:0/4260159855 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8804a020 con 0x55581577ce70 2024-05-27T07:29:37.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.152+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef88042070 con 0x55581577ce70 2024-05-27T07:29:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.152+0000 7fef7a000700 1 -- 172.21.15.12:0/4260159855 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9466+0+0 (secure 0 0 0) 0x7fef88012b70 con 0x55581577ce70 2024-05-27T07:29:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.152+0000 7fef7a000700 1 -- 172.21.15.12:0/4260159855 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fef88012d70 con 0x55581577ce70 2024-05-27T07:29:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 10 client.16335 fetch_fsmap learned FSMap version 104 2024-05-27T07:29:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x555815801ad0 con 0x55581577ce70 2024-05-27T07:29:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fef7a000700 1 -- 172.21.15.12:0/4260159855 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef88042070 con 0x55581577ce70 2024-05-27T07:29:37.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fef7a000700 1 -- 172.21.15.12:0/4260159855 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 104) ==== 4143+0+0 (secure 0 0 0) 0x7fef88011040 con 0x55581577ce70 2024-05-27T07:29:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 10 client.16335 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:29:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 20 client.16335 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815802110 0x555815804530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 4 client.16335 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:29:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.153+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5558156a6570 con 0x555815802110 2024-05-27T07:29:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.154+0000 7fef8fe00700 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815802110 0x555815804530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:37.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.155+0000 7fef8fe00700 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815802110 0x555815804530 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b5bc21f1cde0263 server_cookie=c3d00aac8f9cb4b5 in_seq=0 out_seq=0 2024-05-27T07:29:37.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.155+0000 7fef7a000700 10 client.16335 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.157+0000 7fef7a000700 1 -- 172.21.15.12:0/4260159855 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5558156a6570 con 0x555815802110 2024-05-27T07:29:37.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.157+0000 7fef7a000700 10 client.16335 handle_command_reply: tid=0 2024-05-27T07:29:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.157+0000 7fefa5d55b80 1 client.16335 shutdown 2024-05-27T07:29:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.157+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 msgr2=0x555815781b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.157+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815781b00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fef8800b3f0 tx=0x7fef88007cb0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 shutdown_connections 2024-05-27T07:29:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55581577ce70 0x555815781b00 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558155fde40 0x555815781080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555815802110 0x555815804530 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 --2- 172.21.15.12:0/4260159855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558156e99f0 0x5558157815c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 >> 172.21.15.12:0/4260159855 conn(0x555815697390 msgr2=0x555815786480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fef7a000700 0 client.16335 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 shutdown_connections 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.158+0000 7fefa5d55b80 1 -- 172.21.15.12:0/4260159855 wait complete. 2024-05-27T07:29:37.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.159+0000 7fefa5d55b80 20 client.16335 trim_cache size 0 max 16384 2024-05-27T07:29:37.151 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:29:37.151 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:29:37.151 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:29:37.151 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:29:37.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.159+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef740b2a70 msgr2=0x7fef740b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.159+0000 7fef97200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef740b2a70 0x7fef740b4f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x55581577eff0 tx=0x7fef8800b720 comp rx=0 tx=0).stop 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.159+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 msgr2=0x7fef9010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.159+0000 7fef97200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 0x7fef9010e060 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e970 tx=0x7fef7c00ec80 comp rx=0 tx=0).stop 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 shutdown_connections 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fef740b2a70 0x7fef740b4f80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef90075ad0 0x7fef9010db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef90110c10 0x7fef9010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 --2- 172.21.15.12:0/411428830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef90076420 0x7fef9010e060 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 >> 172.21.15.12:0/411428830 conn(0x7fef90100030 msgr2=0x7fef90101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 shutdown_connections 2024-05-27T07:29:37.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:37.160+0000 7fef97200700 1 -- 172.21.15.12:0/411428830 wait complete. 2024-05-27T07:29:37.462 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 podman[123765]: 2024-05-27 07:29:37.15612763 +0000 UTC m=+0.043415215 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:37.462 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 podman[123765]: 2024-05-27 07:29:37.282948496 +0000 UTC m=+0.170236055 container create 8df9f85d5ce61d4106076af0c712160dcfae4c85f495f0d43df4704f916a4b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:29:37.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:37 smithi012 ceph-mon[31473]: daemon mds.d restarted 2024-05-27T07:29:37.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:37 smithi012 ceph-mon[31473]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:29:37.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:37 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/506586535,v1:172.21.15.12:6835/506586535] up:boot 2024-05-27T07:29:37.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:37 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:37.463 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:37 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:37 smithi162 ceph-mon[35676]: daemon mds.d restarted 2024-05-27T07:29:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:37 smithi162 ceph-mon[35676]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:29:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:37 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/506586535,v1:172.21.15.12:6835/506586535] up:boot 2024-05-27T07:29:37.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:37 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:37.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:37 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:37.728 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 podman[123765]: 2024-05-27 07:29:37.637990126 +0000 UTC m=+0.525277705 container init 8df9f85d5ce61d4106076af0c712160dcfae4c85f495f0d43df4704f916a4b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:29:37.728 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 podman[123765]: 2024-05-27 07:29:37.645900535 +0000 UTC m=+0.533188113 container start 8df9f85d5ce61d4106076af0c712160dcfae4c85f495f0d43df4704f916a4b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:29:37.729 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5258000700 time 2024-05-27T07:29:37.472552+0000 2024-05-27T07:29:37.729 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:37.729 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.729 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f526dc0dba6] 2024-05-27T07:29:37.729 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.729 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.730 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.731 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: clone() 2024-05-27T07:29:37.731 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.474+0000 7f5258000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5258000700 time 2024-05-27T07:29:37.472552+0000 2024-05-27T07:29:37.733 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:37.733 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.733 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.733 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f526dc0dba6] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.734 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: clone() 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: *** Caught signal (Aborted) ** 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: in thread 7f5258000700 thread_name:md_log_replay 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f526c78ad20] 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: gsignal() 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: abort() 2024-05-27T07:29:37.735 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f526dc0dced] 2024-05-27T07:29:37.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:37 smithi062 ceph-mon[35304]: daemon mds.d restarted 2024-05-27T07:29:37.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:37 smithi062 ceph-mon[35304]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:29:37.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:37 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/506586535,v1:172.21.15.12:6835/506586535] up:boot 2024-05-27T07:29:37.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:37 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:37.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:37 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:37.737 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.737 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 17: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.738 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 19: clone() 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.477+0000 7f5258000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: in thread 7f5258000700 thread_name:md_log_replay 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f526c78ad20] 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: gsignal() 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: abort() 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f526dc0dced] 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.739 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.740 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.740 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.740 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.740 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.740 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 17: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 19: clone() 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: -1> 2024-05-27T07:29:37.474+0000 7f5258000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5258000700 time 2024-05-27T07:29:37.472552+0000 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.741 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f526dc0dba6] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.742 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: clone() 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 0> 2024-05-27T07:29:37.477+0000 7f5258000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: in thread 7f5258000700 thread_name:md_log_replay 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f526c78ad20] 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: gsignal() 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: abort() 2024-05-27T07:29:37.743 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f526dc0dced] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.744 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 17: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 19: clone() 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: -1> 2024-05-27T07:29:37.474+0000 7f5258000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5258000700 time 2024-05-27T07:29:37.472552+0000 2024-05-27T07:29:37.745 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f526dc0dba6] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.746 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: clone() 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 0> 2024-05-27T07:29:37.477+0000 7f5258000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: in thread 7f5258000700 thread_name:md_log_replay 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f526c78ad20] 2024-05-27T07:29:37.747 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2: gsignal() 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 3: abort() 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f526dc0dced] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f526dc0ddd1] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e4e27ae23d] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e4e2799971] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e4e2797fa0] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e4e2798651] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e4e272d347] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e4e242987e] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55e4e242ee94] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e4e2439969] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e4e27d248c] 2024-05-27T07:29:37.748 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e4e27db9cb] 2024-05-27T07:29:37.749 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 16: (Thread::entry_wrapper()+0x43) [0x7f526dbdc72d] 2024-05-27T07:29:37.749 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 17: (Thread::_entry_func(void*)+0xd) [0x7f526dbdc749] 2024-05-27T07:29:37.749 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f526c7801ca] 2024-05-27T07:29:37.749 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 19: clone() 2024-05-27T07:29:37.749 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:37.749 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[123681]: 2024-05-27T07:29:38.070 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:37 smithi012 podman[123827]: 2024-05-27 07:29:37.793044177 +0000 UTC m=+0.224113445 container died 3cc8a919febb803012a4a85898482225827943b877697bf5b38fd1bef1302322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-05-27T07:29:38.071 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 bash[123765]: 8df9f85d5ce61d4106076af0c712160dcfae4c85f495f0d43df4704f916a4b3d 2024-05-27T07:29:38.071 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 systemd[1]: Started Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:38.071 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 ceph-mds[123839]: -- 172.21.15.12:0/3905873689 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608c683da40 con 0x5608c6885a80 2024-05-27T07:29:38.071 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: starting mds.j at 2024-05-27T07:29:38.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:38 smithi162 ceph-mon[35676]: pgmap v256: 129 pgs: 129 active+clean; 110 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:29:38.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:38 smithi162 ceph-mon[35676]: daemon mds.j restarted 2024-05-27T07:29:38.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:38 smithi062 ceph-mon[35304]: pgmap v256: 129 pgs: 129 active+clean; 110 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:29:38.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:38 smithi062 ceph-mon[35304]: daemon mds.j restarted 2024-05-27T07:29:38.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:38 smithi012 podman[123827]: 2024-05-27 07:29:38.543948287 +0000 UTC m=+0.975017532 container remove 3cc8a919febb803012a4a85898482225827943b877697bf5b38fd1bef1302322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:29:38.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:38 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:38.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:38 smithi012 ceph-mon[31473]: pgmap v256: 129 pgs: 129 active+clean; 110 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:29:38.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:38 smithi012 ceph-mon[31473]: daemon mds.j restarted 2024-05-27T07:29:39.563 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:29:39.563 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:29:39.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:39 smithi162 ceph-mon[35676]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:29:39.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:39 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/2496845189,v1:172.21.15.12:6835/2496845189] up:boot 2024-05-27T07:29:39.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:39 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:39.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:39 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:39.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:39 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:39.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:39 smithi062 ceph-mon[35304]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:29:39.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:39 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/2496845189,v1:172.21.15.12:6835/2496845189] up:boot 2024-05-27T07:29:39.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:39 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:39.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:39 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:39.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:39 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:39.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:39 smithi012 ceph-mon[31473]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:29:39.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:39 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/2496845189,v1:172.21.15.12:6835/2496845189] up:boot 2024-05-27T07:29:39.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:39 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:39.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:39 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:39.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:39 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:39.741 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7830e00700 time 2024-05-27T07:29:39.606373+0000 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7846833ba6] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.742 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: clone() 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.608+0000 7f7830e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7830e00700 time 2024-05-27T07:29:39.606373+0000 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7846833ba6] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.743 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: clone() 2024-05-27T07:29:39.744 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.745 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: *** Caught signal (Aborted) ** 2024-05-27T07:29:39.745 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: in thread 7f7830e00700 thread_name:md_log_replay 2024-05-27T07:29:39.745 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.745 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f78453b0d20] 2024-05-27T07:29:39.745 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: gsignal() 2024-05-27T07:29:39.745 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: abort() 2024-05-27T07:29:39.745 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7846833ced] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.746 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 19: clone() 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.611+0000 7f7830e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: in thread 7f7830e00700 thread_name:md_log_replay 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f78453b0d20] 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: gsignal() 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: abort() 2024-05-27T07:29:39.747 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7846833ced] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.748 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 19: clone() 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: -1> 2024-05-27T07:29:39.608+0000 7f7830e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7830e00700 time 2024-05-27T07:29:39.606373+0000 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7846833ba6] 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.749 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: clone() 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 0> 2024-05-27T07:29:39.611+0000 7f7830e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:39.750 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: in thread 7f7830e00700 thread_name:md_log_replay 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f78453b0d20] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: gsignal() 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: abort() 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7846833ced] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.751 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 19: clone() 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: -1> 2024-05-27T07:29:39.608+0000 7f7830e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7830e00700 time 2024-05-27T07:29:39.606373+0000 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7846833ba6] 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.752 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.753 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: clone() 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 0> 2024-05-27T07:29:39.611+0000 7f7830e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: in thread 7f7830e00700 thread_name:md_log_replay 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f78453b0d20] 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2: gsignal() 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 3: abort() 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7846833ced] 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7846833dd1] 2024-05-27T07:29:39.754 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5608b56e823d] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5608b56d3971] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 8: (SnapRealm::check_cache() const+0x13e) [0x5608b56d1fa0] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5608b56d2651] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5608b5667347] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5608b536387e] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5608b5368e94] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5608b5373969] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 14: (MDLog::_replay_thread()+0x1f38) [0x5608b570c48c] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5608b57159cb] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 16: (Thread::entry_wrapper()+0x43) [0x7f784680272d] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7846802749] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f78453a61ca] 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 19: clone() 2024-05-27T07:29:39.755 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:39.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[123824]: 2024-05-27T07:29:39.997 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:39 smithi012 podman[123890]: 2024-05-27 07:29:39.749901243 +0000 UTC m=+0.181657671 container died 8df9f85d5ce61d4106076af0c712160dcfae4c85f495f0d43df4704f916a4b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:29:40.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:40.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.598s CPU time. 2024-05-27T07:29:40.567 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:40.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:40 smithi012 ceph-mon[31473]: pgmap v259: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-05-27T07:29:40.720 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:40 smithi012 podman[123920]: 2024-05-27 07:29:40.473917855 +0000 UTC m=+0.749169727 container remove 8df9f85d5ce61d4106076af0c712160dcfae4c85f495f0d43df4704f916a4b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:40.720 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:40 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:40.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:40 smithi162 ceph-mon[35676]: pgmap v259: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-05-27T07:29:40.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:40 smithi062 ceph-mon[35304]: pgmap v259: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-05-27T07:29:41.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:41 smithi012 podman[123958]: 2024-05-27 07:29:41.21237552 +0000 UTC m=+0.685930382 container died 93d9917e851463998fd03ad15c5723b6d8183e40e8de12b77aee705fd88e9f83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=quirky_kapitsa, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:29:42.340 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:42 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:29:42.340 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:42 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.759s CPU time. 2024-05-27T07:29:42.715 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:42 smithi012 ceph-mon[31473]: pgmap v260: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-05-27T07:29:42.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:42 smithi162 ceph-mon[35676]: pgmap v260: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-05-27T07:29:42.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:42 smithi062 ceph-mon[35304]: pgmap v260: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 -- 172.21.15.12:0/3860246277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604108f10 msgr2=0x7f5604109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 --2- 172.21.15.12:0/3860246277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604108f10 0x7f5604109320 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0099c0 tx=0x7f55ec009cd0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 -- 172.21.15.12:0/3860246277 shutdown_connections 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 --2- 172.21.15.12:0/3860246277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5604109860 0x7f56041111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 --2- 172.21.15.12:0/3860246277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56041045e0 0x7f56041049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 --2- 172.21.15.12:0/3860246277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604108f10 0x7f5604109320 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 -- 172.21.15.12:0/3860246277 >> 172.21.15.12:0/3860246277 conn(0x7f5604100030 msgr2=0x7f5604102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 -- 172.21.15.12:0/3860246277 shutdown_connections 2024-05-27T07:29:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.012+0000 7f5609600700 1 -- 172.21.15.12:0/3860246277 wait complete. 2024-05-27T07:29:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.014+0000 7f5609600700 1 Processor -- start 2024-05-27T07:29:43.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.014+0000 7f5609600700 1 -- start start 2024-05-27T07:29:43.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.015+0000 7f5609600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56041045e0 0x7f560410b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.015+0000 7f5609600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5604108f10 0x7f560410b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.015+0000 7f5609600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604109860 0x7f560410bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.016+0000 7f5609600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5604114390 con 0x7f5604109860 2024-05-27T07:29:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.016+0000 7f5602a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56041045e0 0x7f560410b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.016+0000 7f5609600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5604114210 con 0x7f56041045e0 2024-05-27T07:29:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.016+0000 7f5609600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5604114510 con 0x7f5604108f10 2024-05-27T07:29:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.016+0000 7f5602a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56041045e0 0x7f560410b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:47264/0 (socket says 172.21.15.12:47264) 2024-05-27T07:29:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.016+0000 7f5602a00700 1 -- 172.21.15.12:0/1182581993 learned_addr learned my addr 172.21.15.12:0/1182581993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.016+0000 7f55fa000700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5604108f10 0x7f560410b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.017+0000 7f5602a00700 1 -- 172.21.15.12:0/1182581993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5604108f10 msgr2=0x7f560410b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.017+0000 7f5602a00700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5604108f10 0x7f560410b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.017+0000 7f5603400700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604109860 0x7f560410bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.017+0000 7f5602a00700 1 -- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604109860 msgr2=0x7f560410bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.017+0000 7f5602a00700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604109860 0x7f560410bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.017+0000 7f5602a00700 1 -- 172.21.15.12:0/1182581993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56041ad620 con 0x7f56041045e0 2024-05-27T07:29:43.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.018+0000 7f5602a00700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56041045e0 0x7f560410b240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55f400d980 tx=0x7f55f400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.019+0000 7f55fbe00700 1 -- 172.21.15.12:0/1182581993 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4017070 con 0x7f56041045e0 2024-05-27T07:29:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.019+0000 7f55fbe00700 1 -- 172.21.15.12:0/1182581993 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55f4005020 con 0x7f56041045e0 2024-05-27T07:29:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.019+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56041ad8b0 con 0x7f56041045e0 2024-05-27T07:29:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.019+0000 7f55fbe00700 1 -- 172.21.15.12:0/1182581993 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4009ab0 con 0x7f56041045e0 2024-05-27T07:29:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.019+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56041addc0 con 0x7f56041045e0 2024-05-27T07:29:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.021+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5604105a90 con 0x7f56041045e0 2024-05-27T07:29:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.025+0000 7f55fbe00700 1 -- 172.21.15.12:0/1182581993 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f55f40051d0 con 0x7f56041045e0 2024-05-27T07:29:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.026+0000 7f55fbe00700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55e80b2a70 0x7f55e80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.027+0000 7f55fbe00700 1 -- 172.21.15.12:0/1182581993 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7f55f40e2d00 con 0x7f56041045e0 2024-05-27T07:29:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.027+0000 7f55fa000700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55e80b2a70 0x7f55e80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.027+0000 7f55fa000700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55e80b2a70 0x7f55e80b4f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55ec009990 tx=0x7f55ec005fb0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.044+0000 7f55fbe00700 1 -- 172.21.15.12:0/1182581993 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f55f4020020 con 0x7f56041045e0 2024-05-27T07:29:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.414+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56041ae020 con 0x7f56041045e0 2024-05-27T07:29:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.416+0000 7f55fbe00700 1 -- 172.21.15.12:0/1182581993 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6080 (secure 0 0 0) 0x7f55f40aa350 con 0x7f56041045e0 2024-05-27T07:29:43.407 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:29:43.407 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:29:38.407188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16320":{"gid":16320,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6835/506586535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":506586535},{"type":"v1","addr":"172.21.15.12:6835","nonce":506586535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16341":{"gid":16341,"name":"j","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.12:6835/2496845189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2496845189},{"type":"v1","addr":"172.21.15.12:6835","nonce":2496845189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:29:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55e80b2a70 msgr2=0x7f55e80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55e80b2a70 0x7f55e80b4f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55ec009990 tx=0x7f55ec005fb0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56041045e0 msgr2=0x7f560410b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56041045e0 0x7f560410b240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55f400d980 tx=0x7f55f400dc90 comp rx=0 tx=0).stop 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 shutdown_connections 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55e80b2a70 0x7f55e80b4f80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56041045e0 0x7f560410b240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5604109860 0x7f560410bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 --2- 172.21.15.12:0/1182581993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5604108f10 0x7f560410b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.419+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 >> 172.21.15.12:0/1182581993 conn(0x7f5604100030 msgr2=0x7f560410a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.420+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 shutdown_connections 2024-05-27T07:29:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:43.420+0000 7f5609600700 1 -- 172.21.15.12:0/1182581993 wait complete. 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 107 2024-05-27T07:29:43.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:43 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1182581993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:43.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:43 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1182581993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:43.818 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:43 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1182581993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:44.397 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:29:44.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:44 smithi012 ceph-mon[31473]: pgmap v261: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1006 KiB/s wr, 104 op/s 2024-05-27T07:29:44.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:44 smithi162 ceph-mon[35676]: pgmap v261: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1006 KiB/s wr, 104 op/s 2024-05-27T07:29:44.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:44 smithi062 ceph-mon[35304]: pgmap v261: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1006 KiB/s wr, 104 op/s 2024-05-27T07:29:44.897 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:45.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:45 smithi062 ceph-mon[35304]: pgmap v262: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 208 op/s 2024-05-27T07:29:45.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:45 smithi012 ceph-mon[31473]: pgmap v262: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 208 op/s 2024-05-27T07:29:45.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:45 smithi162 ceph-mon[35676]: pgmap v262: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 208 op/s 2024-05-27T07:29:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.251+0000 7ff631c00700 1 -- 172.21.15.12:0/13093542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 msgr2=0x7ff62c117590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.251+0000 7ff631c00700 1 --2- 172.21.15.12:0/13093542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c117590 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff62400b0a0 tx=0x7ff62400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.251+0000 7ff631c00700 1 -- 172.21.15.12:0/13093542 shutdown_connections 2024-05-27T07:29:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.251+0000 7ff631c00700 1 --2- 172.21.15.12:0/13093542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c117590 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.251+0000 7ff631c00700 1 --2- 172.21.15.12:0/13093542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff62c075b50 0x7ff62c075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.251+0000 7ff631c00700 1 --2- 172.21.15.12:0/13093542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0764a0 0x7ff62c114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.251+0000 7ff631c00700 1 -- 172.21.15.12:0/13093542 >> 172.21.15.12:0/13093542 conn(0x7ff62c079d00 msgr2=0x7ff62c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.252+0000 7ff631c00700 1 -- 172.21.15.12:0/13093542 shutdown_connections 2024-05-27T07:29:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.253+0000 7ff631c00700 1 -- 172.21.15.12:0/13093542 wait complete. 2024-05-27T07:29:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.254+0000 7ff631c00700 1 Processor -- start 2024-05-27T07:29:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.254+0000 7ff631c00700 1 -- start start 2024-05-27T07:29:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.254+0000 7ff631c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff62c075b50 0x7ff62c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.254+0000 7ff631c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0764a0 0x7ff62c1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff631c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c1a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff631c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c11a7a0 con 0x7ff62c115170 2024-05-27T07:29:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff631c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c11a620 con 0x7ff62c0764a0 2024-05-27T07:29:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff631c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c11a920 con 0x7ff62c075b50 2024-05-27T07:29:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff631200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c1a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff631200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c1a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41448/0 (socket says 172.21.15.12:41448) 2024-05-27T07:29:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff631200700 1 -- 172.21.15.12:0/1096018878 learned_addr learned my addr 172.21.15.12:0/1096018878 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff62be00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff62c075b50 0x7ff62c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.255+0000 7ff623e00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0764a0 0x7ff62c1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff631200700 1 -- 172.21.15.12:0/1096018878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff62c075b50 msgr2=0x7ff62c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff631200700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff62c075b50 0x7ff62c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff631200700 1 -- 172.21.15.12:0/1096018878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0764a0 msgr2=0x7ff62c1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff631200700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0764a0 0x7ff62c1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff631200700 1 -- 172.21.15.12:0/1096018878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c1a5f90 con 0x7ff62c115170 2024-05-27T07:29:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff62be00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff62c075b50 0x7ff62c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff623e00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0764a0 0x7ff62c1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff631200700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c1a59a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff6240062a0 tx=0x7ff624004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.256+0000 7ff629600700 1 -- 172.21.15.12:0/1096018878 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624021070 con 0x7ff62c115170 2024-05-27T07:29:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.257+0000 7ff629600700 1 -- 172.21.15.12:0/1096018878 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff624007d00 con 0x7ff62c115170 2024-05-27T07:29:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.257+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62c1a6220 con 0x7ff62c115170 2024-05-27T07:29:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.257+0000 7ff629600700 1 -- 172.21.15.12:0/1096018878 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624005280 con 0x7ff62c115170 2024-05-27T07:29:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.257+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62c1a6700 con 0x7ff62c115170 2024-05-27T07:29:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.261+0000 7ff629600700 1 -- 172.21.15.12:0/1096018878 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff624024070 con 0x7ff62c115170 2024-05-27T07:29:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.262+0000 7ff641c5bb80 1 Processor -- start 2024-05-27T07:29:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.263+0000 7ff641c5bb80 1 -- start start 2024-05-27T07:29:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.263+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf0297ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.263+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0298410 0x55dbf02a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298820 0x55dbf02a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dbf01c6e40 con 0x55dbf0298410 2024-05-27T07:29:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dbf01b69e0 con 0x55dbf0298820 2024-05-27T07:29:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dbf01baa20 con 0x55dbf0119e40 2024-05-27T07:29:47.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff629600700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff6100b29f0 0x7ff6100b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff623e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf0297ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff629600700 1 -- 172.21.15.12:0/1096018878 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7ff62400f020 con 0x7ff62c115170 2024-05-27T07:29:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff623e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf0297ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:55464/0 (socket says 172.21.15.12:55464) 2024-05-27T07:29:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.264+0000 7ff623e00700 1 -- 172.21.15.12:0/1553214195 learned_addr learned my addr 172.21.15.12:0/1553214195 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:47.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.265+0000 7ff631200700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff6100b29f0 0x7ff6100b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.265+0000 7ff623e00700 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298820 0x55dbf02a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.266+0000 7ff623e00700 1 -- 172.21.15.12:0/1553214195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298820 msgr2=0x55dbf02a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.266+0000 7ff623e00700 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298820 0x55dbf02a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.266+0000 7ff623e00700 1 -- 172.21.15.12:0/1553214195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0298410 msgr2=0x55dbf02a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.266+0000 7ff623e00700 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0298410 0x55dbf02a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.266+0000 7ff623e00700 1 -- 172.21.15.12:0/1553214195 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbf017f420 con 0x55dbf0119e40 2024-05-27T07:29:47.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.266+0000 7ff631200700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff6100b29f0 0x7ff6100b4f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff6240e15c0 tx=0x7ff624007eb0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.267+0000 7ff623e00700 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf0297ed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6140139a0 tx=0x7ff61400b550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7078660a9f069a25 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.268+0000 7ff621600700 1 -- 172.21.15.12:0/1553214195 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614004830 con 0x55dbf0119e40 2024-05-27T07:29:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.268+0000 7ff621600700 1 -- 172.21.15.12:0/1553214195 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff614005020 con 0x55dbf0119e40 2024-05-27T07:29:47.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.268+0000 7ff621600700 1 -- 172.21.15.12:0/1553214195 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614004e60 con 0x55dbf0119e40 2024-05-27T07:29:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1553214195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 msgr2=0x55dbf0297ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf0297ed0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6140139a0 tx=0x7ff61400b550 comp rx=0 tx=0).stop 2024-05-27T07:29:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1553214195 shutdown_connections 2024-05-27T07:29:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298820 0x55dbf02a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf0297ed0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1553214195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0298410 0x55dbf02a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1553214195 >> 172.21.15.12:0/1553214195 conn(0x55dbf0179fc0 msgr2=0x55dbf0293d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.269+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1553214195 shutdown_connections 2024-05-27T07:29:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.270+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1553214195 wait complete. 2024-05-27T07:29:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.270+0000 7ff641c5bb80 1 -- start start 2024-05-27T07:29:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.271+0000 7ff641c5bb80 1 Processor -- start 2024-05-27T07:29:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0298820 0x55dbf02a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dbf00bcfa0 con 0x55dbf0119e40 2024-05-27T07:29:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dbf0157760 con 0x55dbf0298410 2024-05-27T07:29:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dbf02a5990 con 0x55dbf0298820 2024-05-27T07:29:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff641c5bb80 10 client.0 _ll_register_callbacks cb 0x55dbf0160c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff623e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff623e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41460/0 (socket says 172.21.15.12:41460) 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.272+0000 7ff623e00700 1 -- 172.21.15.12:0/1492116623 learned_addr learned my addr 172.21.15.12:0/1492116623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.273+0000 7ff62be00700 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.273+0000 7ff623e00700 1 -- 172.21.15.12:0/1492116623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0298820 msgr2=0x55dbf02a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.273+0000 7ff623e00700 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0298820 0x55dbf02a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.273+0000 7ff623e00700 1 -- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 msgr2=0x55dbf02a2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.273+0000 7ff623e00700 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.273+0000 7ff623e00700 1 -- 172.21.15.12:0/1492116623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbf01068d0 con 0x55dbf0119e40 2024-05-27T07:29:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.273+0000 7ff623e00700 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a2630 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff61400b880 tx=0x7ff6140132e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde5285647e50e8b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.274+0000 7ff61a000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.275+0000 7ff61a000700 1 -- 172.21.15.12:0/1492116623 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614004750 con 0x55dbf0119e40 2024-05-27T07:29:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.275+0000 7ff61a000700 1 -- 172.21.15.12:0/1492116623 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7ff6140048f0 con 0x55dbf0119e40 2024-05-27T07:29:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.275+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff614005020 con 0x55dbf0119e40 2024-05-27T07:29:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.275+0000 7ff61a000700 1 -- 172.21.15.12:0/1492116623 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7ff614004af0 con 0x55dbf0119e40 2024-05-27T07:29:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.275+0000 7ff641c5bb80 10 client.16353 fetch_fsmap learned FSMap version 107 2024-05-27T07:29:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.275+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55dbf0187060 con 0x55dbf0119e40 2024-05-27T07:29:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff61a000700 1 -- 172.21.15.12:0/1492116623 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff614005020 con 0x55dbf0119e40 2024-05-27T07:29:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff61a000700 1 -- 172.21.15.12:0/1492116623 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 107) ==== 4143+0+0 (secure 0 0 0) 0x7ff61402c4a0 con 0x55dbf0119e40 2024-05-27T07:29:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff641c5bb80 10 client.16353 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:29:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff641c5bb80 10 client.16353 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:29:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff641c5bb80 20 client.16353 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf02b2c20 0x55dbf02b5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff641c5bb80 4 client.16353 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.276+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbf01c8d90 con 0x55dbf02b2c20 2024-05-27T07:29:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.277+0000 7ff62be00700 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf02b2c20 0x55dbf02b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.278+0000 7ff62be00700 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf02b2c20 0x55dbf02b5070 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c01a1e7ce529db06 server_cookie=cf27f8f2d5f5b439 in_seq=0 out_seq=0 2024-05-27T07:29:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.278+0000 7ff61a000700 10 client.16353 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:29:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.281+0000 7ff61a000700 1 -- 172.21.15.12:0/1492116623 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dbf01c8d90 con 0x55dbf02b2c20 2024-05-27T07:29:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.281+0000 7ff61a000700 10 client.16353 handle_command_reply: tid=0 2024-05-27T07:29:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.281+0000 7ff641c5bb80 1 client.16353 shutdown 2024-05-27T07:29:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.281+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 msgr2=0x55dbf02a2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.281+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a2630 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff61400b880 tx=0x7ff6140132e0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 shutdown_connections 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0298820 0x55dbf02a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a2630 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf02b2c20 0x55dbf02b5070 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/1492116623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 >> 172.21.15.12:0/1492116623 conn(0x55dbf0179fc0 msgr2=0x55dbf0292ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff61a000700 0 client.16353 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:29:47.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 shutdown_connections 2024-05-27T07:29:47.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 1 -- 172.21.15.12:0/1492116623 wait complete. 2024-05-27T07:29:47.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.282+0000 7ff641c5bb80 20 client.16353 trim_cache size 0 max 16384 2024-05-27T07:29:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.305+0000 7ff641c5bb80 1 Processor -- start 2024-05-27T07:29:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff641c5bb80 1 -- start start 2024-05-27T07:29:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0179fc0 0x55dbf02a1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c1b6dd0 con 0x55dbf0119e40 2024-05-27T07:29:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c1b51a0 con 0x55dbf0298410 2024-05-27T07:29:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c1b66a0 con 0x55dbf0179fc0 2024-05-27T07:29:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff62be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff62be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41472/0 (socket says 172.21.15.12:41472) 2024-05-27T07:29:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.306+0000 7ff62be00700 1 -- 172.21.15.12:0/2009790815 learned_addr learned my addr 172.21.15.12:0/2009790815 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff623e00700 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0179fc0 0x55dbf02a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff62be00700 1 -- 172.21.15.12:0/2009790815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0179fc0 msgr2=0x55dbf02a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff623e00700 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff62be00700 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0179fc0 0x55dbf02a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff62be00700 1 -- 172.21.15.12:0/2009790815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 msgr2=0x55dbf02a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff62be00700 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff62be00700 1 -- 172.21.15.12:0/2009790815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbf0295350 con 0x55dbf0119e40 2024-05-27T07:29:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.307+0000 7ff623e00700 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.308+0000 7ff62be00700 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a3020 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000c00 tx=0x7ff61c00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2159a33b82714735 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.308+0000 7ff620c00700 1 -- 172.21.15.12:0/2009790815 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c024070 con 0x55dbf0119e40 2024-05-27T07:29:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.308+0000 7ff620c00700 1 -- 172.21.15.12:0/2009790815 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff61c016da0 con 0x55dbf0119e40 2024-05-27T07:29:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.308+0000 7ff620c00700 1 -- 172.21.15.12:0/2009790815 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c01fc10 con 0x55dbf0119e40 2024-05-27T07:29:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.310+0000 7ff641c5bb80 1 -- 172.21.15.12:0/2009790815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 msgr2=0x55dbf02a3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.310+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a3020 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000c00 tx=0x7ff61c00bdf0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.310+0000 7ff641c5bb80 1 -- 172.21.15.12:0/2009790815 shutdown_connections 2024-05-27T07:29:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.310+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.310+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0179fc0 0x55dbf02a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.310+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/2009790815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0119e40 0x55dbf02a3020 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.310+0000 7ff641c5bb80 1 -- 172.21.15.12:0/2009790815 >> 172.21.15.12:0/2009790815 conn(0x55dbf0293ef0 msgr2=0x55dbf029ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.311+0000 7ff641c5bb80 1 -- 172.21.15.12:0/2009790815 shutdown_connections 2024-05-27T07:29:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.311+0000 7ff641c5bb80 1 -- 172.21.15.12:0/2009790815 wait complete. 2024-05-27T07:29:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.311+0000 7ff641c5bb80 1 -- start start 2024-05-27T07:29:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 1 Processor -- start 2024-05-27T07:29:47.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf02ab390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0179fc0 0x55dbf02a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c1b5f70 con 0x55dbf0179fc0 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c1b5870 con 0x55dbf0298410 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dbf01c8d90 con 0x55dbf0119e40 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.312+0000 7ff641c5bb80 10 client.0 _ll_register_callbacks cb 0x55dbf0160c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0179fc0 0x55dbf02a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0179fc0 0x55dbf02a66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41484/0 (socket says 172.21.15.12:41484) 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 -- 172.21.15.12:0/3724206751 learned_addr learned my addr 172.21.15.12:0/3724206751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:47.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff623e00700 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf02ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff623e00700 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 -- 172.21.15.12:0/3724206751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 msgr2=0x55dbf02ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf02ab390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 -- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 msgr2=0x55dbf02a6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.313+0000 7ff62be00700 1 -- 172.21.15.12:0/3724206751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbf02a73f0 con 0x55dbf0179fc0 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.314+0000 7ff62be00700 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0179fc0 0x55dbf02a66f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff61c006e70 tx=0x7ff61c0169e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff11ceb5d7fa376 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.314+0000 7ff61a000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:47.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.314+0000 7ff61a000700 1 -- 172.21.15.12:0/3724206751 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c02c070 con 0x55dbf0179fc0 2024-05-27T07:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.314+0000 7ff61a000700 1 -- 172.21.15.12:0/3724206751 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7ff61c01f480 con 0x55dbf0179fc0 2024-05-27T07:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.315+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff61c016da0 con 0x55dbf0179fc0 2024-05-27T07:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.315+0000 7ff61a000700 1 -- 172.21.15.12:0/3724206751 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7ff61c036050 con 0x55dbf0179fc0 2024-05-27T07:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.315+0000 7ff641c5bb80 10 client.16359 fetch_fsmap learned FSMap version 107 2024-05-27T07:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.315+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55dbf031d5c0 con 0x55dbf0179fc0 2024-05-27T07:29:47.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.315+0000 7ff61a000700 1 -- 172.21.15.12:0/3724206751 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff61c016da0 con 0x55dbf0179fc0 2024-05-27T07:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff61a000700 1 -- 172.21.15.12:0/3724206751 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 107) ==== 4143+0+0 (secure 0 0 0) 0x7ff61c027050 con 0x55dbf0179fc0 2024-05-27T07:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff641c5bb80 10 client.16359 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff641c5bb80 10 client.16359 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff641c5bb80 20 client.16359 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:47.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf0298820 0x55dbf031fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff641c5bb80 4 client.16359 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dbf01c2570 con 0x55dbf0298820 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.316+0000 7ff623e00700 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf0298820 0x55dbf031fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.317+0000 7ff623e00700 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf0298820 0x55dbf031fc80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d29c2c8f4bf5b881 server_cookie=49f7e7d3bd4b901b in_seq=0 out_seq=0 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.317+0000 7ff61a000700 10 client.16359 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.318+0000 7ff61a000700 1 -- 172.21.15.12:0/3724206751 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dbf01c2570 con 0x55dbf0298820 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.318+0000 7ff61a000700 10 client.16359 handle_command_reply: tid=0 2024-05-27T07:29:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.318+0000 7ff641c5bb80 1 client.16359 shutdown 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.318+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0179fc0 msgr2=0x55dbf02a66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.318+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0179fc0 0x55dbf02a66f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff61c006e70 tx=0x7ff61c0169e0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 shutdown_connections 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55dbf0298820 0x55dbf031fc80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dbf0298410 0x55dbf02a6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf0119e40 0x55dbf02ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff641c5bb80 1 --2- 172.21.15.12:0/3724206751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dbf0179fc0 0x55dbf02a66f0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 >> 172.21.15.12:0/3724206751 conn(0x55dbf0293ef0 msgr2=0x55dbf02a2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff61a000700 0 client.16359 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:29:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.319+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 shutdown_connections 2024-05-27T07:29:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.320+0000 7ff641c5bb80 1 -- 172.21.15.12:0/3724206751 wait complete. 2024-05-27T07:29:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.320+0000 7ff641c5bb80 20 client.16359 trim_cache size 0 max 16384 2024-05-27T07:29:47.314 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.321+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff6100b29f0 msgr2=0x7ff6100b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.321+0000 7ff631c00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff6100b29f0 0x7ff6100b4f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff6240e15c0 tx=0x7ff624007eb0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.321+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 msgr2=0x7ff62c1a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.321+0000 7ff631c00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c1a59a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff6240062a0 tx=0x7ff624004ed0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 shutdown_connections 2024-05-27T07:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff6100b29f0 0x7ff6100b4f00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c115170 0x7ff62c1a59a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff62c075b50 0x7ff62c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 --2- 172.21.15.12:0/1096018878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0764a0 0x7ff62c1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 >> 172.21.15.12:0/1096018878 conn(0x7ff62c079d00 msgr2=0x7ff62c100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 shutdown_connections 2024-05-27T07:29:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:47.322+0000 7ff631c00700 1 -- 172.21.15.12:0/1096018878 wait complete. 2024-05-27T07:29:47.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:47 smithi162 ceph-mon[35676]: pgmap v263: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.7 MiB/s wr, 180 op/s 2024-05-27T07:29:47.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:47 smithi062 ceph-mon[35304]: pgmap v263: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.7 MiB/s wr, 180 op/s 2024-05-27T07:29:48.085 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:47 smithi012 ceph-mon[31473]: pgmap v263: 129 pgs: 129 active+clean; 111 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.7 MiB/s wr, 180 op/s 2024-05-27T07:29:48.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:29:48.651 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:49.890 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:49 smithi012 ceph-mon[31473]: pgmap v264: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 2.3 MiB/s wr, 243 op/s 2024-05-27T07:29:49.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:49 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:49.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:49 smithi162 ceph-mon[35676]: pgmap v264: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 2.3 MiB/s wr, 243 op/s 2024-05-27T07:29:49.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:49 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:49.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:49 smithi062 ceph-mon[35304]: pgmap v264: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 2.3 MiB/s wr, 243 op/s 2024-05-27T07:29:49.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:49 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:50.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:29:50.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:50 smithi012 systemd[1]: Stopped Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:50.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:50 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.598s CPU time. 2024-05-27T07:29:50.341 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:50 smithi012 systemd[1]: Starting Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:50.976 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:50 smithi012 podman[124438]: 2024-05-27 07:29:50.754349167 +0000 UTC m=+0.117606091 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.018+0000 7ff02ae00700 1 -- 172.21.15.12:0/2889892965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024075d20 msgr2=0x7ff024076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.018+0000 7ff02ae00700 1 --2- 172.21.15.12:0/2889892965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024075d20 0x7ff024076130 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff014009990 tx=0x7ff014009ca0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.018+0000 7ff02ae00700 1 -- 172.21.15.12:0/2889892965 shutdown_connections 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.018+0000 7ff02ae00700 1 --2- 172.21.15.12:0/2889892965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff024110b90 0x7ff024112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.018+0000 7ff02ae00700 1 --2- 172.21.15.12:0/2889892965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024075d20 0x7ff024076130 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.018+0000 7ff02ae00700 1 --2- 172.21.15.12:0/2889892965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff024076670 0x7ff024110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.018+0000 7ff02ae00700 1 -- 172.21.15.12:0/2889892965 >> 172.21.15.12:0/2889892965 conn(0x7ff024100030 msgr2=0x7ff024102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:51.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.019+0000 7ff02ae00700 1 -- 172.21.15.12:0/2889892965 shutdown_connections 2024-05-27T07:29:51.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.019+0000 7ff02ae00700 1 -- 172.21.15.12:0/2889892965 wait complete. 2024-05-27T07:29:51.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 Processor -- start 2024-05-27T07:29:51.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 -- start start 2024-05-27T07:29:51.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff024075d20 0x7ff02410efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024076670 0x7ff02410f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff024110b90 0x7ff0241acf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff024104d40 con 0x7ff024076670 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff024104bc0 con 0x7ff024075d20 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.021+0000 7ff02ae00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff024104ec0 con 0x7ff024110b90 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024076670 0x7ff02410f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024076670 0x7ff02410f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41500/0 (socket says 172.21.15.12:41500) 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 -- 172.21.15.12:0/3422885037 learned_addr learned my addr 172.21.15.12:0/3422885037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:51.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff028e00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff024110b90 0x7ff0241acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 -- 172.21.15.12:0/3422885037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff024110b90 msgr2=0x7ff0241acf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff024110b90 0x7ff0241acf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 -- 172.21.15.12:0/3422885037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff024075d20 msgr2=0x7ff02410efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023e00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff024075d20 0x7ff02410efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff024075d20 0x7ff02410efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023400700 1 -- 172.21.15.12:0/3422885037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0241ad4d0 con 0x7ff024076670 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff028e00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff024110b90 0x7ff0241acf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:51.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.022+0000 7ff023e00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff024075d20 0x7ff02410efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:51.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.023+0000 7ff023400700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024076670 0x7ff02410f4f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff01000d980 tx=0x7ff01000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:51.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.024+0000 7ff020c00700 1 -- 172.21.15.12:0/3422885037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010017070 con 0x7ff024076670 2024-05-27T07:29:51.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.024+0000 7ff020c00700 1 -- 172.21.15.12:0/3422885037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff010004830 con 0x7ff024076670 2024-05-27T07:29:51.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.024+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0241ad760 con 0x7ff024076670 2024-05-27T07:29:51.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.024+0000 7ff020c00700 1 -- 172.21.15.12:0/3422885037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0100053a0 con 0x7ff024076670 2024-05-27T07:29:51.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.024+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0241adca0 con 0x7ff024076670 2024-05-27T07:29:51.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.029+0000 7ff020c00700 1 -- 172.21.15.12:0/3422885037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff0100049e0 con 0x7ff024076670 2024-05-27T07:29:51.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.030+0000 7ff039a2cb80 1 Processor -- start 2024-05-27T07:29:51.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.030+0000 7ff039a2cb80 1 -- start start 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff020c00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0080b2a70 0x7ff0080b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d307f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d2a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d2b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563690c58e40 con 0x563690d30d30 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563690c489e0 con 0x563690babe40 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563690c4ca20 con 0x563690d2adb0 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.031+0000 7ff020c00700 1 -- 172.21.15.12:0/3422885037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7ff0100e2610 con 0x7ff024076670 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.032+0000 7ff023400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d2a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.032+0000 7ff028e00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0080b2a70 0x7ff0080b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.032+0000 7ff023e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.032+0000 7ff023e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d307f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:47330/0 (socket says 172.21.15.12:47330) 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.032+0000 7ff023e00700 1 -- 172.21.15.12:0/2004489147 learned_addr learned my addr 172.21.15.12:0/2004489147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.032+0000 7ff023400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d2a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41502/0 (socket says 172.21.15.12:41502) 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.032+0000 7ff028e00700 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d2b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.033+0000 7ff028e00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0080b2a70 0x7ff0080b4f80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff024077bc0 tx=0x7ff01c012af0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.033+0000 7ff028e00700 1 -- 172.21.15.12:0/2004489147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 msgr2=0x563690d307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.033+0000 7ff028e00700 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d307f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.033+0000 7ff028e00700 1 -- 172.21.15.12:0/2004489147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 msgr2=0x563690d2a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.033+0000 7ff028e00700 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d2a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.033+0000 7ff028e00700 1 -- 172.21.15.12:0/2004489147 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563690c11420 con 0x563690d2adb0 2024-05-27T07:29:51.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.034+0000 7ff028e00700 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d2b1c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff01c012ef0 tx=0x7ff01c01c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7137a627341db8a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.034+0000 7ff019600700 1 -- 172.21.15.12:0/2004489147 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c015e90 con 0x563690d2adb0 2024-05-27T07:29:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.034+0000 7ff019600700 1 -- 172.21.15.12:0/2004489147 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff01c011040 con 0x563690d2adb0 2024-05-27T07:29:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.035+0000 7ff019600700 1 -- 172.21.15.12:0/2004489147 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c021660 con 0x563690d2adb0 2024-05-27T07:29:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 -- 172.21.15.12:0/2004489147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 msgr2=0x563690d2b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d2b1c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff01c012ef0 tx=0x7ff01c01c000 comp rx=0 tx=0).stop 2024-05-27T07:29:51.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 -- 172.21.15.12:0/2004489147 shutdown_connections 2024-05-27T07:29:51.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d2b1c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/2004489147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d2a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 -- 172.21.15.12:0/2004489147 >> 172.21.15.12:0/2004489147 conn(0x563690c0bfc0 msgr2=0x563690d258d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:51.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.036+0000 7ff039a2cb80 1 -- 172.21.15.12:0/2004489147 shutdown_connections 2024-05-27T07:29:51.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.037+0000 7ff039a2cb80 1 -- 172.21.15.12:0/2004489147 wait complete. 2024-05-27T07:29:51.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.037+0000 7ff039a2cb80 1 -- start start 2024-05-27T07:29:51.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.038+0000 7ff039a2cb80 1 Processor -- start 2024-05-27T07:29:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.038+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d344e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.038+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d34a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.038+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d34f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563690b4efa0 con 0x563690d30d30 2024-05-27T07:29:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563690be9760 con 0x563690babe40 2024-05-27T07:29:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563690d377a0 con 0x563690d2adb0 2024-05-27T07:29:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff039a2cb80 10 client.0 _ll_register_callbacks cb 0x563690bf2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff028e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d34f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff023e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff028e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d34f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41512/0 (socket says 172.21.15.12:41512) 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff028e00700 1 -- 172.21.15.12:0/476205333 learned_addr learned my addr 172.21.15.12:0/476205333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.039+0000 7ff023400700 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d34a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.040+0000 7ff028e00700 1 -- 172.21.15.12:0/476205333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 msgr2=0x563690d34a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.040+0000 7ff028e00700 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d34a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.040+0000 7ff028e00700 1 -- 172.21.15.12:0/476205333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 msgr2=0x563690d344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.040+0000 7ff028e00700 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.040+0000 7ff028e00700 1 -- 172.21.15.12:0/476205333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563690b988d0 con 0x563690d30d30 2024-05-27T07:29:51.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.041+0000 7ff028e00700 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d34f60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff01c018c20 tx=0x7ff01c018c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab09ccb1f496fe16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:51.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.041+0000 7ff00e000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.041+0000 7ff00e000700 1 -- 172.21.15.12:0/476205333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c004ad0 con 0x563690d30d30 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.041+0000 7ff00e000700 1 -- 172.21.15.12:0/476205333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7ff01c004c70 con 0x563690d30d30 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.041+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff01c011040 con 0x563690d30d30 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.042+0000 7ff00e000700 1 -- 172.21.15.12:0/476205333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7ff01c037050 con 0x563690d30d30 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.042+0000 7ff039a2cb80 10 client.16371 fetch_fsmap learned FSMap version 108 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.042+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x563690c19060 con 0x563690d30d30 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff00e000700 1 -- 172.21.15.12:0/476205333 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff01c011040 con 0x563690d30d30 2024-05-27T07:29:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff00e000700 1 -- 172.21.15.12:0/476205333 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 108) ==== 4139+0+0 (secure 0 0 0) 0x7ff01c029400 con 0x563690d30d30 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff039a2cb80 10 client.16371 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff039a2cb80 10 client.16371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff039a2cb80 20 client.16371 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690d44960 0x563690d46e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff039a2cb80 4 client.16371 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.043+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563690c5ad90 con 0x563690d44960 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.044+0000 7ff023e00700 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690d44960 0x563690d46e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.045+0000 7ff023e00700 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690d44960 0x563690d46e10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a240a2b979f6a9be server_cookie=9158108393ea7124 in_seq=0 out_seq=0 2024-05-27T07:29:51.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.045+0000 7ff00e000700 10 client.16371 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:29:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff00e000700 1 -- 172.21.15.12:0/476205333 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563690c5ad90 con 0x563690d44960 2024-05-27T07:29:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff00e000700 10 client.16371 handle_command_reply: tid=0 2024-05-27T07:29:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 client.16371 shutdown 2024-05-27T07:29:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 msgr2=0x563690d34f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d34f60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff01c018c20 tx=0x7ff01c018c50 comp rx=0 tx=0).stop 2024-05-27T07:29:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 shutdown_connections 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690d44960 0x563690d46e10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690babe40 0x563690d344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d30d30 0x563690d34f60 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/476205333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d34a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 >> 172.21.15.12:0/476205333 conn(0x563690c0bfc0 msgr2=0x563690d25580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff00e000700 0 client.16371 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 shutdown_connections 2024-05-27T07:29:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.048+0000 7ff039a2cb80 1 -- 172.21.15.12:0/476205333 wait complete. 2024-05-27T07:29:51.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.049+0000 7ff039a2cb80 20 client.16371 trim_cache size 0 max 16384 2024-05-27T07:29:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.068+0000 7ff039a2cb80 1 Processor -- start 2024-05-27T07:29:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.068+0000 7ff039a2cb80 1 -- start start 2024-05-27T07:29:51.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690babe40 0x563690d2a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d2adb0 0x563690d33440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d36980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b7010 con 0x563690d2adb0 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b5410 con 0x563690d30d30 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b68e0 con 0x563690babe40 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff023400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d2adb0 0x563690d33440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff023400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d2adb0 0x563690d33440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41522/0 (socket says 172.21.15.12:41522) 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.069+0000 7ff023400700 1 -- 172.21.15.12:0/680436144 learned_addr learned my addr 172.21.15.12:0/680436144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:51.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.070+0000 7ff023e00700 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690babe40 0x563690d2a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.070+0000 7ff028e00700 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d36980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.070+0000 7ff023400700 1 -- 172.21.15.12:0/680436144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690babe40 msgr2=0x563690d2a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.070+0000 7ff023400700 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690babe40 0x563690d2a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.070+0000 7ff023400700 1 -- 172.21.15.12:0/680436144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 msgr2=0x563690d36980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.070+0000 7ff023400700 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d36980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.070+0000 7ff023400700 1 -- 172.21.15.12:0/680436144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563690d46d30 con 0x563690d2adb0 2024-05-27T07:29:51.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.071+0000 7ff023400700 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d2adb0 0x563690d33440 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff0100a6750 tx=0x7ff0100a9640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1ede6418bc82a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:51.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.071+0000 7ff018c00700 1 -- 172.21.15.12:0/680436144 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0100961c0 con 0x563690d2adb0 2024-05-27T07:29:51.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.071+0000 7ff018c00700 1 -- 172.21.15.12:0/680436144 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff0100a5ad0 con 0x563690d2adb0 2024-05-27T07:29:51.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.071+0000 7ff018c00700 1 -- 172.21.15.12:0/680436144 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0100dfbd0 con 0x563690d2adb0 2024-05-27T07:29:51.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.072+0000 7ff039a2cb80 1 -- 172.21.15.12:0/680436144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d2adb0 msgr2=0x563690d33440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.072+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d2adb0 0x563690d33440 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff0100a6750 tx=0x7ff0100a9640 comp rx=0 tx=0).stop 2024-05-27T07:29:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 -- 172.21.15.12:0/680436144 shutdown_connections 2024-05-27T07:29:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d36980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690babe40 0x563690d2a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/680436144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690d2adb0 0x563690d33440 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 -- 172.21.15.12:0/680436144 >> 172.21.15.12:0/680436144 conn(0x563690c0bfc0 msgr2=0x563690d344a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 -- 172.21.15.12:0/680436144 shutdown_connections 2024-05-27T07:29:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 -- 172.21.15.12:0/680436144 wait complete. 2024-05-27T07:29:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.073+0000 7ff039a2cb80 1 -- start start 2024-05-27T07:29:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 1 Processor -- start 2024-05-27T07:29:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690babe40 0x563690d3ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d3cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d4b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b61b0 con 0x563690babe40 2024-05-27T07:29:51.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b5ae0 con 0x563690d30d30 2024-05-27T07:29:51.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563690d47400 con 0x563690d2adb0 2024-05-27T07:29:51.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.074+0000 7ff039a2cb80 10 client.0 _ll_register_callbacks cb 0x563690bf2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:51.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.075+0000 7ff023400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d3cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.075+0000 7ff023e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690babe40 0x563690d3ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.075+0000 7ff023400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d3cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:55544/0 (socket says 172.21.15.12:55544) 2024-05-27T07:29:51.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.075+0000 7ff023400700 1 -- 172.21.15.12:0/3928914266 learned_addr learned my addr 172.21.15.12:0/3928914266 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:51.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.075+0000 7ff028e00700 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d4b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.076+0000 7ff023400700 1 -- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 msgr2=0x563690d4b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.076+0000 7ff023400700 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d4b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.076+0000 7ff023400700 1 -- 172.21.15.12:0/3928914266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690babe40 msgr2=0x563690d3ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.076+0000 7ff023400700 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690babe40 0x563690d3ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.076+0000 7ff023400700 1 -- 172.21.15.12:0/3928914266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563690d4cff0 con 0x563690d2adb0 2024-05-27T07:29:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.076+0000 7ff023400700 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d3cf70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff01008e040 tx=0x7ff0100a5980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eb676db7f917193 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.076+0000 7ff00e000700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:29:51.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.077+0000 7ff00e000700 1 -- 172.21.15.12:0/3928914266 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0100a9be0 con 0x563690d2adb0 2024-05-27T07:29:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.077+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0100a5ad0 con 0x563690d2adb0 2024-05-27T07:29:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.077+0000 7ff00e000700 1 -- 172.21.15.12:0/3928914266 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9466+0+0 (secure 0 0 0) 0x7ff0100e2190 con 0x563690d2adb0 2024-05-27T07:29:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.077+0000 7ff00e000700 1 -- 172.21.15.12:0/3928914266 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7ff0100dfdd0 con 0x563690d2adb0 2024-05-27T07:29:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.077+0000 7ff039a2cb80 10 client.26032 fetch_fsmap learned FSMap version 108 2024-05-27T07:29:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.077+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x563690dae860 con 0x563690d2adb0 2024-05-27T07:29:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff00e000700 1 -- 172.21.15.12:0/3928914266 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff0100a5ad0 con 0x563690d2adb0 2024-05-27T07:29:51.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff00e000700 1 -- 172.21.15.12:0/3928914266 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 108) ==== 4139+0+0 (secure 0 0 0) 0x7ff0100b1eb0 con 0x563690d2adb0 2024-05-27T07:29:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff039a2cb80 10 client.26032 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:29:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff039a2cb80 10 client.26032 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:29:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff039a2cb80 20 client.26032 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690daed00 0x563690db1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff039a2cb80 4 client.26032 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.078+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563690c54570 con 0x563690daed00 2024-05-27T07:29:51.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.079+0000 7ff023e00700 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690daed00 0x563690db1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:51.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.080+0000 7ff023e00700 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690daed00 0x563690db1120 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e5e39207678588f7 server_cookie=bd1452fb8e59bffc in_seq=0 out_seq=0 2024-05-27T07:29:51.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.080+0000 7ff00e000700 10 client.26032 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:29:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff00e000700 1 -- 172.21.15.12:0/3928914266 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563690c54570 con 0x563690daed00 2024-05-27T07:29:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff00e000700 10 client.26032 handle_command_reply: tid=0 2024-05-27T07:29:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 client.26032 shutdown 2024-05-27T07:29:51.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 msgr2=0x563690d3cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d3cf70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff01008e040 tx=0x7ff0100a5980 comp rx=0 tx=0).stop 2024-05-27T07:29:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 shutdown_connections 2024-05-27T07:29:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x563690daed00 0x563690db1120 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563690d30d30 0x563690d4b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563690babe40 0x563690d3ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 --2- 172.21.15.12:0/3928914266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563690d2adb0 0x563690d3cf70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 >> 172.21.15.12:0/3928914266 conn(0x563690c0bfc0 msgr2=0x563690d2fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:51.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.081+0000 7ff00e000700 0 client.26032 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:29:51.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.082+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 shutdown_connections 2024-05-27T07:29:51.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.082+0000 7ff039a2cb80 1 -- 172.21.15.12:0/3928914266 wait complete. 2024-05-27T07:29:51.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.082+0000 7ff039a2cb80 20 client.26032 trim_cache size 0 max 16384 2024-05-27T07:29:51.074 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:29:51.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0080b2a70 msgr2=0x7ff0080b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0080b2a70 0x7ff0080b4f80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff024077bc0 tx=0x7ff01c012af0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024076670 msgr2=0x7ff02410f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024076670 0x7ff02410f4f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff01000d980 tx=0x7ff01000dd40 comp rx=0 tx=0).stop 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 shutdown_connections 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0080b2a70 0x7ff0080b4f80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff024075d20 0x7ff02410efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff024110b90 0x7ff0241acf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 --2- 172.21.15.12:0/3422885037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024076670 0x7ff02410f4f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:51.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.083+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 >> 172.21.15.12:0/3422885037 conn(0x7ff024100030 msgr2=0x7ff0241006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.087+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 shutdown_connections 2024-05-27T07:29:51.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:51.087+0000 7ff02ae00700 1 -- 172.21.15.12:0/3422885037 wait complete. 2024-05-27T07:29:51.340 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:50 smithi012 podman[124438]: 2024-05-27 07:29:50.985478618 +0000 UTC m=+0.348735538 container create 5bafe610d886b4c6fbfd5e93e9f2029414983161e6a6051b18f0c412c1c78864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:29:51.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:51 smithi012 ceph-mon[31473]: pgmap v265: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:29:51.772 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:51 smithi012 podman[124438]: 2024-05-27 07:29:51.493962262 +0000 UTC m=+0.857219249 container init 5bafe610d886b4c6fbfd5e93e9f2029414983161e6a6051b18f0c412c1c78864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:29:51.772 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:51 smithi012 podman[124438]: 2024-05-27 07:29:51.509618444 +0000 UTC m=+0.872875383 container start 5bafe610d886b4c6fbfd5e93e9f2029414983161e6a6051b18f0c412c1c78864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:51.772 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:51 smithi012 bash[124438]: 5bafe610d886b4c6fbfd5e93e9f2029414983161e6a6051b18f0c412c1c78864 2024-05-27T07:29:51.772 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:51 smithi012 systemd[1]: Started Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:51.772 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:51 smithi012 ceph-mds[124515]: -- 172.21.15.12:0/3392317282 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596738b5a40 con 0x5596738fda80 2024-05-27T07:29:51.772 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:51 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: starting mds.d at 2024-05-27T07:29:51.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:51 smithi162 ceph-mon[35676]: pgmap v265: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:29:51.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:51 smithi062 ceph-mon[35304]: pgmap v265: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:29:52.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:29:52.487 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:52 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:29:52.487 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:52 smithi012 systemd[1]: Stopped Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:52.487 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:52 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.759s CPU time. 2024-05-27T07:29:52.487 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:52 smithi012 systemd[1]: Starting Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:29:52.840 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:52 smithi012 podman[124614]: 2024-05-27 07:29:52.651473767 +0000 UTC m=+0.056271722 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:52.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:52 smithi012 ceph-mon[31473]: daemon mds.d restarted 2024-05-27T07:29:52.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:52 smithi012 ceph-mon[31473]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:29:52.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:52 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/638524566,v1:172.21.15.12:6835/638524566] up:boot 2024-05-27T07:29:52.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:52 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:52.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:52 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:52.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff31ac00700 time 2024-05-27T07:29:52.496491+0000 2024-05-27T07:29:52.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.842 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff330728ba6] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.843 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: clone() 2024-05-27T07:29:52.844 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.498+0000 7ff31ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff31ac00700 time 2024-05-27T07:29:52.496491+0000 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff330728ba6] 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.845 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: clone() 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.846 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: *** Caught signal (Aborted) ** 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: in thread 7ff31ac00700 thread_name:md_log_replay 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff32f2a5d20] 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: gsignal() 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: abort() 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff330728ced] 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.847 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 19: clone() 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.503+0000 7ff31ac00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: in thread 7ff31ac00700 thread_name:md_log_replay 2024-05-27T07:29:52.848 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff32f2a5d20] 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: gsignal() 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: abort() 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff330728ced] 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.849 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 19: clone() 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:52.850 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: -1> 2024-05-27T07:29:52.498+0000 7ff31ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff31ac00700 time 2024-05-27T07:29:52.496491+0000 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff330728ba6] 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.851 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.852 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: clone() 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 0> 2024-05-27T07:29:52.503+0000 7ff31ac00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: in thread 7ff31ac00700 thread_name:md_log_replay 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff32f2a5d20] 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: gsignal() 2024-05-27T07:29:52.853 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: abort() 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff330728ced] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.854 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 19: clone() 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: -1> 2024-05-27T07:29:52.498+0000 7ff31ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff31ac00700 time 2024-05-27T07:29:52.496491+0000 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.855 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff330728ba6] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.856 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: clone() 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 0> 2024-05-27T07:29:52.503+0000 7ff31ac00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: in thread 7ff31ac00700 thread_name:md_log_replay 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.857 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff32f2a5d20] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2: gsignal() 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 3: abort() 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff330728ced] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff330728dd1] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559637f3523d] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559637f20971] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 8: (SnapRealm::check_cache() const+0x13e) [0x559637f1efa0] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559637f1f651] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559637eb4347] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559637bb087e] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559637bb5e94] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559637bc0969] 2024-05-27T07:29:52.858 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 14: (MDLog::_replay_thread()+0x1f38) [0x559637f5948c] 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559637f629cb] 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 16: (Thread::entry_wrapper()+0x43) [0x7ff3306f772d] 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff3306f7749] 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff32f29b1ca] 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 19: clone() 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d[124502]: 2024-05-27T07:29:52.859 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:52 smithi012 podman[124615]: 2024-05-27 07:29:52.647589623 +0000 UTC m=+0.050474301 container died 5bafe610d886b4c6fbfd5e93e9f2029414983161e6a6051b18f0c412c1c78864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:29:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:52 smithi162 ceph-mon[35676]: daemon mds.d restarted 2024-05-27T07:29:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:52 smithi162 ceph-mon[35676]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:29:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:52 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/638524566,v1:172.21.15.12:6835/638524566] up:boot 2024-05-27T07:29:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:52 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:52.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:52 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:52 smithi062 ceph-mon[35304]: daemon mds.d restarted 2024-05-27T07:29:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:52 smithi062 ceph-mon[35304]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:29:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:52 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/638524566,v1:172.21.15.12:6835/638524566] up:boot 2024-05-27T07:29:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:52 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:52.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:52 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:53.260 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:53 smithi012 podman[124614]: 2024-05-27 07:29:53.109801055 +0000 UTC m=+0.514599032 container create 26784cf42e879354533de2f6859bad7a6dce4419ab587ed11a2b96ca7d07e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:53.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:53 smithi012 podman[124615]: 2024-05-27 07:29:53.539896368 +0000 UTC m=+0.942781089 container remove 5bafe610d886b4c6fbfd5e93e9f2029414983161e6a6051b18f0c412c1c78864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:29:53.840 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:53 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:53.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:53 smithi012 podman[124614]: 2024-05-27 07:29:53.741941163 +0000 UTC m=+1.146739143 container init 26784cf42e879354533de2f6859bad7a6dce4419ab587ed11a2b96ca7d07e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:53.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:53 smithi012 podman[124614]: 2024-05-27 07:29:53.75200489 +0000 UTC m=+1.156802854 container start 26784cf42e879354533de2f6859bad7a6dce4419ab587ed11a2b96ca7d07e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:29:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:54.205 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:53 smithi012 bash[124614]: 26784cf42e879354533de2f6859bad7a6dce4419ab587ed11a2b96ca7d07e2b4 2024-05-27T07:29:54.205 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:53 smithi012 systemd[1]: Started Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:29:54.205 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:53 smithi012 ceph-mds[124676]: -- 172.21.15.12:0/1217910457 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5557c6af9a40 con 0x5557c6b41180 2024-05-27T07:29:54.205 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: starting mds.j at 2024-05-27T07:29:54.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:54 smithi012 ceph-mon[31473]: pgmap v267: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-05-27T07:29:54.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:54 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:54.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:54 smithi012 ceph-mon[31473]: daemon mds.j restarted 2024-05-27T07:29:54.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:54 smithi162 ceph-mon[35676]: pgmap v267: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-05-27T07:29:54.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:54 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:54.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:54 smithi162 ceph-mon[35676]: daemon mds.j restarted 2024-05-27T07:29:54.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:54 smithi062 ceph-mon[35304]: pgmap v267: 129 pgs: 129 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-05-27T07:29:54.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:54 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:54.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:54 smithi062 ceph-mon[35304]: daemon mds.j restarted 2024-05-27T07:29:55.091 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:54 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:55.091 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:29:54 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.650s CPU time. 2024-05-27T07:29:55.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:55 smithi012 ceph-mon[31473]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:29:55.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:55 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/1476847376,v1:172.21.15.12:6835/1476847376] up:boot 2024-05-27T07:29:55.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:55 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:55.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:55 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:55.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:55 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:55.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:55 smithi162 ceph-mon[35676]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:29:55.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:55 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/1476847376,v1:172.21.15.12:6835/1476847376] up:boot 2024-05-27T07:29:55.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:55 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:55.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:55 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:55.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:55 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:55.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:55 smithi062 ceph-mon[35304]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:29:55.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:55 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/1476847376,v1:172.21.15.12:6835/1476847376] up:boot 2024-05-27T07:29:55.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:55 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 1 up:standby 2024-05-27T07:29:55.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:55 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:55.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:55 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 2024-05-27T07:29:56.022 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff566000700 time 2024-05-27T07:29:55.644779+0000 2024-05-27T07:29:56.022 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:56.022 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff57ba83ba6] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.023 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: clone() 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:55.645+0000 7ff566000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff566000700 time 2024-05-27T07:29:55.644779+0000 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.024 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff57ba83ba6] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.025 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: clone() 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: *** Caught signal (Aborted) ** 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: in thread 7ff566000700 thread_name:md_log_replay 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff57a600d20] 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: gsignal() 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: abort() 2024-05-27T07:29:56.026 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff57ba83ced] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.027 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 19: clone() 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:55.650+0000 7ff566000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: in thread 7ff566000700 thread_name:md_log_replay 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff57a600d20] 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: gsignal() 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: abort() 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff57ba83ced] 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.028 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.029 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 19: clone() 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: -1> 2024-05-27T07:29:55.645+0000 7ff566000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff566000700 time 2024-05-27T07:29:55.644779+0000 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff57ba83ba6] 2024-05-27T07:29:56.030 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.031 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.031 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.031 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.031 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.031 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.031 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.032 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: clone() 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 0> 2024-05-27T07:29:55.650+0000 7ff566000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: in thread 7ff566000700 thread_name:md_log_replay 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff57a600d20] 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: gsignal() 2024-05-27T07:29:56.033 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: abort() 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff57ba83ced] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.034 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.035 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.035 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.035 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.035 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 19: clone() 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: -1> 2024-05-27T07:29:55.645+0000 7ff566000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff566000700 time 2024-05-27T07:29:55.644779+0000 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff57ba83ba6] 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.036 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.037 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.038 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.038 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.038 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.038 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.038 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: clone() 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 0> 2024-05-27T07:29:55.650+0000 7ff566000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: in thread 7ff566000700 thread_name:md_log_replay 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff57a600d20] 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2: gsignal() 2024-05-27T07:29:56.039 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 3: abort() 2024-05-27T07:29:56.040 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff57ba83ced] 2024-05-27T07:29:56.040 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff57ba83dd1] 2024-05-27T07:29:56.040 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55579e1b323d] 2024-05-27T07:29:56.040 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55579e19e971] 2024-05-27T07:29:56.040 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 8: (SnapRealm::check_cache() const+0x13e) [0x55579e19cfa0] 2024-05-27T07:29:56.040 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55579e19d651] 2024-05-27T07:29:56.041 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55579e132347] 2024-05-27T07:29:56.041 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55579de2e87e] 2024-05-27T07:29:56.041 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55579de33e94] 2024-05-27T07:29:56.041 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55579de3e969] 2024-05-27T07:29:56.041 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 14: (MDLog::_replay_thread()+0x1f38) [0x55579e1d748c] 2024-05-27T07:29:56.041 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55579e1e09cb] 2024-05-27T07:29:56.041 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 16: (Thread::entry_wrapper()+0x43) [0x7ff57ba5272d] 2024-05-27T07:29:56.042 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff57ba52749] 2024-05-27T07:29:56.042 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff57a5f61ca] 2024-05-27T07:29:56.042 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 19: clone() 2024-05-27T07:29:56.042 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:56.042 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[124650]: 2024-05-27T07:29:56.042 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:56 smithi012 podman[124897]: 2024-05-27 07:29:56.029492284 +0000 UTC m=+0.277081612 container died 26784cf42e879354533de2f6859bad7a6dce4419ab587ed11a2b96ca7d07e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T07:29:56.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:56 smithi162 ceph-mon[35676]: pgmap v269: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 198 op/s 2024-05-27T07:29:56.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:56 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:56.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:56 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:56.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:56 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:29:56.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:56 smithi062 ceph-mon[35304]: pgmap v269: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 198 op/s 2024-05-27T07:29:56.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:56 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:56 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:56.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:56 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:29:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:56 smithi012 ceph-mon[31473]: pgmap v269: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.7 MiB/s wr, 198 op/s 2024-05-27T07:29:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:56 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:56 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:56.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:56 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:29:57.221 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:57 smithi012 podman[124897]: 2024-05-27 07:29:57.013953721 +0000 UTC m=+1.261543028 container remove 26784cf42e879354533de2f6859bad7a6dce4419ab587ed11a2b96ca7d07e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:29:57.221 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:57 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:58.421 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:58 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:29:58.421 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:29:58 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.844s CPU time. 2024-05-27T07:29:58.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:29:58 smithi012 ceph-mon[31473]: pgmap v270: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 KiB/s wr, 97 op/s 2024-05-27T07:29:58.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:29:58 smithi162 ceph-mon[35676]: pgmap v270: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 KiB/s wr, 97 op/s 2024-05-27T07:29:58.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:29:58 smithi062 ceph-mon[35304]: pgmap v270: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 KiB/s wr, 97 op/s 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.125+0000 7fe304e00700 1 -- 172.21.15.12:0/3950593319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30010b580 msgr2=0x7fe300119980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.125+0000 7fe304e00700 1 --2- 172.21.15.12:0/3950593319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30010b580 0x7fe300119980 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009fc0 tx=0x7fe2f800b330 comp rx=0 tx=0).stop 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.125+0000 7fe304e00700 1 -- 172.21.15.12:0/3950593319 shutdown_connections 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.125+0000 7fe304e00700 1 --2- 172.21.15.12:0/3950593319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe30010b580 0x7fe300119980 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.125+0000 7fe304e00700 1 --2- 172.21.15.12:0/3950593319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe300112c20 0x7fe300113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.125+0000 7fe304e00700 1 --2- 172.21.15.12:0/3950593319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe30010ac30 0x7fe30010b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.125+0000 7fe304e00700 1 -- 172.21.15.12:0/3950593319 >> 172.21.15.12:0/3950593319 conn(0x7fe30006f980 msgr2=0x7fe30006fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.126+0000 7fe304e00700 1 -- 172.21.15.12:0/3950593319 shutdown_connections 2024-05-27T07:29:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.126+0000 7fe304e00700 1 -- 172.21.15.12:0/3950593319 wait complete. 2024-05-27T07:29:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.127+0000 7fe304e00700 1 Processor -- start 2024-05-27T07:29:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.127+0000 7fe304e00700 1 -- start start 2024-05-27T07:29:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.128+0000 7fe304e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30010ac30 0x7fe3001a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.128+0000 7fe304e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe30010b580 0x7fe3001a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.128+0000 7fe304e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe300112c20 0x7fe3001a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.128+0000 7fe304e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe30011cc50 con 0x7fe300112c20 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.128+0000 7fe304e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe30011cad0 con 0x7fe30010b580 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.128+0000 7fe304e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe30011cdd0 con 0x7fe30010ac30 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe300112c20 0x7fe3001a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fde00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30010ac30 0x7fe3001a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fd400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe30010b580 0x7fe3001a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe300112c20 0x7fe3001a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37008/0 (socket says 172.21.15.12:37008) 2024-05-27T07:29:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 -- 172.21.15.12:0/709659281 learned_addr learned my addr 172.21.15.12:0/709659281 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 -- 172.21.15.12:0/709659281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30010ac30 msgr2=0x7fe3001a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30010ac30 0x7fe3001a57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 -- 172.21.15.12:0/709659281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe30010b580 msgr2=0x7fe3001a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe30010b580 0x7fe3001a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.129+0000 7fe2fe800700 1 -- 172.21.15.12:0/709659281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3001aa690 con 0x7fe300112c20 2024-05-27T07:29:59.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.130+0000 7fe2fe800700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe300112c20 0x7fe3001a9ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009790 tx=0x7fe2f800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.130+0000 7fe2f6a00700 1 -- 172.21.15.12:0/709659281 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8021070 con 0x7fe300112c20 2024-05-27T07:29:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.130+0000 7fe2f6a00700 1 -- 172.21.15.12:0/709659281 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f8007ac0 con 0x7fe300112c20 2024-05-27T07:29:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.130+0000 7fe2f6a00700 1 -- 172.21.15.12:0/709659281 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8005040 con 0x7fe300112c20 2024-05-27T07:29:59.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.131+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3001aa920 con 0x7fe300112c20 2024-05-27T07:29:59.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.131+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3001aae10 con 0x7fe300112c20 2024-05-27T07:29:59.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.135+0000 7fe2f6a00700 1 -- 172.21.15.12:0/709659281 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe2f8024070 con 0x7fe300112c20 2024-05-27T07:29:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.137+0000 7fe2f6a00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2e40b2a70 0x7fe2e40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.137+0000 7fe2f6a00700 1 -- 172.21.15.12:0/709659281 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9466+0+0 (secure 0 0 0) 0x7fe2f800f020 con 0x7fe300112c20 2024-05-27T07:29:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 Processor -- start 2024-05-27T07:29:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 -- start start 2024-05-27T07:29:59.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe2fde00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2e40b2a70 0x7fe2e40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628aab1e40 0x55628ac367f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac36d30 0x55628ac30870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac311c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55628ab5ee40 con 0x55628aab1e40 2024-05-27T07:29:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55628ab4e9e0 con 0x55628ac36d30 2024-05-27T07:29:59.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.138+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55628ab52a20 con 0x55628ac30db0 2024-05-27T07:29:59.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.139+0000 7fe2fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628aab1e40 0x55628ac367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.139+0000 7fe2fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628aab1e40 0x55628ac367f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37016/0 (socket says 172.21.15.12:37016) 2024-05-27T07:29:59.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.139+0000 7fe2fde00700 1 -- 172.21.15.12:0/2599256078 learned_addr learned my addr 172.21.15.12:0/2599256078 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:59.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.139+0000 7fe2fd400700 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac36d30 0x55628ac30870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.139+0000 7fe2fe800700 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.139+0000 7fe2fde00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2e40b2a70 0x7fe2e40b4f80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0013450 tx=0x7fe2f0013810 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:59.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.140+0000 7fe2fde00700 1 -- 172.21.15.12:0/2599256078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 msgr2=0x55628ac311c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.140+0000 7fe2fde00700 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac311c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.140+0000 7fe2fde00700 1 -- 172.21.15.12:0/2599256078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac36d30 msgr2=0x55628ac30870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.140+0000 7fe2fde00700 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac36d30 0x55628ac30870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.140+0000 7fe2fde00700 1 -- 172.21.15.12:0/2599256078 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55628ab17420 con 0x55628aab1e40 2024-05-27T07:29:59.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.140+0000 7fe2fde00700 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628aab1e40 0x55628ac367f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009630 tx=0x7fe2f001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ce80096d393cb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:59.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.140+0000 7fe2eb400700 1 -- 172.21.15.12:0/2599256078 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000fca0 con 0x55628aab1e40 2024-05-27T07:29:59.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.141+0000 7fe2eb400700 1 -- 172.21.15.12:0/2599256078 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f000fe40 con 0x55628aab1e40 2024-05-27T07:29:59.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.141+0000 7fe2eb400700 1 -- 172.21.15.12:0/2599256078 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0003d00 con 0x55628aab1e40 2024-05-27T07:29:59.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2599256078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628aab1e40 msgr2=0x55628ac367f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628aab1e40 0x55628ac367f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009630 tx=0x7fe2f001d000 comp rx=0 tx=0).stop 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2599256078 shutdown_connections 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628aab1e40 0x55628ac367f0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2599256078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac36d30 0x55628ac30870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2599256078 >> 172.21.15.12:0/2599256078 conn(0x55628ab11fc0 msgr2=0x55628ac2b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2599256078 shutdown_connections 2024-05-27T07:29:59.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.142+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2599256078 wait complete. 2024-05-27T07:29:59.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.143+0000 7fe3138b7b80 1 -- start start 2024-05-27T07:29:59.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.143+0000 7fe3138b7b80 1 Processor -- start 2024-05-27T07:29:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628aab1e40 0x55628ac3a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac3aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac3af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55628aa54fa0 con 0x55628ac36d30 2024-05-27T07:29:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55628aaef760 con 0x55628aab1e40 2024-05-27T07:29:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55628ac45fb0 con 0x55628ac30db0 2024-05-27T07:29:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe3138b7b80 10 client.0 _ll_register_callbacks cb 0x55628aaf8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac3af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac3af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37032/0 (socket says 172.21.15.12:37032) 2024-05-27T07:29:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe2fe800700 1 -- 172.21.15.12:0/2137500350 learned_addr learned my addr 172.21.15.12:0/2137500350 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.144+0000 7fe2fd400700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac3aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.145+0000 7fe2fde00700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628aab1e40 0x55628ac3a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.145+0000 7fe2fe800700 1 -- 172.21.15.12:0/2137500350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 msgr2=0x55628ac3aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.145+0000 7fe2fe800700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac3aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.145+0000 7fe2fe800700 1 -- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628aab1e40 msgr2=0x55628ac3a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.145+0000 7fe2fe800700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628aab1e40 0x55628ac3a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.145+0000 7fe2fe800700 1 -- 172.21.15.12:0/2137500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55628aa9e8d0 con 0x55628ac36d30 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.145+0000 7fe2fde00700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628aab1e40 0x55628ac3a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.146+0000 7fe2fe800700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac3af60 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009b90 tx=0x7fe2f80ab3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3bde318a04f3de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.146+0000 7fe2dfe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.146+0000 7fe2dfe00700 1 -- 172.21.15.12:0/2137500350 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80aae70 con 0x55628ac36d30 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.146+0000 7fe2dfe00700 1 -- 172.21.15.12:0/2137500350 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9466+0+0 (secure 0 0 0) 0x7fe2f8007c70 con 0x55628ac36d30 2024-05-27T07:29:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.146+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2f000fe40 con 0x55628ac36d30 2024-05-27T07:29:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.147+0000 7fe2dfe00700 1 -- 172.21.15.12:0/2137500350 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7fe2f80af220 con 0x55628ac36d30 2024-05-27T07:29:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.147+0000 7fe3138b7b80 10 client.16401 fetch_fsmap learned FSMap version 111 2024-05-27T07:29:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.147+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55628ab1f060 con 0x55628ac36d30 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe2dfe00700 1 -- 172.21.15.12:0/2137500350 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f000fe40 con 0x55628ac36d30 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe2dfe00700 1 -- 172.21.15.12:0/2137500350 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 111) ==== 4139+0+0 (secure 0 0 0) 0x7fe2f8011070 con 0x55628ac36d30 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe3138b7b80 10 client.16401 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe3138b7b80 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe3138b7b80 20 client.16401 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628ac52f70 0x55628ac553e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe3138b7b80 4 client.16401 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.148+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55628ab60d90 con 0x55628ac52f70 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.149+0000 7fe2fde00700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628ac52f70 0x55628ac553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.150+0000 7fe2fde00700 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628ac52f70 0x55628ac553e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed6b615a93df3954 server_cookie=8b5fd651d4624436 in_seq=0 out_seq=0 2024-05-27T07:29:59.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.150+0000 7fe2dfe00700 10 client.16401 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:59.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.152+0000 7fe2dfe00700 1 -- 172.21.15.12:0/2137500350 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55628ab60d90 con 0x55628ac52f70 2024-05-27T07:29:59.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.152+0000 7fe2dfe00700 10 client.16401 handle_command_reply: tid=0 2024-05-27T07:29:59.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.152+0000 7fe3138b7b80 1 client.16401 shutdown 2024-05-27T07:29:59.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 msgr2=0x55628ac3af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac3af60 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009b90 tx=0x7fe2f80ab3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 shutdown_connections 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628ac52f70 0x55628ac553e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628aab1e40 0x55628ac3a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac3af60 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe2dfe00700 0 client.16401 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2137500350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628ac30db0 0x55628ac3aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 >> 172.21.15.12:0/2137500350 conn(0x55628ab11fc0 msgr2=0x55628ac2b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:59.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 shutdown_connections 2024-05-27T07:29:59.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.153+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2137500350 wait complete. 2024-05-27T07:29:59.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.154+0000 7fe3138b7b80 20 client.16401 trim_cache size 0 max 16384 2024-05-27T07:29:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.175+0000 7fe3138b7b80 1 Processor -- start 2024-05-27T07:29:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.175+0000 7fe3138b7b80 1 -- start start 2024-05-27T07:29:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.176+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac30870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.176+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac34c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.176+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac35cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.176+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b6e10 con 0x55628ac36d30 2024-05-27T07:29:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.176+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b5210 con 0x55628ac30db0 2024-05-27T07:29:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.176+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b6740 con 0x55628aab1e40 2024-05-27T07:29:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac35cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fde00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac30870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac35cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37034/0 (socket says 172.21.15.12:37034) 2024-05-27T07:29:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 -- 172.21.15.12:0/2930260849 learned_addr learned my addr 172.21.15.12:0/2930260849 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fd400700 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac34c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 -- 172.21.15.12:0/2930260849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 msgr2=0x55628ac30870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac30870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 -- 172.21.15.12:0/2930260849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 msgr2=0x55628ac34c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac34c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.177+0000 7fe2fe800700 1 -- 172.21.15.12:0/2930260849 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55628ac371e0 con 0x55628ac36d30 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.178+0000 7fe2fe800700 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac35cd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80aae00 tx=0x7fe2f80a6e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3697e4a681ea06ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.178+0000 7fe2eaa00700 1 -- 172.21.15.12:0/2930260849 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80aae70 con 0x55628ac36d30 2024-05-27T07:29:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.178+0000 7fe2eaa00700 1 -- 172.21.15.12:0/2930260849 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f80c5480 con 0x55628ac36d30 2024-05-27T07:29:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.178+0000 7fe2eaa00700 1 -- 172.21.15.12:0/2930260849 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80aa2e0 con 0x55628ac36d30 2024-05-27T07:29:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2930260849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 msgr2=0x55628ac35cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac35cd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80aae00 tx=0x7fe2f80a6e60 comp rx=0 tx=0).stop 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2930260849 shutdown_connections 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac35cd0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac30870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/2930260849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac34c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2930260849 >> 172.21.15.12:0/2930260849 conn(0x55628ab11fc0 msgr2=0x55628ac3a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.179+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2930260849 shutdown_connections 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.180+0000 7fe3138b7b80 1 -- 172.21.15.12:0/2930260849 wait complete. 2024-05-27T07:29:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.180+0000 7fe3138b7b80 1 -- start start 2024-05-27T07:29:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.180+0000 7fe3138b7b80 1 Processor -- start 2024-05-27T07:29:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac4b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac4baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe3138b7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac5a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b6070 con 0x55628ac36d30 2024-05-27T07:29:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b5940 con 0x55628ac30db0 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe3138b7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55628ac31870 con 0x55628aab1e40 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe3138b7b80 10 client.0 _ll_register_callbacks cb 0x55628aaf8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac5a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe2fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac5a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37048/0 (socket says 172.21.15.12:37048) 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe2fe800700 1 -- 172.21.15.12:0/1888269565 learned_addr learned my addr 172.21.15.12:0/1888269565 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.181+0000 7fe2fde00700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac4b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fd400700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac4baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fe800700 1 -- 172.21.15.12:0/1888269565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 msgr2=0x55628ac4b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fe800700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac4b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fe800700 1 -- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 msgr2=0x55628ac4baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fe800700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac4baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fe800700 1 -- 172.21.15.12:0/1888269565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55628ac5aa20 con 0x55628ac36d30 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fd400700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac4baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2fe800700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac5a2c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80ab3f0 tx=0x7fe2f80099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9426c480a9ff6a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2dfe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2dfe00700 1 -- 172.21.15.12:0/1888269565 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f808d070 con 0x55628ac36d30 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe2dfe00700 1 -- 172.21.15.12:0/1888269565 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9466+0+0 (secure 0 0 0) 0x7fe2f80b1420 con 0x55628ac36d30 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.182+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2f80c5480 con 0x55628ac36d30 2024-05-27T07:29:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe2dfe00700 1 -- 172.21.15.12:0/1888269565 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7fe2f80bd050 con 0x55628ac36d30 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 10 client.16407 fetch_fsmap learned FSMap version 111 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55628ac5aa20 con 0x55628ac36d30 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe2dfe00700 1 -- 172.21.15.12:0/1888269565 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f80c5480 con 0x55628ac36d30 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe2dfe00700 1 -- 172.21.15.12:0/1888269565 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 111) ==== 4139+0+0 (secure 0 0 0) 0x7fe2f8011070 con 0x55628ac36d30 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 10 client.16407 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 20 client.16407 populate_metadata read hostname 'smithi012' 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628acbd7b0 0x55628acbfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 4 client.16407 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.183+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55628ab5a570 con 0x55628acbd7b0 2024-05-27T07:29:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.184+0000 7fe2fde00700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628acbd7b0 0x55628acbfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:59.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.185+0000 7fe2fde00700 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628acbd7b0 0x55628acbfbd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5303f75006b4c8e0 server_cookie=38ac807227987608 in_seq=0 out_seq=0 2024-05-27T07:29:59.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.185+0000 7fe2dfe00700 10 client.16407 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:59.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe2dfe00700 1 -- 172.21.15.12:0/1888269565 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55628ab5a570 con 0x55628acbd7b0 2024-05-27T07:29:59.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe2dfe00700 10 client.16407 handle_command_reply: tid=0 2024-05-27T07:29:59.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 client.16407 shutdown 2024-05-27T07:29:59.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 msgr2=0x55628ac5a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac5a2c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80ab3f0 tx=0x7fe2f80099a0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 shutdown_connections 2024-05-27T07:29:59.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55628ac36d30 0x55628ac5a2c0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55628aab1e40 0x55628ac4b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55628acbd7b0 0x55628acbfbd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 --2- 172.21.15.12:0/1888269565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55628ac30db0 0x55628ac4baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe2dfe00700 0 client.16407 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.188+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 >> 172.21.15.12:0/1888269565 conn(0x55628ab11fc0 msgr2=0x55628ac349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.189+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 shutdown_connections 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.189+0000 7fe3138b7b80 1 -- 172.21.15.12:0/1888269565 wait complete. 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.189+0000 7fe3138b7b80 20 client.16407 trim_cache size 0 max 16384 2024-05-27T07:29:59.181 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:29:59.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2e40b2a70 msgr2=0x7fe2e40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2e40b2a70 0x7fe2e40b4f80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0013450 tx=0x7fe2f0013810 comp rx=0 tx=0).stop 2024-05-27T07:29:59.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe300112c20 msgr2=0x7fe3001a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe300112c20 0x7fe3001a9ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009790 tx=0x7fe2f800cce0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 shutdown_connections 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2e40b2a70 0x7fe2e40b4f80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe300112c20 0x7fe3001a9ff0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30010ac30 0x7fe3001a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 --2- 172.21.15.12:0/709659281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe30010b580 0x7fe3001a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 >> 172.21.15.12:0/709659281 conn(0x7fe30006f980 msgr2=0x7fe30010c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 shutdown_connections 2024-05-27T07:29:59.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:29:59.190+0000 7fe304e00700 1 -- 172.21.15.12:0/709659281 wait complete. 2024-05-27T07:30:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:00 smithi162 ceph-mon[35676]: pgmap v271: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-05-27T07:30:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:00 smithi162 ceph-mon[35676]: overall HEALTH_OK 2024-05-27T07:30:00.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:00 smithi062 ceph-mon[35304]: pgmap v271: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-05-27T07:30:00.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:00 smithi062 ceph-mon[35304]: overall HEALTH_OK 2024-05-27T07:30:00.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:00 smithi012 ceph-mon[31473]: pgmap v271: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-05-27T07:30:00.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:00 smithi012 ceph-mon[31473]: overall HEALTH_OK 2024-05-27T07:30:01.103 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:30:01.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:30:01.599 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:02.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:02 smithi162 ceph-mon[35676]: pgmap v272: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-05-27T07:30:02.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:02 smithi062 ceph-mon[35304]: pgmap v272: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-05-27T07:30:02.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:02 smithi012 ceph-mon[31473]: pgmap v272: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-05-27T07:30:04.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:04 smithi162 ceph-mon[35676]: pgmap v273: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:30:04.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:04 smithi062 ceph-mon[35304]: pgmap v273: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:30:04.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:04 smithi012 ceph-mon[31473]: pgmap v273: 129 pgs: 129 active+clean; 114 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:30:05.092 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:30:04 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:30:05.092 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:30:05 smithi012 systemd[1]: Stopped Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:30:05.092 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:30:05 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.650s CPU time. 2024-05-27T07:30:05.092 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:30:05 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-05-27T07:30:05.092 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:30:05 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:30:05.092 INFO:journalctl@ceph.mds.d.smithi012.stdout:May 27 07:30:05 smithi012 systemd[1]: Failed to start Ceph mds.d for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:30:05.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 -- 172.21.15.12:0/1653120093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420107de0 msgr2=0x7f54201081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 --2- 172.21.15.12:0/1653120093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420107de0 0x7f54201081f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f540c0099c0 tx=0x7f540c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 -- 172.21.15.12:0/1653120093 shutdown_connections 2024-05-27T07:30:05.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 --2- 172.21.15.12:0/1653120093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 0x7f54201038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 --2- 172.21.15.12:0/1653120093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5420108730 0x7f5420110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 --2- 172.21.15.12:0/1653120093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420107de0 0x7f54201081f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 -- 172.21.15.12:0/1653120093 >> 172.21.15.12:0/1653120093 conn(0x7f54200fedb0 msgr2=0x7f54201011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:05.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 -- 172.21.15.12:0/1653120093 shutdown_connections 2024-05-27T07:30:05.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.476+0000 7f5426000700 1 -- 172.21.15.12:0/1653120093 wait complete. 2024-05-27T07:30:05.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.477+0000 7f5426000700 1 Processor -- start 2024-05-27T07:30:05.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.478+0000 7f5426000700 1 -- start start 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.478+0000 7f5426000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 0x7f542019d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.478+0000 7f5426000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5420107de0 0x7f542019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.478+0000 7f5426000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420108730 0x7f54201a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.478+0000 7f5426000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5420113240 con 0x7f5420108730 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.478+0000 7f5426000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f54201130c0 con 0x7f54201034b0 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.478+0000 7f5426000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54201133c0 con 0x7f5420107de0 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.479+0000 7f541f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 0x7f542019d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.479+0000 7f541ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5420107de0 0x7f542019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.479+0000 7f541f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 0x7f542019d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:49816/0 (socket says 172.21.15.12:49816) 2024-05-27T07:30:05.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.479+0000 7f541f400700 1 -- 172.21.15.12:0/2548396836 learned_addr learned my addr 172.21.15.12:0/2548396836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:05.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.480+0000 7f541fe00700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420108730 0x7f54201a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.480+0000 7f541f400700 1 -- 172.21.15.12:0/2548396836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5420107de0 msgr2=0x7f542019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.480+0000 7f541f400700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5420107de0 0x7f542019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.480+0000 7f541f400700 1 -- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420108730 msgr2=0x7f54201a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.480+0000 7f541f400700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420108730 0x7f54201a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.480+0000 7f541f400700 1 -- 172.21.15.12:0/2548396836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54201a1e80 con 0x7f54201034b0 2024-05-27T07:30:05.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.480+0000 7f541fe00700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420108730 0x7f54201a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:05.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.481+0000 7f541f400700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 0x7f542019d050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f541000d950 tx=0x7f541000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:05.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.481+0000 7f5417e00700 1 -- 172.21.15.12:0/2548396836 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410016070 con 0x7f54201034b0 2024-05-27T07:30:05.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.481+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54201a2170 con 0x7f54201034b0 2024-05-27T07:30:05.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.482+0000 7f5417e00700 1 -- 172.21.15.12:0/2548396836 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5410009dd0 con 0x7f54201034b0 2024-05-27T07:30:05.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.482+0000 7f5417e00700 1 -- 172.21.15.12:0/2548396836 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410020060 con 0x7f54201034b0 2024-05-27T07:30:05.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.482+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542006d1d0 con 0x7f54201034b0 2024-05-27T07:30:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.488+0000 7f5434bf3b80 1 Processor -- start 2024-05-27T07:30:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.488+0000 7f5434bf3b80 1 -- start start 2024-05-27T07:30:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.488+0000 7f5417e00700 1 -- 172.21.15.12:0/2548396836 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5410019070 con 0x7f54201034b0 2024-05-27T07:30:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926b88ec0 0x55c926d0d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926d0ddb0 0x55c926d078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d08240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c926c35ec0 con 0x55c926d07e30 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c926c25a60 con 0x55c926b88ec0 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c926c29aa0 con 0x55c926d0ddb0 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f541f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926d0ddb0 0x55c926d078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.489+0000 7f541fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926b88ec0 0x55c926d0d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.490+0000 7f541fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926b88ec0 0x55c926d0d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:49826/0 (socket says 172.21.15.12:49826) 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.490+0000 7f541fe00700 1 -- 172.21.15.12:0/3250436871 learned_addr learned my addr 172.21.15.12:0/3250436871 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.490+0000 7f541ea00700 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d08240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.490+0000 7f541fe00700 1 -- 172.21.15.12:0/3250436871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926d0ddb0 msgr2=0x55c926d078f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.490+0000 7f541fe00700 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926d0ddb0 0x55c926d078f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.490+0000 7f541fe00700 1 -- 172.21.15.12:0/3250436871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 msgr2=0x55c926d08240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.491+0000 7f541fe00700 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d08240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.491+0000 7f541fe00700 1 -- 172.21.15.12:0/3250436871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c926bee4a0 con 0x55c926b88ec0 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.491+0000 7f541ea00700 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d08240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.491+0000 7f5417e00700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f54040b2bf0 0x7f54040b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.491+0000 7f5417e00700 1 -- 172.21.15.12:0/2548396836 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9466+0+0 (secure 0 0 0) 0x7f5410010cd0 con 0x7f54201034b0 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.491+0000 7f541fe00700 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926b88ec0 0x55c926d0d870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5418009dd0 tx=0x7f54180077e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d21f22bc2bc9a65d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.491+0000 7f541f400700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f54040b2bf0 0x7f54040b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.492+0000 7f5414c00700 1 -- 172.21.15.12:0/3250436871 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418004bb0 con 0x55c926b88ec0 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.492+0000 7f5414c00700 1 -- 172.21.15.12:0/3250436871 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5418004d50 con 0x55c926b88ec0 2024-05-27T07:30:05.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.492+0000 7f5414c00700 1 -- 172.21.15.12:0/3250436871 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418014670 con 0x55c926b88ec0 2024-05-27T07:30:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3250436871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926b88ec0 msgr2=0x55c926d0d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926b88ec0 0x55c926d0d870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5418009dd0 tx=0x7f54180077e0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f541f400700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f54040b2bf0 0x7f54040b5100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54100a6930 tx=0x7f54100e8110 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3250436871 shutdown_connections 2024-05-27T07:30:05.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926b88ec0 0x55c926d0d870 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d08240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3250436871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926d0ddb0 0x55c926d078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3250436871 >> 172.21.15.12:0/3250436871 conn(0x55c926be9040 msgr2=0x55c926d02950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3250436871 shutdown_connections 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.493+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3250436871 wait complete. 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.494+0000 7f5434bf3b80 1 -- start start 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.494+0000 7f5434bf3b80 1 Processor -- start 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d115e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d11b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d12060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c926b2c020 con 0x55c926d0ddb0 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c926bc67e0 con 0x55c926d07e30 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c926d14880 con 0x55c926b88ec0 2024-05-27T07:30:05.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f5434bf3b80 10 client.0 _ll_register_callbacks cb 0x55c926bcfcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:05.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f541fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d12060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f541fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d12060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39046/0 (socket says 172.21.15.12:39046) 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f541fe00700 1 -- 172.21.15.12:0/2499991733 learned_addr learned my addr 172.21.15.12:0/2499991733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.495+0000 7f541fe00700 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.496+0000 7f541ea00700 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d11b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.496+0000 7f541fe00700 1 -- 172.21.15.12:0/2499991733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 msgr2=0x55c926d115e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.496+0000 7f541fe00700 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d115e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.496+0000 7f541fe00700 1 -- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 msgr2=0x55c926d11b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.496+0000 7f541fe00700 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d11b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.496+0000 7f541fe00700 1 -- 172.21.15.12:0/2499991733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c926b75950 con 0x55c926d0ddb0 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f541fe00700 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d12060 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f54180057c0 tx=0x7f5418005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5776a78c800ab46d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f5409600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f5409600700 1 -- 172.21.15.12:0/2499991733 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418014aa0 con 0x55c926d0ddb0 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f5409600700 1 -- 172.21.15.12:0/2499991733 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9466+0+0 (secure 0 0 0) 0x7f5418014c40 con 0x55c926d0ddb0 2024-05-27T07:30:05.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5418004d50 con 0x55c926d0ddb0 2024-05-27T07:30:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f5409600700 1 -- 172.21.15.12:0/2499991733 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f541802f050 con 0x55c926d0ddb0 2024-05-27T07:30:05.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f5434bf3b80 10 client.16419 fetch_fsmap learned FSMap version 111 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.497+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55c926bf60e0 con 0x55c926d0ddb0 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5409600700 1 -- 172.21.15.12:0/2499991733 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5418004d50 con 0x55c926d0ddb0 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5409600700 1 -- 172.21.15.12:0/2499991733 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 111) ==== 4139+0+0 (secure 0 0 0) 0x7f541801f550 con 0x55c926d0ddb0 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5434bf3b80 10 client.16419 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5434bf3b80 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5434bf3b80 20 client.16419 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d21bc0 0x55c926d24050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5434bf3b80 4 client.16419 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.498+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c926c37e10 con 0x55c926d21bc0 2024-05-27T07:30:05.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.499+0000 7f541ea00700 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d21bc0 0x55c926d24050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.500+0000 7f541ea00700 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d21bc0 0x55c926d24050 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ab256a219f6e23 server_cookie=7de9af627a4de35f in_seq=0 out_seq=0 2024-05-27T07:30:05.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.500+0000 7f5409600700 10 client.16419 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:05.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.504+0000 7f5409600700 1 -- 172.21.15.12:0/2499991733 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c926c37e10 con 0x55c926d21bc0 2024-05-27T07:30:05.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.504+0000 7f5409600700 10 client.16419 handle_command_reply: tid=0 2024-05-27T07:30:05.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.504+0000 7f5434bf3b80 1 client.16419 shutdown 2024-05-27T07:30:05.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 msgr2=0x55c926d12060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d12060 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f54180057c0 tx=0x7f5418005820 comp rx=0 tx=0).stop 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 shutdown_connections 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d12060 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d115e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d21bc0 0x55c926d24050 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5409600700 0 client.16419 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2499991733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d11b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.505+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 >> 172.21.15.12:0/2499991733 conn(0x55c926be9040 msgr2=0x55c926d02600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:05.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.506+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 shutdown_connections 2024-05-27T07:30:05.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.506+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2499991733 wait complete. 2024-05-27T07:30:05.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.506+0000 7f5434bf3b80 20 client.16419 trim_cache size 0 max 16384 2024-05-27T07:30:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.538+0000 7f5434bf3b80 1 Processor -- start 2024-05-27T07:30:05.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.538+0000 7f5434bf3b80 1 -- start start 2024-05-27T07:30:05.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.538+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.538+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d1d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.538+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d11c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c926d0ce20 con 0x55c926d0ddb0 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f54201bb330 con 0x55c926d07e30 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5420074670 con 0x55c926b88ec0 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f541fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d11c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f541fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d11c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39050/0 (socket says 172.21.15.12:39050) 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f541fe00700 1 -- 172.21.15.12:0/3455617369 learned_addr learned my addr 172.21.15.12:0/3455617369 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f541ea00700 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d1d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.539+0000 7f541fe00700 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f541fe00700 1 -- 172.21.15.12:0/3455617369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 msgr2=0x55c926d078f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f541fe00700 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d078f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f541fe00700 1 -- 172.21.15.12:0/3455617369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 msgr2=0x55c926d1d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f541fe00700 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d1d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f541fe00700 1 -- 172.21.15.12:0/3455617369 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c926d08670 con 0x55c926d0ddb0 2024-05-27T07:30:05.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f541fe00700 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d11c10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f54180143e0 tx=0x7f5418009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51e028bf5fdcb90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f540be00700 1 -- 172.21.15.12:0/3455617369 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418025040 con 0x55c926d0ddb0 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.540+0000 7f540be00700 1 -- 172.21.15.12:0/3455617369 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f541801cbd0 con 0x55c926d0ddb0 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.541+0000 7f540be00700 1 -- 172.21.15.12:0/3455617369 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418014aa0 con 0x55c926d0ddb0 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3455617369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 msgr2=0x55c926d11c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d11c10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f54180143e0 tx=0x7f5418009040 comp rx=0 tx=0).stop 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3455617369 shutdown_connections 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d0ddb0 0x55c926d11c10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/3455617369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d07e30 0x55c926d1d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3455617369 >> 172.21.15.12:0/3455617369 conn(0x55c926be9040 msgr2=0x55c926d10900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3455617369 shutdown_connections 2024-05-27T07:30:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.542+0000 7f5434bf3b80 1 -- 172.21.15.12:0/3455617369 wait complete. 2024-05-27T07:30:05.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.543+0000 7f5434bf3b80 1 -- start start 2024-05-27T07:30:05.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.543+0000 7f5434bf3b80 1 Processor -- start 2024-05-27T07:30:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.544+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d19cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.544+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d1a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.544+0000 7f5434bf3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d0ddb0 0x55c926d16ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.544+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5420074da0 con 0x55c926d07e30 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.544+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f54201bac60 con 0x55c926d0ddb0 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.544+0000 7f5434bf3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54200754d0 con 0x55c926b88ec0 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.544+0000 7f5434bf3b80 10 client.0 _ll_register_callbacks cb 0x55c926bcfcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.545+0000 7f541fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d0ddb0 0x55c926d16ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.545+0000 7f541fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d0ddb0 0x55c926d16ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:49858/0 (socket says 172.21.15.12:49858) 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.545+0000 7f541fe00700 1 -- 172.21.15.12:0/2473725484 learned_addr learned my addr 172.21.15.12:0/2473725484 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.545+0000 7f541fe00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d19cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.545+0000 7f541ea00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d1a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f541fe00700 1 -- 172.21.15.12:0/2473725484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 msgr2=0x55c926d19cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f541fe00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d19cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f541fe00700 1 -- 172.21.15.12:0/2473725484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 msgr2=0x55c926d1a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f541fe00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d1a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f541fe00700 1 -- 172.21.15.12:0/2473725484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c926d11010 con 0x55c926d0ddb0 2024-05-27T07:30:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f541ea00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d1a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f541fe00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d0ddb0 0x55c926d16ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5418009310 tx=0x7f5418009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f747a206a2a1f929 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.546+0000 7f5409600700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:30:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.547+0000 7f5409600700 1 -- 172.21.15.12:0/2473725484 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418025070 con 0x55c926d0ddb0 2024-05-27T07:30:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.547+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541801cbd0 con 0x55c926d0ddb0 2024-05-27T07:30:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.547+0000 7f5409600700 1 -- 172.21.15.12:0/2473725484 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9466+0+0 (secure 0 0 0) 0x7f5418029040 con 0x55c926d0ddb0 2024-05-27T07:30:05.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.548+0000 7f5409600700 1 -- 172.21.15.12:0/2473725484 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f541801f670 con 0x55c926d0ddb0 2024-05-27T07:30:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.548+0000 7f5434bf3b80 10 client.35966 fetch_fsmap learned FSMap version 111 2024-05-27T07:30:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.548+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55c926d95740 con 0x55c926d0ddb0 2024-05-27T07:30:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.548+0000 7f5409600700 1 -- 172.21.15.12:0/2473725484 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f541801cbd0 con 0x55c926d0ddb0 2024-05-27T07:30:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f5409600700 1 -- 172.21.15.12:0/2473725484 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 111) ==== 4139+0+0 (secure 0 0 0) 0x7f541802c070 con 0x55c926d0ddb0 2024-05-27T07:30:05.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f5434bf3b80 10 client.35966 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f5434bf3b80 10 client.35966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f5434bf3b80 20 client.35966 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d95cc0 0x55c926d980e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f5434bf3b80 4 client.35966 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c926c315f0 con 0x55c926d95cc0 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.549+0000 7f541ea00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d95cc0 0x55c926d980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.550+0000 7f541ea00700 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d95cc0 0x55c926d980e0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67acef5b767bbc3 server_cookie=ad5b6f0fe64609a3 in_seq=0 out_seq=0 2024-05-27T07:30:05.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.550+0000 7f5409600700 10 client.35966 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5409600700 1 -- 172.21.15.12:0/2473725484 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c926c315f0 con 0x55c926d95cc0 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5409600700 10 client.35966 handle_command_reply: tid=0 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 client.35966 shutdown 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d0ddb0 msgr2=0x55c926d16ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d0ddb0 0x55c926d16ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5418009310 tx=0x7f5418009490 comp rx=0 tx=0).stop 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 shutdown_connections 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c926d95cc0 0x55c926d980e0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c926d0ddb0 0x55c926d16ae0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c926b88ec0 0x55c926d19cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 --2- 172.21.15.12:0/2473725484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c926d07e30 0x55c926d1a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5409600700 0 client.35966 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.554+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 >> 172.21.15.12:0/2473725484 conn(0x55c926be9040 msgr2=0x55c926d10900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.555+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 shutdown_connections 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.555+0000 7f5434bf3b80 1 -- 172.21.15.12:0/2473725484 wait complete. 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.555+0000 7f5434bf3b80 20 client.35966 trim_cache size 0 max 16384 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:30:05.547 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:30:05.548 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "4a152713-eb36-4214-8704-937e0aaf3cf3", 2024-05-27T07:30:05.548 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:30:05.548 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:30:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f54040b2bf0 msgr2=0x7f54040b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f54040b2bf0 0x7f54040b5100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54100a6930 tx=0x7f54100e8110 comp rx=0 tx=0).stop 2024-05-27T07:30:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 msgr2=0x7f542019d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:05.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 0x7f542019d050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f541000d950 tx=0x7f541000dc60 comp rx=0 tx=0).stop 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 shutdown_connections 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f54040b2bf0 0x7f54040b5100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54201034b0 0x7f542019d050 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5420108730 0x7f54201a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 --2- 172.21.15.12:0/2548396836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5420107de0 0x7f542019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 >> 172.21.15.12:0/2548396836 conn(0x7f54200fedb0 msgr2=0x7f54201066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 shutdown_connections 2024-05-27T07:30:05.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:05.556+0000 7f5426000700 1 -- 172.21.15.12:0/2548396836 wait complete. 2024-05-27T07:30:06.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:06 smithi012 ceph-mon[31473]: pgmap v274: 129 pgs: 129 active+clean; 116 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-05-27T07:30:06.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:06 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:30:06.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:06 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:06.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:06 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:30:06.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:06 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:30:06.420 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:06 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:06.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:30:06.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:06 smithi162 ceph-mon[35676]: pgmap v274: 129 pgs: 129 active+clean; 116 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-05-27T07:30:06.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:06 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:30:06.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:06 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:06.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:06 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:30:06.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:06 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:30:06.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:06 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:06.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:06 smithi062 ceph-mon[35304]: pgmap v274: 129 pgs: 129 active+clean; 116 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-05-27T07:30:06.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:06 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:30:06.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:06 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:06.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:06 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:30:06.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:06 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:30:06.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:06 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:07.676 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:07 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:30:07.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:07 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:07.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:07 smithi012 ceph-mon[31473]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:30:07.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:07 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:07.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:07 smithi012 ceph-mon[31473]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:07.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:07 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:30:07.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:07 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:30:07.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:07 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:07.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:07 smithi162 ceph-mon[35676]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:30:07.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:07 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:07.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:07 smithi162 ceph-mon[35676]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:07.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:07 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:30:07.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:07 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:30:07.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:07 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:07.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:07 smithi062 ceph-mon[35304]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:30:07.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:07 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:07.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:07 smithi062 ceph-mon[35304]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:07.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:07 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:30:08.499 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:08 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:30:08.499 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:08 smithi012 systemd[1]: Stopped Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:30:08.499 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:08 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.844s CPU time. 2024-05-27T07:30:08.499 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:08 smithi012 ceph-mon[31473]: pgmap v276: 129 pgs: 129 active+clean; 116 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-05-27T07:30:08.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:08 smithi162 ceph-mon[35676]: pgmap v276: 129 pgs: 129 active+clean; 116 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-05-27T07:30:08.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:08 smithi062 ceph-mon[35304]: pgmap v276: 129 pgs: 129 active+clean; 116 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-05-27T07:30:08.831 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:08 smithi012 systemd[1]: Starting Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:30:09.707 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:09 smithi012 podman[125354]: 2024-05-27 07:30:09.351606118 +0000 UTC m=+0.094229144 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:09.707 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:09 smithi012 podman[125354]: 2024-05-27 07:30:09.477075092 +0000 UTC m=+0.219698114 container create 48b2825848f4b7e706cdd3621f01d40a1f7b776d1add0ae274cb4111dbc60da9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:30:09.708 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:09 smithi012 ceph-mon[31473]: pgmap v277: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-05-27T07:30:09.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:09 smithi162 ceph-mon[35676]: pgmap v277: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-05-27T07:30:09.981 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:09 smithi012 podman[125354]: 2024-05-27 07:30:09.884235107 +0000 UTC m=+0.626858231 container init 48b2825848f4b7e706cdd3621f01d40a1f7b776d1add0ae274cb4111dbc60da9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-27T07:30:09.981 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:09 smithi012 podman[125354]: 2024-05-27 07:30:09.896349178 +0000 UTC m=+0.638972197 container start 48b2825848f4b7e706cdd3621f01d40a1f7b776d1add0ae274cb4111dbc60da9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:30:09.982 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:09 smithi012 bash[125354]: 48b2825848f4b7e706cdd3621f01d40a1f7b776d1add0ae274cb4111dbc60da9 2024-05-27T07:30:09.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:09 smithi062 ceph-mon[35304]: pgmap v277: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-05-27T07:30:10.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.010+0000 7f8bb5800700 1 -- 172.21.15.12:0/1402149856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb0111bc0 msgr2=0x7f8bb0111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.010+0000 7f8bb5800700 1 --2- 172.21.15.12:0/1402149856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb0111bc0 0x7f8bb0111fd0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009990 tx=0x7f8ba8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.010+0000 7f8bb5800700 1 -- 172.21.15.12:0/1402149856 shutdown_connections 2024-05-27T07:30:10.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.010+0000 7f8bb5800700 1 --2- 172.21.15.12:0/1402149856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bb010b270 0x7f8bb01198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.010+0000 7f8bb5800700 1 --2- 172.21.15.12:0/1402149856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb0111bc0 0x7f8bb0111fd0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.010+0000 7f8bb5800700 1 --2- 172.21.15.12:0/1402149856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bb010a920 0x7f8bb010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.010+0000 7f8bb5800700 1 -- 172.21.15.12:0/1402149856 >> 172.21.15.12:0/1402149856 conn(0x7f8bb006f4c0 msgr2=0x7f8bb006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:10.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.011+0000 7f8bb5800700 1 -- 172.21.15.12:0/1402149856 shutdown_connections 2024-05-27T07:30:10.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.011+0000 7f8bb5800700 1 -- 172.21.15.12:0/1402149856 wait complete. 2024-05-27T07:30:10.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.013+0000 7f8bb5800700 1 Processor -- start 2024-05-27T07:30:10.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.013+0000 7f8bb5800700 1 -- start start 2024-05-27T07:30:10.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8bb5800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb010a920 0x7f8bb01a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8bb5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bb010b270 0x7f8bb01a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8bb5800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bb0111bc0 0x7f8bb01a9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8bb5800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb011cb40 con 0x7f8bb010a920 2024-05-27T07:30:10.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8bb5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb011c9c0 con 0x7f8bb010b270 2024-05-27T07:30:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8bb5800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb011ccc0 con 0x7f8bb0111bc0 2024-05-27T07:30:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb010a920 0x7f8bb01a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb010a920 0x7f8bb01a57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39066/0 (socket says 172.21.15.12:39066) 2024-05-27T07:30:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8baea00700 1 -- 172.21.15.12:0/2951758389 learned_addr learned my addr 172.21.15.12:0/2951758389 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.014+0000 7f8baf400700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bb0111bc0 0x7f8bb01a9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.015+0000 7f8bae000700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bb010b270 0x7f8bb01a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.015+0000 7f8baea00700 1 -- 172.21.15.12:0/2951758389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bb0111bc0 msgr2=0x7f8bb01a9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.015+0000 7f8baea00700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bb0111bc0 0x7f8bb01a9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.015+0000 7f8baea00700 1 -- 172.21.15.12:0/2951758389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bb010b270 msgr2=0x7f8bb01a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.015+0000 7f8baea00700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bb010b270 0x7f8bb01a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.015+0000 7f8baea00700 1 -- 172.21.15.12:0/2951758389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb01aa5d0 con 0x7f8bb010a920 2024-05-27T07:30:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.015+0000 7f8baea00700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb010a920 0x7f8bb01a57b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800b5c0 tx=0x7f8ba80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.016+0000 7f8ba7400700 1 -- 172.21.15.12:0/2951758389 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8027070 con 0x7f8bb010a920 2024-05-27T07:30:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.016+0000 7f8ba7400700 1 -- 172.21.15.12:0/2951758389 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ba800fb00 con 0x7f8bb010a920 2024-05-27T07:30:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.016+0000 7f8ba7400700 1 -- 172.21.15.12:0/2951758389 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8022b10 con 0x7f8bb010a920 2024-05-27T07:30:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.016+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb01aa860 con 0x7f8bb010a920 2024-05-27T07:30:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.016+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb01aacf0 con 0x7f8bb010a920 2024-05-27T07:30:10.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.022+0000 7f8ba7400700 1 -- 172.21.15.12:0/2951758389 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8ba800f670 con 0x7f8bb010a920 2024-05-27T07:30:10.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.025+0000 7f8ba7400700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b940b2a70 0x7f8b940b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.025+0000 7f8ba7400700 1 -- 172.21.15.12:0/2951758389 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9642+0+0 (secure 0 0 0) 0x7f8ba80e3c70 con 0x7f8bb010a920 2024-05-27T07:30:10.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.026+0000 7f8bc435db80 1 Processor -- start 2024-05-27T07:30:10.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.026+0000 7f8bc435db80 1 -- start start 2024-05-27T07:30:10.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.026+0000 7f8bae000700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b940b2a70 0x7f8b940b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.026+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x55566975d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.026+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669757080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.026+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x5556697579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8bc435db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55566967ce40 con 0x5556695cfe40 2024-05-27T07:30:10.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8bc435db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55566966c9e0 con 0x5556697575c0 2024-05-27T07:30:10.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8bc435db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555669670a20 con 0x55566975d540 2024-05-27T07:30:10.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x55566975d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x55566975d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39072/0 (socket says 172.21.15.12:39072) 2024-05-27T07:30:10.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8baea00700 1 -- 172.21.15.12:0/1332758871 learned_addr learned my addr 172.21.15.12:0/1332758871 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:10.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8bae000700 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669757080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8baf400700 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x5556697579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.027+0000 7f8bae000700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b940b2a70 0x7f8b940b4f80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8bb006f980 tx=0x7f8b9c00b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:10.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8baea00700 1 -- 172.21.15.12:0/1332758871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 msgr2=0x555669757080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8baea00700 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669757080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8baea00700 1 -- 172.21.15.12:0/1332758871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 msgr2=0x5556697579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8baea00700 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x5556697579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8baea00700 1 -- 172.21.15.12:0/1332758871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555669635420 con 0x5556695cfe40 2024-05-27T07:30:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8baf400700 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x5556697579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8baea00700 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x55566975d000 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80aab10 tx=0x7f8ba80a63e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de5c35a57627a0a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8b9be00700 1 -- 172.21.15.12:0/1332758871 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba80b3290 con 0x5556695cfe40 2024-05-27T07:30:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8b9be00700 1 -- 172.21.15.12:0/1332758871 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ba80aa690 con 0x5556695cfe40 2024-05-27T07:30:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.028+0000 7f8b9be00700 1 -- 172.21.15.12:0/1332758871 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba80ab040 con 0x5556695cfe40 2024-05-27T07:30:10.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.030+0000 7f8bc435db80 1 -- 172.21.15.12:0/1332758871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 msgr2=0x55566975d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.030+0000 7f8bc435db80 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x55566975d000 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80aab10 tx=0x7f8ba80a63e0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.030+0000 7f8bc435db80 1 -- 172.21.15.12:0/1332758871 shutdown_connections 2024-05-27T07:30:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.030+0000 7f8bc435db80 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x5556697579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.030+0000 7f8bc435db80 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x55566975d000 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.030+0000 7f8bc435db80 1 --2- 172.21.15.12:0/1332758871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669757080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.030+0000 7f8bc435db80 1 -- 172.21.15.12:0/1332758871 >> 172.21.15.12:0/1332758871 conn(0x55566962ffc0 msgr2=0x5556697498d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.031+0000 7f8bc435db80 1 -- 172.21.15.12:0/1332758871 shutdown_connections 2024-05-27T07:30:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.031+0000 7f8bc435db80 1 -- 172.21.15.12:0/1332758871 wait complete. 2024-05-27T07:30:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.031+0000 7f8bc435db80 1 -- start start 2024-05-27T07:30:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.032+0000 7f8bc435db80 1 Processor -- start 2024-05-27T07:30:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.033+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669760de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.033+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x555669761320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.033+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669761860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.033+0000 7f8bc435db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555669572fa0 con 0x5556695cfe40 2024-05-27T07:30:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.033+0000 7f8bc435db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55566960d760 con 0x5556697575c0 2024-05-27T07:30:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.033+0000 7f8bc435db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555669763dd0 con 0x55566975d540 2024-05-27T07:30:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.033+0000 7f8bc435db80 10 client.0 _ll_register_callbacks cb 0x555669616c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669760de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baf400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669761860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669760de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39088/0 (socket says 172.21.15.12:39088) 2024-05-27T07:30:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 -- 172.21.15.12:0/3953531292 learned_addr learned my addr 172.21.15.12:0/3953531292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8bae000700 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x555669761320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 -- 172.21.15.12:0/3953531292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 msgr2=0x555669761860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669761860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 -- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 msgr2=0x555669761320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x555669761320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8baea00700 1 -- 172.21.15.12:0/3953531292 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556695bc8d0 con 0x5556695cfe40 2024-05-27T07:30:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.034+0000 7f8bae000700 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x555669761320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.035+0000 7f8baea00700 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669760de0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80e1560 tx=0x7f8ba8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640e426d381ea962 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.035+0000 7f8b98c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:30:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.036+0000 7f8b98c00700 1 -- 172.21.15.12:0/3953531292 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba806f060 con 0x5556695cfe40 2024-05-27T07:30:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.036+0000 7f8b98c00700 1 -- 172.21.15.12:0/3953531292 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9642+0+0 (secure 0 0 0) 0x7f8ba80c5170 con 0x5556695cfe40 2024-05-27T07:30:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.036+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ba80aa690 con 0x5556695cfe40 2024-05-27T07:30:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.036+0000 7f8b98c00700 1 -- 172.21.15.12:0/3953531292 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f8ba80b3290 con 0x5556695cfe40 2024-05-27T07:30:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.036+0000 7f8bc435db80 10 client.16437 fetch_fsmap learned FSMap version 112 2024-05-27T07:30:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.036+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55566963d060 con 0x5556695cfe40 2024-05-27T07:30:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.038+0000 7f8b98c00700 1 -- 172.21.15.12:0/3953531292 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ba80aa690 con 0x5556695cfe40 2024-05-27T07:30:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.039+0000 7f8b98c00700 1 -- 172.21.15.12:0/3953531292 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 112) ==== 3546+0+0 (secure 0 0 0) 0x7f8ba8070250 con 0x5556695cfe40 2024-05-27T07:30:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.039+0000 7f8bc435db80 10 client.16437 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:30:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.039+0000 7f8bc435db80 10 client.16437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.039+0000 7f8bc435db80 20 client.16437 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:10.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.039+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555669770b10 0x555669772f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.039+0000 7f8bc435db80 4 client.16437 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.039+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55566967ed90 con 0x555669770b10 2024-05-27T07:30:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.040+0000 7f8bae000700 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555669770b10 0x555669772f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.041+0000 7f8b98c00700 10 client.16437 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.041+0000 7f8bae000700 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555669770b10 0x555669772f30 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7973605f25ac7eee server_cookie=8b6bee110807de78 in_seq=0 out_seq=0 2024-05-27T07:30:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.043+0000 7f8b98c00700 1 -- 172.21.15.12:0/3953531292 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55566967ed90 con 0x555669770b10 2024-05-27T07:30:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.043+0000 7f8b98c00700 10 client.16437 handle_command_reply: tid=0 2024-05-27T07:30:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.043+0000 7f8bc435db80 1 client.16437 shutdown 2024-05-27T07:30:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 msgr2=0x555669760de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669760de0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80e1560 tx=0x7f8ba8009310 comp rx=0 tx=0).stop 2024-05-27T07:30:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 shutdown_connections 2024-05-27T07:30:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55566975d540 0x555669761860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669760de0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x555669770b10 0x555669772f30 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3953531292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556697575c0 0x555669761320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 >> 172.21.15.12:0/3953531292 conn(0x55566962ffc0 msgr2=0x555669749580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.044+0000 7f8b98c00700 0 client.16437 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.045+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 shutdown_connections 2024-05-27T07:30:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.045+0000 7f8bc435db80 1 -- 172.21.15.12:0/3953531292 wait complete. 2024-05-27T07:30:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.045+0000 7f8bc435db80 20 client.16437 trim_cache size 0 max 16384 2024-05-27T07:30:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.076+0000 7f8bc435db80 1 Processor -- start 2024-05-27T07:30:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.076+0000 7f8bc435db80 1 -- start start 2024-05-27T07:30:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556695cfe40 0x555669757080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556697575c0 0x55566975fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669763280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bc435db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01b6910 con 0x5556697575c0 2024-05-27T07:30:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bc435db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01b5440 con 0x55566975d540 2024-05-27T07:30:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bc435db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01b5b10 con 0x5556695cfe40 2024-05-27T07:30:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556697575c0 0x55566975fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bae000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556697575c0 0x55566975fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39118/0 (socket says 172.21.15.12:39118) 2024-05-27T07:30:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.077+0000 7f8bae000700 1 -- 172.21.15.12:0/3077322658 learned_addr learned my addr 172.21.15.12:0/3077322658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8baea00700 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556695cfe40 0x555669757080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8baf400700 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669763280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8bae000700 1 -- 172.21.15.12:0/3077322658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556695cfe40 msgr2=0x555669757080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8bae000700 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556695cfe40 0x555669757080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8bae000700 1 -- 172.21.15.12:0/3077322658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 msgr2=0x555669763280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8bae000700 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669763280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8bae000700 1 -- 172.21.15.12:0/3077322658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555669761bd0 con 0x5556697575c0 2024-05-27T07:30:10.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8baf400700 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669763280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:10.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.078+0000 7f8bae000700 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556697575c0 0x55566975fdc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c010db0 tx=0x7f8b9c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f9e9ddd2ed4d10 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.079+0000 7f8b9b400700 1 -- 172.21.15.12:0/3077322658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c02e070 con 0x5556697575c0 2024-05-27T07:30:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.079+0000 7f8b9b400700 1 -- 172.21.15.12:0/3077322658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b9c021110 con 0x5556697575c0 2024-05-27T07:30:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.079+0000 7f8b9b400700 1 -- 172.21.15.12:0/3077322658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c028c20 con 0x5556697575c0 2024-05-27T07:30:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 -- 172.21.15.12:0/3077322658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556697575c0 msgr2=0x55566975fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556697575c0 0x55566975fdc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c010db0 tx=0x7f8b9c013000 comp rx=0 tx=0).stop 2024-05-27T07:30:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 -- 172.21.15.12:0/3077322658 shutdown_connections 2024-05-27T07:30:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669763280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556695cfe40 0x555669757080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 --2- 172.21.15.12:0/3077322658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556697575c0 0x55566975fdc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 -- 172.21.15.12:0/3077322658 >> 172.21.15.12:0/3077322658 conn(0x55566962ffc0 msgr2=0x55566975b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 -- 172.21.15.12:0/3077322658 shutdown_connections 2024-05-27T07:30:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.080+0000 7f8bc435db80 1 -- 172.21.15.12:0/3077322658 wait complete. 2024-05-27T07:30:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 -- start start 2024-05-27T07:30:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 Processor -- start 2024-05-27T07:30:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669768f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556697575c0 0x555669769450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669765ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55566967ed90 con 0x5556695cfe40 2024-05-27T07:30:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb01b6240 con 0x55566975d540 2024-05-27T07:30:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55566974ae00 con 0x5556697575c0 2024-05-27T07:30:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.081+0000 7f8bc435db80 10 client.0 _ll_register_callbacks cb 0x555669773510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669768f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669768f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39122/0 (socket says 172.21.15.12:39122) 2024-05-27T07:30:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 -- 172.21.15.12:0/294259098 learned_addr learned my addr 172.21.15.12:0/294259098 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baf400700 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669765ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 -- 172.21.15.12:0/294259098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556697575c0 msgr2=0x555669769450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556697575c0 0x555669769450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 -- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 msgr2=0x555669765ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669765ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baea00700 1 -- 172.21.15.12:0/294259098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556697699c0 con 0x5556695cfe40 2024-05-27T07:30:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.082+0000 7f8baf400700 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669765ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8baea00700 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669768f10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800ffd0 tx=0x7f8ba8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91e3bd3bb5e367b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8b98c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:30:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8b98c00700 1 -- 172.21.15.12:0/294259098 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba806f070 con 0x5556695cfe40 2024-05-27T07:30:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8b98c00700 1 -- 172.21.15.12:0/294259098 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9642+0+0 (secure 0 0 0) 0x7f8ba80a6600 con 0x5556695cfe40 2024-05-27T07:30:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b9c021110 con 0x5556695cfe40 2024-05-27T07:30:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8b98c00700 1 -- 172.21.15.12:0/294259098 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f8ba80a6a40 con 0x5556695cfe40 2024-05-27T07:30:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8bc435db80 10 client.16449 fetch_fsmap learned FSMap version 112 2024-05-27T07:30:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x5556697699c0 con 0x5556695cfe40 2024-05-27T07:30:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.083+0000 7f8b98c00700 1 -- 172.21.15.12:0/294259098 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b9c021110 con 0x5556695cfe40 2024-05-27T07:30:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.084+0000 7f8b98c00700 1 -- 172.21.15.12:0/294259098 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 112) ==== 3546+0+0 (secure 0 0 0) 0x7f8ba80a93b0 con 0x5556695cfe40 2024-05-27T07:30:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.084+0000 7f8bc435db80 10 client.16449 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:30:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.084+0000 7f8bc435db80 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.084+0000 7f8bc435db80 20 client.16449 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.084+0000 7f8bc435db80 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556697dbc10 0x5556697de030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.084+0000 7f8bc435db80 4 client.16449 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:30:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.084+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555669678570 con 0x5556697dbc10 2024-05-27T07:30:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.085+0000 7f8bae000700 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556697dbc10 0x5556697de030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.086+0000 7f8bae000700 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556697dbc10 0x5556697de030 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56bb05b0c5fa97ba server_cookie=7c731880ded0e030 in_seq=0 out_seq=0 2024-05-27T07:30:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.086+0000 7f8b98c00700 10 client.16449 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.091+0000 7f8b98c00700 1 -- 172.21.15.12:0/294259098 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555669678570 con 0x5556697dbc10 2024-05-27T07:30:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.091+0000 7f8b98c00700 10 client.16449 handle_command_reply: tid=0 2024-05-27T07:30:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.091+0000 7f8bc435db80 1 client.16449 shutdown 2024-05-27T07:30:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.091+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 msgr2=0x555669768f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.091+0000 7f8bc435db80 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669768f10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800ffd0 tx=0x7f8ba8005880 comp rx=0 tx=0).stop 2024-05-27T07:30:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.091+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 shutdown_connections 2024-05-27T07:30:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.091+0000 7f8bc435db80 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556697dbc10 0x5556697de030 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8bc435db80 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566975d540 0x555669765ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8bc435db80 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556695cfe40 0x555669768f10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8bc435db80 1 --2- 172.21.15.12:0/294259098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556697575c0 0x555669769450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 >> 172.21.15.12:0/294259098 conn(0x55566962ffc0 msgr2=0x55566975b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8b98c00700 0 client.16449 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 shutdown_connections 2024-05-27T07:30:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8bc435db80 1 -- 172.21.15.12:0/294259098 wait complete. 2024-05-27T07:30:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.092+0000 7f8bc435db80 20 client.16449 trim_cache size 0 max 16384 2024-05-27T07:30:10.195 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:30:10.195 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:30:10.195 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:30:10.196 INFO:teuthology.orchestra.run.smithi012.stdout: "4a152713-eb36-4214-8704-937e0aaf3cf3": { 2024-05-27T07:30:10.196 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:30:10.196 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "4a152713-eb36-4214-8704-937e0aaf3cf3", 2024-05-27T07:30:10.196 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:30:10.196 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:30:10.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:30:10.197 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:30:10.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b940b2a70 msgr2=0x7f8b940b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b940b2a70 0x7f8b940b4f80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8bb006f980 tx=0x7f8b9c00b410 comp rx=0 tx=0).stop 2024-05-27T07:30:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb010a920 msgr2=0x7f8bb01a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb010a920 0x7f8bb01a57b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800b5c0 tx=0x7f8ba80047b0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 shutdown_connections 2024-05-27T07:30:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b940b2a70 0x7f8b940b4f80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bb0111bc0 0x7f8bb01a9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb010a920 0x7f8bb01a57b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 --2- 172.21.15.12:0/2951758389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bb010b270 0x7f8bb01a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 >> 172.21.15.12:0/2951758389 conn(0x7f8bb006f4c0 msgr2=0x7f8bb01105a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 shutdown_connections 2024-05-27T07:30:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:10.093+0000 7f8bb5800700 1 -- 172.21.15.12:0/2951758389 wait complete. 2024-05-27T07:30:10.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:09 smithi012 systemd[1]: Started Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:30:10.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:10 smithi012 ceph-mds[125379]: -- 172.21.15.12:0/1518354925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5576c4aac4e0 con 0x5576c4af7600 2024-05-27T07:30:10.342 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:10 smithi012 ceph-mds[125379]: -- 172.21.15.12:0/1518354925 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5576c4aafa40 con 0x5576c4af7600 2024-05-27T07:30:10.342 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:10 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: starting mds.j at 2024-05-27T07:30:10.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:10 smithi012 ceph-mon[31473]: daemon mds.j restarted 2024-05-27T07:30:10.918 INFO:tasks.cephfs.filesystem:scrub status for tag:4a152713-eb36-4214-8704-937e0aaf3cf3 - {'path': '/', 'tag': '4a152713-eb36-4214-8704-937e0aaf3cf3', 'options': 'recursive,force'} 2024-05-27T07:30:10.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:10 smithi162 ceph-mon[35676]: daemon mds.j restarted 2024-05-27T07:30:10.986 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:10 smithi062 ceph-mon[35304]: daemon mds.j restarted 2024-05-27T07:30:11.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:11 smithi012 ceph-mon[31473]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:30:11.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:11 smithi012 ceph-mon[31473]: mds.? [v2:172.21.15.12:6834/4245490734,v1:172.21.15.12:6835/4245490734] up:boot 2024-05-27T07:30:11.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:11 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby 2024-05-27T07:30:11.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:11 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:11.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:11 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:11 smithi012 ceph-mon[31473]: pgmap v279: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f517f000700 time 2024-05-27T07:30:11.725857+0000 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5194a6dba6] 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.756 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: clone() 2024-05-27T07:30:11.757 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: *** Caught signal (Aborted) ** 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: in thread 7f517f000700 thread_name:md_log_replay 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.728+0000 7f517f000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f517f000700 time 2024-05-27T07:30:11.725857+0000 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5194a6dba6] 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.758 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: clone() 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f51935ead20] 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: gsignal() 2024-05-27T07:30:11.759 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: abort() 2024-05-27T07:30:11.760 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5194a6dced] 2024-05-27T07:30:11.760 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.760 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.760 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.760 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.760 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.760 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 19: clone() 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.731+0000 7f517f000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: in thread 7f517f000700 thread_name:md_log_replay 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.761 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f51935ead20] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: gsignal() 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: abort() 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5194a6dced] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.762 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 19: clone() 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: -1> 2024-05-27T07:30:11.728+0000 7f517f000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f517f000700 time 2024-05-27T07:30:11.725857+0000 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.763 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5194a6dba6] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.764 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.787 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.787 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: clone() 2024-05-27T07:30:11.787 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.787 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 0> 2024-05-27T07:30:11.731+0000 7f517f000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:11.787 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: in thread 7f517f000700 thread_name:md_log_replay 2024-05-27T07:30:11.787 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.787 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f51935ead20] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: gsignal() 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: abort() 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5194a6dced] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.788 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.789 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.789 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.789 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.789 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.789 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 19: clone() 2024-05-27T07:30:11.789 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:11.789 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: -1> 2024-05-27T07:30:11.728+0000 7f517f000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f517f000700 time 2024-05-27T07:30:11.725857+0000 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5194a6dba6] 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.790 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: clone() 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 0> 2024-05-27T07:30:11.731+0000 7f517f000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: in thread 7f517f000700 thread_name:md_log_replay 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.791 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f51935ead20] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2: gsignal() 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 3: abort() 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5194a6dced] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5194a6ddd1] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5576a876323d] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5576a874e971] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 8: (SnapRealm::check_cache() const+0x13e) [0x5576a874cfa0] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5576a874d651] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5576a86e2347] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5576a83de87e] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5576a83e3e94] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5576a83ee969] 2024-05-27T07:30:11.792 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 14: (MDLog::_replay_thread()+0x1f38) [0x5576a878748c] 2024-05-27T07:30:11.793 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5576a87909cb] 2024-05-27T07:30:11.793 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 16: (Thread::entry_wrapper()+0x43) [0x7f5194a3c72d] 2024-05-27T07:30:11.793 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5194a3c749] 2024-05-27T07:30:11.793 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f51935e01ca] 2024-05-27T07:30:11.793 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 19: clone() 2024-05-27T07:30:11.793 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:11.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:11 smithi162 ceph-mon[35676]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:30:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:11 smithi162 ceph-mon[35676]: mds.? [v2:172.21.15.12:6834/4245490734,v1:172.21.15.12:6835/4245490734] up:boot 2024-05-27T07:30:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:11 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby 2024-05-27T07:30:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:11 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:11 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:30:11.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:11 smithi162 ceph-mon[35676]: pgmap v279: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-05-27T07:30:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:11 smithi062 ceph-mon[35304]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:30:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:11 smithi062 ceph-mon[35304]: mds.? [v2:172.21.15.12:6834/4245490734,v1:172.21.15.12:6835/4245490734] up:boot 2024-05-27T07:30:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:11 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby 2024-05-27T07:30:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:11 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:11 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 1 up:standby-replay 2024-05-27T07:30:11.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:11 smithi062 ceph-mon[35304]: pgmap v279: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-05-27T07:30:12.091 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j[125373]: 2024-05-27T07:30:12.091 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:11 smithi012 podman[125469]: 2024-05-27 07:30:11.865132483 +0000 UTC m=+0.035607574 container died 48b2825848f4b7e706cdd3621f01d40a1f7b776d1add0ae274cb4111dbc60da9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:12.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:12 smithi012 podman[125469]: 2024-05-27 07:30:12.569636278 +0000 UTC m=+0.740111381 container remove 48b2825848f4b7e706cdd3621f01d40a1f7b776d1add0ae274cb4111dbc60da9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:12.841 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:12 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:13.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:13 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:30:13.341 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:13 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.738s CPU time. 2024-05-27T07:30:13.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:13 smithi162 ceph-mon[35676]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:30:13.736 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:13 smithi062 ceph-mon[35304]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:30:13.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:13 smithi012 ceph-mon[31473]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:30:14.639 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:14 smithi062 ceph-mon[35304]: pgmap v281: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:30:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:14 smithi162 ceph-mon[35676]: pgmap v281: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:30:14.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:14 smithi012 ceph-mon[31473]: pgmap v281: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:30:16.632 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:16 smithi062 ceph-mon[35304]: pgmap v282: 129 pgs: 129 active+clean; 119 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.8 MiB/s wr, 199 op/s 2024-05-27T07:30:16.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:16 smithi162 ceph-mon[35676]: pgmap v282: 129 pgs: 129 active+clean; 119 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.8 MiB/s wr, 199 op/s 2024-05-27T07:30:16.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:16 smithi012 ceph-mon[31473]: pgmap v282: 129 pgs: 129 active+clean; 119 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.8 MiB/s wr, 199 op/s 2024-05-27T07:30:18.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:18 smithi012 ceph-mon[31473]: pgmap v283: 129 pgs: 129 active+clean; 119 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 883 KiB/s wr, 94 op/s 2024-05-27T07:30:18.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:18 smithi162 ceph-mon[35676]: pgmap v283: 129 pgs: 129 active+clean; 119 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 883 KiB/s wr, 94 op/s 2024-05-27T07:30:18.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:18 smithi062 ceph-mon[35304]: pgmap v283: 129 pgs: 129 active+clean; 119 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 883 KiB/s wr, 94 op/s 2024-05-27T07:30:20.536 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:20 smithi012 ceph-mon[31473]: pgmap v284: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-05-27T07:30:20.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:20 smithi162 ceph-mon[35676]: pgmap v284: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-05-27T07:30:20.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:20 smithi062 ceph-mon[35304]: pgmap v284: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-05-27T07:30:21.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:30:21.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:30:21.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:30:21.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:30:21.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:30:21.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:30:22.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:22 smithi162 ceph-mon[35676]: pgmap v285: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-05-27T07:30:22.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:22 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:30:22.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:22 smithi062 ceph-mon[35304]: pgmap v285: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-05-27T07:30:22.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:22 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:30:22.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:22 smithi012 ceph-mon[31473]: pgmap v285: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-05-27T07:30:22.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:22 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:30:23.592 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:23 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:30:23.592 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:23 smithi012 systemd[1]: Stopped Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:30:23.592 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:23 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Consumed 3.738s CPU time. 2024-05-27T07:30:23.592 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:23 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-05-27T07:30:23.592 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:23 smithi012 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:30:23.592 INFO:journalctl@ceph.mds.j.smithi012.stdout:May 27 07:30:23 smithi012 systemd[1]: Failed to start Ceph mds.j for 55c251ba-1bf7-11ef-bc9b-c7b262605968. 2024-05-27T07:30:24.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:24 smithi162 ceph-mon[35676]: pgmap v286: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-05-27T07:30:24.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:24 smithi062 ceph-mon[35304]: pgmap v286: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-05-27T07:30:24.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:24 smithi012 ceph-mon[31473]: pgmap v286: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-05-27T07:30:26.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:26 smithi162 ceph-mon[35676]: pgmap v287: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-05-27T07:30:26.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:26 smithi062 ceph-mon[35304]: pgmap v287: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-05-27T07:30:26.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:26 smithi012 ceph-mon[31473]: pgmap v287: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-05-27T07:30:27.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:27 smithi162 ceph-mon[35676]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:27.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:27 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:27.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:27 smithi062 ceph-mon[35304]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:27.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:27 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:27.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:27 smithi012 ceph-mon[31473]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:27.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:27 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:28 smithi162 ceph-mon[35676]: pgmap v288: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 885 KiB/s wr, 103 op/s 2024-05-27T07:30:28.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:28 smithi062 ceph-mon[35304]: pgmap v288: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 885 KiB/s wr, 103 op/s 2024-05-27T07:30:28.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:28 smithi012 ceph-mon[31473]: pgmap v288: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 885 KiB/s wr, 103 op/s 2024-05-27T07:30:29.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:29 smithi012 ceph-mon[31473]: pgmap v289: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 150 op/s 2024-05-27T07:30:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:29 smithi162 ceph-mon[35676]: pgmap v289: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 150 op/s 2024-05-27T07:30:29.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:29 smithi062 ceph-mon[35304]: pgmap v289: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 150 op/s 2024-05-27T07:30:31.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:31 smithi162 ceph-mon[35676]: pgmap v290: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 100 op/s 2024-05-27T07:30:31.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:31 smithi062 ceph-mon[35304]: pgmap v290: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 100 op/s 2024-05-27T07:30:32.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:31 smithi012 ceph-mon[31473]: pgmap v290: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 100 op/s 2024-05-27T07:30:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:33 smithi162 ceph-mon[35676]: pgmap v291: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 100 op/s 2024-05-27T07:30:33.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:33 smithi062 ceph-mon[35304]: pgmap v291: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 100 op/s 2024-05-27T07:30:34.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:33 smithi012 ceph-mon[31473]: pgmap v291: 129 pgs: 129 active+clean; 122 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 100 op/s 2024-05-27T07:30:35.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:35 smithi162 ceph-mon[35676]: pgmap v292: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-05-27T07:30:36.000 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:35 smithi062 ceph-mon[35304]: pgmap v292: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-05-27T07:30:36.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:35 smithi012 ceph-mon[31473]: pgmap v292: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-05-27T07:30:36.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:36 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:36.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:36 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:37.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:36 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:37.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:37 smithi162 ceph-mon[35676]: pgmap v293: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 908 KiB/s wr, 94 op/s 2024-05-27T07:30:37.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:37 smithi062 ceph-mon[35304]: pgmap v293: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 908 KiB/s wr, 94 op/s 2024-05-27T07:30:38.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:37 smithi012 ceph-mon[31473]: pgmap v293: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 908 KiB/s wr, 94 op/s 2024-05-27T07:30:39.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:39 smithi162 ceph-mon[35676]: pgmap v294: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 141 op/s 2024-05-27T07:30:39.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:39 smithi062 ceph-mon[35304]: pgmap v294: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 141 op/s 2024-05-27T07:30:40.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:39 smithi012 ceph-mon[31473]: pgmap v294: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 141 op/s 2024-05-27T07:30:40.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:30:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:41.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:41 smithi012 ceph-mon[31473]: pgmap v295: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-05-27T07:30:41.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:41 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:41.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:41 smithi162 ceph-mon[35676]: pgmap v295: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-05-27T07:30:41.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:41 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:41.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:41 smithi062 ceph-mon[35304]: pgmap v295: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-05-27T07:30:41.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:41 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:43.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:43 smithi162 ceph-mon[35676]: pgmap v296: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-05-27T07:30:43.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:43 smithi062 ceph-mon[35304]: pgmap v296: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-05-27T07:30:44.007 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:43 smithi012 ceph-mon[31473]: pgmap v296: 129 pgs: 129 active+clean; 124 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 -- 172.21.15.12:0/1758280178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c076c20 msgr2=0x7ff01c077030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 --2- 172.21.15.12:0/1758280178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c076c20 0x7ff01c077030 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff010009990 tx=0x7ff010009ca0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 -- 172.21.15.12:0/1758280178 shutdown_connections 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 --2- 172.21.15.12:0/1758280178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff01c110bb0 0x7ff01c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 --2- 172.21.15.12:0/1758280178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c076c20 0x7ff01c077030 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 --2- 172.21.15.12:0/1758280178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c077570 0x7ff01c110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 -- 172.21.15.12:0/1758280178 >> 172.21.15.12:0/1758280178 conn(0x7ff01c0ffe90 msgr2=0x7ff01c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 -- 172.21.15.12:0/1758280178 shutdown_connections 2024-05-27T07:30:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.015+0000 7ff024600700 1 -- 172.21.15.12:0/1758280178 wait complete. 2024-05-27T07:30:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.017+0000 7ff024600700 1 Processor -- start 2024-05-27T07:30:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.017+0000 7ff024600700 1 -- start start 2024-05-27T07:30:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.017+0000 7ff024600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c076c20 0x7ff01c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.017+0000 7ff024600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c077570 0x7ff01c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff024600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff01c110bb0 0x7ff01c1a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff024600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c116590 con 0x7ff01c077570 2024-05-27T07:30:44.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff024600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c116410 con 0x7ff01c076c20 2024-05-27T07:30:44.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff024600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c116710 con 0x7ff01c110bb0 2024-05-27T07:30:44.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff021c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c076c20 0x7ff01c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff021200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c077570 0x7ff01c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff021c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c076c20 0x7ff01c19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:38606/0 (socket says 172.21.15.12:38606) 2024-05-27T07:30:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff021c00700 1 -- 172.21.15.12:0/2070394757 learned_addr learned my addr 172.21.15.12:0/2070394757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.018+0000 7ff021200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c077570 0x7ff01c19d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39228/0 (socket says 172.21.15.12:39228) 2024-05-27T07:30:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff021c00700 1 -- 172.21.15.12:0/2070394757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff01c110bb0 msgr2=0x7ff01c1a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff022600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff01c110bb0 0x7ff01c1a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff021c00700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff01c110bb0 0x7ff01c1a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff021c00700 1 -- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c077570 msgr2=0x7ff01c19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff021c00700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c077570 0x7ff01c19d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff021c00700 1 -- 172.21.15.12:0/2070394757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c1a1e70 con 0x7ff01c076c20 2024-05-27T07:30:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff022600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff01c110bb0 0x7ff01c1a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.019+0000 7ff021200700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c077570 0x7ff01c19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.020+0000 7ff021c00700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c076c20 0x7ff01c19cfa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff010000c00 tx=0x7ff0100057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.020+0000 7ff01a000700 1 -- 172.21.15.12:0/2070394757 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010027070 con 0x7ff01c076c20 2024-05-27T07:30:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.020+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01c1a2100 con 0x7ff01c076c20 2024-05-27T07:30:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.021+0000 7ff01a000700 1 -- 172.21.15.12:0/2070394757 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff01000fb00 con 0x7ff01c076c20 2024-05-27T07:30:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.021+0000 7ff01a000700 1 -- 172.21.15.12:0/2070394757 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010022c40 con 0x7ff01c076c20 2024-05-27T07:30:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.021+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01c1a9a80 con 0x7ff01c076c20 2024-05-27T07:30:44.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.026+0000 7ff033131b80 1 Processor -- start 2024-05-27T07:30:44.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.026+0000 7ff033131b80 1 -- start start 2024-05-27T07:30:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.027+0000 7ff01a000700 1 -- 172.21.15.12:0/2070394757 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff01002a070 con 0x7ff01c076c20 2024-05-27T07:30:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.027+0000 7ff033131b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a18330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.027+0000 7ff033131b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a18870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.027+0000 7ff033131b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a25110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.027+0000 7ff033131b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559a72946e40 con 0x559a72899e40 2024-05-27T07:30:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.027+0000 7ff033131b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559a729369e0 con 0x559a729859f0 2024-05-27T07:30:44.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.027+0000 7ff033131b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559a7293aa20 con 0x559a72a18e70 2024-05-27T07:30:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a18330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a18330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:38610/0 (socket says 172.21.15.12:38610) 2024-05-27T07:30:44.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 -- 172.21.15.12:0/3327732749 learned_addr learned my addr 172.21.15.12:0/3327732749 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 -- 172.21.15.12:0/3327732749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 msgr2=0x559a72a25110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 --2- 172.21.15.12:0/3327732749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a25110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 -- 172.21.15.12:0/3327732749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 msgr2=0x559a72a18870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:30:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 --2- 172.21.15.12:0/3327732749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a18870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.028+0000 7ff022600700 1 -- 172.21.15.12:0/3327732749 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a728ff420 con 0x559a729859f0 2024-05-27T07:30:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.029+0000 7ff022600700 1 --2- 172.21.15.12:0/3327732749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a18330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff014009ca0 tx=0x7ff014007500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc56ca835abba97c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.029+0000 7ff01a000700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0040b2af0 0x7ff0040b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.030+0000 7ff01a000700 1 -- 172.21.15.12:0/2070394757 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7ff0100e3a40 con 0x7ff01c076c20 2024-05-27T07:30:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.030+0000 7ff00ea00700 1 -- 172.21.15.12:0/3327732749 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014019070 con 0x559a729859f0 2024-05-27T07:30:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.030+0000 7ff00ea00700 1 -- 172.21.15.12:0/3327732749 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff014007e40 con 0x559a729859f0 2024-05-27T07:30:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.030+0000 7ff00ea00700 1 -- 172.21.15.12:0/3327732749 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014013920 con 0x559a729859f0 2024-05-27T07:30:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.030+0000 7ff021c00700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0040b2af0 0x7ff0040b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 -- 172.21.15.12:0/3327732749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 msgr2=0x559a72a18330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff021c00700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0040b2af0 0x7ff0040b5000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x559a72a15a20 tx=0x7ff010009310 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 --2- 172.21.15.12:0/3327732749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a18330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff014009ca0 tx=0x7ff014007500 comp rx=0 tx=0).stop 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 -- 172.21.15.12:0/3327732749 shutdown_connections 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 --2- 172.21.15.12:0/3327732749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a18330 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 --2- 172.21.15.12:0/3327732749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a25110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 --2- 172.21.15.12:0/3327732749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a18870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 -- 172.21.15.12:0/3327732749 >> 172.21.15.12:0/3327732749 conn(0x559a72933390 msgr2=0x559a72a144c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 -- 172.21.15.12:0/3327732749 shutdown_connections 2024-05-27T07:30:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.032+0000 7ff033131b80 1 -- 172.21.15.12:0/3327732749 wait complete. 2024-05-27T07:30:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 -- start start 2024-05-27T07:30:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 Processor -- start 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a22640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a22b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a230c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559a7283cfa0 con 0x559a72899e40 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559a728d7760 con 0x559a729859f0 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.033+0000 7ff033131b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559a72a25b80 con 0x559a72a18e70 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.034+0000 7ff033131b80 10 client.0 _ll_register_callbacks cb 0x559a728e0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.034+0000 7ff022600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a22640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.034+0000 7ff022600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a22640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39238/0 (socket says 172.21.15.12:39238) 2024-05-27T07:30:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.034+0000 7ff022600700 1 -- 172.21.15.12:0/720293183 learned_addr learned my addr 172.21.15.12:0/720293183 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.034+0000 7ff021200700 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a22b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.035+0000 7ff022600700 1 -- 172.21.15.12:0/720293183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 msgr2=0x559a72a230c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.035+0000 7ff022600700 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.035+0000 7ff022600700 1 -- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 msgr2=0x559a72a22b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.035+0000 7ff022600700 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a22b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.035+0000 7ff022600700 1 -- 172.21.15.12:0/720293183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a728868d0 con 0x559a72899e40 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.035+0000 7ff022600700 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a22640 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff0140093e0 tx=0x7ff014013460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9169bfc325b55d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.035+0000 7ff003400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.036+0000 7ff003400700 1 -- 172.21.15.12:0/720293183 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014025060 con 0x559a72899e40 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.036+0000 7ff003400700 1 -- 172.21.15.12:0/720293183 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7ff014020420 con 0x559a72899e40 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.036+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559a72948d90 con 0x559a72899e40 2024-05-27T07:30:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.036+0000 7ff003400700 1 -- 172.21.15.12:0/720293183 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7ff01402c050 con 0x559a72899e40 2024-05-27T07:30:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.036+0000 7ff033131b80 10 client.16458 fetch_fsmap learned FSMap version 117 2024-05-27T07:30:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.036+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=4+}) -- 0x559a72907060 con 0x559a72899e40 2024-05-27T07:30:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff003400700 1 -- 172.21.15.12:0/720293183 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x559a72948d90 con 0x559a72899e40 2024-05-27T07:30:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff003400700 1 -- 172.21.15.12:0/720293183 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 117) ==== 2945+0+0 (secure 0 0 0) 0x7ff01401f450 con 0x559a72899e40 2024-05-27T07:30:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff033131b80 10 client.16458 fetch_fsmap finished waiting for FSMap version 117 2024-05-27T07:30:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff033131b80 10 client.16458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff033131b80 20 client.16458 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff033131b80 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a32e50 0x559a72a352a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff033131b80 4 client.16458 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.037+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a72942570 con 0x559a72a32e50 2024-05-27T07:30:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.038+0000 7ff021200700 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a32e50 0x559a72a352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.039+0000 7ff021200700 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a32e50 0x559a72a352a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed01acf45e03f54 server_cookie=611887539e598b2 in_seq=0 out_seq=0 2024-05-27T07:30:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.039+0000 7ff003400700 10 client.16458 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff003400700 1 -- 172.21.15.12:0/720293183 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559a72942570 con 0x559a72a32e50 2024-05-27T07:30:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff003400700 10 client.16458 handle_command_reply: tid=0 2024-05-27T07:30:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 client.16458 shutdown 2024-05-27T07:30:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 msgr2=0x559a72a22640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a22640 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff0140093e0 tx=0x7ff014013460 comp rx=0 tx=0).stop 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 shutdown_connections 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a22640 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a32e50 0x559a72a352a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 --2- 172.21.15.12:0/720293183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a729859f0 0x559a72a22b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff003400700 0 client.16458 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 >> 172.21.15.12:0/720293183 conn(0x559a72933390 msgr2=0x559a72a183b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.042+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 shutdown_connections 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.043+0000 7ff033131b80 1 -- 172.21.15.12:0/720293183 wait complete. 2024-05-27T07:30:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.043+0000 7ff033131b80 20 client.16458 trim_cache size 0 max 16384 2024-05-27T07:30:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 Processor -- start 2024-05-27T07:30:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 -- start start 2024-05-27T07:30:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72899e40 0x559a72a1ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a729859f0 0x559a72a1c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a3aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c1b1df0 con 0x559a729859f0 2024-05-27T07:30:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c1b1020 con 0x559a72899e40 2024-05-27T07:30:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.076+0000 7ff033131b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c1b16f0 con 0x559a72a18e70 2024-05-27T07:30:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a729859f0 0x559a72a1c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a729859f0 0x559a72a1c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39254/0 (socket says 172.21.15.12:39254) 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 -- 172.21.15.12:0/127877954 learned_addr learned my addr 172.21.15.12:0/127877954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff022600700 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72899e40 0x559a72a1ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 -- 172.21.15.12:0/127877954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 msgr2=0x559a72a3aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a3aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 -- 172.21.15.12:0/127877954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72899e40 msgr2=0x559a72a1ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72899e40 0x559a72a1ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.077+0000 7ff021200700 1 -- 172.21.15.12:0/127877954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a72a15850 con 0x559a729859f0 2024-05-27T07:30:44.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.078+0000 7ff021200700 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a729859f0 0x559a72a1c5f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff00800bfd0 tx=0x7ff008005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed886368f44f3d23 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:44.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.078+0000 7ff00e000700 1 -- 172.21.15.12:0/127877954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008013390 con 0x559a729859f0 2024-05-27T07:30:44.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.078+0000 7ff00e000700 1 -- 172.21.15.12:0/127877954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff008004cc0 con 0x559a729859f0 2024-05-27T07:30:44.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.078+0000 7ff00e000700 1 -- 172.21.15.12:0/127877954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0080115e0 con 0x559a729859f0 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.079+0000 7ff033131b80 1 -- 172.21.15.12:0/127877954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a729859f0 msgr2=0x559a72a1c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.079+0000 7ff033131b80 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a729859f0 0x559a72a1c5f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff00800bfd0 tx=0x7ff008005d20 comp rx=0 tx=0).stop 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.080+0000 7ff033131b80 1 -- 172.21.15.12:0/127877954 shutdown_connections 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.080+0000 7ff033131b80 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72899e40 0x559a72a1ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.080+0000 7ff033131b80 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a72a18e70 0x559a72a3aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.080+0000 7ff033131b80 1 --2- 172.21.15.12:0/127877954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a729859f0 0x559a72a1c5f0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.080+0000 7ff033131b80 1 -- 172.21.15.12:0/127877954 >> 172.21.15.12:0/127877954 conn(0x559a72933390 msgr2=0x559a72a1bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.080+0000 7ff033131b80 1 -- 172.21.15.12:0/127877954 shutdown_connections 2024-05-27T07:30:44.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.080+0000 7ff033131b80 1 -- 172.21.15.12:0/127877954 wait complete. 2024-05-27T07:30:44.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.081+0000 7ff033131b80 1 -- start start 2024-05-27T07:30:44.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.081+0000 7ff033131b80 1 Processor -- start 2024-05-27T07:30:44.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff033131b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a2ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff033131b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a729859f0 0x559a72a2b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff033131b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72a18e70 0x559a72a27300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff033131b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559a72a10950 con 0x559a72899e40 2024-05-27T07:30:44.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff033131b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559a72948d90 con 0x559a72a18e70 2024-05-27T07:30:44.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff033131b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559a72a23750 con 0x559a729859f0 2024-05-27T07:30:44.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff033131b80 10 client.0 _ll_register_callbacks cb 0x559a728e0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:44.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff022600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a2ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff022600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a2ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39256/0 (socket says 172.21.15.12:39256) 2024-05-27T07:30:44.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff022600700 1 -- 172.21.15.12:0/2314468580 learned_addr learned my addr 172.21.15.12:0/2314468580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:44.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.082+0000 7ff021200700 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a729859f0 0x559a72a2b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.083+0000 7ff022600700 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72a18e70 0x559a72a27300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.083+0000 7ff022600700 1 -- 172.21.15.12:0/2314468580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a729859f0 msgr2=0x559a72a2b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:44.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.083+0000 7ff022600700 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a729859f0 0x559a72a2b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.083+0000 7ff022600700 1 -- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72a18e70 msgr2=0x559a72a27300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:44.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.083+0000 7ff022600700 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72a18e70 0x559a72a27300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.083+0000 7ff022600700 1 -- 172.21.15.12:0/2314468580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a72a25710 con 0x559a72899e40 2024-05-27T07:30:44.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.084+0000 7ff022600700 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a2ae90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff014009ca0 tx=0x7ff014007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b9cc54b347b58a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:44.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.084+0000 7ff003400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:30:44.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.084+0000 7ff003400700 1 -- 172.21.15.12:0/2314468580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014025040 con 0x559a72899e40 2024-05-27T07:30:44.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.084+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff008004cc0 con 0x559a72899e40 2024-05-27T07:30:44.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.084+0000 7ff003400700 1 -- 172.21.15.12:0/2314468580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7ff0140272b0 con 0x559a72899e40 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff003400700 1 -- 172.21.15.12:0/2314468580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7ff014019030 con 0x559a72899e40 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 10 client.16464 fetch_fsmap learned FSMap version 117 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=4+}) -- 0x559a72a9e0f0 con 0x559a72899e40 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff003400700 1 -- 172.21.15.12:0/2314468580 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff008004cc0 con 0x559a72899e40 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff003400700 1 -- 172.21.15.12:0/2314468580 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 117) ==== 2945+0+0 (secure 0 0 0) 0x7ff014027820 con 0x559a72899e40 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 10 client.16464 fetch_fsmap finished waiting for FSMap version 117 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 10 client.16464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 20 client.16464 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a9e690 0x559a72aa0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 4 client.16464 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:30:44.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.085+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559a727b9250 con 0x559a72a9e690 2024-05-27T07:30:44.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.086+0000 7ff021200700 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a9e690 0x559a72aa0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:44.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.087+0000 7ff021200700 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a9e690 0x559a72aa0ab0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaeab57a02b89fc6 server_cookie=8d1ff93027c3f42d in_seq=0 out_seq=0 2024-05-27T07:30:44.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.087+0000 7ff003400700 10 client.16464 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:44.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.088+0000 7ff003400700 1 -- 172.21.15.12:0/2314468580 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559a727b9250 con 0x559a72a9e690 2024-05-27T07:30:44.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.088+0000 7ff003400700 10 client.16464 handle_command_reply: tid=0 2024-05-27T07:30:44.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.088+0000 7ff033131b80 1 client.16464 shutdown 2024-05-27T07:30:44.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.088+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 msgr2=0x559a72a2ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.088+0000 7ff033131b80 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a2ae90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff014009ca0 tx=0x7ff014007500 comp rx=0 tx=0).stop 2024-05-27T07:30:44.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 shutdown_connections 2024-05-27T07:30:44.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a72a9e690 0x559a72aa0ab0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a72a18e70 0x559a72a27300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a72899e40 0x559a72a2ae90 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 --2- 172.21.15.12:0/2314468580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a729859f0 0x559a72a2b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 >> 172.21.15.12:0/2314468580 conn(0x559a72933390 msgr2=0x559a72a212b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:44.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff003400700 0 client.16464 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:30:44.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 shutdown_connections 2024-05-27T07:30:44.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 1 -- 172.21.15.12:0/2314468580 wait complete. 2024-05-27T07:30:44.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.089+0000 7ff033131b80 20 client.16464 trim_cache size 0 max 16384 2024-05-27T07:30:44.082 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:30:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:30:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:30:44.082 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:30:44.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.090+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0040b2af0 msgr2=0x7ff0040b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.090+0000 7ff024600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0040b2af0 0x7ff0040b5000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x559a72a15a20 tx=0x7ff010009310 comp rx=0 tx=0).stop 2024-05-27T07:30:44.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.090+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c076c20 msgr2=0x7ff01c19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:44.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.090+0000 7ff024600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c076c20 0x7ff01c19cfa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff010000c00 tx=0x7ff0100057c0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 shutdown_connections 2024-05-27T07:30:44.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff0040b2af0 0x7ff0040b5000 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff01c076c20 0x7ff01c19cfa0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff01c110bb0 0x7ff01c1a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 --2- 172.21.15.12:0/2070394757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff01c077570 0x7ff01c19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:44.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 >> 172.21.15.12:0/2070394757 conn(0x7ff01c0ffe90 msgr2=0x7ff01c100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:44.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 shutdown_connections 2024-05-27T07:30:44.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:44.091+0000 7ff024600700 1 -- 172.21.15.12:0/2070394757 wait complete. 2024-05-27T07:30:45.314 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:30:45.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:30:45.830 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:45.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:45 smithi162 ceph-mon[35676]: pgmap v297: 129 pgs: 129 active+clean; 125 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-05-27T07:30:45.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:45 smithi062 ceph-mon[35304]: pgmap v297: 129 pgs: 129 active+clean; 125 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-05-27T07:30:45.988 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:45 smithi012 ceph-mon[31473]: pgmap v297: 129 pgs: 129 active+clean; 125 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-05-27T07:30:47.911 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:47 smithi012 ceph-mon[31473]: pgmap v298: 129 pgs: 129 active+clean; 125 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 976 KiB/s wr, 97 op/s 2024-05-27T07:30:47.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:47 smithi162 ceph-mon[35676]: pgmap v298: 129 pgs: 129 active+clean; 125 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 976 KiB/s wr, 97 op/s 2024-05-27T07:30:47.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:47 smithi062 ceph-mon[35304]: pgmap v298: 129 pgs: 129 active+clean; 125 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 976 KiB/s wr, 97 op/s 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 -- 172.21.15.12:0/2693527605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a010e8f0 msgr2=0x7f89a0110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 --2- 172.21.15.12:0/2693527605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a010e8f0 0x7f89a0110db0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f899800b0a0 tx=0x7f899800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 -- 172.21.15.12:0/2693527605 shutdown_connections 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 --2- 172.21.15.12:0/2693527605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a010e8f0 0x7f89a0110db0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 --2- 172.21.15.12:0/2693527605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89a0105510 0x7f89a0105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 --2- 172.21.15.12:0/2693527605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a0105e60 0x7f89a010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 -- 172.21.15.12:0/2693527605 >> 172.21.15.12:0/2693527605 conn(0x7f89a00fedb0 msgr2=0x7f89a01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:48.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 -- 172.21.15.12:0/2693527605 shutdown_connections 2024-05-27T07:30:48.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.327+0000 7f89a8200700 1 -- 172.21.15.12:0/2693527605 wait complete. 2024-05-27T07:30:48.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.330+0000 7f89a8200700 1 Processor -- start 2024-05-27T07:30:48.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.330+0000 7f89a8200700 1 -- start start 2024-05-27T07:30:48.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.330+0000 7f89a8200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89a0105510 0x7f89a01a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:48.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.330+0000 7f89a8200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a0105e60 0x7f89a01a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:48.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a8200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a010e8f0 0x7f89a019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:48.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a8200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f89a0114330 con 0x7f89a0105e60 2024-05-27T07:30:48.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a8200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f89a01141b0 con 0x7f89a010e8f0 2024-05-27T07:30:48.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a8200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f89a01144b0 con 0x7f89a0105510 2024-05-27T07:30:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a0105e60 0x7f89a01a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a4e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a0105e60 0x7f89a01a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39282/0 (socket says 172.21.15.12:39282) 2024-05-27T07:30:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a4e00700 1 -- 172.21.15.12:0/2660231200 learned_addr learned my addr 172.21.15.12:0/2660231200 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a5800700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89a0105510 0x7f89a01a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:48.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.331+0000 7f89a6200700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a010e8f0 0x7f89a019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f89a4e00700 1 -- 172.21.15.12:0/2660231200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89a0105510 msgr2=0x7f89a01a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f89a4e00700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89a0105510 0x7f89a01a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f89a4e00700 1 -- 172.21.15.12:0/2660231200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a010e8f0 msgr2=0x7f89a019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f89a4e00700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a010e8f0 0x7f89a019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f89a4e00700 1 -- 172.21.15.12:0/2660231200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a019b1b0 con 0x7f89a0105e60 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f89a4e00700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a0105e60 0x7f89a01a0df0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f898c00d950 tx=0x7f898c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f899e000700 1 -- 172.21.15.12:0/2660231200 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c016070 con 0x7f89a0105e60 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f899e000700 1 -- 172.21.15.12:0/2660231200 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f898c00fd20 con 0x7f89a0105e60 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a019b440 con 0x7f89a0105e60 2024-05-27T07:30:48.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.332+0000 7f899e000700 1 -- 172.21.15.12:0/2660231200 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c00e3b0 con 0x7f89a0105e60 2024-05-27T07:30:48.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.333+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a01a7740 con 0x7f89a0105e60 2024-05-27T07:30:48.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.334+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a00512b0 con 0x7f89a0105e60 2024-05-27T07:30:48.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.337+0000 7f899e000700 1 -- 172.21.15.12:0/2660231200 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f898c019070 con 0x7f89a0105e60 2024-05-27T07:30:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.340+0000 7f899e000700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f89880b29f0 0x7f89880b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.340+0000 7f899e000700 1 -- 172.21.15.12:0/2660231200 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f898c0e24a0 con 0x7f89a0105e60 2024-05-27T07:30:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.340+0000 7f89a5800700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f89880b29f0 0x7f89880b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.341+0000 7f89a5800700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f89880b29f0 0x7f89880b4f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f89a019bc10 tx=0x7f899400b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.343+0000 7f899e000700 1 -- 172.21.15.12:0/2660231200 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f898c0a9a10 con 0x7f89a0105e60 2024-05-27T07:30:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.718+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89a002e5e0 con 0x7f89a0105e60 2024-05-27T07:30:48.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.724+0000 7f899e000700 1 -- 172.21.15.12:0/2660231200 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+4412 (secure 0 0 0) 0x7f898c0a9310 con 0x7f89a0105e60 2024-05-27T07:30:48.717 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:30:48.717 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:40.577882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:30:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f89880b29f0 msgr2=0x7f89880b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f89880b29f0 0x7f89880b4f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f89a019bc10 tx=0x7f899400b410 comp rx=0 tx=0).stop 2024-05-27T07:30:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a0105e60 msgr2=0x7f89a01a0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a0105e60 0x7f89a01a0df0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f898c00d950 tx=0x7f898c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:30:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 shutdown_connections 2024-05-27T07:30:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f89880b29f0 0x7f89880b4f00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a010e8f0 0x7f89a019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89a0105510 0x7f89a01a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 --2- 172.21.15.12:0/2660231200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89a0105e60 0x7f89a01a0df0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.728+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 >> 172.21.15.12:0/2660231200 conn(0x7f89a00fedb0 msgr2=0x7f89a00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:48.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.730+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 shutdown_connections 2024-05-27T07:30:48.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:48.730+0000 7f89a8200700 1 -- 172.21.15.12:0/2660231200 wait complete. 2024-05-27T07:30:48.724 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 117 2024-05-27T07:30:49.724 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:30:49.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:49 smithi162 ceph-mon[35676]: pgmap v299: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-05-27T07:30:49.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:49 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2660231200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:30:49.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:49 smithi062 ceph-mon[35304]: pgmap v299: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-05-27T07:30:49.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:49 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2660231200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:30:50.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:49 smithi012 ceph-mon[31473]: pgmap v299: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-05-27T07:30:50.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:49 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2660231200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:30:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:51.961 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:51 smithi012 ceph-mon[31473]: pgmap v300: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-05-27T07:30:51.961 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:51 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:51.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:51 smithi062 ceph-mon[35304]: pgmap v300: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-05-27T07:30:51.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:51 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:52.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:51 smithi162 ceph-mon[35676]: pgmap v300: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-05-27T07:30:52.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:51 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 -- 172.21.15.12:0/1796383544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc108f10 msgr2=0x7f7fcc109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/1796383544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc108f10 0x7f7fcc109320 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7fb80099c0 tx=0x7f7fb8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 -- 172.21.15.12:0/1796383544 shutdown_connections 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/1796383544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1045e0 0x7f7fcc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/1796383544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fcc109860 0x7f7fcc1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/1796383544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc108f10 0x7f7fcc109320 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 -- 172.21.15.12:0/1796383544 >> 172.21.15.12:0/1796383544 conn(0x7f7fcc100030 msgr2=0x7f7fcc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.803+0000 7f7fd0c00700 1 -- 172.21.15.12:0/1796383544 shutdown_connections 2024-05-27T07:30:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.804+0000 7f7fd0c00700 1 -- 172.21.15.12:0/1796383544 wait complete. 2024-05-27T07:30:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.805+0000 7f7fd0c00700 1 Processor -- start 2024-05-27T07:30:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.805+0000 7f7fd0c00700 1 -- start start 2024-05-27T07:30:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fd0c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fcc1045e0 0x7f7fcc19ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fd0c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc108f10 0x7f7fcc19b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fd0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc109860 0x7f7fcc19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fd0c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7fcc114320 con 0x7f7fcc109860 2024-05-27T07:30:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fd0c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7fcc1141a0 con 0x7f7fcc108f10 2024-05-27T07:30:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fd0c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fcc1144a0 con 0x7f7fcc1045e0 2024-05-27T07:30:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc109860 0x7f7fcc19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc109860 0x7f7fcc19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60992/0 (socket says 172.21.15.12:60992) 2024-05-27T07:30:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.806+0000 7f7fca800700 1 -- 172.21.15.12:0/463632733 learned_addr learned my addr 172.21.15.12:0/463632733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.807+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fcc1045e0 0x7f7fcc19ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.807+0000 7f7fc9400700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc108f10 0x7f7fcc19b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.807+0000 7f7fca800700 1 -- 172.21.15.12:0/463632733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fcc1045e0 msgr2=0x7f7fcc19ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.807+0000 7f7fca800700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fcc1045e0 0x7f7fcc19ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.807+0000 7f7fca800700 1 -- 172.21.15.12:0/463632733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc108f10 msgr2=0x7f7fcc19b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.807+0000 7f7fca800700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc108f10 0x7f7fcc19b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.807+0000 7f7fca800700 1 -- 172.21.15.12:0/463632733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fcc19fcb0 con 0x7f7fcc109860 2024-05-27T07:30:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.808+0000 7f7fca800700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc109860 0x7f7fcc19f610 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400d980 tx=0x7f7fc400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.808+0000 7f7fc2a00700 1 -- 172.21.15.12:0/463632733 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4017070 con 0x7f7fcc109860 2024-05-27T07:30:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.808+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fcc19ffa0 con 0x7f7fcc109860 2024-05-27T07:30:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.808+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fcc0759e0 con 0x7f7fcc109860 2024-05-27T07:30:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.808+0000 7f7fc2a00700 1 -- 172.21.15.12:0/463632733 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fc4009bc0 con 0x7f7fcc109860 2024-05-27T07:30:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.808+0000 7f7fc2a00700 1 -- 172.21.15.12:0/463632733 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4010b80 con 0x7f7fcc109860 2024-05-27T07:30:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.813+0000 7f7fc2a00700 1 -- 172.21.15.12:0/463632733 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7fc4009730 con 0x7f7fcc109860 2024-05-27T07:30:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.815+0000 7f7fc2a00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7fb00b2a70 0x7f7fb00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.815+0000 7f7fc2a00700 1 -- 172.21.15.12:0/463632733 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f7fc40e2690 con 0x7f7fcc109860 2024-05-27T07:30:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.815+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7fb00b2a70 0x7f7fb00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.815+0000 7f7fdf70db80 1 Processor -- start 2024-05-27T07:30:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.815+0000 7f7fdf70db80 1 -- start start 2024-05-27T07:30:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.816+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bbd19f0 0x55f05bc64330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.816+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc64870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.816+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bc64e70 0x55f05bc71110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.816+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f05bb92e40 con 0x55f05bc64e70 2024-05-27T07:30:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.816+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f05bb829e0 con 0x55f05bbd19f0 2024-05-27T07:30:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.816+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f05bb86a20 con 0x55f05bae5e40 2024-05-27T07:30:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bc64e70 0x55f05bc71110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bc64e70 0x55f05bc71110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32776/0 (socket says 172.21.15.12:32776) 2024-05-27T07:30:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 -- 172.21.15.12:0/2949613314 learned_addr learned my addr 172.21.15.12:0/2949613314 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fc9400700 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc64870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 -- 172.21.15.12:0/2949613314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 msgr2=0x55f05bc64870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc64870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 -- 172.21.15.12:0/2949613314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bbd19f0 msgr2=0x55f05bc64330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bbd19f0 0x55f05bc64330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fca800700 1 -- 172.21.15.12:0/2949613314 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f05bb4b420 con 0x55f05bc64e70 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.817+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7fb00b2a70 0x7f7fb00b4f80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0060b0 tx=0x7f7fbc007b80 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.818+0000 7f7fca800700 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bc64e70 0x55f05bc71110 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7fc40a5580 tx=0x7f7fc4009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e6904c7a7f3188 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.819+0000 7f7fb7400700 1 -- 172.21.15.12:0/2949613314 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc40910d0 con 0x55f05bc64e70 2024-05-27T07:30:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.819+0000 7f7fb7400700 1 -- 172.21.15.12:0/2949613314 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fc40a9500 con 0x55f05bc64e70 2024-05-27T07:30:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.819+0000 7f7fb7400700 1 -- 172.21.15.12:0/2949613314 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc40a99c0 con 0x55f05bc64e70 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.820+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2949613314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bc64e70 msgr2=0x55f05bc71110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.820+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bc64e70 0x55f05bc71110 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7fc40a5580 tx=0x7f7fc4009aa0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.820+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2949613314 shutdown_connections 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.820+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bbd19f0 0x55f05bc64330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.820+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bc64e70 0x55f05bc71110 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.820+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2949613314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc64870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.820+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2949613314 >> 172.21.15.12:0/2949613314 conn(0x55f05bb7f390 msgr2=0x55f05bc604c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.821+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2949613314 shutdown_connections 2024-05-27T07:30:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.821+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2949613314 wait complete. 2024-05-27T07:30:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.821+0000 7f7fdf70db80 1 -- start start 2024-05-27T07:30:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 1 Processor -- start 2024-05-27T07:30:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc6e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bbd19f0 0x55f05bc6eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc64e70 0x55f05bc6f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f05ba88fa0 con 0x55f05bbd19f0 2024-05-27T07:30:52.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f05bb23760 con 0x55f05bc64e70 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f05bc71b20 con 0x55f05bae5e40 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.822+0000 7f7fdf70db80 10 client.0 _ll_register_callbacks cb 0x55f05bb2cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bbd19f0 0x55f05bc6eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bbd19f0 0x55f05bc6eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32788/0 (socket says 172.21.15.12:32788) 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 -- 172.21.15.12:0/2023139578 learned_addr learned my addr 172.21.15.12:0/2023139578 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fca800700 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc64e70 0x55f05bc6f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc6e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 -- 172.21.15.12:0/2023139578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 msgr2=0x55f05bc6e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc6e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 -- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc64e70 msgr2=0x55f05bc6f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc64e70 0x55f05bc6f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.823+0000 7f7fc9400700 1 -- 172.21.15.12:0/2023139578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f05bad28d0 con 0x55f05bbd19f0 2024-05-27T07:30:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fc9400700 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bbd19f0 0x55f05bc6eb90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc06c670 tx=0x7f7fb8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1c7f51f56691d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fabe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fabe00700 1 -- 172.21.15.12:0/2023139578 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8003680 con 0x55f05bbd19f0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fabe00700 1 -- 172.21.15.12:0/2023139578 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f7fb8003820 con 0x55f05bbd19f0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fc40a9500 con 0x55f05bbd19f0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fabe00700 1 -- 172.21.15.12:0/2023139578 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f7fb8033050 con 0x55f05bbd19f0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fdf70db80 10 client.16482 fetch_fsmap learned FSMap version 118 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.824+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=4+}) -- 0x55f05bb53060 con 0x55f05bbd19f0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fabe00700 1 -- 172.21.15.12:0/2023139578 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fc40a9500 con 0x55f05bbd19f0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fabe00700 1 -- 172.21.15.12:0/2023139578 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 118) ==== 2941+0+0 (secure 0 0 0) 0x7f7fb800f990 con 0x55f05bbd19f0 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fdf70db80 10 client.16482 fetch_fsmap finished waiting for FSMap version 118 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fdf70db80 10 client.16482 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fdf70db80 20 client.16482 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bc8feb0 0x55f05bc92360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fdf70db80 4 client.16482 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.825+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f05bb94d90 con 0x55f05bc8feb0 2024-05-27T07:30:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.826+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bc8feb0 0x55f05bc92360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.827+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bc8feb0 0x55f05bc92360 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=82f44ba8dd59775 server_cookie=c3e34700ed0b2110 in_seq=0 out_seq=0 2024-05-27T07:30:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.827+0000 7f7fabe00700 10 client.16482 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:30:52.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.829+0000 7f7fabe00700 1 -- 172.21.15.12:0/2023139578 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f05bb94d90 con 0x55f05bc8feb0 2024-05-27T07:30:52.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.829+0000 7f7fabe00700 10 client.16482 handle_command_reply: tid=0 2024-05-27T07:30:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.829+0000 7f7fdf70db80 1 client.16482 shutdown 2024-05-27T07:30:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bbd19f0 msgr2=0x55f05bc6eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bbd19f0 0x55f05bc6eb90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc06c670 tx=0x7f7fb8009680 comp rx=0 tx=0).stop 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 shutdown_connections 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bc8feb0 0x55f05bc92360 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc64e70 0x55f05bc6f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bae5e40 0x55f05bc6e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/2023139578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bbd19f0 0x55f05bc6eb90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fabe00700 0 client.16482 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 >> 172.21.15.12:0/2023139578 conn(0x55f05bb7f390 msgr2=0x55f05bc643b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 shutdown_connections 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 1 -- 172.21.15.12:0/2023139578 wait complete. 2024-05-27T07:30:52.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.830+0000 7f7fdf70db80 20 client.16482 trim_cache size 0 max 16384 2024-05-27T07:30:52.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.851+0000 7f7fdf70db80 1 Processor -- start 2024-05-27T07:30:52.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.851+0000 7f7fdf70db80 1 -- start start 2024-05-27T07:30:52.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 0x55f05bc6d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc97b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac00e410 con 0x55f05bae5e40 2024-05-27T07:30:52.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac00d670 con 0x55f05bc60590 2024-05-27T07:30:52.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac00dd40 con 0x55f05bc64e70 2024-05-27T07:30:52.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32796/0 (socket says 172.21.15.12:32796) 2024-05-27T07:30:52.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.852+0000 7f7fc9e00700 1 -- 172.21.15.12:0/443338685 learned_addr learned my addr 172.21.15.12:0/443338685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fc9400700 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 0x55f05bc6d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fca800700 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc97b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fc9e00700 1 -- 172.21.15.12:0/443338685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 msgr2=0x55f05bc6d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 0x55f05bc6d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fc9e00700 1 -- 172.21.15.12:0/443338685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 msgr2=0x55f05bc97b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc97b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fc9e00700 1 -- 172.21.15.12:0/443338685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f05bc70ef0 con 0x55f05bae5e40 2024-05-27T07:30:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.853+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69e90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x55f05bc6ded0 tx=0x7f7fbc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a1b753d58fb6a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:52.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.854+0000 7f7fb6a00700 1 -- 172.21.15.12:0/443338685 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc025440 con 0x55f05bae5e40 2024-05-27T07:30:52.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.854+0000 7f7fb6a00700 1 -- 172.21.15.12:0/443338685 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fbc025a70 con 0x55f05bae5e40 2024-05-27T07:30:52.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.854+0000 7f7fb6a00700 1 -- 172.21.15.12:0/443338685 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc023bc0 con 0x55f05bae5e40 2024-05-27T07:30:52.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 -- 172.21.15.12:0/443338685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 msgr2=0x55f05bc69e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69e90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x55f05bc6ded0 tx=0x7f7fbc011000 comp rx=0 tx=0).stop 2024-05-27T07:30:52.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 -- 172.21.15.12:0/443338685 shutdown_connections 2024-05-27T07:30:52.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc97b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69e90 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/443338685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 0x55f05bc6d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 -- 172.21.15.12:0/443338685 >> 172.21.15.12:0/443338685 conn(0x55f05bb7f390 msgr2=0x55f05bc5fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:52.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 -- 172.21.15.12:0/443338685 shutdown_connections 2024-05-27T07:30:52.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.855+0000 7f7fdf70db80 1 -- 172.21.15.12:0/443338685 wait complete. 2024-05-27T07:30:52.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.856+0000 7f7fdf70db80 1 -- start start 2024-05-27T07:30:52.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.856+0000 7f7fdf70db80 1 Processor -- start 2024-05-27T07:30:52.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc69580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fdf70db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 0x55f05bc69ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f05bb94d90 con 0x55f05bae5e40 2024-05-27T07:30:52.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f05bc6e430 con 0x55f05bc60590 2024-05-27T07:30:52.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fdf70db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f05bc61000 con 0x55f05bc64e70 2024-05-27T07:30:52.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fdf70db80 10 client.0 _ll_register_callbacks cb 0x55f05bb2cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:52.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fc9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32812/0 (socket says 172.21.15.12:32812) 2024-05-27T07:30:52.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fc9e00700 1 -- 172.21.15.12:0/815741925 learned_addr learned my addr 172.21.15.12:0/815741925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:52.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fc9400700 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc69580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.857+0000 7f7fc9e00700 1 -- 172.21.15.12:0/815741925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 msgr2=0x55f05bc69ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 0x55f05bc69ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fc9e00700 1 -- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 msgr2=0x55f05bc69580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:52.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc69580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fc9e00700 1 -- 172.21.15.12:0/815741925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f05bc751a0 con 0x55f05bae5e40 2024-05-27T07:30:52.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fc9400700 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc69580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:52.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fc9e00700 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69040 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009010 tx=0x7f7fbc023520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a1f729304dccc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:52.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fabe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:30:52.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.858+0000 7f7fabe00700 1 -- 172.21.15.12:0/815741925 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc025640 con 0x55f05bae5e40 2024-05-27T07:30:52.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.859+0000 7f7fabe00700 1 -- 172.21.15.12:0/815741925 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f7fbc0257e0 con 0x55f05bae5e40 2024-05-27T07:30:52.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.859+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fbc025a70 con 0x55f05bae5e40 2024-05-27T07:30:52.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.859+0000 7f7fabe00700 1 -- 172.21.15.12:0/815741925 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f7fbc017040 con 0x55f05bae5e40 2024-05-27T07:30:52.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.859+0000 7f7fdf70db80 10 client.16488 fetch_fsmap learned FSMap version 118 2024-05-27T07:30:52.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.859+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=4+}) -- 0x55f05bcfaaf0 con 0x55f05bae5e40 2024-05-27T07:30:52.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.859+0000 7f7fabe00700 1 -- 172.21.15.12:0/815741925 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fbc025a70 con 0x55f05bae5e40 2024-05-27T07:30:52.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.859+0000 7f7fabe00700 1 -- 172.21.15.12:0/815741925 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 118) ==== 2941+0+0 (secure 0 0 0) 0x7f7fbc016670 con 0x55f05bae5e40 2024-05-27T07:30:52.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.860+0000 7f7fdf70db80 10 client.16488 fetch_fsmap finished waiting for FSMap version 118 2024-05-27T07:30:52.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.860+0000 7f7fdf70db80 10 client.16488 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:30:52.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.860+0000 7f7fdf70db80 20 client.16488 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:52.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.860+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bcfb1c0 0x55f05bcfd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:52.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.860+0000 7f7fdf70db80 4 client.16488 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:52.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.860+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f05bb8e570 con 0x55f05bcfb1c0 2024-05-27T07:30:52.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.860+0000 7f7fc9400700 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bcfb1c0 0x55f05bcfd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:52.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.862+0000 7f7fc9400700 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bcfb1c0 0x55f05bcfd5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70621c8ab0f8db0e server_cookie=a79952d867b97530 in_seq=0 out_seq=0 2024-05-27T07:30:52.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.862+0000 7f7fabe00700 10 client.16488 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:30:52.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.862+0000 7f7fabe00700 1 -- 172.21.15.12:0/815741925 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f05bb8e570 con 0x55f05bcfb1c0 2024-05-27T07:30:52.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fabe00700 10 client.16488 handle_command_reply: tid=0 2024-05-27T07:30:52.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 client.16488 shutdown 2024-05-27T07:30:52.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 msgr2=0x55f05bc69040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69040 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009010 tx=0x7f7fbc023520 comp rx=0 tx=0).stop 2024-05-27T07:30:52.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 shutdown_connections 2024-05-27T07:30:52.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f05bc64e70 0x55f05bc69ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f05bae5e40 0x55f05bc69040 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55f05bcfb1c0 0x55f05bcfd5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 --2- 172.21.15.12:0/815741925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f05bc60590 0x55f05bc69580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 >> 172.21.15.12:0/815741925 conn(0x55f05bb7f390 msgr2=0x55f05bc5fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:52.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fabe00700 0 client.16488 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:30:52.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 shutdown_connections 2024-05-27T07:30:52.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.863+0000 7f7fdf70db80 1 -- 172.21.15.12:0/815741925 wait complete. 2024-05-27T07:30:52.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fdf70db80 20 client.16488 trim_cache size 0 max 16384 2024-05-27T07:30:52.864 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:30:52.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7fb00b2a70 msgr2=0x7f7fb00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7fb00b2a70 0x7f7fb00b4f80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0060b0 tx=0x7f7fbc007b80 comp rx=0 tx=0).stop 2024-05-27T07:30:52.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc109860 msgr2=0x7f7fcc19f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:52.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc109860 0x7f7fcc19f610 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400d980 tx=0x7f7fc400dc90 comp rx=0 tx=0).stop 2024-05-27T07:30:52.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 shutdown_connections 2024-05-27T07:30:52.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f7fb00b2a70 0x7f7fb00b4f80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fcc109860 0x7f7fcc19f610 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fcc1045e0 0x7f7fcc19ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 --2- 172.21.15.12:0/463632733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc108f10 0x7f7fcc19b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:52.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.864+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 >> 172.21.15.12:0/463632733 conn(0x7f7fcc100030 msgr2=0x7f7fcc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:52.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.865+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 shutdown_connections 2024-05-27T07:30:52.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:52.865+0000 7f7fd0c00700 1 -- 172.21.15.12:0/463632733 wait complete. 2024-05-27T07:30:53.729 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:30:53.987 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:53 smithi062 ceph-mon[35304]: pgmap v301: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-05-27T07:30:54.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:53 smithi012 ceph-mon[31473]: pgmap v301: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-05-27T07:30:54.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:53 smithi162 ceph-mon[35676]: pgmap v301: 129 pgs: 129 active+clean; 126 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-05-27T07:30:54.232 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:56.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:55 smithi012 ceph-mon[31473]: pgmap v302: 129 pgs: 129 active+clean; 127 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-27T07:30:56.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:55 smithi162 ceph-mon[35676]: pgmap v302: 129 pgs: 129 active+clean; 127 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-27T07:30:56.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:55 smithi062 ceph-mon[35304]: pgmap v302: 129 pgs: 129 active+clean; 127 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-27T07:30:56.766 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:56 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:56.766 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:56 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:56.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.809+0000 7fa5bbe00700 1 -- 172.21.15.12:0/1207772130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 msgr2=0x7fa5bc108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.809+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/1207772130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc108010 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80099c0 tx=0x7fa5a8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.809+0000 7fa5bbe00700 1 -- 172.21.15.12:0/1207772130 shutdown_connections 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.809+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/1207772130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5bc1032d0 0x7fa5bc1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.809+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/1207772130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5bc108550 0x7fa5bc10feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.809+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/1207772130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc108010 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.809+0000 7fa5bbe00700 1 -- 172.21.15.12:0/1207772130 >> 172.21.15.12:0/1207772130 conn(0x7fa5bc078ab0 msgr2=0x7fa5bc078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.810+0000 7fa5bbe00700 1 -- 172.21.15.12:0/1207772130 shutdown_connections 2024-05-27T07:30:56.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.810+0000 7fa5bbe00700 1 -- 172.21.15.12:0/1207772130 wait complete. 2024-05-27T07:30:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.811+0000 7fa5bbe00700 1 Processor -- start 2024-05-27T07:30:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.811+0000 7fa5bbe00700 1 -- start start 2024-05-27T07:30:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.811+0000 7fa5bbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5bc1032d0 0x7fa5bc1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.811+0000 7fa5bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5bbe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5bc108550 0x7fa5bc19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5bbe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc1141d0 con 0x7fa5bc107c00 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5bbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc114050 con 0x7fa5bc1032d0 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5bbe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc114350 con 0x7fa5bc108550 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5ba000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5ba000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc1a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32818/0 (socket says 172.21.15.12:32818) 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5ba000700 1 -- 172.21.15.12:0/295481526 learned_addr learned my addr 172.21.15.12:0/295481526 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5bb400700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5bc108550 0x7fa5bc19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.812+0000 7fa5baa00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5bc1032d0 0x7fa5bc1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5ba000700 1 -- 172.21.15.12:0/295481526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5bc108550 msgr2=0x7fa5bc19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5ba000700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5bc108550 0x7fa5bc19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5ba000700 1 -- 172.21.15.12:0/295481526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5bc1032d0 msgr2=0x7fa5bc1a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5ba000700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5bc1032d0 0x7fa5bc1a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5ba000700 1 -- 172.21.15.12:0/295481526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5bc19b030 con 0x7fa5bc107c00 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5bb400700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5bc108550 0x7fa5bc19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5baa00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5bc1032d0 0x7fa5bc1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.813+0000 7fa5ba000700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc1a0d10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800b580 tx=0x7fa5a800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.814+0000 7fa5b3400700 1 -- 172.21.15.12:0/295481526 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8027070 con 0x7fa5bc107c00 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.814+0000 7fa5b3400700 1 -- 172.21.15.12:0/295481526 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5a800fda0 con 0x7fa5bc107c00 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.814+0000 7fa5b3400700 1 -- 172.21.15.12:0/295481526 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80173f0 con 0x7fa5bc107c00 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.814+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5bc19b2c0 con 0x7fa5bc107c00 2024-05-27T07:30:56.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.814+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5bc19b750 con 0x7fa5bc107c00 2024-05-27T07:30:56.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.819+0000 7fa5b3400700 1 -- 172.21.15.12:0/295481526 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa5a802a070 con 0x7fa5bc107c00 2024-05-27T07:30:56.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.820+0000 7fa5d06b5b80 1 Processor -- start 2024-05-27T07:30:56.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.820+0000 7fa5d06b5b80 1 -- start start 2024-05-27T07:30:56.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.820+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e5a89f0 0x56054e63c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.820+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e4bce40 0x56054e63c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.820+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 0x56054e64c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.820+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56054e569e40 con 0x56054e4bce40 2024-05-27T07:30:56.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.820+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56054e5599e0 con 0x56054e5a89f0 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56054e55da20 con 0x56054e63cf90 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5bb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e5a89f0 0x56054e63c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5bb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e5a89f0 0x56054e63c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41782/0 (socket says 172.21.15.12:41782) 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5bb400700 1 -- 172.21.15.12:0/476333751 learned_addr learned my addr 172.21.15.12:0/476333751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5bb400700 1 -- 172.21.15.12:0/476333751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 msgr2=0x56054e64c6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5bb400700 1 --2- 172.21.15.12:0/476333751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 0x56054e64c6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5bb400700 1 -- 172.21.15.12:0/476333751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e4bce40 msgr2=0x56054e63c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:56.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.821+0000 7fa5bb400700 1 --2- 172.21.15.12:0/476333751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e4bce40 0x56054e63c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.822+0000 7fa5bb400700 1 -- 172.21.15.12:0/476333751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56054e522420 con 0x56054e5a89f0 2024-05-27T07:30:56.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.822+0000 7fa5bb400700 1 --2- 172.21.15.12:0/476333751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e5a89f0 0x56054e63c470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e160 tx=0x7fa5b400e470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0da620ddf514b3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:56.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.822+0000 7fa5a7e00700 1 -- 172.21.15.12:0/476333751 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400edb0 con 0x56054e5a89f0 2024-05-27T07:30:56.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.823+0000 7fa5a7e00700 1 -- 172.21.15.12:0/476333751 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b4004bb0 con 0x56054e5a89f0 2024-05-27T07:30:56.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.823+0000 7fa5a7e00700 1 -- 172.21.15.12:0/476333751 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4004420 con 0x56054e5a89f0 2024-05-27T07:30:56.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.823+0000 7fa5b3400700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a00b2b40 0x7fa5a00b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5b3400700 1 -- 172.21.15.12:0/295481526 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fa5a80e38f0 con 0x7fa5bc107c00 2024-05-27T07:30:56.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/476333751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e5a89f0 msgr2=0x56054e63c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/476333751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e5a89f0 0x56054e63c470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e160 tx=0x7fa5b400e470 comp rx=0 tx=0).stop 2024-05-27T07:30:56.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5baa00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a00b2b40 0x7fa5a00b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/476333751 shutdown_connections 2024-05-27T07:30:56.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/476333751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e5a89f0 0x56054e63c470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/476333751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 0x56054e64c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/476333751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e4bce40 0x56054e63c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.824+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/476333751 >> 172.21.15.12:0/476333751 conn(0x56054e556390 msgr2=0x56054e637710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:56.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.825+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/476333751 shutdown_connections 2024-05-27T07:30:56.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.825+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/476333751 wait complete. 2024-05-27T07:30:56.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.825+0000 7fa5baa00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a00b2b40 0x7fa5a00b5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x56054e638ba0 tx=0x7fa5ac00b2e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:56.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.826+0000 7fa5d06b5b80 1 -- start start 2024-05-27T07:30:56.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.826+0000 7fa5d06b5b80 1 Processor -- start 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e649d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e64a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 0x56054e64a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56054e45ffa0 con 0x56054e5a89f0 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56054e4fa760 con 0x56054e4bce40 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56054e64d130 con 0x56054e63cf90 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5d06b5b80 10 client.0 _ll_register_callbacks cb 0x56054e503c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:56.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e649d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e649d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41790/0 (socket says 172.21.15.12:41790) 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.827+0000 7fa5baa00700 1 -- 172.21.15.12:0/3524274947 learned_addr learned my addr 172.21.15.12:0/3524274947 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5ba000700 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e64a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5baa00700 1 -- 172.21.15.12:0/3524274947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 msgr2=0x56054e64a790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5baa00700 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 0x56054e64a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5baa00700 1 -- 172.21.15.12:0/3524274947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 msgr2=0x56054e64a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5baa00700 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e64a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5baa00700 1 -- 172.21.15.12:0/3524274947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56054e4a98d0 con 0x56054e4bce40 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5baa00700 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e649d10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00fe30 tx=0x7fa5ac0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd33e4475ed3f625 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.828+0000 7fa5a4c00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.829+0000 7fa5a4c00700 1 -- 172.21.15.12:0/3524274947 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac022070 con 0x56054e4bce40 2024-05-27T07:30:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.829+0000 7fa5a4c00700 1 -- 172.21.15.12:0/3524274947 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fa5ac01e650 con 0x56054e4bce40 2024-05-27T07:30:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.829+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b4004bb0 con 0x56054e4bce40 2024-05-27T07:30:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.829+0000 7fa5a4c00700 1 -- 172.21.15.12:0/3524274947 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa5ac01e850 con 0x56054e4bce40 2024-05-27T07:30:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.829+0000 7fa5d06b5b80 10 client.36029 fetch_fsmap learned FSMap version 119 2024-05-27T07:30:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.829+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56054e52a060 con 0x56054e4bce40 2024-05-27T07:30:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5a4c00700 1 -- 172.21.15.12:0/3524274947 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b4004bb0 con 0x56054e4bce40 2024-05-27T07:30:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5a4c00700 1 -- 172.21.15.12:0/3524274947 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa5ac026890 con 0x56054e4bce40 2024-05-27T07:30:56.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5d06b5b80 10 client.36029 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5d06b5b80 10 client.36029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5d06b5b80 20 client.36029 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5d06b5b80 4 client.36029 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.830+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56054e56bd90 con 0x56054e6410e0 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.831+0000 7fa5ba000700 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.833+0000 7fa5ba000700 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6415b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f74f52eb147b792 server_cookie=51dceba13670da1b in_seq=0 out_seq=0 2024-05-27T07:30:56.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.833+0000 7fa5a4c00700 10 client.36029 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:30:56.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.834+0000 7fa5a4c00700 1 -- 172.21.15.12:0/3524274947 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56054e56bd90 con 0x56054e6410e0 2024-05-27T07:30:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.834+0000 7fa5a4c00700 10 client.36029 handle_command_reply: tid=0 2024-05-27T07:30:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.835+0000 7fa5d06b5b80 1 client.36029 shutdown 2024-05-27T07:30:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.835+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 msgr2=0x56054e649d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.835+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e649d10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00fe30 tx=0x7fa5ac0047c0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.835+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 shutdown_connections 2024-05-27T07:30:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.835+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6415b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5a4c00700 0 client.36029 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:30:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e649d10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e63cf90 0x56054e64a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/3524274947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e64a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 >> 172.21.15.12:0/3524274947 conn(0x56054e556390 msgr2=0x56054e63c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 shutdown_connections 2024-05-27T07:30:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/3524274947 wait complete. 2024-05-27T07:30:56.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.836+0000 7fa5d06b5b80 20 client.36029 trim_cache size 0 max 16384 2024-05-27T07:30:56.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.869+0000 7fa5d06b5b80 1 Processor -- start 2024-05-27T07:30:56.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.870+0000 7fa5d06b5b80 1 -- start start 2024-05-27T07:30:56.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.870+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e4bce40 0x56054e6454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.870+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e637740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.870+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e63cf90 0x56054e656400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.870+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc1ad6a0 con 0x56054e5a89f0 2024-05-27T07:30:56.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.870+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc1ac900 con 0x56054e63cf90 2024-05-27T07:30:56.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.870+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc1acfd0 con 0x56054e4bce40 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5ba000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e637740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e63cf90 0x56054e656400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e63cf90 0x56054e656400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41800/0 (socket says 172.21.15.12:41800) 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 -- 172.21.15.12:0/2835984112 learned_addr learned my addr 172.21.15.12:0/2835984112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5baa00700 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e4bce40 0x56054e6454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 -- 172.21.15.12:0/2835984112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e4bce40 msgr2=0x56054e6454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e4bce40 0x56054e6454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 -- 172.21.15.12:0/2835984112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 msgr2=0x56054e637740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e637740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.871+0000 7fa5bb400700 1 -- 172.21.15.12:0/2835984112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56054e64ab00 con 0x56054e63cf90 2024-05-27T07:30:56.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.872+0000 7fa5bb400700 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e63cf90 0x56054e656400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007740 tx=0x7fa5b4007d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8552e594f573f05f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:56.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.872+0000 7fa5a7400700 1 -- 172.21.15.12:0/2835984112 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4022030 con 0x56054e63cf90 2024-05-27T07:30:56.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.872+0000 7fa5a7400700 1 -- 172.21.15.12:0/2835984112 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b4003a00 con 0x56054e63cf90 2024-05-27T07:30:56.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.873+0000 7fa5a7400700 1 -- 172.21.15.12:0/2835984112 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4004290 con 0x56054e63cf90 2024-05-27T07:30:56.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.874+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2835984112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e63cf90 msgr2=0x56054e656400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.874+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e63cf90 0x56054e656400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007740 tx=0x7fa5b4007d60 comp rx=0 tx=0).stop 2024-05-27T07:30:56.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.876+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2835984112 shutdown_connections 2024-05-27T07:30:56.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.876+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e63cf90 0x56054e656400 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.876+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e4bce40 0x56054e6454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.876+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2835984112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e5a89f0 0x56054e637740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.876+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2835984112 >> 172.21.15.12:0/2835984112 conn(0x56054e556390 msgr2=0x56054e63b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:56.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.877+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2835984112 shutdown_connections 2024-05-27T07:30:56.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.877+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2835984112 wait complete. 2024-05-27T07:30:56.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.878+0000 7fa5d06b5b80 1 -- start start 2024-05-27T07:30:56.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.879+0000 7fa5d06b5b80 1 Processor -- start 2024-05-27T07:30:56.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.879+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e667190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.879+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e5a89f0 0x56054e6676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.879+0000 7fa5d06b5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e63cf90 0x56054e661640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.879+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56054e64ae20 con 0x56054e63cf90 2024-05-27T07:30:56.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.879+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56054e56bd90 con 0x56054e4bce40 2024-05-27T07:30:56.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.879+0000 7fa5d06b5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56054e649030 con 0x56054e5a89f0 2024-05-27T07:30:56.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5d06b5b80 10 client.0 _ll_register_callbacks cb 0x56054e503c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:56.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5bb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e63cf90 0x56054e661640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e667190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e667190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41804/0 (socket says 172.21.15.12:41804) 2024-05-27T07:30:56.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5baa00700 1 -- 172.21.15.12:0/2906995720 learned_addr learned my addr 172.21.15.12:0/2906995720 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:30:56.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5baa00700 1 -- 172.21.15.12:0/2906995720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e5a89f0 msgr2=0x56054e6676d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:30:56.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5baa00700 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e5a89f0 0x56054e6676d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.880+0000 7fa5baa00700 1 -- 172.21.15.12:0/2906995720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e63cf90 msgr2=0x56054e661640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:56.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.881+0000 7fa5baa00700 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e63cf90 0x56054e661640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.881+0000 7fa5baa00700 1 -- 172.21.15.12:0/2906995720 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56054e661de0 con 0x56054e4bce40 2024-05-27T07:30:56.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.881+0000 7fa5bb400700 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e63cf90 0x56054e661640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:56.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.881+0000 7fa5baa00700 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e667190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac000c00 tx=0x7fa5ac026f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87c0c9e0e53ee38 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:56.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.881+0000 7fa5a4c00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:30:56.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.882+0000 7fa5a4c00700 1 -- 172.21.15.12:0/2906995720 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac036030 con 0x56054e4bce40 2024-05-27T07:30:56.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.882+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b4003a00 con 0x56054e4bce40 2024-05-27T07:30:56.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.882+0000 7fa5a4c00700 1 -- 172.21.15.12:0/2906995720 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fa5ac01ec00 con 0x56054e4bce40 2024-05-27T07:30:56.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.883+0000 7fa5a4c00700 1 -- 172.21.15.12:0/2906995720 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa5ac042050 con 0x56054e4bce40 2024-05-27T07:30:56.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.883+0000 7fa5d06b5b80 10 client.36035 fetch_fsmap learned FSMap version 119 2024-05-27T07:30:56.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.883+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56054e6d9870 con 0x56054e4bce40 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.884+0000 7fa5a4c00700 1 -- 172.21.15.12:0/2906995720 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b4003a00 con 0x56054e4bce40 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.884+0000 7fa5a4c00700 1 -- 172.21.15.12:0/2906995720 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa5ac026690 con 0x56054e4bce40 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.884+0000 7fa5d06b5b80 10 client.36035 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.884+0000 7fa5d06b5b80 10 client.36035 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.884+0000 7fa5d06b5b80 20 client.36035 populate_metadata read hostname 'smithi012' 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.885+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6dbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.885+0000 7fa5d06b5b80 4 client.36035 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:56.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.885+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56054e565570 con 0x56054e6410e0 2024-05-27T07:30:56.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.886+0000 7fa5ba000700 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6dbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:56.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.887+0000 7fa5ba000700 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6dbec0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f13d1e9d82593b9d server_cookie=61f56a476654ba3f in_seq=0 out_seq=0 2024-05-27T07:30:56.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.887+0000 7fa5a4c00700 10 client.36035 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:30:56.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5a4c00700 1 -- 172.21.15.12:0/2906995720 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56054e565570 con 0x56054e6410e0 2024-05-27T07:30:56.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5a4c00700 10 client.36035 handle_command_reply: tid=0 2024-05-27T07:30:56.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 client.36035 shutdown 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 msgr2=0x56054e667190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e667190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac000c00 tx=0x7fa5ac026f20 comp rx=0 tx=0).stop 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 shutdown_connections 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56054e6410e0 0x56054e6dbec0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56054e4bce40 0x56054e667190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56054e63cf90 0x56054e661640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 --2- 172.21.15.12:0/2906995720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56054e5a89f0 0x56054e6676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.888+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 >> 172.21.15.12:0/2906995720 conn(0x56054e556390 msgr2=0x56054e649af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:56.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.889+0000 7fa5a4c00700 0 client.36035 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:30:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.891+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 shutdown_connections 2024-05-27T07:30:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.891+0000 7fa5d06b5b80 1 -- 172.21.15.12:0/2906995720 wait complete. 2024-05-27T07:30:56.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.892+0000 7fa5d06b5b80 20 client.36035 trim_cache size 0 max 16384 2024-05-27T07:30:56.884 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:30:56.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a00b2b40 msgr2=0x7fa5a00b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a00b2b40 0x7fa5a00b5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x56054e638ba0 tx=0x7fa5ac00b2e0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 msgr2=0x7fa5bc1a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:56.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc1a0d10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800b580 tx=0x7fa5a800f460 comp rx=0 tx=0).stop 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 shutdown_connections 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a00b2b40 0x7fa5a00b5050 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5bc1032d0 0x7fa5bc1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5bc108550 0x7fa5bc19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 --2- 172.21.15.12:0/295481526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5bc107c00 0x7fa5bc1a0d10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 >> 172.21.15.12:0/295481526 conn(0x7fa5bc078ab0 msgr2=0x7fa5bc1019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 shutdown_connections 2024-05-27T07:30:56.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:30:56.893+0000 7fa5bbe00700 1 -- 172.21.15.12:0/295481526 wait complete. 2024-05-27T07:30:57.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:56 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:57.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:56 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:57.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:56 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:30:57.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:56 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:58.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:57 smithi012 ceph-mon[31473]: pgmap v303: 129 pgs: 129 active+clean; 127 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 114 op/s 2024-05-27T07:30:58.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:57 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:58.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:57 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:58.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:57 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:30:58.209 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:30:58.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:57 smithi162 ceph-mon[35676]: pgmap v303: 129 pgs: 129 active+clean; 127 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 114 op/s 2024-05-27T07:30:58.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:57 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:58.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:57 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:58.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:57 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:30:58.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:57 smithi062 ceph-mon[35304]: pgmap v303: 129 pgs: 129 active+clean; 127 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 114 op/s 2024-05-27T07:30:58.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:57 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:58.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:57 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:58.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:57 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:30:58.693 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:00.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:30:59 smithi012 ceph-mon[31473]: pgmap v304: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 176 op/s 2024-05-27T07:31:00.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:30:59 smithi162 ceph-mon[35676]: pgmap v304: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 176 op/s 2024-05-27T07:31:00.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:30:59 smithi062 ceph-mon[35304]: pgmap v304: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 176 op/s 2024-05-27T07:31:01.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 -- 172.21.15.12:0/2716647309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 msgr2=0x7f999810c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 --2- 172.21.15.12:0/2716647309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 0x7f999810c9b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f9988009990 tx=0x7f9988009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 -- 172.21.15.12:0/2716647309 shutdown_connections 2024-05-27T07:31:01.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 --2- 172.21.15.12:0/2716647309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106eb0 0x7f9998113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 --2- 172.21.15.12:0/2716647309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 0x7f999810c9b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 --2- 172.21.15.12:0/2716647309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9998106560 0x7f9998106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 -- 172.21.15.12:0/2716647309 >> 172.21.15.12:0/2716647309 conn(0x7f9998102040 msgr2=0x7f9998104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:01.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 -- 172.21.15.12:0/2716647309 shutdown_connections 2024-05-27T07:31:01.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.236+0000 7f999e800700 1 -- 172.21.15.12:0/2716647309 wait complete. 2024-05-27T07:31:01.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.238+0000 7f999e800700 1 Processor -- start 2024-05-27T07:31:01.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.240+0000 7f999e800700 1 -- start start 2024-05-27T07:31:01.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.240+0000 7f999e800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106560 0x7f9998110770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.240+0000 7f999e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9998106eb0 0x7f9998110cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.240+0000 7f999e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 0x7f99981111f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.241+0000 7f999e800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9998116630 con 0x7f999810c5a0 2024-05-27T07:31:01.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.241+0000 7f999e800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99981164b0 con 0x7f9998106eb0 2024-05-27T07:31:01.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.241+0000 7f999e800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99981167b0 con 0x7f9998106560 2024-05-27T07:31:01.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.241+0000 7f9997400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106560 0x7f9998110770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.241+0000 7f9997400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106560 0x7f9998110770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:53926/0 (socket says 172.21.15.12:53926) 2024-05-27T07:31:01.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.241+0000 7f9997400700 1 -- 172.21.15.12:0/1923192833 learned_addr learned my addr 172.21.15.12:0/1923192833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:01.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.242+0000 7f9997e00700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 0x7f99981111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.242+0000 7f998fe00700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9998106eb0 0x7f9998110cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.242+0000 7f9997400700 1 -- 172.21.15.12:0/1923192833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9998106eb0 msgr2=0x7f9998110cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.242+0000 7f9997400700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9998106eb0 0x7f9998110cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.242+0000 7f9997400700 1 -- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 msgr2=0x7f99981111f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.242+0000 7f9997400700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 0x7f99981111f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.242+0000 7f9997400700 1 -- 172.21.15.12:0/1923192833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99981ad530 con 0x7f9998106560 2024-05-27T07:31:01.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.243+0000 7f9997400700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106560 0x7f9998110770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f998800b5c0 tx=0x7f99880047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:01.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.243+0000 7f9994c00700 1 -- 172.21.15.12:0/1923192833 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988004830 con 0x7f9998106560 2024-05-27T07:31:01.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.244+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99981ad7c0 con 0x7f9998106560 2024-05-27T07:31:01.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.244+0000 7f9994c00700 1 -- 172.21.15.12:0/1923192833 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9988005020 con 0x7f9998106560 2024-05-27T07:31:01.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.244+0000 7f9994c00700 1 -- 172.21.15.12:0/1923192833 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998800f950 con 0x7f9998106560 2024-05-27T07:31:01.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.244+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99981adc90 con 0x7f9998106560 2024-05-27T07:31:01.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.249+0000 7f9994c00700 1 -- 172.21.15.12:0/1923192833 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f99880049d0 con 0x7f9998106560 2024-05-27T07:31:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.249+0000 7f99ad435b80 1 Processor -- start 2024-05-27T07:31:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.250+0000 7f99ad435b80 1 -- start start 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.250+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118cdc99f0 0x56118ce5c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.250+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce5c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.250+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce69110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.250+0000 7f99ad435b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56118cd8ae40 con 0x56118cdc99f0 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.250+0000 7f99ad435b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56118cd7a9e0 con 0x56118ccdde40 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.250+0000 7f99ad435b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56118cd7ea20 con 0x56118ce5ce70 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118cdc99f0 0x56118ce5c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118cdc99f0 0x56118ce5c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32878/0 (socket says 172.21.15.12:32878) 2024-05-27T07:31:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 -- 172.21.15.12:0/2503230561 learned_addr learned my addr 172.21.15.12:0/2503230561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:01.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997400700 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce5c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f998fe00700 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce69110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 -- 172.21.15.12:0/2503230561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 msgr2=0x56118ce69110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:01.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce69110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 -- 172.21.15.12:0/2503230561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 msgr2=0x56118ce5c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:01.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce5c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997e00700 1 -- 172.21.15.12:0/2503230561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56118cd43420 con 0x56118cdc99f0 2024-05-27T07:31:01.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9994c00700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f997c0b29f0 0x7f997c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f998fe00700 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce69110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:01.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.251+0000 7f9997400700 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce5c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:01.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.252+0000 7f9994c00700 1 -- 172.21.15.12:0/1923192833 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f99880e3990 con 0x7f9998106560 2024-05-27T07:31:01.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.252+0000 7f9997e00700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f997c0b29f0 0x7f997c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.252+0000 7f9997e00700 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118cdc99f0 0x56118ce5c330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9990009f20 tx=0x7f9990009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c758c55c0caa7d90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:01.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.252+0000 7f998cc00700 1 -- 172.21.15.12:0/2503230561 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999000c250 con 0x56118cdc99f0 2024-05-27T07:31:01.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.253+0000 7f998cc00700 1 -- 172.21.15.12:0/2503230561 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99900168b0 con 0x56118cdc99f0 2024-05-27T07:31:01.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.253+0000 7f998cc00700 1 -- 172.21.15.12:0/2503230561 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999000aad0 con 0x56118cdc99f0 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 -- 172.21.15.12:0/2503230561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118cdc99f0 msgr2=0x56118ce5c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118cdc99f0 0x56118ce5c330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9990009f20 tx=0x7f9990009f50 comp rx=0 tx=0).stop 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f9997e00700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f997c0b29f0 0x7f997c0b4f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f999001d000 tx=0x7f999000ae60 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 -- 172.21.15.12:0/2503230561 shutdown_connections 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce69110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118cdc99f0 0x56118ce5c330 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2503230561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce5c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 -- 172.21.15.12:0/2503230561 >> 172.21.15.12:0/2503230561 conn(0x56118cd77390 msgr2=0x56118ce584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 -- 172.21.15.12:0/2503230561 shutdown_connections 2024-05-27T07:31:01.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.254+0000 7f99ad435b80 1 -- 172.21.15.12:0/2503230561 wait complete. 2024-05-27T07:31:01.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.255+0000 7f99ad435b80 1 -- start start 2024-05-27T07:31:01.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 1 Processor -- start 2024-05-27T07:31:01.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce66680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce66bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ce5ce70 0x56118ce67100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56118cc80fa0 con 0x56118ccdde40 2024-05-27T07:31:01.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56118cd1b760 con 0x56118ce5ce70 2024-05-27T07:31:01.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56118ce69b80 con 0x56118cdc99f0 2024-05-27T07:31:01.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f99ad435b80 10 client.0 _ll_register_callbacks cb 0x56118cd24c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.256+0000 7f9997400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce66680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce66680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32884/0 (socket says 172.21.15.12:32884) 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 -- 172.21.15.12:0/657221941 learned_addr learned my addr 172.21.15.12:0/657221941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997e00700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ce5ce70 0x56118ce67100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f998fe00700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce66bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 -- 172.21.15.12:0/657221941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 msgr2=0x56118ce66bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce66bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 -- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ce5ce70 msgr2=0x56118ce67100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ce5ce70 0x56118ce67100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 -- 172.21.15.12:0/657221941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56118ccca8d0 con 0x56118ccdde40 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997e00700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ce5ce70 0x56118ce67100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.257+0000 7f9997400700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce66680 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9988000c00 tx=0x7f9988009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd148692290e05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.258+0000 7f9985600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:01.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.258+0000 7f9985600700 1 -- 172.21.15.12:0/657221941 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99880a6080 con 0x56118ccdde40 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.258+0000 7f9985600700 1 -- 172.21.15.12:0/657221941 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f99880aab50 con 0x56118ccdde40 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.258+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99900168b0 con 0x56118ccdde40 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.258+0000 7f9985600700 1 -- 172.21.15.12:0/657221941 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f99880095c0 con 0x56118ccdde40 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.258+0000 7f99ad435b80 10 client.16515 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.258+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56118cd4b060 con 0x56118ccdde40 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f9985600700 1 -- 172.21.15.12:0/657221941 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99900168b0 con 0x56118ccdde40 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f9985600700 1 -- 172.21.15.12:0/657221941 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f99880b3230 con 0x56118ccdde40 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f99ad435b80 10 client.16515 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f99ad435b80 10 client.16515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f99ad435b80 20 client.16515 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f99ad435b80 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118ce87b20 0x56118ce89f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f99ad435b80 4 client.16515 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:01.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56118cd8cd90 con 0x56118ce87b20 2024-05-27T07:31:01.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.259+0000 7f998fe00700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118ce87b20 0x56118ce89f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.261+0000 7f998fe00700 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118ce87b20 0x56118ce89f90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ac702e094001383 server_cookie=316ba999e0e98a82 in_seq=0 out_seq=0 2024-05-27T07:31:01.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.261+0000 7f9985600700 10 client.16515 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f9985600700 1 -- 172.21.15.12:0/657221941 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56118cd8cd90 con 0x56118ce87b20 2024-05-27T07:31:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f9985600700 10 client.16515 handle_command_reply: tid=0 2024-05-27T07:31:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 client.16515 shutdown 2024-05-27T07:31:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 msgr2=0x56118ce66680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce66680 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9988000c00 tx=0x7f9988009310 comp rx=0 tx=0).stop 2024-05-27T07:31:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 shutdown_connections 2024-05-27T07:31:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118ce87b20 0x56118ce89f90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ce5ce70 0x56118ce67100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce66680 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 --2- 172.21.15.12:0/657221941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce66bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.263+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 >> 172.21.15.12:0/657221941 conn(0x56118cd77390 msgr2=0x56118ce5c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.264+0000 7f9985600700 0 client.16515 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.264+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 shutdown_connections 2024-05-27T07:31:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.264+0000 7f99ad435b80 1 -- 172.21.15.12:0/657221941 wait complete. 2024-05-27T07:31:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.264+0000 7f99ad435b80 20 client.16515 trim_cache size 0 max 16384 2024-05-27T07:31:01.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.285+0000 7f99ad435b80 1 Processor -- start 2024-05-27T07:31:01.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.285+0000 7f99ad435b80 1 -- start start 2024-05-27T07:31:01.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.285+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce61e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.285+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce5c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ce5ce70 0x56118ce8f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f99ad435b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99981a3e70 con 0x56118ce5ce70 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f99ad435b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99981a30d0 con 0x56118ccdde40 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f99ad435b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99981a37a0 con 0x56118cdc99f0 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f9997e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ce5ce70 0x56118ce8f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f9997e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ce5ce70 0x56118ce8f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32900/0 (socket says 172.21.15.12:32900) 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f9997e00700 1 -- 172.21.15.12:0/367845855 learned_addr learned my addr 172.21.15.12:0/367845855 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f9997400700 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce61e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.286+0000 7f998fe00700 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce5c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.287+0000 7f9997e00700 1 -- 172.21.15.12:0/367845855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 msgr2=0x56118ce5c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.287+0000 7f9997e00700 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce5c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.287+0000 7f9997e00700 1 -- 172.21.15.12:0/367845855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 msgr2=0x56118ce61e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.287+0000 7f9997e00700 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce61e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.287+0000 7f9997e00700 1 -- 172.21.15.12:0/367845855 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56118ce87900 con 0x56118ce5ce70 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.287+0000 7f9997e00700 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ce5ce70 0x56118ce8f790 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9990009f90 tx=0x7f9990026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5ae5d3bfe4bab4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.288+0000 7f9987e00700 1 -- 172.21.15.12:0/367845855 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990031070 con 0x56118ce5ce70 2024-05-27T07:31:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.288+0000 7f9987e00700 1 -- 172.21.15.12:0/367845855 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f999000c250 con 0x56118ce5ce70 2024-05-27T07:31:01.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.288+0000 7f9987e00700 1 -- 172.21.15.12:0/367845855 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999000a760 con 0x56118ce5ce70 2024-05-27T07:31:01.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 -- 172.21.15.12:0/367845855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ce5ce70 msgr2=0x56118ce8f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ce5ce70 0x56118ce8f790 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9990009f90 tx=0x7f9990026000 comp rx=0 tx=0).stop 2024-05-27T07:31:01.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 -- 172.21.15.12:0/367845855 shutdown_connections 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118ccdde40 0x56118ce61e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ce5ce70 0x56118ce8f790 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 --2- 172.21.15.12:0/367845855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118cdc99f0 0x56118ce5c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 -- 172.21.15.12:0/367845855 >> 172.21.15.12:0/367845855 conn(0x56118cd77390 msgr2=0x56118ce5c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 -- 172.21.15.12:0/367845855 shutdown_connections 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.289+0000 7f99ad435b80 1 -- 172.21.15.12:0/367845855 wait complete. 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.290+0000 7f99ad435b80 1 -- start start 2024-05-27T07:31:01.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.290+0000 7f99ad435b80 1 Processor -- start 2024-05-27T07:31:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118cdc99f0 0x56118ce61700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f99ad435b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce61c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f99ad435b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56118ce59600 con 0x56118ccdde40 2024-05-27T07:31:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f99ad435b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56118cd8cd90 con 0x56118cdc99f0 2024-05-27T07:31:01.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f99ad435b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56118ce72c20 con 0x56118ce5ce70 2024-05-27T07:31:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f99ad435b80 10 client.0 _ll_register_callbacks cb 0x56118cd24c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f9997400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f9997400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32914/0 (socket says 172.21.15.12:32914) 2024-05-27T07:31:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f9997400700 1 -- 172.21.15.12:0/2211429596 learned_addr learned my addr 172.21.15.12:0/2211429596 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.291+0000 7f998fe00700 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118cdc99f0 0x56118ce61700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.292+0000 7f9997400700 1 -- 172.21.15.12:0/2211429596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 msgr2=0x56118ce61c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.292+0000 7f9997400700 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce61c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.292+0000 7f9997400700 1 -- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118cdc99f0 msgr2=0x56118ce61700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.292+0000 7f9997400700 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118cdc99f0 0x56118ce61700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.292+0000 7f9997400700 1 -- 172.21.15.12:0/2211429596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56118ce67470 con 0x56118ccdde40 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.292+0000 7f9997400700 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce611c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f99880a98f0 tx=0x7f99880aad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2003b2ac25804ddb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.292+0000 7f9985600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.293+0000 7f9985600700 1 -- 172.21.15.12:0/2211429596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99880a6080 con 0x56118ccdde40 2024-05-27T07:31:01.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.293+0000 7f9985600700 1 -- 172.21.15.12:0/2211429596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f99880a9b50 con 0x56118ccdde40 2024-05-27T07:31:01.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.293+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f999000c250 con 0x56118ccdde40 2024-05-27T07:31:01.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.293+0000 7f9985600700 1 -- 172.21.15.12:0/2211429596 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f99880051d0 con 0x56118ccdde40 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.293+0000 7f99ad435b80 10 client.16521 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.293+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56118ce54240 con 0x56118ccdde40 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f9985600700 1 -- 172.21.15.12:0/2211429596 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f999000c250 con 0x56118ccdde40 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f9985600700 1 -- 172.21.15.12:0/2211429596 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f998800faf0 con 0x56118ccdde40 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f99ad435b80 10 client.16521 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f99ad435b80 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f99ad435b80 20 client.16521 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118cef2860 0x56118cef4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f99ad435b80 4 client.16521 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:01.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.294+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56118cd86570 con 0x56118cef2860 2024-05-27T07:31:01.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.295+0000 7f998fe00700 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118cef2860 0x56118cef4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:01.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.296+0000 7f998fe00700 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118cef2860 0x56118cef4c80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c01644d2ebbb9d1 server_cookie=fd5b9238984f457f in_seq=0 out_seq=0 2024-05-27T07:31:01.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.296+0000 7f9985600700 10 client.16521 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:01.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f9985600700 1 -- 172.21.15.12:0/2211429596 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56118cd86570 con 0x56118cef2860 2024-05-27T07:31:01.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f9985600700 10 client.16521 handle_command_reply: tid=0 2024-05-27T07:31:01.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 client.16521 shutdown 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 msgr2=0x56118ce611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce611c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f99880a98f0 tx=0x7f99880aad80 comp rx=0 tx=0).stop 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 shutdown_connections 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118ce5ce70 0x56118ce61c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118ccdde40 0x56118ce611c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118cef2860 0x56118cef4c80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 --2- 172.21.15.12:0/2211429596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118cdc99f0 0x56118ce61700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 >> 172.21.15.12:0/2211429596 conn(0x56118cd77390 msgr2=0x56118ce71c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:01.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f9985600700 0 client.16521 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:01.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.297+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 shutdown_connections 2024-05-27T07:31:01.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.298+0000 7f99ad435b80 1 -- 172.21.15.12:0/2211429596 wait complete. 2024-05-27T07:31:01.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.298+0000 7f99ad435b80 20 client.16521 trim_cache size 0 max 16384 2024-05-27T07:31:01.291 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:31:01.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.299+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f997c0b29f0 msgr2=0x7f997c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.299+0000 7f999e800700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f997c0b29f0 0x7f997c0b4f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f999001d000 tx=0x7f999000ae60 comp rx=0 tx=0).stop 2024-05-27T07:31:01.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.299+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106560 msgr2=0x7f9998110770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:01.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.299+0000 7f999e800700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106560 0x7f9998110770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f998800b5c0 tx=0x7f99880047b0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 shutdown_connections 2024-05-27T07:31:01.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f997c0b29f0 0x7f997c0b4f00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f999810c5a0 0x7f99981111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9998106560 0x7f9998110770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 --2- 172.21.15.12:0/1923192833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9998106eb0 0x7f9998110cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:01.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 >> 172.21.15.12:0/1923192833 conn(0x7f9998102040 msgr2=0x7f99981035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:01.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 shutdown_connections 2024-05-27T07:31:01.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:01.300+0000 7f999e800700 1 -- 172.21.15.12:0/1923192833 wait complete. 2024-05-27T07:31:02.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:01 smithi012 ceph-mon[31473]: pgmap v305: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 995 KiB/s wr, 115 op/s 2024-05-27T07:31:02.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:01 smithi162 ceph-mon[35676]: pgmap v305: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 995 KiB/s wr, 115 op/s 2024-05-27T07:31:02.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:01 smithi062 ceph-mon[35304]: pgmap v305: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 995 KiB/s wr, 115 op/s 2024-05-27T07:31:03.311 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:31:03.311 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:31:03.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:03 smithi012 ceph-mon[31473]: pgmap v306: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 995 KiB/s wr, 115 op/s 2024-05-27T07:31:03.816 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:04.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:03 smithi162 ceph-mon[35676]: pgmap v306: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 995 KiB/s wr, 115 op/s 2024-05-27T07:31:04.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:03 smithi062 ceph-mon[35304]: pgmap v306: 129 pgs: 129 active+clean; 128 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 995 KiB/s wr, 115 op/s 2024-05-27T07:31:05.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:05 smithi012 ceph-mon[31473]: pgmap v307: 129 pgs: 129 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-05-27T07:31:06.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:05 smithi162 ceph-mon[35676]: pgmap v307: 129 pgs: 129 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-05-27T07:31:06.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:05 smithi062 ceph-mon[35304]: pgmap v307: 129 pgs: 129 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-05-27T07:31:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 -- 172.21.15.12:0/564359938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 msgr2=0x7fba98110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 --2- 172.21.15.12:0/564359938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba98110fd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fba9000b0d0 tx=0x7fba9000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 -- 172.21.15.12:0/564359938 shutdown_connections 2024-05-27T07:31:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 --2- 172.21.15.12:0/564359938 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba98110fd0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 --2- 172.21.15.12:0/564359938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba981043f0 0x7fba98104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 --2- 172.21.15.12:0/564359938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba98108d20 0x7fba98109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 -- 172.21.15.12:0/564359938 >> 172.21.15.12:0/564359938 conn(0x7fba98079d00 msgr2=0x7fba9807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 -- 172.21.15.12:0/564359938 shutdown_connections 2024-05-27T07:31:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.322+0000 7fba97200700 1 -- 172.21.15.12:0/564359938 wait complete. 2024-05-27T07:31:06.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.324+0000 7fba97200700 1 Processor -- start 2024-05-27T07:31:06.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.325+0000 7fba97200700 1 -- start start 2024-05-27T07:31:06.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.325+0000 7fba97200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba981043f0 0x7fba98199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.325+0000 7fba97200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba98108d20 0x7fba98199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.325+0000 7fba97200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba981a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.325+0000 7fba97200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba981141a0 con 0x7fba98109670 2024-05-27T07:31:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.325+0000 7fba97200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba98114020 con 0x7fba98108d20 2024-05-27T07:31:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.325+0000 7fba97200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba98114320 con 0x7fba981043f0 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba981a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba981a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57222/0 (socket says 172.21.15.12:57222) 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 -- 172.21.15.12:0/1115924867 learned_addr learned my addr 172.21.15.12:0/1115924867 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba95e00700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba981043f0 0x7fba98199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba95400700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba98108d20 0x7fba98199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 -- 172.21.15.12:0/1115924867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba981043f0 msgr2=0x7fba98199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba981043f0 0x7fba98199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 -- 172.21.15.12:0/1115924867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba98108d20 msgr2=0x7fba98199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba98108d20 0x7fba98199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.326+0000 7fba96800700 1 -- 172.21.15.12:0/1115924867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba981a0c90 con 0x7fba98109670 2024-05-27T07:31:06.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.327+0000 7fba96800700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba981a06d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fba900079d0 tx=0x7fba90007ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:06.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.327+0000 7fba8ea00700 1 -- 172.21.15.12:0/1115924867 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9001f070 con 0x7fba98109670 2024-05-27T07:31:06.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.327+0000 7fba8ea00700 1 -- 172.21.15.12:0/1115924867 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba90007d80 con 0x7fba98109670 2024-05-27T07:31:06.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.327+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba981a0f20 con 0x7fba98109670 2024-05-27T07:31:06.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.327+0000 7fba8ea00700 1 -- 172.21.15.12:0/1115924867 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9001b690 con 0x7fba98109670 2024-05-27T07:31:06.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.327+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba981a1450 con 0x7fba98109670 2024-05-27T07:31:06.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.331+0000 7fba8ea00700 1 -- 172.21.15.12:0/1115924867 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fba90022070 con 0x7fba98109670 2024-05-27T07:31:06.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.333+0000 7fbaab867b80 1 Processor -- start 2024-05-27T07:31:06.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.333+0000 7fbaab867b80 1 -- start start 2024-05-27T07:31:06.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d82193ba70 0x55d8219cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8219d9340 0x55d8219db760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fbaab867b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d8218fcec0 con 0x55d82193ba70 2024-05-27T07:31:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fbaab867b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d8218eca60 con 0x55d82184fec0 2024-05-27T07:31:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fbaab867b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8218f0aa0 con 0x55d8219d9340 2024-05-27T07:31:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fba8ea00700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fba7c0b2a70 0x7fba7c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fba95e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fba95e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:43282/0 (socket says 172.21.15.12:43282) 2024-05-27T07:31:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.334+0000 7fba95e00700 1 -- 172.21.15.12:0/1584096073 learned_addr learned my addr 172.21.15.12:0/1584096073 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba96800700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fba7c0b2a70 0x7fba7c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba95400700 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d82193ba70 0x55d8219cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba8ea00700 1 -- 172.21.15.12:0/1115924867 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7fba9000f020 con 0x7fba98109670 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba95e00700 1 -- 172.21.15.12:0/1584096073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8219d9340 msgr2=0x55d8219db760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba95e00700 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8219d9340 0x55d8219db760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba95e00700 1 -- 172.21.15.12:0/1584096073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d82193ba70 msgr2=0x55d8219cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba95e00700 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d82193ba70 0x55d8219cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.335+0000 7fba95e00700 1 -- 172.21.15.12:0/1584096073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8218b54a0 con 0x55d82184fec0 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.336+0000 7fba96800700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fba7c0b2a70 0x7fba7c0b4f80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba900062a0 tx=0x7fba90006230 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.336+0000 7fba95e00700 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cfd10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba8800bd70 tx=0x7fba8800f640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dd7668759bb59ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:06.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.336+0000 7fba83400700 1 -- 172.21.15.12:0/1584096073 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88015070 con 0x55d82184fec0 2024-05-27T07:31:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.337+0000 7fba83400700 1 -- 172.21.15.12:0/1584096073 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba8800fe50 con 0x55d82184fec0 2024-05-27T07:31:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.337+0000 7fba83400700 1 -- 172.21.15.12:0/1584096073 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88015070 con 0x55d82184fec0 2024-05-27T07:31:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 -- 172.21.15.12:0/1584096073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 msgr2=0x55d8219cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cfd10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba8800bd70 tx=0x7fba8800f640 comp rx=0 tx=0).stop 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 -- 172.21.15.12:0/1584096073 shutdown_connections 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8219d9340 0x55d8219db760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d82193ba70 0x55d8219cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 --2- 172.21.15.12:0/1584096073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cfd10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 -- 172.21.15.12:0/1584096073 >> 172.21.15.12:0/1584096073 conn(0x55d8218e9410 msgr2=0x55d8219ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 -- 172.21.15.12:0/1584096073 shutdown_connections 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.338+0000 7fbaab867b80 1 -- 172.21.15.12:0/1584096073 wait complete. 2024-05-27T07:31:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.339+0000 7fbaab867b80 1 -- start start 2024-05-27T07:31:06.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.339+0000 7fbaab867b80 1 Processor -- start 2024-05-27T07:31:06.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219d62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219d6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fbaab867b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d8217f3020 con 0x55d8219d9340 2024-05-27T07:31:06.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fbaab867b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d82188d7e0 con 0x55d82184fec0 2024-05-27T07:31:06.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fbaab867b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8219ece90 con 0x55d82193ba70 2024-05-27T07:31:06.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fbaab867b80 10 client.0 _ll_register_callbacks cb 0x55d821896cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:06.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fba95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219d6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fba95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219d6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57246/0 (socket says 172.21.15.12:57246) 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fba95400700 1 -- 172.21.15.12:0/4033629891 learned_addr learned my addr 172.21.15.12:0/4033629891 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.340+0000 7fba95400700 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219d62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba95e00700 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba95400700 1 -- 172.21.15.12:0/4033629891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 msgr2=0x55d8219d6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba95400700 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba95400700 1 -- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 msgr2=0x55d8219d62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba95400700 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219d62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba95400700 1 -- 172.21.15.12:0/4033629891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d82183c950 con 0x55d8219d9340 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba95400700 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219d6d60 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fba84012c50 tx=0x7fba84009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cffc54003df865e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:06.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.341+0000 7fba77e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.342+0000 7fba77e00700 1 -- 172.21.15.12:0/4033629891 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84012110 con 0x55d8219d9340 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.342+0000 7fba77e00700 1 -- 172.21.15.12:0/4033629891 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7fba8400f8e0 con 0x55d8219d9340 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.342+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba8800fe50 con 0x55d8219d9340 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.342+0000 7fba77e00700 1 -- 172.21.15.12:0/4033629891 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fba84026050 con 0x55d8219d9340 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.342+0000 7fbaab867b80 10 client.16533 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.342+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55d8218bd0e0 con 0x55d8219d9340 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.342+0000 7fba77e00700 1 -- 172.21.15.12:0/4033629891 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba8800fe50 con 0x55d8219d9340 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fba77e00700 1 -- 172.21.15.12:0/4033629891 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fba84010960 con 0x55d8219d9340 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fbaab867b80 10 client.16533 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fbaab867b80 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fbaab867b80 20 client.16533 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fbaab867b80 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d8219fa240 0x55d8219fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fbaab867b80 4 client.16533 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:06.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8218fee10 con 0x55d8219fa240 2024-05-27T07:31:06.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.343+0000 7fba95e00700 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d8219fa240 0x55d8219fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.345+0000 7fba95e00700 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d8219fa240 0x55d8219fc6b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb50a1c188182779 server_cookie=2feb1b6328fc555 in_seq=0 out_seq=0 2024-05-27T07:31:06.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.345+0000 7fba77e00700 10 client.16533 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:06.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fba77e00700 1 -- 172.21.15.12:0/4033629891 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d8218fee10 con 0x55d8219fa240 2024-05-27T07:31:06.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fba77e00700 10 client.16533 handle_command_reply: tid=0 2024-05-27T07:31:06.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 client.16533 shutdown 2024-05-27T07:31:06.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 msgr2=0x55d8219d6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219d6d60 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fba84012c50 tx=0x7fba84009730 comp rx=0 tx=0).stop 2024-05-27T07:31:06.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 shutdown_connections 2024-05-27T07:31:06.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d8219fa240 0x55d8219fc6b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219d62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219d6d60 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 --2- 172.21.15.12:0/4033629891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 >> 172.21.15.12:0/4033629891 conn(0x55d8218e9410 msgr2=0x55d8219ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.347+0000 7fba77e00700 0 client.16533 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.348+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 shutdown_connections 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.348+0000 7fbaab867b80 1 -- 172.21.15.12:0/4033629891 wait complete. 2024-05-27T07:31:06.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.348+0000 7fbaab867b80 20 client.16533 trim_cache size 0 max 16384 2024-05-27T07:31:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.374+0000 7fbaab867b80 1 Processor -- start 2024-05-27T07:31:06.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.374+0000 7fbaab867b80 1 -- start start 2024-05-27T07:31:06.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fbaab867b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d8219d5460 con 0x55d8219d9340 2024-05-27T07:31:06.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fbaab867b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba98074ca0 con 0x55d82184fec0 2024-05-27T07:31:06.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fbaab867b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba98073f00 con 0x55d82193ba70 2024-05-27T07:31:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fba95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fba95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57254/0 (socket says 172.21.15.12:57254) 2024-05-27T07:31:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.375+0000 7fba95400700 1 -- 172.21.15.12:0/2871846829 learned_addr learned my addr 172.21.15.12:0/2871846829 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:06.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95e00700 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95400700 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95400700 1 -- 172.21.15.12:0/2871846829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 msgr2=0x55d8219d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95400700 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95400700 1 -- 172.21.15.12:0/2871846829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 msgr2=0x55d8219cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95400700 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95400700 1 -- 172.21.15.12:0/2871846829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8219d70d0 con 0x55d8219d9340 2024-05-27T07:31:06.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.376+0000 7fba95e00700 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.377+0000 7fba95400700 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219f5f60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fba8402afd0 tx=0x7fba84001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ab4baa0b3f9adc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.377+0000 7fba82a00700 1 -- 172.21.15.12:0/2871846829 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8400f8e0 con 0x55d8219d9340 2024-05-27T07:31:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.378+0000 7fba82a00700 1 -- 172.21.15.12:0/2871846829 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba84005070 con 0x55d8219d9340 2024-05-27T07:31:06.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.378+0000 7fba82a00700 1 -- 172.21.15.12:0/2871846829 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84010690 con 0x55d8219d9340 2024-05-27T07:31:06.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.380+0000 7fbaab867b80 1 -- 172.21.15.12:0/2871846829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 msgr2=0x55d8219f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.380+0000 7fbaab867b80 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219f5f60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fba8402afd0 tx=0x7fba84001e50 comp rx=0 tx=0).stop 2024-05-27T07:31:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.381+0000 7fbaab867b80 1 -- 172.21.15.12:0/2871846829 shutdown_connections 2024-05-27T07:31:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.381+0000 7fbaab867b80 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d8219cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.381+0000 7fbaab867b80 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219f5f60 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.381+0000 7fbaab867b80 1 --2- 172.21.15.12:0/2871846829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d8219d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.381+0000 7fbaab867b80 1 -- 172.21.15.12:0/2871846829 >> 172.21.15.12:0/2871846829 conn(0x55d8218e9410 msgr2=0x55d8219d7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:06.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.381+0000 7fbaab867b80 1 -- 172.21.15.12:0/2871846829 shutdown_connections 2024-05-27T07:31:06.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.381+0000 7fbaab867b80 1 -- 172.21.15.12:0/2871846829 wait complete. 2024-05-27T07:31:06.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.382+0000 7fbaab867b80 1 -- start start 2024-05-27T07:31:06.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.382+0000 7fbaab867b80 1 Processor -- start 2024-05-27T07:31:06.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d821a5e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d821a5e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fbaab867b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fbaab867b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba980745d0 con 0x55d8219d9340 2024-05-27T07:31:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fbaab867b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d8219cb230 con 0x55d82184fec0 2024-05-27T07:31:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fbaab867b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8219eec20 con 0x55d82193ba70 2024-05-27T07:31:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fbaab867b80 10 client.0 _ll_register_callbacks cb 0x55d821896cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fba95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fba95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57270/0 (socket says 172.21.15.12:57270) 2024-05-27T07:31:06.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.383+0000 7fba95400700 1 -- 172.21.15.12:0/546563650 learned_addr learned my addr 172.21.15.12:0/546563650 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.384+0000 7fba95e00700 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d821a5e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.384+0000 7fba95400700 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d821a5e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.384+0000 7fba95400700 1 -- 172.21.15.12:0/546563650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 msgr2=0x55d821a5e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.384+0000 7fba95400700 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d821a5e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.384+0000 7fba95400700 1 -- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 msgr2=0x55d821a5e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.384+0000 7fba95400700 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d821a5e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.384+0000 7fba95400700 1 -- 172.21.15.12:0/546563650 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8219eee20 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.385+0000 7fba95400700 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219ee5f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fba84012420 tx=0x7fba8402af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ad593982c80649 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.385+0000 7fba77e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.385+0000 7fba77e00700 1 -- 172.21.15.12:0/546563650 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba840054e0 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.385+0000 7fba77e00700 1 -- 172.21.15.12:0/546563650 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7fba8400f8e0 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.385+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba84005070 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fba77e00700 1 -- 172.21.15.12:0/546563650 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fba84005680 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 10 client.16539 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55d821a65850 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fba77e00700 1 -- 172.21.15.12:0/546563650 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba84005070 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fba77e00700 1 -- 172.21.15.12:0/546563650 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fba84038070 con 0x55d8219d9340 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 10 client.16539 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:06.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 20 client.16539 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d821a65d20 0x55d821a68140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 4 client.16539 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:31:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.386+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d8218f85f0 con 0x55d821a65d20 2024-05-27T07:31:06.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.387+0000 7fba95e00700 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d821a65d20 0x55d821a68140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:06.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.388+0000 7fba95e00700 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d821a65d20 0x55d821a68140 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf46f8401b8f958a server_cookie=d5e6ac5a5ee38d8f in_seq=0 out_seq=0 2024-05-27T07:31:06.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.388+0000 7fba77e00700 10 client.16539 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.392+0000 7fba77e00700 1 -- 172.21.15.12:0/546563650 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d8218f85f0 con 0x55d821a65d20 2024-05-27T07:31:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.392+0000 7fba77e00700 10 client.16539 handle_command_reply: tid=0 2024-05-27T07:31:06.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.392+0000 7fbaab867b80 1 client.16539 shutdown 2024-05-27T07:31:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 msgr2=0x55d8219ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219ee5f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fba84012420 tx=0x7fba8402af30 comp rx=0 tx=0).stop 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 shutdown_connections 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d821a65d20 0x55d821a68140 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d82184fec0 0x55d821a5e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d8219d9340 0x55d8219ee5f0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 --2- 172.21.15.12:0/546563650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d82193ba70 0x55d821a5e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fba77e00700 0 client.16539 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 >> 172.21.15.12:0/546563650 conn(0x55d8218e9410 msgr2=0x55d8219ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.396+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 shutdown_connections 2024-05-27T07:31:06.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.397+0000 7fbaab867b80 1 -- 172.21.15.12:0/546563650 wait complete. 2024-05-27T07:31:06.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.397+0000 7fbaab867b80 20 client.16539 trim_cache size 0 max 16384 2024-05-27T07:31:06.390 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:31:06.390 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:31:06.390 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "26851543-25ed-4864-af4d-09c0a08dd863", 2024-05-27T07:31:06.390 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:31:06.390 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:31:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fba7c0b2a70 msgr2=0x7fba7c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fba7c0b2a70 0x7fba7c0b4f80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba900062a0 tx=0x7fba90006230 comp rx=0 tx=0).stop 2024-05-27T07:31:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 msgr2=0x7fba981a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba981a06d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fba900079d0 tx=0x7fba90007ab0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 shutdown_connections 2024-05-27T07:31:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fba7c0b2a70 0x7fba7c0b4f80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba98109670 0x7fba981a06d0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba981043f0 0x7fba98199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 --2- 172.21.15.12:0/1115924867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba98108d20 0x7fba98199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 >> 172.21.15.12:0/1115924867 conn(0x7fba98079d00 msgr2=0x7fba98101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.398+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 shutdown_connections 2024-05-27T07:31:06.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:06.399+0000 7fba97200700 1 -- 172.21.15.12:0/1115924867 wait complete. 2024-05-27T07:31:07.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:06 smithi012 ceph-mon[31473]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:31:07.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:06 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:31:07.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:06 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:07.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:06 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:31:07.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:06 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:31:07.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:06 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:31:07.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:06 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:07.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:06 smithi162 ceph-mon[35676]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:31:07.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:06 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:31:07.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:06 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:07.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:06 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:31:07.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:06 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:31:07.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:06 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:31:07.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:06 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:07.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:06 smithi062 ceph-mon[35304]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:31:07.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:06 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:31:07.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:06 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:07.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:06 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:31:07.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:06 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:31:07.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:06 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:31:07.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:06 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:07.339 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:31:07.842 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:08.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:07 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:08.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:07 smithi012 ceph-mon[31473]: pgmap v309: 129 pgs: 129 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-05-27T07:31:08.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:07 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:08.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:07 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:08.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:07 smithi162 ceph-mon[35676]: pgmap v309: 129 pgs: 129 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-05-27T07:31:08.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:07 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:08.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:07 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:08.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:07 smithi062 ceph-mon[35304]: pgmap v309: 129 pgs: 129 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-05-27T07:31:08.237 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:07 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:10.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:09 smithi012 ceph-mon[31473]: pgmap v310: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:31:10.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 -- 172.21.15.12:0/2084450614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657010e900 msgr2=0x7f6570110de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 --2- 172.21.15.12:0/2084450614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657010e900 0x7f6570110de0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f656800b0a0 tx=0x7f656800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 -- 172.21.15.12:0/2084450614 shutdown_connections 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 --2- 172.21.15.12:0/2084450614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f657010e900 0x7f6570110de0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 --2- 172.21.15.12:0/2084450614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65701054d0 0x7f65701058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 --2- 172.21.15.12:0/2084450614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6570105e20 0x7f657010e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 -- 172.21.15.12:0/2084450614 >> 172.21.15.12:0/2084450614 conn(0x7f65700fef20 msgr2=0x7f6570101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 -- 172.21.15.12:0/2084450614 shutdown_connections 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.177+0000 7f6575c00700 1 -- 172.21.15.12:0/2084450614 wait complete. 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.178+0000 7f6575c00700 1 Processor -- start 2024-05-27T07:31:10.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.178+0000 7f6575c00700 1 -- start start 2024-05-27T07:31:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f6575c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65701054d0 0x7f65701031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f6575c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6570105e20 0x7f6570103700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f6575c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657010e900 0x7f6570103c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f6575c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f65701143a0 con 0x7f6570105e20 2024-05-27T07:31:10.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f6575c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6570114220 con 0x7f65701054d0 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f6575c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6570114520 con 0x7f657010e900 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6570105e20 0x7f6570103700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6570105e20 0x7f6570103700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57300/0 (socket says 172.21.15.12:57300) 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.179+0000 7f656e000700 1 -- 172.21.15.12:0/3917799703 learned_addr learned my addr 172.21.15.12:0/3917799703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656ea00700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65701054d0 0x7f65701031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656f400700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657010e900 0x7f6570103c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656e000700 1 -- 172.21.15.12:0/3917799703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657010e900 msgr2=0x7f6570103c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656e000700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657010e900 0x7f6570103c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656e000700 1 -- 172.21.15.12:0/3917799703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65701054d0 msgr2=0x7f65701031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656e000700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65701054d0 0x7f65701031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656e000700 1 -- 172.21.15.12:0/3917799703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65701a90d0 con 0x7f6570105e20 2024-05-27T07:31:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.180+0000 7f656e000700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6570105e20 0x7f6570103700 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f655c00d950 tx=0x7f655c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.181+0000 7f6567400700 1 -- 172.21.15.12:0/3917799703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c016070 con 0x7f6570105e20 2024-05-27T07:31:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.181+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65701a93c0 con 0x7f6570105e20 2024-05-27T07:31:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.181+0000 7f6567400700 1 -- 172.21.15.12:0/3917799703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f655c010460 con 0x7f6570105e20 2024-05-27T07:31:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.181+0000 7f6567400700 1 -- 172.21.15.12:0/3917799703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c00f620 con 0x7f6570105e20 2024-05-27T07:31:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.181+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65701a9950 con 0x7f6570105e20 2024-05-27T07:31:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.187+0000 7f6567400700 1 -- 172.21.15.12:0/3917799703 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f655c019070 con 0x7f6570105e20 2024-05-27T07:31:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.187+0000 7f65846bbb80 1 Processor -- start 2024-05-27T07:31:10.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.188+0000 7f65846bbb80 1 -- start start 2024-05-27T07:31:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.188+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.188+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a98144870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.188+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a981451c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.188+0000 7f65846bbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557a98072e40 con 0x557a98144db0 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.188+0000 7f65846bbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557a980629e0 con 0x557a9814ad30 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.188+0000 7f65846bbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557a98066a20 con 0x557a97fc5e40 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.189+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a981451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.189+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a981451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57308/0 (socket says 172.21.15.12:57308) 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.189+0000 7f656e000700 1 -- 172.21.15.12:0/2688860488 learned_addr learned my addr 172.21.15.12:0/2688860488 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:10.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.190+0000 7f656ea00700 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a98144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.190+0000 7f6567400700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f65540b2a70 0x7f65540b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.190+0000 7f656e000700 1 -- 172.21.15.12:0/2688860488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 msgr2=0x557a9814a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.190+0000 7f656f400700 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656e000700 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656e000700 1 -- 172.21.15.12:0/2688860488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 msgr2=0x557a98144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:10.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656e000700 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a98144870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f6567400700 1 -- 172.21.15.12:0/3917799703 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7f655c0e2620 con 0x7f6570105e20 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656e000700 1 -- 172.21.15.12:0/2688860488 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a9802b420 con 0x557a98144db0 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656f400700 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656f400700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f65540b2a70 0x7f65540b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656ea00700 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a98144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.191+0000 7f656e000700 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a981451c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f655c0df880 tx=0x7f655c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9efe891035f3f6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.192+0000 7f655be00700 1 -- 172.21.15.12:0/2688860488 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c020070 con 0x557a98144db0 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.192+0000 7f655be00700 1 -- 172.21.15.12:0/2688860488 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f655c0a8fb0 con 0x557a98144db0 2024-05-27T07:31:10.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.192+0000 7f655be00700 1 -- 172.21.15.12:0/2688860488 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c0a94a0 con 0x557a98144db0 2024-05-27T07:31:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 -- 172.21.15.12:0/2688860488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 msgr2=0x557a981451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f656f400700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f65540b2a70 0x7f65540b4f80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6568009e60 tx=0x7f65680099e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a981451c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f655c0df880 tx=0x7f655c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 -- 172.21.15.12:0/2688860488 shutdown_connections 2024-05-27T07:31:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a981451c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 --2- 172.21.15.12:0/2688860488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a98144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 -- 172.21.15.12:0/2688860488 >> 172.21.15.12:0/2688860488 conn(0x557a98025fc0 msgr2=0x557a9813f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 -- 172.21.15.12:0/2688860488 shutdown_connections 2024-05-27T07:31:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.193+0000 7f65846bbb80 1 -- 172.21.15.12:0/2688860488 wait complete. 2024-05-27T07:31:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.194+0000 7f65846bbb80 1 -- start start 2024-05-27T07:31:10.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.194+0000 7f65846bbb80 1 Processor -- start 2024-05-27T07:31:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.195+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.195+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a9814ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.195+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a9814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.195+0000 7f65846bbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557a97f68fa0 con 0x557a98144db0 2024-05-27T07:31:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.195+0000 7f65846bbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557a98003760 con 0x557a9814ad30 2024-05-27T07:31:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.195+0000 7f65846bbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557a98151740 con 0x557a97fc5e40 2024-05-27T07:31:10.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f65846bbb80 10 client.0 _ll_register_callbacks cb 0x557a9800cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a9814ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a9814ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57322/0 (socket says 172.21.15.12:57322) 2024-05-27T07:31:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 -- 172.21.15.12:0/131159133 learned_addr learned my addr 172.21.15.12:0/131159133 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 -- 172.21.15.12:0/131159133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 msgr2=0x557a9814e510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 -- 172.21.15.12:0/131159133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 msgr2=0x557a9814ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:10.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a9814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.196+0000 7f656e000700 1 -- 172.21.15.12:0/131159133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a97fb28d0 con 0x557a98144db0 2024-05-27T07:31:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.197+0000 7f656e000700 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a9814ea50 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f655c0e4040 tx=0x7f655c091180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbebbdfe749b6ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.197+0000 7f6558c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:10.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.197+0000 7f6558c00700 1 -- 172.21.15.12:0/131159133 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c020040 con 0x557a98144db0 2024-05-27T07:31:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.197+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f655c0a8fb0 con 0x557a98144db0 2024-05-27T07:31:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.197+0000 7f6558c00700 1 -- 172.21.15.12:0/131159133 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7f655c0a9700 con 0x557a98144db0 2024-05-27T07:31:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.198+0000 7f6558c00700 1 -- 172.21.15.12:0/131159133 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f655c0a8df0 con 0x557a98144db0 2024-05-27T07:31:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.198+0000 7f65846bbb80 10 client.16551 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:10.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.198+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x557a98033060 con 0x557a98144db0 2024-05-27T07:31:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.198+0000 7f6558c00700 1 -- 172.21.15.12:0/131159133 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f655c0a8fb0 con 0x557a98144db0 2024-05-27T07:31:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.199+0000 7f6558c00700 1 -- 172.21.15.12:0/131159133 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f655c0b1e70 con 0x557a98144db0 2024-05-27T07:31:10.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.199+0000 7f65846bbb80 10 client.16551 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.199+0000 7f65846bbb80 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.199+0000 7f65846bbb80 20 client.16551 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.199+0000 7f65846bbb80 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a9815e540 0x557a981609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.199+0000 7f65846bbb80 4 client.16551 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.199+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a98074d90 con 0x557a9815e540 2024-05-27T07:31:10.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.200+0000 7f656ea00700 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a9815e540 0x557a981609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.202+0000 7f656ea00700 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a9815e540 0x557a981609c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b12743dc4d3be88 server_cookie=763cb8dbc027c51f in_seq=0 out_seq=0 2024-05-27T07:31:10.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.202+0000 7f6558c00700 10 client.16551 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:10.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.204+0000 7f6558c00700 1 -- 172.21.15.12:0/131159133 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557a98074d90 con 0x557a9815e540 2024-05-27T07:31:10.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.204+0000 7f6558c00700 10 client.16551 handle_command_reply: tid=0 2024-05-27T07:31:10.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.204+0000 7f65846bbb80 1 client.16551 shutdown 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 msgr2=0x557a9814ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a9814ea50 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f655c0e4040 tx=0x7f655c091180 comp rx=0 tx=0).stop 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 shutdown_connections 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a9815e540 0x557a981609c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a9814ad30 0x557a9814ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a97fc5e40 0x557a9814e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 --2- 172.21.15.12:0/131159133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a9814ea50 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 >> 172.21.15.12:0/131159133 conn(0x557a98025fc0 msgr2=0x557a9813f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f6558c00700 0 client.16551 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:10.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 shutdown_connections 2024-05-27T07:31:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 1 -- 172.21.15.12:0/131159133 wait complete. 2024-05-27T07:31:10.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.205+0000 7f65846bbb80 20 client.16551 trim_cache size 0 max 16384 2024-05-27T07:31:10.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.231+0000 7f65846bbb80 1 Processor -- start 2024-05-27T07:31:10.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.231+0000 7f65846bbb80 1 -- start start 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.232+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98144870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.232+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a98144db0 0x557a981509b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.232+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a9814ad30 0x557a98148c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.232+0000 7f65846bbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b1f30 con 0x557a9814ad30 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.232+0000 7f65846bbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b1190 con 0x557a97fc5e40 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.232+0000 7f65846bbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b1860 con 0x557a98144db0 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.232+0000 7f656f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a9814ad30 0x557a98148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a9814ad30 0x557a98148c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57332/0 (socket says 172.21.15.12:57332) 2024-05-27T07:31:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656f400700 1 -- 172.21.15.12:0/415926556 learned_addr learned my addr 172.21.15.12:0/415926556 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:10.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656ea00700 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:09 smithi162 ceph-mon[35676]: pgmap v310: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656e000700 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a98144db0 0x557a981509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656f400700 1 -- 172.21.15.12:0/415926556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a98144db0 msgr2=0x557a981509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656f400700 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a98144db0 0x557a981509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656f400700 1 -- 172.21.15.12:0/415926556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 msgr2=0x557a98144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656f400700 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98144870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.233+0000 7f656f400700 1 -- 172.21.15.12:0/415926556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a98139130 con 0x557a9814ad30 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.234+0000 7f656f400700 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a9814ad30 0x557a98148c10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f65680098d0 tx=0x7f656800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5a16b1d5aa74df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.234+0000 7f655b400700 1 -- 172.21.15.12:0/415926556 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656802b030 con 0x557a9814ad30 2024-05-27T07:31:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.234+0000 7f655b400700 1 -- 172.21.15.12:0/415926556 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6568004db0 con 0x557a9814ad30 2024-05-27T07:31:10.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.234+0000 7f655b400700 1 -- 172.21.15.12:0/415926556 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65680279d0 con 0x557a9814ad30 2024-05-27T07:31:10.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.235+0000 7f65846bbb80 1 -- 172.21.15.12:0/415926556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a9814ad30 msgr2=0x557a98148c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.235+0000 7f65846bbb80 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a9814ad30 0x557a98148c10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f65680098d0 tx=0x7f656800b7b0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.235+0000 7f65846bbb80 1 -- 172.21.15.12:0/415926556 shutdown_connections 2024-05-27T07:31:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.235+0000 7f65846bbb80 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.235+0000 7f65846bbb80 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a9814ad30 0x557a98148c10 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.235+0000 7f65846bbb80 1 --2- 172.21.15.12:0/415926556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a98144db0 0x557a981509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.235+0000 7f65846bbb80 1 -- 172.21.15.12:0/415926556 >> 172.21.15.12:0/415926556 conn(0x557a98025fc0 msgr2=0x557a9814ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.236+0000 7f65846bbb80 1 -- 172.21.15.12:0/415926556 shutdown_connections 2024-05-27T07:31:10.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.236+0000 7f65846bbb80 1 -- 172.21.15.12:0/415926556 wait complete. 2024-05-27T07:31:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.236+0000 7f65846bbb80 1 -- start start 2024-05-27T07:31:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 1 Processor -- start 2024-05-27T07:31:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98156d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a98165790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a9814ad30 0x557a98165cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557a981451c0 con 0x557a98144db0 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557a9814ca80 con 0x557a97fc5e40 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557a9814a320 con 0x557a9814ad30 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.237+0000 7f65846bbb80 10 client.0 _ll_register_callbacks cb 0x557a9800cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a98165790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a98165790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57348/0 (socket says 172.21.15.12:57348) 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 -- 172.21.15.12:0/3913184646 learned_addr learned my addr 172.21.15.12:0/3913184646 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656ea00700 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656f400700 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a9814ad30 0x557a98165cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 -- 172.21.15.12:0/3913184646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a9814ad30 msgr2=0x557a98165cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a9814ad30 0x557a98165cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 -- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 msgr2=0x557a98156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.238+0000 7f656e000700 1 -- 172.21.15.12:0/3913184646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a98166210 con 0x557a98144db0 2024-05-27T07:31:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f656e000700 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a98165790 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f655c0a5520 tx=0x7f655c0a59f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee3ca1d9e46386c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f6558c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f6558c00700 1 -- 172.21.15.12:0/3913184646 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c020040 con 0x557a98144db0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6568004db0 con 0x557a98144db0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f6558c00700 1 -- 172.21.15.12:0/3913184646 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7f655c0a8430 con 0x557a98144db0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f6558c00700 1 -- 172.21.15.12:0/3913184646 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f655c0a8fb0 con 0x557a98144db0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f65846bbb80 10 client.16557 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.239+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x557a98166210 con 0x557a98144db0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f6558c00700 1 -- 172.21.15.12:0/3913184646 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6568004db0 con 0x557a98144db0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f6558c00700 1 -- 172.21.15.12:0/3913184646 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f655c0ca070 con 0x557a98144db0 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f65846bbb80 10 client.16557 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f65846bbb80 10 client.16557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f65846bbb80 20 client.16557 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f65846bbb80 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a981c8d90 0x557a981cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f65846bbb80 4 client.16557 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.240+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557a9806e570 con 0x557a981c8d90 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.241+0000 7f656ea00700 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a981c8d90 0x557a981cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.242+0000 7f656ea00700 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a981c8d90 0x557a981cb1b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2e7f17ec7a344f3 server_cookie=7612e7b8ce0c67d6 in_seq=0 out_seq=0 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.242+0000 7f6558c00700 10 client.16557 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f6558c00700 1 -- 172.21.15.12:0/3913184646 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557a9806e570 con 0x557a981c8d90 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f6558c00700 10 client.16557 handle_command_reply: tid=0 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 client.16557 shutdown 2024-05-27T07:31:10.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 msgr2=0x557a98165790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a98165790 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f655c0a5520 tx=0x7f655c0a59f0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 shutdown_connections 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557a981c8d90 0x557a981cb1b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a97fc5e40 0x557a98156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a9814ad30 0x557a98165cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 --2- 172.21.15.12:0/3913184646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557a98144db0 0x557a98165790 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 >> 172.21.15.12:0/3913184646 conn(0x557a98025fc0 msgr2=0x557a9814ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f6558c00700 0 client.16557 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 shutdown_connections 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.243+0000 7f65846bbb80 1 -- 172.21.15.12:0/3913184646 wait complete. 2024-05-27T07:31:10.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f65846bbb80 20 client.16557 trim_cache size 0 max 16384 2024-05-27T07:31:10.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:09 smithi062 ceph-mon[35304]: pgmap v310: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:31:10.238 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:31:10.238 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:31:10.238 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:31:10.238 INFO:teuthology.orchestra.run.smithi012.stdout: "26851543-25ed-4864-af4d-09c0a08dd863": { 2024-05-27T07:31:10.238 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:31:10.238 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "26851543-25ed-4864-af4d-09c0a08dd863", 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f65540b2a70 msgr2=0x7f65540b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f65540b2a70 0x7f65540b4f80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6568009e60 tx=0x7f65680099e0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6570105e20 msgr2=0x7f6570103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:10.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6570105e20 0x7f6570103700 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f655c00d950 tx=0x7f655c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 shutdown_connections 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f65540b2a70 0x7f65540b4f80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65701054d0 0x7f65701031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657010e900 0x7f6570103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 --2- 172.21.15.12:0/3917799703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6570105e20 0x7f6570103700 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.244+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 >> 172.21.15.12:0/3917799703 conn(0x7f65700fef20 msgr2=0x7f65700ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.245+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 shutdown_connections 2024-05-27T07:31:10.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:10.245+0000 7f6575c00700 1 -- 172.21.15.12:0/3917799703 wait complete. 2024-05-27T07:31:11.097 INFO:tasks.cephfs.filesystem:scrub status for tag:26851543-25ed-4864-af4d-09c0a08dd863 - {'path': '/', 'tag': '26851543-25ed-4864-af4d-09c0a08dd863', 'options': 'recursive,force'} 2024-05-27T07:31:12.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:11 smithi012 ceph-mon[31473]: pgmap v311: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:31:12.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:11 smithi162 ceph-mon[35676]: pgmap v311: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:31:12.238 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:11 smithi062 ceph-mon[35304]: pgmap v311: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:31:13.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:13 smithi162 ceph-mon[35676]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:31:13.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:13 smithi062 ceph-mon[35304]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:31:13.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:13 smithi012 ceph-mon[31473]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:31:14.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:14 smithi162 ceph-mon[35676]: pgmap v313: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 733 KiB/s wr, 88 op/s 2024-05-27T07:31:14.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:14 smithi062 ceph-mon[35304]: pgmap v313: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 733 KiB/s wr, 88 op/s 2024-05-27T07:31:14.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:14 smithi012 ceph-mon[31473]: pgmap v313: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 733 KiB/s wr, 88 op/s 2024-05-27T07:31:16.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:16 smithi162 ceph-mon[35676]: pgmap v314: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-05-27T07:31:16.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:16 smithi062 ceph-mon[35304]: pgmap v314: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-05-27T07:31:16.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:16 smithi012 ceph-mon[31473]: pgmap v314: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-05-27T07:31:18.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:18 smithi162 ceph-mon[35676]: pgmap v315: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-05-27T07:31:18.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:18 smithi062 ceph-mon[35304]: pgmap v315: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-05-27T07:31:18.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:18 smithi012 ceph-mon[31473]: pgmap v315: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-05-27T07:31:20.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:20 smithi162 ceph-mon[35676]: pgmap v316: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-27T07:31:20.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:20 smithi062 ceph-mon[35304]: pgmap v316: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-27T07:31:20.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:20 smithi012 ceph-mon[31473]: pgmap v316: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-27T07:31:21.675 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:31:21.675 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:31:21.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:31:21.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:31:21.737 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:31:21.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:31:22.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:22 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:31:22.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:22 smithi162 ceph-mon[35676]: pgmap v317: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-27T07:31:22.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:22 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:31:22.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:22 smithi062 ceph-mon[35304]: pgmap v317: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-27T07:31:22.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:22 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:31:22.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:22 smithi012 ceph-mon[31473]: pgmap v317: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-27T07:31:24.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:24 smithi162 ceph-mon[35676]: pgmap v318: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-05-27T07:31:24.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:24 smithi062 ceph-mon[35304]: pgmap v318: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-05-27T07:31:24.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:24 smithi012 ceph-mon[31473]: pgmap v318: 129 pgs: 129 active+clean; 131 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-05-27T07:31:26.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:26 smithi162 ceph-mon[35676]: pgmap v319: 129 pgs: 129 active+clean; 133 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-05-27T07:31:26.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:26 smithi062 ceph-mon[35304]: pgmap v319: 129 pgs: 129 active+clean; 133 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-05-27T07:31:26.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:26 smithi012 ceph-mon[31473]: pgmap v319: 129 pgs: 129 active+clean; 133 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-05-27T07:31:28.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:28 smithi162 ceph-mon[35676]: pgmap v320: 129 pgs: 129 active+clean; 133 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 993 KiB/s wr, 104 op/s 2024-05-27T07:31:28.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:28 smithi062 ceph-mon[35304]: pgmap v320: 129 pgs: 129 active+clean; 133 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 993 KiB/s wr, 104 op/s 2024-05-27T07:31:28.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:28 smithi012 ceph-mon[31473]: pgmap v320: 129 pgs: 129 active+clean; 133 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 993 KiB/s wr, 104 op/s 2024-05-27T07:31:29.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:29 smithi012 ceph-mon[31473]: pgmap v321: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-05-27T07:31:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:29 smithi162 ceph-mon[35676]: pgmap v321: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-05-27T07:31:29.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:29 smithi062 ceph-mon[35304]: pgmap v321: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-05-27T07:31:31.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:31 smithi162 ceph-mon[35676]: pgmap v322: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 939 KiB/s wr, 104 op/s 2024-05-27T07:31:31.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:31 smithi062 ceph-mon[35304]: pgmap v322: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 939 KiB/s wr, 104 op/s 2024-05-27T07:31:32.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:31 smithi012 ceph-mon[31473]: pgmap v322: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 939 KiB/s wr, 104 op/s 2024-05-27T07:31:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:33 smithi162 ceph-mon[35676]: pgmap v323: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 939 KiB/s wr, 104 op/s 2024-05-27T07:31:33.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:33 smithi062 ceph-mon[35304]: pgmap v323: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 939 KiB/s wr, 104 op/s 2024-05-27T07:31:34.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:33 smithi012 ceph-mon[31473]: pgmap v323: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 939 KiB/s wr, 104 op/s 2024-05-27T07:31:35.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:35 smithi162 ceph-mon[35676]: pgmap v324: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-05-27T07:31:35.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:35 smithi062 ceph-mon[35304]: pgmap v324: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-05-27T07:31:36.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:35 smithi012 ceph-mon[31473]: pgmap v324: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-05-27T07:31:37.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:37 smithi162 ceph-mon[35676]: pgmap v325: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 880 KiB/s wr, 95 op/s 2024-05-27T07:31:38.005 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:37 smithi062 ceph-mon[35304]: pgmap v325: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 880 KiB/s wr, 95 op/s 2024-05-27T07:31:38.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:37 smithi012 ceph-mon[31473]: pgmap v325: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 880 KiB/s wr, 95 op/s 2024-05-27T07:31:39.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:39 smithi162 ceph-mon[35676]: pgmap v326: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-05-27T07:31:39.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:39 smithi062 ceph-mon[35304]: pgmap v326: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-05-27T07:31:40.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:39 smithi012 ceph-mon[31473]: pgmap v326: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-05-27T07:31:41.098 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:31:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:41.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:41 smithi162 ceph-mon[35676]: pgmap v327: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 885 KiB/s wr, 94 op/s 2024-05-27T07:31:41.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:41 smithi062 ceph-mon[35304]: pgmap v327: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 885 KiB/s wr, 94 op/s 2024-05-27T07:31:42.003 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:41 smithi012 ceph-mon[31473]: pgmap v327: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 885 KiB/s wr, 94 op/s 2024-05-27T07:31:43.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:43 smithi162 ceph-mon[35676]: pgmap v328: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 885 KiB/s wr, 94 op/s 2024-05-27T07:31:43.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:43 smithi062 ceph-mon[35304]: pgmap v328: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 885 KiB/s wr, 94 op/s 2024-05-27T07:31:43.991 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:43 smithi012 ceph-mon[31473]: pgmap v328: 129 pgs: 129 active+clean; 136 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 885 KiB/s wr, 94 op/s 2024-05-27T07:31:44.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.179+0000 7f11e3e00700 1 -- 172.21.15.12:0/848555984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 msgr2=0x7f11dc102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.179+0000 7f11e3e00700 1 --2- 172.21.15.12:0/848555984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc102ee0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f11d000b0d0 tx=0x7f11d000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.180+0000 7f11e3e00700 1 -- 172.21.15.12:0/848555984 shutdown_connections 2024-05-27T07:31:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.180+0000 7f11e3e00700 1 --2- 172.21.15.12:0/848555984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11dc109840 0x7f11dc1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.180+0000 7f11e3e00700 1 --2- 172.21.15.12:0/848555984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc102ee0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.180+0000 7f11e3e00700 1 --2- 172.21.15.12:0/848555984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11dc108ef0 0x7f11dc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.180+0000 7f11e3e00700 1 -- 172.21.15.12:0/848555984 >> 172.21.15.12:0/848555984 conn(0x7f11dc0fe4d0 msgr2=0x7f11dc100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.180+0000 7f11e3e00700 1 -- 172.21.15.12:0/848555984 shutdown_connections 2024-05-27T07:31:44.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.180+0000 7f11e3e00700 1 -- 172.21.15.12:0/848555984 wait complete. 2024-05-27T07:31:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.181+0000 7f11e3e00700 1 Processor -- start 2024-05-27T07:31:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.181+0000 7f11e3e00700 1 -- start start 2024-05-27T07:31:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11dc108ef0 0x7f11dc19d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e3e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11dc109840 0x7f11dc1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc114320 con 0x7f11dc102ad0 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1141a0 con 0x7f11dc108ef0 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e3e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1144a0 con 0x7f11dc109840 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41886/0 (socket says 172.21.15.12:41886) 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e1400700 1 -- 172.21.15.12:0/337609192 learned_addr learned my addr 172.21.15.12:0/337609192 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e1e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11dc109840 0x7f11dc1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.182+0000 7f11e0a00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11dc108ef0 0x7f11dc19d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e1400700 1 -- 172.21.15.12:0/337609192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11dc109840 msgr2=0x7f11dc1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e1400700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11dc109840 0x7f11dc1a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e1400700 1 -- 172.21.15.12:0/337609192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11dc108ef0 msgr2=0x7f11dc19d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e1400700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11dc108ef0 0x7f11dc19d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e1400700 1 -- 172.21.15.12:0/337609192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc1a1d70 con 0x7f11dc102ad0 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e1e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11dc109840 0x7f11dc1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e0a00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11dc108ef0 0x7f11dc19d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e1400700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc19cf50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f11d0007dc0 tx=0x7f11d0007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11da000700 1 -- 172.21.15.12:0/337609192 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d001f030 con 0x7f11dc102ad0 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11da000700 1 -- 172.21.15.12:0/337609192 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11d001bb90 con 0x7f11dc102ad0 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc1a2000 con 0x7f11dc102ad0 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.183+0000 7f11da000700 1 -- 172.21.15.12:0/337609192 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0012bb0 con 0x7f11dc102ad0 2024-05-27T07:31:44.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.184+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc1a2590 con 0x7f11dc102ad0 2024-05-27T07:31:44.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.188+0000 7f11da000700 1 -- 172.21.15.12:0/337609192 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f11d0022030 con 0x7f11dc102ad0 2024-05-27T07:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.189+0000 7f11f28e2b80 1 Processor -- start 2024-05-27T07:31:44.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.189+0000 7f11f28e2b80 1 -- start start 2024-05-27T07:31:44.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.190+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a57fe40 0x56194a7047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.190+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a704d30 0x56194a6fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.190+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a6ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.190+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56194a62ce40 con 0x56194a6fedb0 2024-05-27T07:31:44.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.190+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56194a61c9e0 con 0x56194a704d30 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.190+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56194a620a20 con 0x56194a57fe40 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.190+0000 7f11da000700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f11c40b2a70 0x7f11c40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a6ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11da000700 1 -- 172.21.15.12:0/337609192 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f11d000f020 con 0x7f11dc102ad0 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e1e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f11c40b2a70 0x7f11c40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a6ff1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41902/0 (socket says 172.21.15.12:41902) 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 -- 172.21.15.12:0/76708335 learned_addr learned my addr 172.21.15.12:0/76708335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e1400700 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a704d30 0x56194a6fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e1e00700 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a57fe40 0x56194a7047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 -- 172.21.15.12:0/76708335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a57fe40 msgr2=0x56194a7047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:44.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a57fe40 0x56194a7047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 -- 172.21.15.12:0/76708335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a704d30 msgr2=0x56194a6fe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a704d30 0x56194a6fe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.191+0000 7f11e0a00700 1 -- 172.21.15.12:0/76708335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56194a5e5420 con 0x56194a6fedb0 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.192+0000 7f11e1e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f11c40b2a70 0x7f11c40b4f80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11d4005fd0 tx=0x7f11d400bb10 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.192+0000 7f11e0a00700 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a6ff1c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f11c8008c40 tx=0x7f11c8008f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef94b62eda3a7e79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.192+0000 7f11cea00700 1 -- 172.21.15.12:0/76708335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8014070 con 0x56194a6fedb0 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.192+0000 7f11cea00700 1 -- 172.21.15.12:0/76708335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11c800ed70 con 0x56194a6fedb0 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.192+0000 7f11cea00700 1 -- 172.21.15.12:0/76708335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80055b0 con 0x56194a6fedb0 2024-05-27T07:31:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.193+0000 7f11f28e2b80 1 -- 172.21.15.12:0/76708335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 msgr2=0x56194a6ff1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.193+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a6ff1c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f11c8008c40 tx=0x7f11c8008f50 comp rx=0 tx=0).stop 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.194+0000 7f11f28e2b80 1 -- 172.21.15.12:0/76708335 shutdown_connections 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.194+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a6ff1c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.194+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a57fe40 0x56194a7047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.194+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/76708335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a704d30 0x56194a6fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.194+0000 7f11f28e2b80 1 -- 172.21.15.12:0/76708335 >> 172.21.15.12:0/76708335 conn(0x56194a5dffc0 msgr2=0x56194a6f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.195+0000 7f11f28e2b80 1 -- 172.21.15.12:0/76708335 shutdown_connections 2024-05-27T07:31:44.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.195+0000 7f11f28e2b80 1 -- 172.21.15.12:0/76708335 wait complete. 2024-05-27T07:31:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.195+0000 7f11f28e2b80 1 -- start start 2024-05-27T07:31:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 1 Processor -- start 2024-05-27T07:31:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a708580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a708ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a709000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56194a522fa0 con 0x56194a6fedb0 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56194a5bd760 con 0x56194a57fe40 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56194a70b800 con 0x56194a704d30 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.196+0000 7f11f28e2b80 10 client.0 _ll_register_callbacks cb 0x56194a5c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a708ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a708ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41910/0 (socket says 172.21.15.12:41910) 2024-05-27T07:31:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 -- 172.21.15.12:0/2920256415 learned_addr learned my addr 172.21.15.12:0/2920256415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:44.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e1400700 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a708580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 -- 172.21.15.12:0/2920256415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 msgr2=0x56194a709000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:44.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a709000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 -- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 msgr2=0x56194a708580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a708580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.197+0000 7f11e0a00700 1 -- 172.21.15.12:0/2920256415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56194a56c8d0 con 0x56194a6fedb0 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.198+0000 7f11e0a00700 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a708ac0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f11c8008c10 tx=0x7f11c80056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf4045aa6c65148 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.198+0000 7f11e1400700 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a708580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.198+0000 7f11c3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.198+0000 7f11c3400700 1 -- 172.21.15.12:0/2920256415 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8027060 con 0x56194a6fedb0 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.198+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c800ed70 con 0x56194a6fedb0 2024-05-27T07:31:44.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.198+0000 7f11c3400700 1 -- 172.21.15.12:0/2920256415 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f11c8004960 con 0x56194a6fedb0 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.199+0000 7f11c3400700 1 -- 172.21.15.12:0/2920256415 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f11c802e050 con 0x56194a6fedb0 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.199+0000 7f11f28e2b80 10 client.16569 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.199+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56194a5ed060 con 0x56194a6fedb0 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.199+0000 7f11c3400700 1 -- 172.21.15.12:0/2920256415 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11c800ed70 con 0x56194a6fedb0 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.200+0000 7f11c3400700 1 -- 172.21.15.12:0/2920256415 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f11c8010bd0 con 0x56194a6fedb0 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.200+0000 7f11f28e2b80 10 client.16569 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.200+0000 7f11f28e2b80 10 client.16569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.200+0000 7f11f28e2b80 20 client.16569 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.200+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a721250 0x56194a7236c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.200+0000 7f11f28e2b80 4 client.16569 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:44.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.200+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56194a62ed90 con 0x56194a721250 2024-05-27T07:31:44.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.201+0000 7f11e1400700 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a721250 0x56194a7236c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.202+0000 7f11e1400700 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a721250 0x56194a7236c0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45c0acf7c235f430 server_cookie=d9f8842ead7db2a9 in_seq=0 out_seq=0 2024-05-27T07:31:44.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.202+0000 7f11c3400700 10 client.16569 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.204+0000 7f11c3400700 1 -- 172.21.15.12:0/2920256415 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56194a62ed90 con 0x56194a721250 2024-05-27T07:31:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.204+0000 7f11c3400700 10 client.16569 handle_command_reply: tid=0 2024-05-27T07:31:44.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.204+0000 7f11f28e2b80 1 client.16569 shutdown 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 msgr2=0x56194a708ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a708ac0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f11c8008c10 tx=0x7f11c80056f0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 shutdown_connections 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a721250 0x56194a7236c0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a708580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a709000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/2920256415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6fedb0 0x56194a708ac0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11c3400700 0 client.16569 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:44.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.205+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 >> 172.21.15.12:0/2920256415 conn(0x56194a5dffc0 msgr2=0x56194a6f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.206+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 shutdown_connections 2024-05-27T07:31:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.206+0000 7f11f28e2b80 1 -- 172.21.15.12:0/2920256415 wait complete. 2024-05-27T07:31:44.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.206+0000 7f11f28e2b80 20 client.16569 trim_cache size 0 max 16384 2024-05-27T07:31:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.232+0000 7f11f28e2b80 1 Processor -- start 2024-05-27T07:31:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.233+0000 7f11f28e2b80 1 -- start start 2024-05-27T07:31:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.233+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a6fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.233+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a704d30 0x56194a706150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.233+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a6f9b60 0x56194a71d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.233+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1ad8e0 con 0x56194a704d30 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.233+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1acb40 con 0x56194a57fe40 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.233+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1ad210 con 0x56194a6f9b60 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.234+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a704d30 0x56194a706150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.234+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a704d30 0x56194a706150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41918/0 (socket says 172.21.15.12:41918) 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.234+0000 7f11e0a00700 1 -- 172.21.15.12:0/1652946421 learned_addr learned my addr 172.21.15.12:0/1652946421 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.234+0000 7f11e1400700 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a6fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.234+0000 7f11e1e00700 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a6f9b60 0x56194a71d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.235+0000 7f11e0a00700 1 -- 172.21.15.12:0/1652946421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a6f9b60 msgr2=0x56194a71d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.235+0000 7f11e0a00700 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a6f9b60 0x56194a71d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.235+0000 7f11e0a00700 1 -- 172.21.15.12:0/1652946421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 msgr2=0x56194a6fe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.235+0000 7f11e0a00700 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a6fe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.235+0000 7f11e0a00700 1 -- 172.21.15.12:0/1652946421 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56194a6ff880 con 0x56194a704d30 2024-05-27T07:31:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.235+0000 7f11e1e00700 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a6f9b60 0x56194a71d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.235+0000 7f11e1400700 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a6fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.236+0000 7f11e0a00700 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a704d30 0x56194a706150 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f11c8008c40 tx=0x7f11c8005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a9948a4808df05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.236+0000 7f11ce000700 1 -- 172.21.15.12:0/1652946421 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8027040 con 0x56194a704d30 2024-05-27T07:31:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.237+0000 7f11ce000700 1 -- 172.21.15.12:0/1652946421 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11c800eb70 con 0x56194a704d30 2024-05-27T07:31:44.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.237+0000 7f11ce000700 1 -- 172.21.15.12:0/1652946421 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8014070 con 0x56194a704d30 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 -- 172.21.15.12:0/1652946421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a704d30 msgr2=0x56194a706150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a704d30 0x56194a706150 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f11c8008c40 tx=0x7f11c8005ae0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 -- 172.21.15.12:0/1652946421 shutdown_connections 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a6fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a6f9b60 0x56194a71d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/1652946421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a704d30 0x56194a706150 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 -- 172.21.15.12:0/1652946421 >> 172.21.15.12:0/1652946421 conn(0x56194a5dffc0 msgr2=0x56194a708360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:44.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 -- 172.21.15.12:0/1652946421 shutdown_connections 2024-05-27T07:31:44.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.238+0000 7f11f28e2b80 1 -- 172.21.15.12:0/1652946421 wait complete. 2024-05-27T07:31:44.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.239+0000 7f11f28e2b80 1 -- start start 2024-05-27T07:31:44.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.239+0000 7f11f28e2b80 1 Processor -- start 2024-05-27T07:31:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a719550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6f9b60 0x56194a719a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11f28e2b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a716320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56194a706af0 con 0x56194a6f9b60 2024-05-27T07:31:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56194a6ff1c0 con 0x56194a57fe40 2024-05-27T07:31:44.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11f28e2b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56194a6ff650 con 0x56194a704d30 2024-05-27T07:31:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11f28e2b80 10 client.0 _ll_register_callbacks cb 0x56194a5c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6f9b60 0x56194a719a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6f9b60 0x56194a719a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41920/0 (socket says 172.21.15.12:41920) 2024-05-27T07:31:44.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.240+0000 7f11e0a00700 1 -- 172.21.15.12:0/928480724 learned_addr learned my addr 172.21.15.12:0/928480724 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e1400700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a719550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e1e00700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a716320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e0a00700 1 -- 172.21.15.12:0/928480724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 msgr2=0x56194a716320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e0a00700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a716320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e0a00700 1 -- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 msgr2=0x56194a719550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e0a00700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a719550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e0a00700 1 -- 172.21.15.12:0/928480724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56194a716a60 con 0x56194a6f9b60 2024-05-27T07:31:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e1e00700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a716320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11e0a00700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6f9b60 0x56194a719a90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f11c8018520 tx=0x7f11c80056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882c41ff82932994 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.241+0000 7f11c3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11c3400700 1 -- 172.21.15.12:0/928480724 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8014040 con 0x56194a6f9b60 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11c3400700 1 -- 172.21.15.12:0/928480724 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f11c8037040 con 0x56194a6f9b60 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c800eb70 con 0x56194a6f9b60 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11c3400700 1 -- 172.21.15.12:0/928480724 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f11c8045050 con 0x56194a6f9b60 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11f28e2b80 10 client.16575 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56194a78c680 con 0x56194a6f9b60 2024-05-27T07:31:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11c3400700 1 -- 172.21.15.12:0/928480724 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11c800eb70 con 0x56194a6f9b60 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.242+0000 7f11c3400700 1 -- 172.21.15.12:0/928480724 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f11c8004020 con 0x56194a6f9b60 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.243+0000 7f11f28e2b80 10 client.16575 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.243+0000 7f11f28e2b80 10 client.16575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.243+0000 7f11f28e2b80 20 client.16575 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.243+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a78cb10 0x56194a78ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.243+0000 7f11f28e2b80 4 client.16575 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.243+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56194a628570 con 0x56194a78cb10 2024-05-27T07:31:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.243+0000 7f11e1400700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a78cb10 0x56194a78ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.245+0000 7f11e1400700 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a78cb10 0x56194a78ef30 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8f27bfc8b8dddfb server_cookie=a25a2adb920169ac in_seq=0 out_seq=0 2024-05-27T07:31:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.245+0000 7f11c3400700 10 client.16575 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.245+0000 7f11c3400700 1 -- 172.21.15.12:0/928480724 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56194a628570 con 0x56194a78cb10 2024-05-27T07:31:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.245+0000 7f11c3400700 10 client.16575 handle_command_reply: tid=0 2024-05-27T07:31:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.245+0000 7f11f28e2b80 1 client.16575 shutdown 2024-05-27T07:31:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.245+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6f9b60 msgr2=0x56194a719a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.245+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6f9b60 0x56194a719a90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f11c8018520 tx=0x7f11c80056f0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 shutdown_connections 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56194a78cb10 0x56194a78ef30 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56194a57fe40 0x56194a719550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56194a704d30 0x56194a716320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 --2- 172.21.15.12:0/928480724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56194a6f9b60 0x56194a719a90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 >> 172.21.15.12:0/928480724 conn(0x56194a5dffc0 msgr2=0x56194a703e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11c3400700 0 client.16575 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 shutdown_connections 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 1 -- 172.21.15.12:0/928480724 wait complete. 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.246+0000 7f11f28e2b80 20 client.16575 trim_cache size 0 max 16384 2024-05-27T07:31:44.240 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:31:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:31:44.241 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:31:44.241 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:31:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f11c40b2a70 msgr2=0x7f11c40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f11c40b2a70 0x7f11c40b4f80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11d4005fd0 tx=0x7f11d400bb10 comp rx=0 tx=0).stop 2024-05-27T07:31:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 msgr2=0x7f11dc19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc19cf50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f11d0007dc0 tx=0x7f11d0007ea0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 shutdown_connections 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f11c40b2a70 0x7f11c40b4f80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11dc109840 0x7f11dc1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11dc102ad0 0x7f11dc19cf50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 --2- 172.21.15.12:0/337609192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11dc108ef0 0x7f11dc19d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 >> 172.21.15.12:0/337609192 conn(0x7f11dc0fe4d0 msgr2=0x7f11dc0ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 shutdown_connections 2024-05-27T07:31:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:44.247+0000 7f11e3e00700 1 -- 172.21.15.12:0/337609192 wait complete. 2024-05-27T07:31:45.178 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:31:45.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:31:45.663 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:45.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:45 smithi162 ceph-mon[35676]: pgmap v329: 129 pgs: 129 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-05-27T07:31:45.977 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:45 smithi012 ceph-mon[31473]: pgmap v329: 129 pgs: 129 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-05-27T07:31:45.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:45 smithi062 ceph-mon[35304]: pgmap v329: 129 pgs: 129 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-05-27T07:31:47.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:47 smithi162 ceph-mon[35676]: pgmap v330: 129 pgs: 129 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 915 KiB/s wr, 95 op/s 2024-05-27T07:31:47.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:47 smithi062 ceph-mon[35304]: pgmap v330: 129 pgs: 129 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 915 KiB/s wr, 95 op/s 2024-05-27T07:31:48.042 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:47 smithi012 ceph-mon[31473]: pgmap v330: 129 pgs: 129 active+clean; 137 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 915 KiB/s wr, 95 op/s 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 -- 172.21.15.12:0/1676296365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105450 msgr2=0x7f4a64105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 --2- 172.21.15.12:0/1676296365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105450 0x7f4a64105860 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009990 tx=0x7f4a54009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 -- 172.21.15.12:0/1676296365 shutdown_connections 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 --2- 172.21.15.12:0/1676296365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a6410e8c0 0x7f4a64110dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 --2- 172.21.15.12:0/1676296365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105450 0x7f4a64105860 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 --2- 172.21.15.12:0/1676296365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64105da0 0x7f4a6410e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 -- 172.21.15.12:0/1676296365 >> 172.21.15.12:0/1676296365 conn(0x7f4a640fede0 msgr2=0x7f4a64101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 -- 172.21.15.12:0/1676296365 shutdown_connections 2024-05-27T07:31:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.721+0000 7f4a6b200700 1 -- 172.21.15.12:0/1676296365 wait complete. 2024-05-27T07:31:48.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.723+0000 7f4a6b200700 1 Processor -- start 2024-05-27T07:31:48.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.723+0000 7f4a6b200700 1 -- start start 2024-05-27T07:31:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.724+0000 7f4a6b200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a64105450 0x7f4a641a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.724+0000 7f4a6b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105da0 0x7f4a641a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.724+0000 7f4a6b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6410e8c0 0x7f4a641ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.724+0000 7f4a6b200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4a64114370 con 0x7f4a64105da0 2024-05-27T07:31:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.724+0000 7f4a6b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4a641141f0 con 0x7f4a6410e8c0 2024-05-27T07:31:48.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.724+0000 7f4a6b200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a641144f0 con 0x7f4a64105450 2024-05-27T07:31:48.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.725+0000 7f4a63e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a64105450 0x7f4a641a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:48.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.725+0000 7f4a63400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105da0 0x7f4a641a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:48.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.725+0000 7f4a63e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a64105450 0x7f4a641a2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:57246/0 (socket says 172.21.15.12:57246) 2024-05-27T07:31:48.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.725+0000 7f4a63e00700 1 -- 172.21.15.12:0/295147789 learned_addr learned my addr 172.21.15.12:0/295147789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:48.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.725+0000 7f4a69200700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6410e8c0 0x7f4a641ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:48.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.726+0000 7f4a63400700 1 -- 172.21.15.12:0/295147789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a64105450 msgr2=0x7f4a641a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:48.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.726+0000 7f4a63400700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a64105450 0x7f4a641a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:48.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.726+0000 7f4a63400700 1 -- 172.21.15.12:0/295147789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6410e8c0 msgr2=0x7f4a641ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:48.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.726+0000 7f4a63400700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6410e8c0 0x7f4a641ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:48.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.726+0000 7f4a63400700 1 -- 172.21.15.12:0/295147789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a641ad610 con 0x7f4a64105da0 2024-05-27T07:31:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.726+0000 7f4a63400700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105da0 0x7f4a641a2760 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4a64076b30 tx=0x7f4a5000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.727+0000 7f4a60c00700 1 -- 172.21.15.12:0/295147789 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50017070 con 0x7f4a64105da0 2024-05-27T07:31:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.727+0000 7f4a60c00700 1 -- 172.21.15.12:0/295147789 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a50009dc0 con 0x7f4a64105da0 2024-05-27T07:31:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.727+0000 7f4a60c00700 1 -- 172.21.15.12:0/295147789 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50010cf0 con 0x7f4a64105da0 2024-05-27T07:31:48.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.728+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a641ad900 con 0x7f4a64105da0 2024-05-27T07:31:48.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.728+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a641addc0 con 0x7f4a64105da0 2024-05-27T07:31:48.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.729+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a640762c0 con 0x7f4a64105da0 2024-05-27T07:31:48.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.735+0000 7f4a60c00700 1 -- 172.21.15.12:0/295147789 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4a5001a070 con 0x7f4a64105da0 2024-05-27T07:31:48.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.737+0000 7f4a60c00700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a480b29f0 0x7f4a480b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:48.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.737+0000 7f4a60c00700 1 -- 172.21.15.12:0/295147789 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f4a500e2400 con 0x7f4a64105da0 2024-05-27T07:31:48.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.738+0000 7f4a63e00700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a480b29f0 0x7f4a480b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:48.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.739+0000 7f4a63e00700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a480b29f0 0x7f4a480b4f00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4a54006010 tx=0x7f4a5400b560 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:48.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:48.740+0000 7f4a60c00700 1 -- 172.21.15.12:0/295147789 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4a500a9970 con 0x7f4a64105da0 2024-05-27T07:31:49.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.107+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a6402e5e0 con 0x7f4a64105da0 2024-05-27T07:31:49.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.108+0000 7f4a60c00700 1 -- 172.21.15.12:0/295147789 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f4a500a9270 con 0x7f4a64105da0 2024-05-27T07:31:49.101 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:31:49.102 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.111+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a480b29f0 msgr2=0x7f4a480b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.111+0000 7f4a6b200700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a480b29f0 0x7f4a480b4f00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4a54006010 tx=0x7f4a5400b560 comp rx=0 tx=0).stop 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.111+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105da0 msgr2=0x7f4a641a2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.111+0000 7f4a6b200700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105da0 0x7f4a641a2760 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4a64076b30 tx=0x7f4a5000dc30 comp rx=0 tx=0).stop 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 shutdown_connections 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4a480b29f0 0x7f4a480b4f00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6410e8c0 0x7f4a641ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a64105450 0x7f4a641a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 --2- 172.21.15.12:0/295147789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a64105da0 0x7f4a641a2760 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 >> 172.21.15.12:0/295147789 conn(0x7f4a640fede0 msgr2=0x7f4a64103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 shutdown_connections 2024-05-27T07:31:49.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:49.112+0000 7f4a6b200700 1 -- 172.21.15.12:0/295147789 wait complete. 2024-05-27T07:31:49.106 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:31:49.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:49 smithi012 ceph-mon[31473]: pgmap v331: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-05-27T07:31:49.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:49 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/295147789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:49.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:49 smithi162 ceph-mon[35676]: pgmap v331: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-05-27T07:31:49.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:49 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/295147789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:49.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:49 smithi062 ceph-mon[35304]: pgmap v331: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-05-27T07:31:49.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:49 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/295147789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:50.000 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:31:50.506 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:51.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:51 smithi012 ceph-mon[31473]: pgmap v332: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1010 KiB/s wr, 104 op/s 2024-05-27T07:31:51.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:51 smithi162 ceph-mon[35676]: pgmap v332: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1010 KiB/s wr, 104 op/s 2024-05-27T07:31:51.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:51 smithi062 ceph-mon[35304]: pgmap v332: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1010 KiB/s wr, 104 op/s 2024-05-27T07:31:53.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 -- 172.21.15.12:0/1073786242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b810c5a0 msgr2=0x7fb6b810c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 --2- 172.21.15.12:0/1073786242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b810c5a0 0x7fb6b810c9b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009990 tx=0x7fb6ac009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 -- 172.21.15.12:0/1073786242 shutdown_connections 2024-05-27T07:31:53.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 --2- 172.21.15.12:0/1073786242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b8106eb0 0x7fb6b8113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 --2- 172.21.15.12:0/1073786242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b810c5a0 0x7fb6b810c9b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 --2- 172.21.15.12:0/1073786242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6b8106560 0x7fb6b8106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 -- 172.21.15.12:0/1073786242 >> 172.21.15.12:0/1073786242 conn(0x7fb6b8102040 msgr2=0x7fb6b8104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.075+0000 7fb6bf400700 1 -- 172.21.15.12:0/1073786242 shutdown_connections 2024-05-27T07:31:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.077+0000 7fb6bf400700 1 -- 172.21.15.12:0/1073786242 wait complete. 2024-05-27T07:31:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.078+0000 7fb6bf400700 1 Processor -- start 2024-05-27T07:31:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.078+0000 7fb6bf400700 1 -- start start 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.078+0000 7fb6bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b8106560 0x7fb6b819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.078+0000 7fb6bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6b8106eb0 0x7fb6b819d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.078+0000 7fb6bf400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b810c5a0 0x7fb6b81a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.078+0000 7fb6bf400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b81165b0 con 0x7fb6b8106560 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bf400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b8116430 con 0x7fb6b8106eb0 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bf400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b8116730 con 0x7fb6b810c5a0 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b8106560 0x7fb6b819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b8106560 0x7fb6b819cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49196/0 (socket says 172.21.15.12:49196) 2024-05-27T07:31:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 -- 172.21.15.12:0/3715203474 learned_addr learned my addr 172.21.15.12:0/3715203474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6b8106eb0 0x7fb6b819d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 -- 172.21.15.12:0/3715203474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b810c5a0 msgr2=0x7fb6b81a17b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:31:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b810c5a0 0x7fb6b81a17b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 -- 172.21.15.12:0/3715203474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6b8106eb0 msgr2=0x7fb6b819d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6b8106eb0 0x7fb6b819d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.079+0000 7fb6bca00700 1 -- 172.21.15.12:0/3715203474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b81a1e50 con 0x7fb6b8106560 2024-05-27T07:31:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.080+0000 7fb6bca00700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b8106560 0x7fb6b819cf80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0098e0 tx=0x7fb6ac009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:53.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.080+0000 7fb6b5600700 1 -- 172.21.15.12:0/3715203474 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac027070 con 0x7fb6b8106560 2024-05-27T07:31:53.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.080+0000 7fb6b5600700 1 -- 172.21.15.12:0/3715203474 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6ac022470 con 0x7fb6b8106560 2024-05-27T07:31:53.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.081+0000 7fb6b5600700 1 -- 172.21.15.12:0/3715203474 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac017650 con 0x7fb6b8106560 2024-05-27T07:31:53.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.081+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b81a20e0 con 0x7fb6b8106560 2024-05-27T07:31:53.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.081+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b81a2570 con 0x7fb6b8106560 2024-05-27T07:31:53.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.086+0000 7fb6b5600700 1 -- 172.21.15.12:0/3715203474 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb6ac009d10 con 0x7fb6b8106560 2024-05-27T07:31:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.088+0000 7fb6b5600700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb6a00b2a70 0x7fb6a00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.088+0000 7fb6b5600700 1 -- 172.21.15.12:0/3715203474 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7fb6ac0e35d0 con 0x7fb6b8106560 2024-05-27T07:31:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.089+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb6a00b2a70 0x7fb6a00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.089+0000 7fb6cdf94b80 1 Processor -- start 2024-05-27T07:31:53.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6cdf94b80 1 -- start start 2024-05-27T07:31:53.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb6a00b2a70 0x7fb6a00b4f80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb6b81056f0 tx=0x7fb6a4009260 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d615330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d615870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d622110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a14d543e40 con 0x55a14d615e70 2024-05-27T07:31:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a14d5339e0 con 0x55a14d496e40 2024-05-27T07:31:53.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.090+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a14d537a20 con 0x55a14d5829f0 2024-05-27T07:31:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d622110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d622110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49212/0 (socket says 172.21.15.12:49212) 2024-05-27T07:31:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 -- 172.21.15.12:0/1313182723 learned_addr learned my addr 172.21.15.12:0/1313182723 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bca00700 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d615330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d615870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 -- 172.21.15.12:0/1313182723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 msgr2=0x55a14d615330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d615330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 -- 172.21.15.12:0/1313182723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 msgr2=0x55a14d615870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d615870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.091+0000 7fb6bd400700 1 -- 172.21.15.12:0/1313182723 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14d4fc420 con 0x55a14d615e70 2024-05-27T07:31:53.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.092+0000 7fb6bd400700 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d622110 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000e400 tx=0x7fb6b0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36321b974e9b3b2f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.092+0000 7fb6aa000700 1 -- 172.21.15.12:0/1313182723 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0004bb0 con 0x55a14d615e70 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.092+0000 7fb6aa000700 1 -- 172.21.15.12:0/1313182723 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6b0004d50 con 0x55a14d615e70 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.092+0000 7fb6aa000700 1 -- 172.21.15.12:0/1313182723 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b00042c0 con 0x55a14d615e70 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/1313182723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 msgr2=0x55a14d622110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d622110 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000e400 tx=0x7fb6b0009920 comp rx=0 tx=0).stop 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/1313182723 shutdown_connections 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d622110 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d615330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/1313182723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d615870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/1313182723 >> 172.21.15.12:0/1313182723 conn(0x55a14d530390 msgr2=0x55a14d6114c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/1313182723 shutdown_connections 2024-05-27T07:31:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.094+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/1313182723 wait complete. 2024-05-27T07:31:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.095+0000 7fb6cdf94b80 1 -- start start 2024-05-27T07:31:53.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.095+0000 7fb6cdf94b80 1 Processor -- start 2024-05-27T07:31:53.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d496e40 0x55a14d61f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d615e70 0x55a14d620160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a14d439fa0 con 0x55a14d496e40 2024-05-27T07:31:53.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a14d4d4760 con 0x55a14d615e70 2024-05-27T07:31:53.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a14d622b20 con 0x55a14d5829f0 2024-05-27T07:31:53.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6cdf94b80 10 client.0 _ll_register_callbacks cb 0x55a14d4ddc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:53.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.096+0000 7fb6bca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d496e40 0x55a14d61f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.097+0000 7fb6b7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.097+0000 7fb6bd400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d615e70 0x55a14d620160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.097+0000 7fb6b7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:45100/0 (socket says 172.21.15.12:45100) 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.097+0000 7fb6b7e00700 1 -- 172.21.15.12:0/2152112392 learned_addr learned my addr 172.21.15.12:0/2152112392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.098+0000 7fb6b7e00700 1 -- 172.21.15.12:0/2152112392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d615e70 msgr2=0x55a14d620160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.098+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d615e70 0x55a14d620160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.098+0000 7fb6b7e00700 1 -- 172.21.15.12:0/2152112392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d496e40 msgr2=0x55a14d61f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.098+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d496e40 0x55a14d61f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.098+0000 7fb6b7e00700 1 -- 172.21.15.12:0/2152112392 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a14d4838d0 con 0x55a14d5829f0 2024-05-27T07:31:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.098+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61fc20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8075b70 tx=0x7fb6a40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0c458c892340c8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.098+0000 7fb69ea00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:31:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.099+0000 7fb69ea00700 1 -- 172.21.15.12:0/2152112392 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4004220 con 0x55a14d5829f0 2024-05-27T07:31:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.099+0000 7fb69ea00700 1 -- 172.21.15.12:0/2152112392 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7fb6a40043c0 con 0x55a14d5829f0 2024-05-27T07:31:53.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.099+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b0004d50 con 0x55a14d5829f0 2024-05-27T07:31:53.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.100+0000 7fb69ea00700 1 -- 172.21.15.12:0/2152112392 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fb6a4003680 con 0x55a14d5829f0 2024-05-27T07:31:53.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.100+0000 7fb6cdf94b80 10 client.26176 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:53.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.100+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a14d504060 con 0x55a14d5829f0 2024-05-27T07:31:53.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb69ea00700 1 -- 172.21.15.12:0/2152112392 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6b0004d50 con 0x55a14d5829f0 2024-05-27T07:31:53.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb69ea00700 1 -- 172.21.15.12:0/2152112392 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fb6a4029890 con 0x55a14d5829f0 2024-05-27T07:31:53.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb6cdf94b80 10 client.26176 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:53.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb6cdf94b80 10 client.26176 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:31:53.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb6cdf94b80 20 client.26176 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:53.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6387a0 0x55a14d63ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb6cdf94b80 4 client.26176 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:53.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.101+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a14d545d90 con 0x55a14d6387a0 2024-05-27T07:31:53.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.102+0000 7fb6bca00700 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6387a0 0x55a14d63ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.103+0000 7fb6bca00700 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6387a0 0x55a14d63ac50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fff66ea697df49f7 server_cookie=fee6608d92046aa6 in_seq=0 out_seq=0 2024-05-27T07:31:53.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.103+0000 7fb69ea00700 10 client.26176 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:31:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb69ea00700 1 -- 172.21.15.12:0/2152112392 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a14d545d90 con 0x55a14d6387a0 2024-05-27T07:31:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb69ea00700 10 client.26176 handle_command_reply: tid=0 2024-05-27T07:31:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 client.26176 shutdown 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 msgr2=0x55a14d61fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61fc20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8075b70 tx=0x7fb6a40047c0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 shutdown_connections 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6387a0 0x55a14d63ac50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d615e70 0x55a14d620160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d496e40 0x55a14d61f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2152112392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61fc20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb69ea00700 0 client.26176 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:31:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.105+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 >> 172.21.15.12:0/2152112392 conn(0x55a14d530390 msgr2=0x55a14d615370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.106+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 shutdown_connections 2024-05-27T07:31:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.106+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2152112392 wait complete. 2024-05-27T07:31:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.106+0000 7fb6cdf94b80 20 client.26176 trim_cache size 0 max 16384 2024-05-27T07:31:53.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.129+0000 7fb6cdf94b80 1 Processor -- start 2024-05-27T07:31:53.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.130+0000 7fb6cdf94b80 1 -- start start 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.130+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d61e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.130+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.130+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d6195f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.130+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a14d611680 con 0x55a14d615e70 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.130+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b81ad930 con 0x55a14d496e40 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.130+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b81acb90 con 0x55a14d5829f0 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d6195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d61e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d6195f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49242/0 (socket says 172.21.15.12:49242) 2024-05-27T07:31:53.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 -- 172.21.15.12:0/2341332034 learned_addr learned my addr 172.21.15.12:0/2341332034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 -- 172.21.15.12:0/2341332034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 msgr2=0x55a14d61aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 -- 172.21.15.12:0/2341332034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 msgr2=0x55a14d61e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d61e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.131+0000 7fb6bd400700 1 -- 172.21.15.12:0/2341332034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14d621f80 con 0x55a14d615e70 2024-05-27T07:31:53.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.132+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:53.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.132+0000 7fb6bd400700 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d6195f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000e3a0 tx=0x7fb6b0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5286ae524623f699 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:53.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.132+0000 7fb6a9600700 1 -- 172.21.15.12:0/2341332034 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0022070 con 0x55a14d615e70 2024-05-27T07:31:53.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.133+0000 7fb6a9600700 1 -- 172.21.15.12:0/2341332034 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6b0009350 con 0x55a14d615e70 2024-05-27T07:31:53.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.133+0000 7fb6a9600700 1 -- 172.21.15.12:0/2341332034 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0007b60 con 0x55a14d615e70 2024-05-27T07:31:53.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2341332034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 msgr2=0x55a14d6195f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d6195f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000e3a0 tx=0x7fb6b0009260 comp rx=0 tx=0).stop 2024-05-27T07:31:53.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2341332034 shutdown_connections 2024-05-27T07:31:53.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d496e40 0x55a14d61e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d6195f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/2341332034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d5829f0 0x55a14d61aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2341332034 >> 172.21.15.12:0/2341332034 conn(0x55a14d530390 msgr2=0x55a14d618ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2341332034 shutdown_connections 2024-05-27T07:31:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/2341332034 wait complete. 2024-05-27T07:31:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.134+0000 7fb6cdf94b80 1 -- start start 2024-05-27T07:31:53.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.135+0000 7fb6cdf94b80 1 Processor -- start 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.135+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d496e40 0x55a14d628330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.135+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d5829f0 0x55a14d6236e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.135+0000 7fb6cdf94b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d623c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.135+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a14d61f4c0 con 0x55a14d615e70 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.135+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b81ad260 con 0x55a14d5829f0 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.135+0000 7fb6cdf94b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a14d63b230 con 0x55a14d496e40 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.136+0000 7fb6cdf94b80 10 client.0 _ll_register_callbacks cb 0x55a14d4ddc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:53.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.136+0000 7fb6bd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d623c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.136+0000 7fb6bca00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d496e40 0x55a14d628330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.136+0000 7fb6bca00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d496e40 0x55a14d628330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:45110/0 (socket says 172.21.15.12:45110) 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.136+0000 7fb6bca00700 1 -- 172.21.15.12:0/3494571156 learned_addr learned my addr 172.21.15.12:0/3494571156 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.136+0000 7fb6b7e00700 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d5829f0 0x55a14d6236e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb6bd400700 1 -- 172.21.15.12:0/3494571156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d496e40 msgr2=0x55a14d628330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb6bd400700 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d496e40 0x55a14d628330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb6bd400700 1 -- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d5829f0 msgr2=0x55a14d6236e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb6bd400700 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d5829f0 0x55a14d6236e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb6bd400700 1 -- 172.21.15.12:0/3494571156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a14d624210 con 0x55a14d615e70 2024-05-27T07:31:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb6bd400700 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d623c20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0009790 tx=0x7fb6b0007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb9087ebad895a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb69ea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.137+0000 7fb69ea00700 1 -- 172.21.15.12:0/3494571156 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b002b070 con 0x55a14d615e70 2024-05-27T07:31:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.138+0000 7fb69ea00700 1 -- 172.21.15.12:0/3494571156 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7fb6b0003a00 con 0x55a14d615e70 2024-05-27T07:31:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.138+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b0009350 con 0x55a14d615e70 2024-05-27T07:31:53.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.138+0000 7fb69ea00700 1 -- 172.21.15.12:0/3494571156 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fb6b0022030 con 0x55a14d615e70 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.138+0000 7fb6cdf94b80 10 client.16599 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.138+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a14d6a3010 con 0x55a14d615e70 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.138+0000 7fb69ea00700 1 -- 172.21.15.12:0/3494571156 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb6b0009350 con 0x55a14d615e70 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.138+0000 7fb69ea00700 1 -- 172.21.15.12:0/3494571156 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fb6b0005900 con 0x55a14d615e70 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.139+0000 7fb6cdf94b80 10 client.16599 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.139+0000 7fb6cdf94b80 10 client.16599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.139+0000 7fb6cdf94b80 20 client.16599 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.139+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6a3460 0x55a14d6a5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.139+0000 7fb6cdf94b80 4 client.16599 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.139+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a14d53f570 con 0x55a14d6a3460 2024-05-27T07:31:53.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.139+0000 7fb6bca00700 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6a3460 0x55a14d6a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.141+0000 7fb69ea00700 10 client.16599 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:31:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.141+0000 7fb6bca00700 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6a3460 0x55a14d6a5880 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b584a844da76ba54 server_cookie=120f605716beb253 in_seq=0 out_seq=0 2024-05-27T07:31:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.141+0000 7fb69ea00700 1 -- 172.21.15.12:0/3494571156 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a14d53f570 con 0x55a14d6a3460 2024-05-27T07:31:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.141+0000 7fb69ea00700 10 client.16599 handle_command_reply: tid=0 2024-05-27T07:31:53.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.141+0000 7fb6cdf94b80 1 client.16599 shutdown 2024-05-27T07:31:53.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 msgr2=0x55a14d623c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d623c20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0009790 tx=0x7fb6b0007ef0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 shutdown_connections 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a14d615e70 0x55a14d623c20 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a14d496e40 0x55a14d628330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55a14d6a3460 0x55a14d6a5880 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 --2- 172.21.15.12:0/3494571156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a14d5829f0 0x55a14d6236e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb69ea00700 0 client.16599 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 >> 172.21.15.12:0/3494571156 conn(0x55a14d530390 msgr2=0x55a14d620d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 shutdown_connections 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 1 -- 172.21.15.12:0/3494571156 wait complete. 2024-05-27T07:31:53.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.142+0000 7fb6cdf94b80 20 client.16599 trim_cache size 0 max 16384 2024-05-27T07:31:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:31:53.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb6a00b2a70 msgr2=0x7fb6a00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb6a00b2a70 0x7fb6a00b4f80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb6b81056f0 tx=0x7fb6a4009260 comp rx=0 tx=0).stop 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b8106560 msgr2=0x7fb6b819cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b8106560 0x7fb6b819cf80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0098e0 tx=0x7fb6ac009550 comp rx=0 tx=0).stop 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 shutdown_connections 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb6a00b2a70 0x7fb6a00b4f80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b810c5a0 0x7fb6b81a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6b8106560 0x7fb6b819cf80 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 --2- 172.21.15.12:0/3715203474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6b8106eb0 0x7fb6b819d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.143+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 >> 172.21.15.12:0/3715203474 conn(0x7fb6b8102040 msgr2=0x7fb6b81039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.144+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 shutdown_connections 2024-05-27T07:31:53.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:53.144+0000 7fb6bf400700 1 -- 172.21.15.12:0/3715203474 wait complete. 2024-05-27T07:31:54.254 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:31:54.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:54 smithi012 ceph-mon[31473]: pgmap v333: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1010 KiB/s wr, 104 op/s 2024-05-27T07:31:54.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:54 smithi162 ceph-mon[35676]: pgmap v333: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1010 KiB/s wr, 104 op/s 2024-05-27T07:31:54.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:54 smithi062 ceph-mon[35304]: pgmap v333: 129 pgs: 129 active+clean; 138 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1010 KiB/s wr, 104 op/s 2024-05-27T07:31:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:56.096 INFO:tasks.workunit.client.0.smithi012.stdout:50000 write 0x39280 thru 0x48ee3 (0xfc64 bytes) 2024-05-27T07:31:56.098 INFO:tasks.workunit.client.0.smithi012.stdout:All 50000 operations completed A-OK! 2024-05-27T07:31:56.103 INFO:tasks.workunit.client.0.smithi012.stderr:+ ./fsx -z 10MB -N 50000 -p 10000 -l 10485760 2024-05-27T07:31:56.108 INFO:tasks.workunit.client.0.smithi012.stdout:Seed set to 1 2024-05-27T07:31:56.324 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-05-27T07:31:56.325 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-05-27T07:31:56.325 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support clone range, disabling! 2024-05-27T07:31:56.325 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support dedupe range, disabling! 2024-05-27T07:31:56.338 INFO:tasks.workunit.client.0.smithi012.stdout:skipping zero length punch hole 2024-05-27T07:31:56.347 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0x81c806 2024-05-27T07:31:56.671 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:56 smithi012 ceph-mon[31473]: pgmap v334: 129 pgs: 129 active+clean; 139 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-05-27T07:31:56.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:56 smithi162 ceph-mon[35676]: pgmap v334: 129 pgs: 129 active+clean; 139 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-05-27T07:31:56.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:56 smithi062 ceph-mon[35304]: pgmap v334: 129 pgs: 129 active+clean; 139 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-05-27T07:31:56.800 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0x9bc582 2024-05-27T07:31:56.897 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0x9eecbc 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 -- 172.21.15.12:0/2422458084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106570 msgr2=0x7fced0106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 --2- 172.21.15.12:0/2422458084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106570 0x7fced0106980 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009990 tx=0x7fceb8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 -- 172.21.15.12:0/2422458084 shutdown_connections 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 --2- 172.21.15.12:0/2422458084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced010c5b0 0x7fced010c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 --2- 172.21.15.12:0/2422458084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fced0106ec0 0x7fced0113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 --2- 172.21.15.12:0/2422458084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106570 0x7fced0106980 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 -- 172.21.15.12:0/2422458084 >> 172.21.15.12:0/2422458084 conn(0x7fced0102070 msgr2=0x7fced01044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 -- 172.21.15.12:0/2422458084 shutdown_connections 2024-05-27T07:31:57.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.115+0000 7fced7200700 1 -- 172.21.15.12:0/2422458084 wait complete. 2024-05-27T07:31:57.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.116+0000 7fced7200700 1 Processor -- start 2024-05-27T07:31:57.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fced7200700 1 -- start start 2024-05-27T07:31:57.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fced7200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fced0106570 0x7fced019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fced7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106ec0 0x7fced019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fced7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced010c5b0 0x7fced01a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fced7200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fced0116610 con 0x7fced0106ec0 2024-05-27T07:31:57.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fced7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fced0116490 con 0x7fced010c5b0 2024-05-27T07:31:57.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fced7200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fced0116790 con 0x7fced0106570 2024-05-27T07:31:57.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.117+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106ec0 0x7fced019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106ec0 0x7fced019d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49276/0 (socket says 172.21.15.12:49276) 2024-05-27T07:31:57.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 -- 172.21.15.12:0/831104597 learned_addr learned my addr 172.21.15.12:0/831104597 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcecfe00700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fced0106570 0x7fced019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fced5200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced010c5b0 0x7fced01a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 -- 172.21.15.12:0/831104597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fced0106570 msgr2=0x7fced019d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fced0106570 0x7fced019d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 -- 172.21.15.12:0/831104597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced010c5b0 msgr2=0x7fced01a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced010c5b0 0x7fced01a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 -- 172.21.15.12:0/831104597 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced01a1ec0 con 0x7fced0106ec0 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcecfe00700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fced0106570 0x7fced019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.118+0000 7fcec7e00700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106ec0 0x7fced019d640 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fceb80098e0 tx=0x7fceb8003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.119+0000 7fcecd600700 1 -- 172.21.15.12:0/831104597 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8027070 con 0x7fced0106ec0 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.119+0000 7fcecd600700 1 -- 172.21.15.12:0/831104597 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fceb800fc00 con 0x7fced0106ec0 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.119+0000 7fcecd600700 1 -- 172.21.15.12:0/831104597 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8022ce0 con 0x7fced0106ec0 2024-05-27T07:31:57.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.119+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced01a2150 con 0x7fced0106ec0 2024-05-27T07:31:57.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.119+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced01a2680 con 0x7fced0106ec0 2024-05-27T07:31:57.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.123+0000 7fcecd600700 1 -- 172.21.15.12:0/831104597 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fceb802a070 con 0x7fced0106ec0 2024-05-27T07:31:57.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.124+0000 7fcee5dfbb80 1 Processor -- start 2024-05-27T07:31:57.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcee5dfbb80 1 -- start start 2024-05-27T07:31:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e91870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e9e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55eff2dbfe40 con 0x55eff2dfe9f0 2024-05-27T07:31:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55eff2daf9e0 con 0x55eff2d12e40 2024-05-27T07:31:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eff2db3a20 con 0x55eff2e91e70 2024-05-27T07:31:57.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.125+0000 7fcecd600700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fceb40b2a70 0x7fceb40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.126+0000 7fcecd600700 1 -- 172.21.15.12:0/831104597 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7fceb80e3a30 con 0x7fced0106ec0 2024-05-27T07:31:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.126+0000 7fced5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.126+0000 7fcecfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e91870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.126+0000 7fced5200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49280/0 (socket says 172.21.15.12:49280) 2024-05-27T07:31:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.126+0000 7fced5200700 1 -- 172.21.15.12:0/3133116427 learned_addr learned my addr 172.21.15.12:0/3133116427 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.126+0000 7fced5200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fceb40b2a70 0x7fceb40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fcec7e00700 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e9e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fced5200700 1 -- 172.21.15.12:0/3133116427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 msgr2=0x55eff2e9e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fced5200700 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e9e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fced5200700 1 -- 172.21.15.12:0/3133116427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 msgr2=0x55eff2e91870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fced5200700 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e91870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fced5200700 1 -- 172.21.15.12:0/3133116427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff2d78420 con 0x55eff2dfe9f0 2024-05-27T07:31:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fced5200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fceb40b2a70 0x7fceb40b4f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcec80149d0 tx=0x7fcec8006640 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:57.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.127+0000 7fced5200700 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91330 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fcec8014fd0 tx=0x7fcec8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f8ea2451c82664 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:57.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.128+0000 7fcec5600700 1 -- 172.21.15.12:0/3133116427 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec801c070 con 0x55eff2dfe9f0 2024-05-27T07:31:57.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.128+0000 7fcec5600700 1 -- 172.21.15.12:0/3133116427 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcec8018cd0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.128+0000 7fcec5600700 1 -- 172.21.15.12:0/3133116427 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec800a070 con 0x55eff2dfe9f0 2024-05-27T07:31:57.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3133116427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 msgr2=0x55eff2e91330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91330 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fcec8014fd0 tx=0x7fcec8019000 comp rx=0 tx=0).stop 2024-05-27T07:31:57.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3133116427 shutdown_connections 2024-05-27T07:31:57.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e9e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91330 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3133116427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e91870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3133116427 >> 172.21.15.12:0/3133116427 conn(0x55eff2dac390 msgr2=0x55eff2e8d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:57.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.129+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3133116427 shutdown_connections 2024-05-27T07:31:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.130+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3133116427 wait complete. 2024-05-27T07:31:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.130+0000 7fcee5dfbb80 1 -- start start 2024-05-27T07:31:57.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 1 Processor -- start 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 0x55eff2e9b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e9bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2e9c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55eff2cb5fa0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55eff2d50760 con 0x55eff2e91e70 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eff2e9eac0 con 0x55eff2d12e40 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.131+0000 7fcee5dfbb80 10 client.0 _ll_register_callbacks cb 0x55eff2d59c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e9bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e9bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49286/0 (socket says 172.21.15.12:49286) 2024-05-27T07:31:57.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 -- 172.21.15.12:0/229507876 learned_addr learned my addr 172.21.15.12:0/229507876 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:57.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fced5200700 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2e9c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcecfe00700 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 0x55eff2e9b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 -- 172.21.15.12:0/229507876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 msgr2=0x55eff2e9b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 0x55eff2e9b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 -- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 msgr2=0x55eff2e9c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2e9c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.132+0000 7fcec7e00700 1 -- 172.21.15.12:0/229507876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff2cff8d0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.133+0000 7fcec7e00700 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e9bbf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fceb80a7ba0 tx=0x7fceb80a61f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9865d9da2c2594a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.133+0000 7fcebe000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.133+0000 7fcebe000700 1 -- 172.21.15.12:0/229507876 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80aa930 con 0x55eff2dfe9f0 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.133+0000 7fcebe000700 1 -- 172.21.15.12:0/229507876 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7fceb80b1070 con 0x55eff2dfe9f0 2024-05-27T07:31:57.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.133+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec8018cd0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.134+0000 7fcebe000700 1 -- 172.21.15.12:0/229507876 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fceb80aece0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.134+0000 7fcee5dfbb80 10 client.16611 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.134+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55eff2d80060 con 0x55eff2dfe9f0 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.134+0000 7fcebe000700 1 -- 172.21.15.12:0/229507876 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcec8018cd0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.135+0000 7fcebe000700 1 -- 172.21.15.12:0/229507876 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fceb80bb070 con 0x55eff2dfe9f0 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.135+0000 7fcee5dfbb80 10 client.16611 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.135+0000 7fcee5dfbb80 10 client.16611 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:31:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.135+0000 7fcee5dfbb80 20 client.16611 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:57.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.135+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2eabad0 0x55eff2eadf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.135+0000 7fcee5dfbb80 4 client.16611 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:57.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.135+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eff2dc1d90 con 0x55eff2eabad0 2024-05-27T07:31:57.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.136+0000 7fcecfe00700 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2eabad0 0x55eff2eadf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.137+0000 7fcecfe00700 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2eabad0 0x55eff2eadf20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f5827afabcbf146 server_cookie=40ff9b289649dc7a in_seq=0 out_seq=0 2024-05-27T07:31:57.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.137+0000 7fcebe000700 10 client.16611 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:31:57.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.139+0000 7fcebe000700 1 -- 172.21.15.12:0/229507876 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eff2dc1d90 con 0x55eff2eabad0 2024-05-27T07:31:57.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.139+0000 7fcebe000700 10 client.16611 handle_command_reply: tid=0 2024-05-27T07:31:57.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.139+0000 7fcee5dfbb80 1 client.16611 shutdown 2024-05-27T07:31:57.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.139+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 msgr2=0x55eff2e9bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.139+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e9bbf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fceb80a7ba0 tx=0x7fceb80a61f0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 shutdown_connections 2024-05-27T07:31:57.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2eabad0 0x55eff2eadf20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2e9c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 0x55eff2e9b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/229507876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e9bbf0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcebe000700 0 client.16611 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:31:57.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 >> 172.21.15.12:0/229507876 conn(0x55eff2dac390 msgr2=0x55eff2e913b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:57.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 shutdown_connections 2024-05-27T07:31:57.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/229507876 wait complete. 2024-05-27T07:31:57.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.140+0000 7fcee5dfbb80 20 client.16611 trim_cache size 0 max 16384 2024-05-27T07:31:57.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.159+0000 7fcee5dfbb80 1 Processor -- start 2024-05-27T07:31:57.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.159+0000 7fcee5dfbb80 1 -- start start 2024-05-27T07:31:57.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.159+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 0x55eff2e96e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2eb36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fced01b1c30 con 0x55eff2dfe9f0 2024-05-27T07:31:57.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fced01b0e90 con 0x55eff2e91e70 2024-05-27T07:31:57.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fced01b1560 con 0x55eff2d12e40 2024-05-27T07:31:57.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49300/0 (socket says 172.21.15.12:49300) 2024-05-27T07:31:57.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcec7e00700 1 -- 172.21.15.12:0/123793424 learned_addr learned my addr 172.21.15.12:0/123793424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:57.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fced5200700 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2eb36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.160+0000 7fcec7e00700 1 -- 172.21.15.12:0/123793424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 msgr2=0x55eff2e96e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.161+0000 7fcec7e00700 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 0x55eff2e96e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.161+0000 7fcec7e00700 1 -- 172.21.15.12:0/123793424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 msgr2=0x55eff2eb36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.161+0000 7fcec7e00700 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2eb36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.161+0000 7fcec7e00700 1 -- 172.21.15.12:0/123793424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff2e89240 con 0x55eff2dfe9f0 2024-05-27T07:31:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.161+0000 7fcec7e00700 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91830 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fceb80aa8c0 tx=0x7fceb80cd330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d228ffe63ad05431 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.161+0000 7fcec4c00700 1 -- 172.21.15.12:0/123793424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80aabe0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.162+0000 7fcec4c00700 1 -- 172.21.15.12:0/123793424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fceb8090e50 con 0x55eff2dfe9f0 2024-05-27T07:31:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.162+0000 7fcec4c00700 1 -- 172.21.15.12:0/123793424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80aae20 con 0x55eff2dfe9f0 2024-05-27T07:31:57.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/123793424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 msgr2=0x55eff2e91830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91830 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fceb80aa8c0 tx=0x7fceb80cd330 comp rx=0 tx=0).stop 2024-05-27T07:31:57.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/123793424 shutdown_connections 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2e91e70 0x55eff2eb36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2d12e40 0x55eff2e96e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/123793424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e91830 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/123793424 >> 172.21.15.12:0/123793424 conn(0x55eff2dac390 msgr2=0x55eff2e91120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/123793424 shutdown_connections 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.163+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/123793424 wait complete. 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.164+0000 7fcee5dfbb80 1 -- start start 2024-05-27T07:31:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.164+0000 7fcee5dfbb80 1 Processor -- start 2024-05-27T07:31:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e96210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e96750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcee5dfbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e96c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55eff2e9cb40 con 0x55eff2dfe9f0 2024-05-27T07:31:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55eff2dc1d90 con 0x55eff2d12e40 2024-05-27T07:31:57.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcee5dfbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eff2e9a9d0 con 0x55eff2e91e70 2024-05-27T07:31:57.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcee5dfbb80 10 client.0 _ll_register_callbacks cb 0x55eff2d59c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:57.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e96750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcec7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e96750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49316/0 (socket says 172.21.15.12:49316) 2024-05-27T07:31:57.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.165+0000 7fcec7e00700 1 -- 172.21.15.12:0/3159148457 learned_addr learned my addr 172.21.15.12:0/3159148457 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:31:57.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fced5200700 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e96c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcecfe00700 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e96210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcec7e00700 1 -- 172.21.15.12:0/3159148457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 msgr2=0x55eff2e96c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcec7e00700 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e96c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcec7e00700 1 -- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 msgr2=0x55eff2e96210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcec7e00700 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e96210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcec7e00700 1 -- 172.21.15.12:0/3159148457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff2e9c4a0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcec7e00700 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e96750 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fceb80b3270 tx=0x7fceb80aa5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fb41998136744 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.166+0000 7fcebe000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcebe000700 1 -- 172.21.15.12:0/3159148457 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80a67f0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcebe000700 1 -- 172.21.15.12:0/3159148457 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7fceb80b10b0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceb8090e50 con 0x55eff2dfe9f0 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcebe000700 1 -- 172.21.15.12:0/3159148457 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fceb80b3700 con 0x55eff2dfe9f0 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcee5dfbb80 10 client.16617 fetch_fsmap learned FSMap version 119 2024-05-27T07:31:57.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55eff2e9c4a0 con 0x55eff2dfe9f0 2024-05-27T07:31:57.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcebe000700 1 -- 172.21.15.12:0/3159148457 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fceb8090e50 con 0x55eff2dfe9f0 2024-05-27T07:31:57.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.167+0000 7fcebe000700 1 -- 172.21.15.12:0/3159148457 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fceb8005110 con 0x55eff2dfe9f0 2024-05-27T07:31:57.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.168+0000 7fcee5dfbb80 10 client.16617 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:31:57.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.168+0000 7fcee5dfbb80 10 client.16617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:31:57.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.168+0000 7fcee5dfbb80 20 client.16617 populate_metadata read hostname 'smithi012' 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.168+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2f165f0 0x55eff2f18a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.168+0000 7fcee5dfbb80 4 client.16617 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.168+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eff2dbb570 con 0x55eff2f165f0 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.168+0000 7fcecfe00700 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2f165f0 0x55eff2f18a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.170+0000 7fcebe000700 10 client.16617 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.169+0000 7fcecfe00700 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2f165f0 0x55eff2f18a10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74ae2c6d5b16a015 server_cookie=597bb689ac2cc447 in_seq=0 out_seq=0 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.170+0000 7fcebe000700 1 -- 172.21.15.12:0/3159148457 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eff2dbb570 con 0x55eff2f165f0 2024-05-27T07:31:57.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.170+0000 7fcebe000700 10 client.16617 handle_command_reply: tid=0 2024-05-27T07:31:57.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.170+0000 7fcee5dfbb80 1 client.16617 shutdown 2024-05-27T07:31:57.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 msgr2=0x55eff2e96750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e96750 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fceb80b3270 tx=0x7fceb80aa5a0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 shutdown_connections 2024-05-27T07:31:57.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55eff2f165f0 0x55eff2f18a10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eff2d12e40 0x55eff2e96210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eff2e91e70 0x55eff2e96c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 --2- 172.21.15.12:0/3159148457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eff2dfe9f0 0x55eff2e96750 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 >> 172.21.15.12:0/3159148457 conn(0x55eff2dac390 msgr2=0x55eff2e9b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:57.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.171+0000 7fcebe000700 0 client.16617 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:31:57.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.172+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 shutdown_connections 2024-05-27T07:31:57.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.172+0000 7fcee5dfbb80 1 -- 172.21.15.12:0/3159148457 wait complete. 2024-05-27T07:31:57.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.173+0000 7fcee5dfbb80 20 client.16617 trim_cache size 0 max 16384 2024-05-27T07:31:57.167 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:31:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.175+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fceb40b2a70 msgr2=0x7fceb40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.175+0000 7fced7200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fceb40b2a70 0x7fceb40b4f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcec80149d0 tx=0x7fcec8006640 comp rx=0 tx=0).stop 2024-05-27T07:31:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.175+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106ec0 msgr2=0x7fced019d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.175+0000 7fced7200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106ec0 0x7fced019d640 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fceb80098e0 tx=0x7fceb8003680 comp rx=0 tx=0).stop 2024-05-27T07:31:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 shutdown_connections 2024-05-27T07:31:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fceb40b2a70 0x7fceb40b4f80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced010c5b0 0x7fced01a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fced0106570 0x7fced019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 --2- 172.21.15.12:0/831104597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fced0106ec0 0x7fced019d640 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 >> 172.21.15.12:0/831104597 conn(0x7fced0102070 msgr2=0x7fced0104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 shutdown_connections 2024-05-27T07:31:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:31:57.176+0000 7fced7200700 1 -- 172.21.15.12:0/831104597 wait complete. 2024-05-27T07:31:58.048 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:31:58.581 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:58.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:58 smithi162 ceph-mon[35676]: pgmap v335: 129 pgs: 129 active+clean; 139 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-05-27T07:31:58.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:58 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:58.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:58 smithi012 ceph-mon[31473]: pgmap v335: 129 pgs: 129 active+clean; 139 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-05-27T07:31:58.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:58 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:58.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:58 smithi062 ceph-mon[35304]: pgmap v335: 129 pgs: 129 active+clean; 139 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-05-27T07:31:58.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:58 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:59.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:59 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:59.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:59 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:59.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:31:59 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:31:59.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:59 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:59.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:59 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:59.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:31:59 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:31:59.788 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:59 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:59.788 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:59 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:59.788 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:31:59 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:32:00.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:00 smithi012 ceph-mon[31473]: pgmap v336: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 175 op/s 2024-05-27T07:32:00.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:00 smithi162 ceph-mon[35676]: pgmap v336: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 175 op/s 2024-05-27T07:32:00.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:00 smithi062 ceph-mon[35304]: pgmap v336: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 175 op/s 2024-05-27T07:32:01.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 -- 172.21.15.12:0/2333029290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f589010e8d0 msgr2=0x7f5890110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 --2- 172.21.15.12:0/2333029290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f589010e8d0 0x7f5890110db0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5888009990 tx=0x7f5888009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 -- 172.21.15.12:0/2333029290 shutdown_connections 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 --2- 172.21.15.12:0/2333029290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f589010e8d0 0x7f5890110db0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 --2- 172.21.15.12:0/2333029290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58901054f0 0x7f5890105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 --2- 172.21.15.12:0/2333029290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5890105e40 0x7f589010e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 -- 172.21.15.12:0/2333029290 >> 172.21.15.12:0/2333029290 conn(0x7f58900fedb0 msgr2=0x7f58901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 -- 172.21.15.12:0/2333029290 shutdown_connections 2024-05-27T07:32:01.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.133+0000 7f5896600700 1 -- 172.21.15.12:0/2333029290 wait complete. 2024-05-27T07:32:01.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.134+0000 7f5896600700 1 Processor -- start 2024-05-27T07:32:01.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.135+0000 7f5896600700 1 -- start start 2024-05-27T07:32:01.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.135+0000 7f5896600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58901054f0 0x7f58901031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.135+0000 7f5896600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5890105e40 0x7f5890103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.135+0000 7f5896600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f589010e8d0 0x7f5890103c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.135+0000 7f5896600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f58901143a0 con 0x7f58901054f0 2024-05-27T07:32:01.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.135+0000 7f5896600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5890114220 con 0x7f5890105e40 2024-05-27T07:32:01.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.135+0000 7f5896600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5890114520 con 0x7f589010e8d0 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58901054f0 0x7f58901031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58901054f0 0x7f58901031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49330/0 (socket says 172.21.15.12:49330) 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 -- 172.21.15.12:0/1728394467 learned_addr learned my addr 172.21.15.12:0/1728394467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588fe00700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f589010e8d0 0x7f5890103c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 -- 172.21.15.12:0/1728394467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f589010e8d0 msgr2=0x7f5890103c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f589010e8d0 0x7f5890103c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 -- 172.21.15.12:0/1728394467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5890105e40 msgr2=0x7f5890103720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5890105e40 0x7f5890103720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.136+0000 7f588f400700 1 -- 172.21.15.12:0/1728394467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58901a90b0 con 0x7f58901054f0 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.137+0000 7f588f400700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58901054f0 0x7f58901031e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f588000ba50 tx=0x7f588000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.137+0000 7f588cc00700 1 -- 172.21.15.12:0/1728394467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880019070 con 0x7f58901054f0 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.137+0000 7f588cc00700 1 -- 172.21.15.12:0/1728394467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5880014440 con 0x7f58901054f0 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.138+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58901a93a0 con 0x7f58901054f0 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.138+0000 7f588cc00700 1 -- 172.21.15.12:0/1728394467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800075d0 con 0x7f58901054f0 2024-05-27T07:32:01.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.138+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58901a98e0 con 0x7f58901054f0 2024-05-27T07:32:01.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.144+0000 7f588cc00700 1 -- 172.21.15.12:0/1728394467 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f588001c070 con 0x7f58901054f0 2024-05-27T07:32:01.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.144+0000 7f58a50a7b80 1 Processor -- start 2024-05-27T07:32:01.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.145+0000 7f58a50a7b80 1 -- start start 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.146+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d4d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.146+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d4d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.146+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24d4de70 0x557c24d5a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.146+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557c24c7be40 con 0x557c24cba9f0 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.146+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557c24c6b9e0 con 0x557c24d4de70 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.146+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557c24c6fa20 con 0x557c24bcee40 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.147+0000 7f588f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d4d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.147+0000 7f588f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d4d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:45184/0 (socket says 172.21.15.12:45184) 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.147+0000 7f588f400700 1 -- 172.21.15.12:0/637902099 learned_addr learned my addr 172.21.15.12:0/637902099 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:01.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.147+0000 7f5886a00700 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24d4de70 0x557c24d5a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.147+0000 7f588fe00700 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d4d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.148+0000 7f588f400700 1 -- 172.21.15.12:0/637902099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24d4de70 msgr2=0x557c24d5a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.148+0000 7f588f400700 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24d4de70 0x557c24d5a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.148+0000 7f588f400700 1 -- 172.21.15.12:0/637902099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 msgr2=0x557c24d4d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.148+0000 7f588f400700 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d4d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.148+0000 7f588f400700 1 -- 172.21.15.12:0/637902099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c24c34420 con 0x557c24bcee40 2024-05-27T07:32:01.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.149+0000 7f588cc00700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f58740b2af0 0x7f58740b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.149+0000 7f588cc00700 1 -- 172.21.15.12:0/1728394467 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f58800e1fd0 con 0x7f58901054f0 2024-05-27T07:32:01.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.149+0000 7f588f400700 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d4d870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f58800df300 tx=0x7f58800ea3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=666b349df6ebc0f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.149+0000 7f588fe00700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f58740b2af0 0x7f58740b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.150+0000 7f5884c00700 1 -- 172.21.15.12:0/637902099 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800a4e10 con 0x557c24bcee40 2024-05-27T07:32:01.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.150+0000 7f5884c00700 1 -- 172.21.15.12:0/637902099 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58800a8a60 con 0x557c24bcee40 2024-05-27T07:32:01.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.150+0000 7f5884c00700 1 -- 172.21.15.12:0/637902099 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800a86c0 con 0x557c24bcee40 2024-05-27T07:32:01.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.151+0000 7f58a50a7b80 1 -- 172.21.15.12:0/637902099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 msgr2=0x557c24d4d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.151+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d4d870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f58800df300 tx=0x7f58800ea3c0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.151+0000 7f588fe00700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f58740b2af0 0x7f58740b5000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5888005f50 tx=0x7f5888005e80 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.152+0000 7f58a50a7b80 1 -- 172.21.15.12:0/637902099 shutdown_connections 2024-05-27T07:32:01.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.152+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24d4de70 0x557c24d5a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.152+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d4d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.152+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/637902099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d4d870 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.152+0000 7f58a50a7b80 1 -- 172.21.15.12:0/637902099 >> 172.21.15.12:0/637902099 conn(0x557c24c68390 msgr2=0x557c24d494c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:01.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.152+0000 7f58a50a7b80 1 -- 172.21.15.12:0/637902099 shutdown_connections 2024-05-27T07:32:01.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.153+0000 7f58a50a7b80 1 -- 172.21.15.12:0/637902099 wait complete. 2024-05-27T07:32:01.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.153+0000 7f58a50a7b80 1 -- start start 2024-05-27T07:32:01.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.154+0000 7f58a50a7b80 1 Processor -- start 2024-05-27T07:32:01.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.154+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.154+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d57d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d58260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557c24b71fa0 con 0x557c24cba9f0 2024-05-27T07:32:01.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557c24c0c760 con 0x557c24bcee40 2024-05-27T07:32:01.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557c24d5ab80 con 0x557c24d4de70 2024-05-27T07:32:01.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f58a50a7b80 10 client.0 _ll_register_callbacks cb 0x557c24c15c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:01.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f588f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f588f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:47908/0 (socket says 172.21.15.12:47908) 2024-05-27T07:32:01.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.155+0000 7f588f400700 1 -- 172.21.15.12:0/1213140903 learned_addr learned my addr 172.21.15.12:0/1213140903 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:01.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.156+0000 7f5886a00700 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d57d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.156+0000 7f588fe00700 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d58260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.156+0000 7f588f400700 1 -- 172.21.15.12:0/1213140903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 msgr2=0x557c24d58260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.156+0000 7f588f400700 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d58260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.156+0000 7f588f400700 1 -- 172.21.15.12:0/1213140903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 msgr2=0x557c24d57d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.156+0000 7f588f400700 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d57d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.156+0000 7f588f400700 1 -- 172.21.15.12:0/1213140903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c24bbb8d0 con 0x557c24bcee40 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.157+0000 7f588f400700 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d577e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f588000bfd0 tx=0x7f58800a53f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ec1b9f9262be6d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.157+0000 7f587d600700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:32:01.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.157+0000 7f587d600700 1 -- 172.21.15.12:0/1213140903 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800a88a0 con 0x557c24bcee40 2024-05-27T07:32:01.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.157+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58800a8a60 con 0x557c24bcee40 2024-05-27T07:32:01.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.157+0000 7f587d600700 1 -- 172.21.15.12:0/1213140903 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f58800a91b0 con 0x557c24bcee40 2024-05-27T07:32:01.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.158+0000 7f587d600700 1 -- 172.21.15.12:0/1213140903 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f58800a8d20 con 0x557c24bcee40 2024-05-27T07:32:01.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.158+0000 7f58a50a7b80 10 client.36137 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:01.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.158+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x557c24c3c060 con 0x557c24bcee40 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.158+0000 7f587d600700 1 -- 172.21.15.12:0/1213140903 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58800a8a60 con 0x557c24bcee40 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.158+0000 7f587d600700 1 -- 172.21.15.12:0/1213140903 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f58800b1920 con 0x557c24bcee40 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.159+0000 7f58a50a7b80 10 client.36137 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.159+0000 7f58a50a7b80 10 client.36137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.159+0000 7f58a50a7b80 20 client.36137 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.159+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d69df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.159+0000 7f58a50a7b80 4 client.36137 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.159+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c24c7dd90 con 0x557c24d679d0 2024-05-27T07:32:01.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.159+0000 7f5886a00700 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d69df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.160+0000 7f5886a00700 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d69df0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed9f0ffd154017b server_cookie=e3ad99baa5dc9b16 in_seq=0 out_seq=0 2024-05-27T07:32:01.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.160+0000 7f587d600700 10 client.36137 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:01.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f587d600700 1 -- 172.21.15.12:0/1213140903 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557c24c7dd90 con 0x557c24d679d0 2024-05-27T07:32:01.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f587d600700 10 client.36137 handle_command_reply: tid=0 2024-05-27T07:32:01.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 client.36137 shutdown 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 msgr2=0x557c24d577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d577e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f588000bfd0 tx=0x7f58800a53f0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 shutdown_connections 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d69df0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d577e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f587d600700 0 client.36137 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d58260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/1213140903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d57d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.163+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 >> 172.21.15.12:0/1213140903 conn(0x557c24c68390 msgr2=0x557c24d4d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.164+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 shutdown_connections 2024-05-27T07:32:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.164+0000 7f58a50a7b80 1 -- 172.21.15.12:0/1213140903 wait complete. 2024-05-27T07:32:01.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.164+0000 7f58a50a7b80 20 client.36137 trim_cache size 0 max 16384 2024-05-27T07:32:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.185+0000 7f58a50a7b80 1 Processor -- start 2024-05-27T07:32:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.185+0000 7f58a50a7b80 1 -- start start 2024-05-27T07:32:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d49660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24cba9f0 0x557c24d52f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24d4de70 0x557c24d63c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f58901b1e40 con 0x557c24d4de70 2024-05-27T07:32:01.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f58901b10a0 con 0x557c24cba9f0 2024-05-27T07:32:01.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58901b1770 con 0x557c24bcee40 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f588fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24d4de70 0x557c24d63c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f588fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24d4de70 0x557c24d63c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49358/0 (socket says 172.21.15.12:49358) 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.186+0000 7f588fe00700 1 -- 172.21.15.12:0/2841626242 learned_addr learned my addr 172.21.15.12:0/2841626242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.187+0000 7f588f400700 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d49660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.187+0000 7f5886a00700 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24cba9f0 0x557c24d52f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.187+0000 7f588fe00700 1 -- 172.21.15.12:0/2841626242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 msgr2=0x557c24d49660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.187+0000 7f588fe00700 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d49660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.187+0000 7f588fe00700 1 -- 172.21.15.12:0/2841626242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24cba9f0 msgr2=0x557c24d52f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.187+0000 7f588fe00700 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24cba9f0 0x557c24d52f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.187+0000 7f588fe00700 1 -- 172.21.15.12:0/2841626242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c24d48b10 con 0x557c24d4de70 2024-05-27T07:32:01.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.188+0000 7f588fe00700 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24d4de70 0x557c24d63c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5888009990 tx=0x7f5888023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd3784455ac318c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.188+0000 7f587fe00700 1 -- 172.21.15.12:0/2841626242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588800b9d0 con 0x557c24d4de70 2024-05-27T07:32:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.188+0000 7f587fe00700 1 -- 172.21.15.12:0/2841626242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5888034070 con 0x557c24d4de70 2024-05-27T07:32:01.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.188+0000 7f587fe00700 1 -- 172.21.15.12:0/2841626242 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588800fd60 con 0x557c24d4de70 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 -- 172.21.15.12:0/2841626242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24d4de70 msgr2=0x557c24d63c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24d4de70 0x557c24d63c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5888009990 tx=0x7f5888023000 comp rx=0 tx=0).stop 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 -- 172.21.15.12:0/2841626242 shutdown_connections 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24d4de70 0x557c24d63c20 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24bcee40 0x557c24d49660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/2841626242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24cba9f0 0x557c24d52f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 -- 172.21.15.12:0/2841626242 >> 172.21.15.12:0/2841626242 conn(0x557c24c68390 msgr2=0x557c24d48f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 -- 172.21.15.12:0/2841626242 shutdown_connections 2024-05-27T07:32:01.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.189+0000 7f58a50a7b80 1 -- 172.21.15.12:0/2841626242 wait complete. 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.190+0000 7f58a50a7b80 1 -- start start 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.190+0000 7f58a50a7b80 1 Processor -- start 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d52f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f58a50a7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d518e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557c24d5aa60 con 0x557c24cba9f0 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557c24c7dd90 con 0x557c24bcee40 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f58a50a7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557c24d588f0 con 0x557c24d4de70 2024-05-27T07:32:01.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f58a50a7b80 10 client.0 _ll_register_callbacks cb 0x557c24c15c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f5886a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f5886a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49372/0 (socket says 172.21.15.12:49372) 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f5886a00700 1 -- 172.21.15.12:0/993021573 learned_addr learned my addr 172.21.15.12:0/993021573 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f588f400700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d52f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.191+0000 7f588fe00700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f5886a00700 1 -- 172.21.15.12:0/993021573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 msgr2=0x557c24d518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f5886a00700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f5886a00700 1 -- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 msgr2=0x557c24d52f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f5886a00700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d52f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f5886a00700 1 -- 172.21.15.12:0/993021573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c24d52080 con 0x557c24cba9f0 2024-05-27T07:32:01.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f588fe00700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f5886a00700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d512e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5878005950 tx=0x7f587800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d561c26d34c3a785 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f587d600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f587d600700 1 -- 172.21.15.12:0/993021573 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878023640 con 0x557c24cba9f0 2024-05-27T07:32:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5888034070 con 0x557c24cba9f0 2024-05-27T07:32:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.192+0000 7f587d600700 1 -- 172.21.15.12:0/993021573 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f5878011440 con 0x557c24cba9f0 2024-05-27T07:32:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f587d600700 1 -- 172.21.15.12:0/993021573 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f58780237e0 con 0x557c24cba9f0 2024-05-27T07:32:01.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 10 client.16635 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x557c24d5feb0 con 0x557c24cba9f0 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f587d600700 1 -- 172.21.15.12:0/993021573 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5888034070 con 0x557c24cba9f0 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f587d600700 1 -- 172.21.15.12:0/993021573 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5878021070 con 0x557c24cba9f0 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 10 client.16635 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 20 client.16635 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d60620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 4 client.16635 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.193+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c24c77570 con 0x557c24d679d0 2024-05-27T07:32:01.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.194+0000 7f588f400700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d60620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:01.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.195+0000 7f588f400700 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d60620 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5dec689920087fd server_cookie=75bd3cb2dec4e5c2 in_seq=0 out_seq=0 2024-05-27T07:32:01.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.195+0000 7f587d600700 10 client.16635 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f587d600700 1 -- 172.21.15.12:0/993021573 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c24c77570 con 0x557c24d679d0 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f587d600700 10 client.16635 handle_command_reply: tid=0 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 client.16635 shutdown 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 msgr2=0x557c24d512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d512e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5878005950 tx=0x7f587800bf90 comp rx=0 tx=0).stop 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 shutdown_connections 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x557c24d679d0 0x557c24d60620 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c24bcee40 0x557c24d52f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c24d4de70 0x557c24d518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 --2- 172.21.15.12:0/993021573 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557c24cba9f0 0x557c24d512e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 >> 172.21.15.12:0/993021573 conn(0x557c24c68390 msgr2=0x557c24d575c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f587d600700 0 client.16635 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 shutdown_connections 2024-05-27T07:32:01.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.196+0000 7f58a50a7b80 1 -- 172.21.15.12:0/993021573 wait complete. 2024-05-27T07:32:01.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.197+0000 7f58a50a7b80 20 client.16635 trim_cache size 0 max 16384 2024-05-27T07:32:01.192 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:32:01.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.197+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f58740b2af0 msgr2=0x7f58740b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.197+0000 7f5896600700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f58740b2af0 0x7f58740b5000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5888005f50 tx=0x7f5888005e80 comp rx=0 tx=0).stop 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.197+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58901054f0 msgr2=0x7f58901031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.197+0000 7f5896600700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58901054f0 0x7f58901031e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f588000ba50 tx=0x7f588000be10 comp rx=0 tx=0).stop 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 shutdown_connections 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f58740b2af0 0x7f58740b5000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f589010e8d0 0x7f5890103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58901054f0 0x7f58901031e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 --2- 172.21.15.12:0/1728394467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5890105e40 0x7f5890103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 >> 172.21.15.12:0/1728394467 conn(0x7f58900fedb0 msgr2=0x7f58900ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 shutdown_connections 2024-05-27T07:32:01.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:01.198+0000 7f5896600700 1 -- 172.21.15.12:0/1728394467 wait complete. 2024-05-27T07:32:02.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:02 smithi162 ceph-mon[35676]: pgmap v337: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-27T07:32:02.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:02 smithi062 ceph-mon[35304]: pgmap v337: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-27T07:32:02.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:02 smithi012 ceph-mon[31473]: pgmap v337: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-27T07:32:03.085 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:32:03.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:32:03.587 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:04.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:04 smithi162 ceph-mon[35676]: pgmap v338: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-27T07:32:04.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:04 smithi062 ceph-mon[35304]: pgmap v338: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-27T07:32:04.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:04 smithi012 ceph-mon[31473]: pgmap v338: 129 pgs: 129 active+clean; 150 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-27T07:32:05.668 INFO:tasks.workunit.client.0.smithi012.stdout:fallocating to largest ever: 0x9f2a70 2024-05-27T07:32:06.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 -- 172.21.15.12:0/1506048825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010af40 msgr2=0x7f77f010b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1506048825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010af40 0x7f77f010b350 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0099c0 tx=0x7f77dc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 -- 172.21.15.12:0/1506048825 shutdown_connections 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1506048825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f0106610 0x7f77f0106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1506048825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77f010b890 0x7f77f01131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1506048825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010af40 0x7f77f010b350 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 -- 172.21.15.12:0/1506048825 >> 172.21.15.12:0/1506048825 conn(0x7f77f0102020 msgr2=0x7f77f0104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 -- 172.21.15.12:0/1506048825 shutdown_connections 2024-05-27T07:32:06.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.252+0000 7f77f4c00700 1 -- 172.21.15.12:0/1506048825 wait complete. 2024-05-27T07:32:06.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 Processor -- start 2024-05-27T07:32:06.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 -- start start 2024-05-27T07:32:06.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f0106610 0x7f77f019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77f010af40 0x7f77f019d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010b890 0x7f77f01a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f77f0116370 con 0x7f77f010b890 2024-05-27T07:32:06.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f77f01161f0 con 0x7f77f0106610 2024-05-27T07:32:06.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.254+0000 7f77f4c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f77f01164f0 con 0x7f77f010af40 2024-05-27T07:32:06.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010b890 0x7f77f01a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010b890 0x7f77f01a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34408/0 (socket says 172.21.15.12:34408) 2024-05-27T07:32:06.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 -- 172.21.15.12:0/1886163990 learned_addr learned my addr 172.21.15.12:0/1886163990 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77ef400700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f0106610 0x7f77f019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77eea00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77f010af40 0x7f77f019d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 -- 172.21.15.12:0/1886163990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77f010af40 msgr2=0x7f77f019d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77f010af40 0x7f77f019d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 -- 172.21.15.12:0/1886163990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f0106610 msgr2=0x7f77f019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f0106610 0x7f77f019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77efe00700 1 -- 172.21.15.12:0/1886163990 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f01a1bc0 con 0x7f77f010b890 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.255+0000 7f77eea00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77f010af40 0x7f77f019d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.256+0000 7f77efe00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010b890 0x7f77f01a15d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f77e8007ee0 tx=0x7f77e800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.256+0000 7f77e7e00700 1 -- 172.21.15.12:0/1886163990 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e800dbd0 con 0x7f77f010b890 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.256+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f01a1eb0 con 0x7f77f010b890 2024-05-27T07:32:06.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.256+0000 7f77e7e00700 1 -- 172.21.15.12:0/1886163990 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77e800dd70 con 0x7f77f010b890 2024-05-27T07:32:06.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.256+0000 7f77e7e00700 1 -- 172.21.15.12:0/1886163990 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e80058d0 con 0x7f77f010b890 2024-05-27T07:32:06.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.256+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f01a9880 con 0x7f77f010b890 2024-05-27T07:32:06.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.261+0000 7f77e7e00700 1 -- 172.21.15.12:0/1886163990 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f77e801a020 con 0x7f77f010b890 2024-05-27T07:32:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.262+0000 7f7804d9bb80 1 Processor -- start 2024-05-27T07:32:06.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.262+0000 7f7804d9bb80 1 -- start start 2024-05-27T07:32:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.262+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 0x5617b0899f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.262+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a490 0x5617b08a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a8a0 0x5617b08a70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5617b07c8ec0 con 0x5617b089a8a0 2024-05-27T07:32:06.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617b07b8a60 con 0x5617b089a490 2024-05-27T07:32:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5617b07bcaa0 con 0x5617b071bec0 2024-05-27T07:32:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f77e7e00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77d40b29f0 0x7f77d40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f77eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a8a0 0x5617b08a70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f77eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a8a0 0x5617b08a70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34410/0 (socket says 172.21.15.12:34410) 2024-05-27T07:32:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f77eea00700 1 -- 172.21.15.12:0/1845174168 learned_addr learned my addr 172.21.15.12:0/1845174168 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:06.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f77ef400700 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a490 0x5617b08a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.263+0000 7f77e7e00700 1 -- 172.21.15.12:0/1886163990 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f77e80e2210 con 0x7f77f010b890 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.264+0000 7f77eea00700 1 -- 172.21.15.12:0/1845174168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 msgr2=0x5617b0899f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.264+0000 7f77eea00700 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 0x5617b0899f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.264+0000 7f77eea00700 1 -- 172.21.15.12:0/1845174168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a490 msgr2=0x5617b08a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.264+0000 7f77eea00700 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a490 0x5617b08a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.264+0000 7f77eea00700 1 -- 172.21.15.12:0/1845174168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b07814a0 con 0x5617b089a8a0 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.264+0000 7f77eea00700 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a8a0 0x5617b08a70a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009fd0 tx=0x7f77dc005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cc486f8020548b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.264+0000 7f77efe00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77d40b29f0 0x7f77d40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.265+0000 7f77e4c00700 1 -- 172.21.15.12:0/1845174168 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc005200 con 0x5617b089a8a0 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.265+0000 7f77e4c00700 1 -- 172.21.15.12:0/1845174168 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77dc00b790 con 0x5617b089a8a0 2024-05-27T07:32:06.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.265+0000 7f77e4c00700 1 -- 172.21.15.12:0/1845174168 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc003a00 con 0x5617b089a8a0 2024-05-27T07:32:06.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.266+0000 7f7804d9bb80 1 -- 172.21.15.12:0/1845174168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a8a0 msgr2=0x5617b08a70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.266+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a8a0 0x5617b08a70a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009fd0 tx=0x7f77dc005f00 comp rx=0 tx=0).stop 2024-05-27T07:32:06.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.266+0000 7f77efe00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77d40b29f0 0x7f77d40b4f00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f77e80a5160 tx=0x7f77e800df20 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 -- 172.21.15.12:0/1845174168 shutdown_connections 2024-05-27T07:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a8a0 0x5617b08a70a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 0x5617b0899f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/1845174168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a490 0x5617b08a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 -- 172.21.15.12:0/1845174168 >> 172.21.15.12:0/1845174168 conn(0x5617b077c040 msgr2=0x5617b0895db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 -- 172.21.15.12:0/1845174168 shutdown_connections 2024-05-27T07:32:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 -- 172.21.15.12:0/1845174168 wait complete. 2024-05-27T07:32:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.267+0000 7f7804d9bb80 1 -- start start 2024-05-27T07:32:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 1 Processor -- start 2024-05-27T07:32:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b071bec0 0x5617b08a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a490 0x5617b08a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08a5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5617b06bf020 con 0x5617b071bec0 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617b07597e0 con 0x5617b089a8a0 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5617b08a79b0 con 0x5617b089a490 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.268+0000 7f7804d9bb80 10 client.0 _ll_register_callbacks cb 0x5617b0762cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.269+0000 7f77eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b071bec0 0x5617b08a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.269+0000 7f77eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b071bec0 0x5617b08a45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34420/0 (socket says 172.21.15.12:34420) 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.269+0000 7f77eea00700 1 -- 172.21.15.12:0/871736252 learned_addr learned my addr 172.21.15.12:0/871736252 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.269+0000 7f77ef400700 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a490 0x5617b08a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.269+0000 7f77eea00700 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.269+0000 7f77eea00700 1 -- 172.21.15.12:0/871736252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a490 msgr2=0x5617b08a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.269+0000 7f77eea00700 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a490 0x5617b08a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.270+0000 7f77eea00700 1 -- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 msgr2=0x5617b08a5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.270+0000 7f77eea00700 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08a5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.270+0000 7f77eea00700 1 -- 172.21.15.12:0/871736252 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b0708950 con 0x5617b071bec0 2024-05-27T07:32:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.270+0000 7f77eea00700 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b071bec0 0x5617b08a45d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009990 tx=0x7f77dc0092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9806fa43844f30e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.270+0000 7f77d9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.271+0000 7f77d9600700 1 -- 172.21.15.12:0/871736252 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc003680 con 0x5617b071bec0 2024-05-27T07:32:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.271+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77dc00b790 con 0x5617b071bec0 2024-05-27T07:32:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.271+0000 7f77d9600700 1 -- 172.21.15.12:0/871736252 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f77dc003820 con 0x5617b071bec0 2024-05-27T07:32:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.271+0000 7f77d9600700 1 -- 172.21.15.12:0/871736252 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f77dc036050 con 0x5617b071bec0 2024-05-27T07:32:06.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.271+0000 7f7804d9bb80 10 client.16647 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.271+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5617b07890e0 con 0x5617b071bec0 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.271+0000 7f77d9600700 1 -- 172.21.15.12:0/871736252 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77dc00b790 con 0x5617b071bec0 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f77d9600700 1 -- 172.21.15.12:0/871736252 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f77dc02e020 con 0x5617b071bec0 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f7804d9bb80 10 client.16647 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f7804d9bb80 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f7804d9bb80 20 client.16647 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b08b48d0 0x5617b08b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f7804d9bb80 4 client.16647 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:06.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617b07cae10 con 0x5617b08b48d0 2024-05-27T07:32:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.272+0000 7f77ef400700 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b08b48d0 0x5617b08b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.273+0000 7f77ef400700 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b08b48d0 0x5617b08b6cf0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0b0fcd244a0e9 server_cookie=80e7c8e8ded15c79 in_seq=0 out_seq=0 2024-05-27T07:32:06.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.273+0000 7f77d9600700 10 client.16647 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.277+0000 7f77d9600700 1 -- 172.21.15.12:0/871736252 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5617b07cae10 con 0x5617b08b48d0 2024-05-27T07:32:06.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.277+0000 7f77d9600700 10 client.16647 handle_command_reply: tid=0 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.277+0000 7f7804d9bb80 1 client.16647 shutdown 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b071bec0 msgr2=0x5617b08a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b071bec0 0x5617b08a45d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009990 tx=0x7f77dc0092a0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 shutdown_connections 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b08b48d0 0x5617b08b6cf0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08a5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b071bec0 0x5617b08a45d0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/871736252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a490 0x5617b08a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 >> 172.21.15.12:0/871736252 conn(0x5617b077c040 msgr2=0x5617b0895070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f77d9600700 0 client.16647 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 shutdown_connections 2024-05-27T07:32:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 1 -- 172.21.15.12:0/871736252 wait complete. 2024-05-27T07:32:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.278+0000 7f7804d9bb80 20 client.16647 trim_cache size 0 max 16384 2024-05-27T07:32:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.303+0000 7f7804d9bb80 1 Processor -- start 2024-05-27T07:32:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.303+0000 7f7804d9bb80 1 -- start start 2024-05-27T07:32:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 0x5617b089fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a490 0x5617b08b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08bc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5617b08a49f0 con 0x5617b089a490 2024-05-27T07:32:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f77f01b1a60 con 0x5617b089a8a0 2024-05-27T07:32:06.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f77f01b0cc0 con 0x5617b071bec0 2024-05-27T07:32:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f77ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a490 0x5617b08b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.304+0000 7f77ef400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a490 0x5617b08b0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34430/0 (socket says 172.21.15.12:34430) 2024-05-27T07:32:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77ef400700 1 -- 172.21.15.12:0/108104402 learned_addr learned my addr 172.21.15.12:0/108104402 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77eea00700 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08bc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77eea00700 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 0x5617b089fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77ef400700 1 -- 172.21.15.12:0/108104402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 msgr2=0x5617b089fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77ef400700 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 0x5617b089fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77ef400700 1 -- 172.21.15.12:0/108104402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 msgr2=0x5617b08bc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77ef400700 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08bc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.305+0000 7f77ef400700 1 -- 172.21.15.12:0/108104402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b08973d0 con 0x5617b089a490 2024-05-27T07:32:06.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.306+0000 7f77ef400700 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a490 0x5617b08b0ab0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x5617b0896a30 tx=0x7f77e0010710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63c4fb4ca3c1590 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.306+0000 7f77dbe00700 1 -- 172.21.15.12:0/108104402 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0020030 con 0x5617b089a490 2024-05-27T07:32:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.307+0000 7f77dbe00700 1 -- 172.21.15.12:0/108104402 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77e0004cc0 con 0x5617b089a490 2024-05-27T07:32:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.307+0000 7f77dbe00700 1 -- 172.21.15.12:0/108104402 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0005620 con 0x5617b089a490 2024-05-27T07:32:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 -- 172.21.15.12:0/108104402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a490 msgr2=0x5617b08b0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a490 0x5617b08b0ab0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x5617b0896a30 tx=0x7f77e0010710 comp rx=0 tx=0).stop 2024-05-27T07:32:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 -- 172.21.15.12:0/108104402 shutdown_connections 2024-05-27T07:32:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b089a8a0 0x5617b08bc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b071bec0 0x5617b089fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/108104402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b089a490 0x5617b08b0ab0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 -- 172.21.15.12:0/108104402 >> 172.21.15.12:0/108104402 conn(0x5617b077c040 msgr2=0x5617b089d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 -- 172.21.15.12:0/108104402 shutdown_connections 2024-05-27T07:32:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 -- 172.21.15.12:0/108104402 wait complete. 2024-05-27T07:32:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.309+0000 7f7804d9bb80 1 -- start start 2024-05-27T07:32:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 1 Processor -- start 2024-05-27T07:32:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b071bec0 0x5617b08ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a8a0 0x5617b08a9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b08a9b80 0x5617b08a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f77f01b1390 con 0x5617b08a9b80 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617b091f3d0 con 0x5617b071bec0 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5617b0930560 con 0x5617b089a8a0 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.310+0000 7f7804d9bb80 10 client.0 _ll_register_callbacks cb 0x5617b0762cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.311+0000 7f77eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b08a9b80 0x5617b08a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.311+0000 7f77eea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b08a9b80 0x5617b08a9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34434/0 (socket says 172.21.15.12:34434) 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.311+0000 7f77eea00700 1 -- 172.21.15.12:0/2131808633 learned_addr learned my addr 172.21.15.12:0/2131808633 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.311+0000 7f77eea00700 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b071bec0 0x5617b08ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.311+0000 7f77ef400700 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a8a0 0x5617b08a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77eea00700 1 -- 172.21.15.12:0/2131808633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a8a0 msgr2=0x5617b08a9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77eea00700 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a8a0 0x5617b08a9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77eea00700 1 -- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b071bec0 msgr2=0x5617b08ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77eea00700 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b071bec0 0x5617b08ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77eea00700 1 -- 172.21.15.12:0/2131808633 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b09306e0 con 0x5617b08a9b80 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77eea00700 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b08a9b80 0x5617b08a9f90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0096c0 tx=0x7f77dc0092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1cb07e029c3105 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77d9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77d9600700 1 -- 172.21.15.12:0/2131808633 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00b790 con 0x5617b08a9b80 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f77d9600700 1 -- 172.21.15.12:0/2131808633 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f77dc00ee40 con 0x5617b08a9b80 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.312+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77e0004cc0 con 0x5617b08a9b80 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f77d9600700 1 -- 172.21.15.12:0/2131808633 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f77dc00f460 con 0x5617b08a9b80 2024-05-27T07:32:06.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 10 client.16653 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5617b0931960 con 0x5617b08a9b80 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f77d9600700 1 -- 172.21.15.12:0/2131808633 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77e0004cc0 con 0x5617b08a9b80 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f77d9600700 1 -- 172.21.15.12:0/2131808633 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f77dc02e040 con 0x5617b08a9b80 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 10 client.16653 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 20 client.16653 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b0931ea0 0x5617b09342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 4 client.16653 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.313+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5617b07c45f0 con 0x5617b0931ea0 2024-05-27T07:32:06.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.314+0000 7f77ef400700 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b0931ea0 0x5617b09342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:06.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.316+0000 7f77ef400700 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b0931ea0 0x5617b09342c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=845bb305af1fe366 server_cookie=8192b02c849002d1 in_seq=0 out_seq=0 2024-05-27T07:32:06.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.316+0000 7f77d9600700 10 client.16653 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f77d9600700 1 -- 172.21.15.12:0/2131808633 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5617b07c45f0 con 0x5617b0931ea0 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f77d9600700 10 client.16653 handle_command_reply: tid=0 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 client.16653 shutdown 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b08a9b80 msgr2=0x5617b08a9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b08a9b80 0x5617b08a9f90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0096c0 tx=0x7f77dc0092a0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 shutdown_connections 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5617b0931ea0 0x5617b09342c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617b071bec0 0x5617b08ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5617b08a9b80 0x5617b08a9f90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 --2- 172.21.15.12:0/2131808633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617b089a8a0 0x5617b08a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f77d9600700 0 client.16653 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 >> 172.21.15.12:0/2131808633 conn(0x5617b077c040 msgr2=0x5617b08a6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 shutdown_connections 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 1 -- 172.21.15.12:0/2131808633 wait complete. 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.320+0000 7f7804d9bb80 20 client.16653 trim_cache size 0 max 16384 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:32:06.316 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "7a2d36ba-9366-4b7d-8741-b5c7694ea71c", 2024-05-27T07:32:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:32:06.317 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:32:06.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77d40b29f0 msgr2=0x7f77d40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77d40b29f0 0x7f77d40b4f00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f77e80a5160 tx=0x7f77e800df20 comp rx=0 tx=0).stop 2024-05-27T07:32:06.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010b890 msgr2=0x7f77f01a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:06.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010b890 0x7f77f01a15d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f77e8007ee0 tx=0x7f77e800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 shutdown_connections 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77d40b29f0 0x7f77d40b4f00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f0106610 0x7f77f019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77f010b890 0x7f77f01a15d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 --2- 172.21.15.12:0/1886163990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77f010af40 0x7f77f019d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.321+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 >> 172.21.15.12:0/1886163990 conn(0x7f77f0102020 msgr2=0x7f77f010a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.322+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 shutdown_connections 2024-05-27T07:32:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:06.322+0000 7f77f4c00700 1 -- 172.21.15.12:0/1886163990 wait complete. 2024-05-27T07:32:06.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:06 smithi012 ceph-mon[31473]: pgmap v339: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 206 op/s 2024-05-27T07:32:06.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:06 smithi162 ceph-mon[35676]: pgmap v339: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 206 op/s 2024-05-27T07:32:06.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:06 smithi062 ceph-mon[35304]: pgmap v339: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 206 op/s 2024-05-27T07:32:07.227 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:32:07.505 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:32:07.505 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.505 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:32:07.505 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:32:07.699 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:07.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:32:07.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:32:07.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:07 smithi062 ceph-mon[35304]: pgmap v341: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 178 op/s 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:32:07.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:07 smithi012 ceph-mon[31473]: pgmap v341: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 178 op/s 2024-05-27T07:32:07.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:32:07.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:07 smithi162 ceph-mon[35676]: pgmap v341: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 178 op/s 2024-05-27T07:32:08.486 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:08 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:08.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:08 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:08.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:08 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:09.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:09 smithi012 ceph-mon[31473]: pgmap v342: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:32:09.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:09 smithi062 ceph-mon[35304]: pgmap v342: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:32:09.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:09 smithi162 ceph-mon[35676]: pgmap v342: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:32:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 -- 172.21.15.12:0/2608129203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8109670 msgr2=0x7fb1f8110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 --2- 172.21.15.12:0/2608129203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8109670 0x7fb1f8110fd0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400b0d0 tx=0x7fb1f400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 -- 172.21.15.12:0/2608129203 shutdown_connections 2024-05-27T07:32:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 --2- 172.21.15.12:0/2608129203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8109670 0x7fb1f8110fd0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 --2- 172.21.15.12:0/2608129203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1f81043f0 0x7fb1f8104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 --2- 172.21.15.12:0/2608129203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f8108d20 0x7fb1f8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 -- 172.21.15.12:0/2608129203 >> 172.21.15.12:0/2608129203 conn(0x7fb1f8079d00 msgr2=0x7fb1f807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 -- 172.21.15.12:0/2608129203 shutdown_connections 2024-05-27T07:32:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.308+0000 7fb1ff200700 1 -- 172.21.15.12:0/2608129203 wait complete. 2024-05-27T07:32:10.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.310+0000 7fb1ff200700 1 Processor -- start 2024-05-27T07:32:10.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.311+0000 7fb1ff200700 1 -- start start 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.311+0000 7fb1ff200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f81043f0 0x7fb1f8075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.311+0000 7fb1ff200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8108d20 0x7fb1f8075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.311+0000 7fb1ff200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1f8109670 0x7fb1f8076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.311+0000 7fb1ff200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81141a0 con 0x7fb1f8108d20 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.311+0000 7fb1ff200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f8114020 con 0x7fb1f81043f0 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.311+0000 7fb1ff200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f8114320 con 0x7fb1f8109670 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8108d20 0x7fb1f8075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8108d20 0x7fb1f8075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34458/0 (socket says 172.21.15.12:34458) 2024-05-27T07:32:10.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 -- 172.21.15.12:0/3204634567 learned_addr learned my addr 172.21.15.12:0/3204634567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fde00700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f81043f0 0x7fb1f8075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fe800700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1f8109670 0x7fb1f8076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 -- 172.21.15.12:0/3204634567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1f8109670 msgr2=0x7fb1f8076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1f8109670 0x7fb1f8076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 -- 172.21.15.12:0/3204634567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f81043f0 msgr2=0x7fb1f8075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f81043f0 0x7fb1f8075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.312+0000 7fb1fd400700 1 -- 172.21.15.12:0/3204634567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f81a8e40 con 0x7fb1f8108d20 2024-05-27T07:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.313+0000 7fb1fd400700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8108d20 0x7fb1f8075ff0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800c9e0 tx=0x7fb1e800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.313+0000 7fb1f2a00700 1 -- 172.21.15.12:0/3204634567 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8007aa0 con 0x7fb1f8108d20 2024-05-27T07:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.313+0000 7fb1f2a00700 1 -- 172.21.15.12:0/3204634567 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1e8007c40 con 0x7fb1f8108d20 2024-05-27T07:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.313+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f81a9130 con 0x7fb1f8108d20 2024-05-27T07:32:10.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.313+0000 7fb1f2a00700 1 -- 172.21.15.12:0/3204634567 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8018660 con 0x7fb1f8108d20 2024-05-27T07:32:10.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.314+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f81a9670 con 0x7fb1f8108d20 2024-05-27T07:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.319+0000 7fb1f2a00700 1 -- 172.21.15.12:0/3204634567 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb1e8018800 con 0x7fb1f8108d20 2024-05-27T07:32:10.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.319+0000 7fb20f2b4b80 1 Processor -- start 2024-05-27T07:32:10.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.320+0000 7fb20f2b4b80 1 -- start start 2024-05-27T07:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.320+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c291750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.320+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c291c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.320+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c29d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.320+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56525c1bee40 con 0x56525c1fd9f0 2024-05-27T07:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.320+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56525c1ae9e0 con 0x56525c111e40 2024-05-27T07:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.320+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56525c1b2a20 con 0x56525c29b2c0 2024-05-27T07:32:10.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c291750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c291750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34464/0 (socket says 172.21.15.12:34464) 2024-05-27T07:32:10.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1fe800700 1 -- 172.21.15.12:0/4161099158 learned_addr learned my addr 172.21.15.12:0/4161099158 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:10.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1f2a00700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb1e00b29f0 0x7fb1e00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1fde00700 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c291c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1fd400700 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c29d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1fe800700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb1e00b29f0 0x7fb1e00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1f2a00700 1 -- 172.21.15.12:0/3204634567 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9642+0+0 (secure 0 0 0) 0x7fb1e80e2a60 con 0x7fb1f8108d20 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.321+0000 7fb1fe800700 1 -- 172.21.15.12:0/4161099158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 msgr2=0x56525c29d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.322+0000 7fb1fe800700 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c29d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.322+0000 7fb1fe800700 1 -- 172.21.15.12:0/4161099158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 msgr2=0x56525c291c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.322+0000 7fb1fe800700 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c291c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.322+0000 7fb1fe800700 1 -- 172.21.15.12:0/4161099158 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56525c177420 con 0x56525c1fd9f0 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.322+0000 7fb1fd400700 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c29d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:10.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.322+0000 7fb1fde00700 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c291c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:10.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.322+0000 7fb1fe800700 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c291750 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009560 tx=0x7fb1f4009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb3beedbca9bb56 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:10.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.323+0000 7fb1fe800700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb1e00b29f0 0x7fb1e00b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4024000 tx=0x7fb1f4026000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:10.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.323+0000 7fb1e7400700 1 -- 172.21.15.12:0/4161099158 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4007dc0 con 0x56525c1fd9f0 2024-05-27T07:32:10.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.323+0000 7fb1e7400700 1 -- 172.21.15.12:0/4161099158 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1f4020910 con 0x56525c1fd9f0 2024-05-27T07:32:10.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.323+0000 7fb1e7400700 1 -- 172.21.15.12:0/4161099158 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4028760 con 0x56525c1fd9f0 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4161099158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 msgr2=0x56525c291750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c291750 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009560 tx=0x7fb1f4009590 comp rx=0 tx=0).stop 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4161099158 shutdown_connections 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c29d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c291750 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4161099158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c291c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4161099158 >> 172.21.15.12:0/4161099158 conn(0x56525c1ab390 msgr2=0x56525c28c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:10.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.324+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4161099158 shutdown_connections 2024-05-27T07:32:10.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.325+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4161099158 wait complete. 2024-05-27T07:32:10.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.325+0000 7fb20f2b4b80 1 -- start start 2024-05-27T07:32:10.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.325+0000 7fb20f2b4b80 1 Processor -- start 2024-05-27T07:32:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.326+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c298250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.326+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c298790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.326+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c298cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.326+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56525c0b4fa0 con 0x56525c1fd9f0 2024-05-27T07:32:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.326+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56525c14f760 con 0x56525c111e40 2024-05-27T07:32:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.326+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56525c29dd40 con 0x56525c29b2c0 2024-05-27T07:32:10.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.326+0000 7fb20f2b4b80 10 client.0 _ll_register_callbacks cb 0x56525c158c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:10.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c298790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c298790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34480/0 (socket says 172.21.15.12:34480) 2024-05-27T07:32:10.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 -- 172.21.15.12:0/1978324004 learned_addr learned my addr 172.21.15.12:0/1978324004 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:10.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fde00700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c298250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fe800700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c298cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 -- 172.21.15.12:0/1978324004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 msgr2=0x56525c298cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c298cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 -- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 msgr2=0x56525c298250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c298250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fd400700 1 -- 172.21.15.12:0/1978324004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56525c0fe8d0 con 0x56525c1fd9f0 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.327+0000 7fb1fde00700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c298250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb1fd400700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c298790 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800c9b0 tx=0x7fb1e80a9720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8409e9113d2364e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb1dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb1dbe00700 1 -- 172.21.15.12:0/1978324004 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e806f570 con 0x56525c1fd9f0 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb1dbe00700 1 -- 172.21.15.12:0/1978324004 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9642+0+0 (secure 0 0 0) 0x7fb1e80a9d50 con 0x56525c1fd9f0 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1f4020910 con 0x56525c1fd9f0 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb1dbe00700 1 -- 172.21.15.12:0/1978324004 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fb1e80a5070 con 0x56525c1fd9f0 2024-05-27T07:32:10.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb20f2b4b80 10 client.16665 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.328+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56525c17f060 con 0x56525c1fd9f0 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb1dbe00700 1 -- 172.21.15.12:0/1978324004 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1f4020910 con 0x56525c1fd9f0 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb1dbe00700 1 -- 172.21.15.12:0/1978324004 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fb1e80b22b0 con 0x56525c1fd9f0 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb20f2b4b80 10 client.16665 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb20f2b4b80 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb20f2b4b80 20 client.16665 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c2aaee0 0x56525c2ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb20f2b4b80 4 client.16665 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:10.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.329+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56525c1c0d90 con 0x56525c2aaee0 2024-05-27T07:32:10.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.330+0000 7fb1fde00700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c2aaee0 0x56525c2ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.331+0000 7fb1fde00700 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c2aaee0 0x56525c2ad390 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44ad1bf56f5a0282 server_cookie=77ef8b81bfa39cd9 in_seq=0 out_seq=0 2024-05-27T07:32:10.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.331+0000 7fb1dbe00700 10 client.16665 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:10.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb1dbe00700 1 -- 172.21.15.12:0/1978324004 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56525c1c0d90 con 0x56525c2aaee0 2024-05-27T07:32:10.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb1dbe00700 10 client.16665 handle_command_reply: tid=0 2024-05-27T07:32:10.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 client.16665 shutdown 2024-05-27T07:32:10.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 msgr2=0x56525c298790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c298790 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800c9b0 tx=0x7fb1e80a9720 comp rx=0 tx=0).stop 2024-05-27T07:32:10.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 shutdown_connections 2024-05-27T07:32:10.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c2aaee0 0x56525c2ad390 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c298250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb1dbe00700 0 client.16665 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c298cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/1978324004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c1fd9f0 0x56525c298790 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.333+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 >> 172.21.15.12:0/1978324004 conn(0x56525c1ab390 msgr2=0x56525c28c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.334+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 shutdown_connections 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.334+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/1978324004 wait complete. 2024-05-27T07:32:10.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.334+0000 7fb20f2b4b80 20 client.16665 trim_cache size 0 max 16384 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.365+0000 7fb20f2b4b80 1 Processor -- start 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.365+0000 7fb20f2b4b80 1 -- start start 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c111e40 0x56525c290b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c1fd9f0 0x56525c2a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c2b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b1a90 con 0x56525c111e40 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b0cf0 con 0x56525c1fd9f0 2024-05-27T07:32:10.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f81b13c0 con 0x56525c29b2c0 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb1fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c111e40 0x56525c290b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb1fde00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c111e40 0x56525c290b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34496/0 (socket says 172.21.15.12:34496) 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb1fde00700 1 -- 172.21.15.12:0/245904853 learned_addr learned my addr 172.21.15.12:0/245904853 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb1fd400700 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c1fd9f0 0x56525c2a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb1fe800700 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c2b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.366+0000 7fb1fde00700 1 -- 172.21.15.12:0/245904853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 msgr2=0x56525c2b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1fde00700 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c2b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1fde00700 1 -- 172.21.15.12:0/245904853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c1fd9f0 msgr2=0x56525c2a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1fde00700 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c1fd9f0 0x56525c2a6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1fde00700 1 -- 172.21.15.12:0/245904853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56525c292390 con 0x56525c111e40 2024-05-27T07:32:10.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1fe800700 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c2b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1fd400700 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c1fd9f0 0x56525c2a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1fde00700 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c111e40 0x56525c290b50 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x56525c2adb30 tx=0x7fb1ec008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e527f4bfb25539 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1e6a00700 1 -- 172.21.15.12:0/245904853 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec0120d0 con 0x56525c111e40 2024-05-27T07:32:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1e6a00700 1 -- 172.21.15.12:0/245904853 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1ec012270 con 0x56525c111e40 2024-05-27T07:32:10.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.367+0000 7fb1e6a00700 1 -- 172.21.15.12:0/245904853 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec005020 con 0x56525c111e40 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.368+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/245904853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c111e40 msgr2=0x56525c290b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.368+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c111e40 0x56525c290b50 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x56525c2adb30 tx=0x7fb1ec008f90 comp rx=0 tx=0).stop 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/245904853 shutdown_connections 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c29b2c0 0x56525c2b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c111e40 0x56525c290b50 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/245904853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c1fd9f0 0x56525c2a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/245904853 >> 172.21.15.12:0/245904853 conn(0x56525c1ab390 msgr2=0x56525c2937c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/245904853 shutdown_connections 2024-05-27T07:32:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/245904853 wait complete. 2024-05-27T07:32:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.369+0000 7fb20f2b4b80 1 -- start start 2024-05-27T07:32:10.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.370+0000 7fb20f2b4b80 1 Processor -- start 2024-05-27T07:32:10.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.370+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c2a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c1fd9f0 0x56525c2a3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb20f2b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c29b2c0 0x56525c29f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56525c28b3a0 con 0x56525c29b2c0 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56525c2913f0 con 0x56525c111e40 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb20f2b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56525c1c0d90 con 0x56525c1fd9f0 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb20f2b4b80 10 client.0 _ll_register_callbacks cb 0x56525c158c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb1fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c29b2c0 0x56525c29f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb1fe800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c29b2c0 0x56525c29f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34512/0 (socket says 172.21.15.12:34512) 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb1fe800700 1 -- 172.21.15.12:0/4133437789 learned_addr learned my addr 172.21.15.12:0/4133437789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.371+0000 7fb1fd400700 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c1fd9f0 0x56525c2a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1fde00700 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c2a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1fe800700 1 -- 172.21.15.12:0/4133437789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c1fd9f0 msgr2=0x56525c2a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1fe800700 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c1fd9f0 0x56525c2a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1fe800700 1 -- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 msgr2=0x56525c2a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1fe800700 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c2a30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1fe800700 1 -- 172.21.15.12:0/4133437789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56525c29fcb0 con 0x56525c29b2c0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1fe800700 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c29b2c0 0x56525c29f580 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x56525c28da90 tx=0x7fb1f4007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6070a0ed8653ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.372+0000 7fb1dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.373+0000 7fb1dbe00700 1 -- 172.21.15.12:0/4133437789 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4028400 con 0x56525c29b2c0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.373+0000 7fb1dbe00700 1 -- 172.21.15.12:0/4133437789 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9642+0+0 (secure 0 0 0) 0x7fb1f40285a0 con 0x56525c29b2c0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.373+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1ec012270 con 0x56525c29b2c0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.373+0000 7fb1dbe00700 1 -- 172.21.15.12:0/4133437789 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fb1f4045050 con 0x56525c29b2c0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.373+0000 7fb20f2b4b80 10 client.16671 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.373+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56525c316350 con 0x56525c29b2c0 2024-05-27T07:32:10.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.373+0000 7fb1dbe00700 1 -- 172.21.15.12:0/4133437789 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1ec012270 con 0x56525c29b2c0 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb1dbe00700 1 -- 172.21.15.12:0/4133437789 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fb1f4027040 con 0x56525c29b2c0 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb20f2b4b80 10 client.16671 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb20f2b4b80 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb20f2b4b80 20 client.16671 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c316570 0x56525c318990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb20f2b4b80 4 client.16671 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56525c1ba570 con 0x56525c316570 2024-05-27T07:32:10.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.374+0000 7fb1fde00700 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c316570 0x56525c318990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:10.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.375+0000 7fb1fde00700 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c316570 0x56525c318990 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61060b0ca36842e1 server_cookie=e2218c5c088960af in_seq=0 out_seq=0 2024-05-27T07:32:10.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.375+0000 7fb1dbe00700 10 client.16671 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.376+0000 7fb1dbe00700 1 -- 172.21.15.12:0/4133437789 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56525c1ba570 con 0x56525c316570 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.376+0000 7fb1dbe00700 10 client.16671 handle_command_reply: tid=0 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.376+0000 7fb20f2b4b80 1 client.16671 shutdown 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.376+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c29b2c0 msgr2=0x56525c29f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.376+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c29b2c0 0x56525c29f580 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x56525c28da90 tx=0x7fb1f4007e70 comp rx=0 tx=0).stop 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 shutdown_connections 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56525c316570 0x56525c318990 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56525c111e40 0x56525c2a30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56525c29b2c0 0x56525c29f580 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 --2- 172.21.15.12:0/4133437789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56525c1fd9f0 0x56525c2a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb1dbe00700 0 client.16671 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:10.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 >> 172.21.15.12:0/4133437789 conn(0x56525c1ab390 msgr2=0x56525c2937c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:10.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 shutdown_connections 2024-05-27T07:32:10.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 1 -- 172.21.15.12:0/4133437789 wait complete. 2024-05-27T07:32:10.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.377+0000 7fb20f2b4b80 20 client.16671 trim_cache size 0 max 16384 2024-05-27T07:32:10.371 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: "7a2d36ba-9366-4b7d-8741-b5c7694ea71c": { 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "7a2d36ba-9366-4b7d-8741-b5c7694ea71c", 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:32:10.372 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb1e00b29f0 msgr2=0x7fb1e00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb1e00b29f0 0x7fb1e00b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4024000 tx=0x7fb1f4026000 comp rx=0 tx=0).stop 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8108d20 msgr2=0x7fb1f8075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8108d20 0x7fb1f8075ff0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800c9e0 tx=0x7fb1e800cda0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 shutdown_connections 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb1e00b29f0 0x7fb1e00b4f00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f81043f0 0x7fb1f8075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1f8109670 0x7fb1f8076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 --2- 172.21.15.12:0/3204634567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1f8108d20 0x7fb1f8075ff0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:10.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 >> 172.21.15.12:0/3204634567 conn(0x7fb1f8079d00 msgr2=0x7fb1f8102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:10.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 shutdown_connections 2024-05-27T07:32:10.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:10.378+0000 7fb1ff200700 1 -- 172.21.15.12:0/3204634567 wait complete. 2024-05-27T07:32:11.223 INFO:tasks.cephfs.filesystem:scrub status for tag:7a2d36ba-9366-4b7d-8741-b5c7694ea71c - {'path': '/', 'tag': '7a2d36ba-9366-4b7d-8741-b5c7694ea71c', 'options': 'recursive,force'} 2024-05-27T07:32:11.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:11 smithi162 ceph-mon[35676]: pgmap v343: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:32:11.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:11 smithi062 ceph-mon[35304]: pgmap v343: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:32:12.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:11 smithi012 ceph-mon[31473]: pgmap v343: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:32:13.286 INFO:tasks.workunit.client.0.smithi012.stdout:copying to largest ever: 0x9fdd84 2024-05-27T07:32:13.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:13 smithi162 ceph-mon[35676]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:32:13.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:13 smithi062 ceph-mon[35304]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:32:13.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:13 smithi012 ceph-mon[31473]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:32:14.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:14 smithi162 ceph-mon[35676]: pgmap v345: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 917 KiB/s wr, 112 op/s 2024-05-27T07:32:14.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:14 smithi062 ceph-mon[35304]: pgmap v345: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 917 KiB/s wr, 112 op/s 2024-05-27T07:32:14.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:14 smithi012 ceph-mon[31473]: pgmap v345: 129 pgs: 129 active+clean; 160 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 917 KiB/s wr, 112 op/s 2024-05-27T07:32:16.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:16 smithi162 ceph-mon[35676]: pgmap v346: 129 pgs: 129 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-05-27T07:32:16.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:16 smithi062 ceph-mon[35304]: pgmap v346: 129 pgs: 129 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-05-27T07:32:16.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:16 smithi012 ceph-mon[31473]: pgmap v346: 129 pgs: 129 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 MiB/s wr, 204 op/s 2024-05-27T07:32:18.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:18 smithi162 ceph-mon[35676]: pgmap v347: 129 pgs: 129 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-05-27T07:32:18.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:18 smithi062 ceph-mon[35304]: pgmap v347: 129 pgs: 129 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-05-27T07:32:18.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:18 smithi012 ceph-mon[31473]: pgmap v347: 129 pgs: 129 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-05-27T07:32:20.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:20 smithi162 ceph-mon[35676]: pgmap v348: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-27T07:32:20.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:20 smithi062 ceph-mon[35304]: pgmap v348: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-27T07:32:20.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:20 smithi012 ceph-mon[31473]: pgmap v348: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-27T07:32:21.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:32:21.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:32:21.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:32:21.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:32:21.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:32:21.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:32:22.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:22 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:32:22.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:22 smithi162 ceph-mon[35676]: pgmap v349: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-27T07:32:22.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:22 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:32:22.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:22 smithi062 ceph-mon[35304]: pgmap v349: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-27T07:32:22.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:22 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:32:22.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:22 smithi012 ceph-mon[31473]: pgmap v349: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-27T07:32:24.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:24 smithi162 ceph-mon[35676]: pgmap v350: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 142 op/s 2024-05-27T07:32:24.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:24 smithi062 ceph-mon[35304]: pgmap v350: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 142 op/s 2024-05-27T07:32:24.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:24 smithi012 ceph-mon[31473]: pgmap v350: 129 pgs: 129 active+clean; 156 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 142 op/s 2024-05-27T07:32:26.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:26 smithi162 ceph-mon[35676]: pgmap v351: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:32:26.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:26 smithi062 ceph-mon[35304]: pgmap v351: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:32:26.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:26 smithi012 ceph-mon[31473]: pgmap v351: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:32:27.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:27 smithi162 ceph-mon[35676]: pgmap v352: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 945 KiB/s wr, 119 op/s 2024-05-27T07:32:27.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:27 smithi062 ceph-mon[35304]: pgmap v352: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 945 KiB/s wr, 119 op/s 2024-05-27T07:32:27.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:27 smithi012 ceph-mon[31473]: pgmap v352: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 945 KiB/s wr, 119 op/s 2024-05-27T07:32:29.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:29 smithi162 ceph-mon[35676]: pgmap v353: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:32:29.988 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:29 smithi062 ceph-mon[35304]: pgmap v353: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:32:30.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:29 smithi012 ceph-mon[31473]: pgmap v353: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:32:31.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:31 smithi162 ceph-mon[35676]: pgmap v354: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 945 KiB/s wr, 123 op/s 2024-05-27T07:32:31.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:31 smithi062 ceph-mon[35304]: pgmap v354: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 945 KiB/s wr, 123 op/s 2024-05-27T07:32:32.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:31 smithi012 ceph-mon[31473]: pgmap v354: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 945 KiB/s wr, 123 op/s 2024-05-27T07:32:34.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:34 smithi162 ceph-mon[35676]: pgmap v355: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 945 KiB/s wr, 123 op/s 2024-05-27T07:32:34.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:34 smithi062 ceph-mon[35304]: pgmap v355: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 945 KiB/s wr, 123 op/s 2024-05-27T07:32:34.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:34 smithi012 ceph-mon[31473]: pgmap v355: 129 pgs: 129 active+clean; 159 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 945 KiB/s wr, 123 op/s 2024-05-27T07:32:36.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:36 smithi162 ceph-mon[35676]: pgmap v356: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-05-27T07:32:36.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:36 smithi062 ceph-mon[35304]: pgmap v356: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-05-27T07:32:36.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:36 smithi012 ceph-mon[31473]: pgmap v356: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-05-27T07:32:38.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:38 smithi162 ceph-mon[35676]: pgmap v357: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:38.738 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:38 smithi062 ceph-mon[35304]: pgmap v357: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:38.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:38 smithi012 ceph-mon[31473]: pgmap v357: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:40.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:40 smithi162 ceph-mon[35676]: pgmap v358: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-05-27T07:32:40.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:40 smithi062 ceph-mon[35304]: pgmap v358: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-05-27T07:32:40.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:40 smithi012 ceph-mon[31473]: pgmap v358: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-05-27T07:32:41.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:32:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:42.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:42 smithi012 ceph-mon[31473]: pgmap v359: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:42.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:42 smithi162 ceph-mon[35676]: pgmap v359: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:42.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:42 smithi062 ceph-mon[35304]: pgmap v359: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.228+0000 7f32dca00700 1 -- 172.21.15.12:0/940509047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d81085b0 msgr2=0x7f32d810fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.228+0000 7f32dca00700 1 --2- 172.21.15.12:0/940509047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d81085b0 0x7f32d810fef0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f32d000b0a0 tx=0x7f32d000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.228+0000 7f32dca00700 1 -- 172.21.15.12:0/940509047 shutdown_connections 2024-05-27T07:32:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.228+0000 7f32dca00700 1 --2- 172.21.15.12:0/940509047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d81085b0 0x7f32d810fef0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.228+0000 7f32dca00700 1 --2- 172.21.15.12:0/940509047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32d81039b0 0x7f32d8103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.228+0000 7f32dca00700 1 --2- 172.21.15.12:0/940509047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d8107c60 0x7f32d8108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.228+0000 7f32dca00700 1 -- 172.21.15.12:0/940509047 >> 172.21.15.12:0/940509047 conn(0x7f32d80783a0 msgr2=0x7f32d80787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.229+0000 7f32dca00700 1 -- 172.21.15.12:0/940509047 shutdown_connections 2024-05-27T07:32:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.229+0000 7f32dca00700 1 -- 172.21.15.12:0/940509047 wait complete. 2024-05-27T07:32:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.230+0000 7f32dca00700 1 Processor -- start 2024-05-27T07:32:44.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.230+0000 7f32dca00700 1 -- start start 2024-05-27T07:32:44.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.231+0000 7f32dca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d81039b0 0x7f32d81a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.231+0000 7f32dca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d8107c60 0x7f32d81a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.231+0000 7f32dca00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32d81085b0 0x7f32d819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.231+0000 7f32dca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f32d81130b0 con 0x7f32d8107c60 2024-05-27T07:32:44.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.231+0000 7f32dca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f32d8112f30 con 0x7f32d81039b0 2024-05-27T07:32:44.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.231+0000 7f32dca00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32d8113230 con 0x7f32d81085b0 2024-05-27T07:32:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.232+0000 7f32d5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d8107c60 0x7f32d81a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.232+0000 7f32d5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d8107c60 0x7f32d81a2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35598/0 (socket says 172.21.15.12:35598) 2024-05-27T07:32:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.232+0000 7f32d5e00700 1 -- 172.21.15.12:0/866873973 learned_addr learned my addr 172.21.15.12:0/866873973 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:44.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.232+0000 7f32d6800700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d81039b0 0x7f32d81a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.233+0000 7f32d7200700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32d81085b0 0x7f32d819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.233+0000 7f32d5e00700 1 -- 172.21.15.12:0/866873973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32d81085b0 msgr2=0x7f32d819cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.233+0000 7f32d5e00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32d81085b0 0x7f32d819cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.233+0000 7f32d5e00700 1 -- 172.21.15.12:0/866873973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d81039b0 msgr2=0x7f32d81a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.233+0000 7f32d5e00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d81039b0 0x7f32d81a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.233+0000 7f32d5e00700 1 -- 172.21.15.12:0/866873973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d819d230 con 0x7f32d8107c60 2024-05-27T07:32:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.234+0000 7f32d5e00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d8107c60 0x7f32d81a2ef0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f32c400d950 tx=0x7f32c400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.234+0000 7f32cf400700 1 -- 172.21.15.12:0/866873973 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4017070 con 0x7f32d8107c60 2024-05-27T07:32:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.234+0000 7f32cf400700 1 -- 172.21.15.12:0/866873973 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32c4009dc0 con 0x7f32d8107c60 2024-05-27T07:32:44.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.235+0000 7f32cf400700 1 -- 172.21.15.12:0/866873973 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c400f3c0 con 0x7f32d8107c60 2024-05-27T07:32:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.235+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d819d520 con 0x7f32d8107c60 2024-05-27T07:32:44.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.235+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d81a9760 con 0x7f32d8107c60 2024-05-27T07:32:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.241+0000 7f32ebfcab80 1 Processor -- start 2024-05-27T07:32:44.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32ebfcab80 1 -- start start 2024-05-27T07:32:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32cf400700 1 -- 172.21.15.12:0/866873973 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f32c401a070 con 0x7f32d8107c60 2024-05-27T07:32:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131f2fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560131e5ee40 con 0x560131db1e40 2024-05-27T07:32:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560131e4e9e0 con 0x560131f30820 2024-05-27T07:32:44.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.242+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560131e52a20 con 0x560131f30410 2024-05-27T07:32:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131f2fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131f2fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35604/0 (socket says 172.21.15.12:35604) 2024-05-27T07:32:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d7200700 1 -- 172.21.15.12:0/1324928405 learned_addr learned my addr 172.21.15.12:0/1324928405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:44.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d6800700 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d7200700 1 -- 172.21.15.12:0/1324928405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 msgr2=0x560131f3aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d7200700 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d7200700 1 -- 172.21.15.12:0/1324928405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 msgr2=0x560131f3d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.243+0000 7f32d5e00700 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.244+0000 7f32d7200700 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.244+0000 7f32d7200700 1 -- 172.21.15.12:0/1324928405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560131e17420 con 0x560131db1e40 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.244+0000 7f32d7200700 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131f2fed0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009db0 tx=0x7f32d0009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a929f255608fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.245+0000 7f32c3e00700 1 -- 172.21.15.12:0/1324928405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0020070 con 0x560131db1e40 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.245+0000 7f32c3e00700 1 -- 172.21.15.12:0/1324928405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32d0009480 con 0x560131db1e40 2024-05-27T07:32:44.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.245+0000 7f32c3e00700 1 -- 172.21.15.12:0/1324928405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d001c440 con 0x560131db1e40 2024-05-27T07:32:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.245+0000 7f32cf400700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f32bc0b2a70 0x7f32bc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32cf400700 1 -- 172.21.15.12:0/866873973 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f32c40e2150 con 0x7f32d8107c60 2024-05-27T07:32:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32d6800700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f32bc0b2a70 0x7f32bc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32ebfcab80 1 -- 172.21.15.12:0/1324928405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 msgr2=0x560131f2fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131f2fed0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009db0 tx=0x7f32d0009de0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32ebfcab80 1 -- 172.21.15.12:0/1324928405 shutdown_connections 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131f2fed0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/1324928405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.246+0000 7f32ebfcab80 1 -- 172.21.15.12:0/1324928405 >> 172.21.15.12:0/1324928405 conn(0x560131e11fc0 msgr2=0x560131f2bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.247+0000 7f32d6800700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f32bc0b2a70 0x7f32bc0b4f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x560131f31430 tx=0x7f32c800a380 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.247+0000 7f32ebfcab80 1 -- 172.21.15.12:0/1324928405 shutdown_connections 2024-05-27T07:32:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.247+0000 7f32ebfcab80 1 -- 172.21.15.12:0/1324928405 wait complete. 2024-05-27T07:32:44.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.248+0000 7f32ebfcab80 1 -- start start 2024-05-27T07:32:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.248+0000 7f32ebfcab80 1 Processor -- start 2024-05-27T07:32:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131db1e40 0x560131f3a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560131d54fa0 con 0x560131f30820 2024-05-27T07:32:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560131def760 con 0x560131db1e40 2024-05-27T07:32:44.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560131f3d930 con 0x560131f30410 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32ebfcab80 10 client.0 _ll_register_callbacks cb 0x560131df8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32d6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131db1e40 0x560131f3a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.249+0000 7f32d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35610/0 (socket says 172.21.15.12:35610) 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131db1e40 0x560131f3a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:55932/0 (socket says 172.21.15.12:55932) 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d7200700 1 -- 172.21.15.12:0/24259362 learned_addr learned my addr 172.21.15.12:0/24259362 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d5e00700 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d7200700 1 -- 172.21.15.12:0/24259362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 msgr2=0x560131f3aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d7200700 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d7200700 1 -- 172.21.15.12:0/24259362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131db1e40 msgr2=0x560131f3a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d7200700 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131db1e40 0x560131f3a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.250+0000 7f32d7200700 1 -- 172.21.15.12:0/24259362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560131d9e8d0 con 0x560131f30820 2024-05-27T07:32:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.251+0000 7f32d7200700 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3b020 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f32d000b830 tx=0x7f32d0007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a51fa2a0eb93ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.251+0000 7f32c0c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.252+0000 7f32c0c00700 1 -- 172.21.15.12:0/24259362 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0029030 con 0x560131f30820 2024-05-27T07:32:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.252+0000 7f32c0c00700 1 -- 172.21.15.12:0/24259362 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f32d0004d20 con 0x560131f30820 2024-05-27T07:32:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.252+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560131e60d90 con 0x560131f30820 2024-05-27T07:32:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.252+0000 7f32c0c00700 1 -- 172.21.15.12:0/24259362 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f32d0020040 con 0x560131f30820 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.252+0000 7f32ebfcab80 10 client.16683 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x560131e1f060 con 0x560131f30820 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32c0c00700 1 -- 172.21.15.12:0/24259362 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x560131e60d90 con 0x560131f30820 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32c0c00700 1 -- 172.21.15.12:0/24259362 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f32d0011070 con 0x560131f30820 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32ebfcab80 10 client.16683 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32ebfcab80 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32ebfcab80 20 client.16683 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:44.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131f4a880 0x560131f4cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32ebfcab80 4 client.16683 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.253+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560131e5a570 con 0x560131f4a880 2024-05-27T07:32:44.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.254+0000 7f32d6800700 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131f4a880 0x560131f4cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.255+0000 7f32d6800700 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131f4a880 0x560131f4cd30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a318c34e511c1308 server_cookie=d51a742a231b009 in_seq=0 out_seq=0 2024-05-27T07:32:44.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.255+0000 7f32c0c00700 10 client.16683 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:44.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.257+0000 7f32c0c00700 1 -- 172.21.15.12:0/24259362 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560131e5a570 con 0x560131f4a880 2024-05-27T07:32:44.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.257+0000 7f32c0c00700 10 client.16683 handle_command_reply: tid=0 2024-05-27T07:32:44.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.257+0000 7f32ebfcab80 1 client.16683 shutdown 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 msgr2=0x560131f3b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3b020 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f32d000b830 tx=0x7f32d0007d90 comp rx=0 tx=0).stop 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 shutdown_connections 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131f4a880 0x560131f4cd30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131db1e40 0x560131f3a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32c0c00700 0 client.16683 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3b020 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/24259362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131f3aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 >> 172.21.15.12:0/24259362 conn(0x560131e11fc0 msgr2=0x560131f2b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:44.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 shutdown_connections 2024-05-27T07:32:44.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 1 -- 172.21.15.12:0/24259362 wait complete. 2024-05-27T07:32:44.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.258+0000 7f32ebfcab80 20 client.16683 trim_cache size 0 max 16384 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 Processor -- start 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 -- start start 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131db1e40 0x560131f35d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30410 0x560131f3af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f32b800e540 con 0x560131f30820 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f32b800d7a0 con 0x560131f30410 2024-05-27T07:32:44.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.284+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32b800de70 con 0x560131db1e40 2024-05-27T07:32:44.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35620/0 (socket says 172.21.15.12:35620) 2024-05-27T07:32:44.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 -- 172.21.15.12:0/466834515 learned_addr learned my addr 172.21.15.12:0/466834515 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:44.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d5e00700 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30410 0x560131f3af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 -- 172.21.15.12:0/466834515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131db1e40 msgr2=0x560131f35d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131db1e40 0x560131f35d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 -- 172.21.15.12:0/466834515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30410 msgr2=0x560131f3af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30410 0x560131f3af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.285+0000 7f32d7200700 1 -- 172.21.15.12:0/466834515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560131f3b5b0 con 0x560131f30820 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.286+0000 7f32d7200700 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3ca10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f32d001c630 tx=0x7f32d0008fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fff1a632a468ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.286+0000 7f32c3400700 1 -- 172.21.15.12:0/466834515 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0029070 con 0x560131f30820 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.286+0000 7f32c3400700 1 -- 172.21.15.12:0/466834515 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32d0011560 con 0x560131f30820 2024-05-27T07:32:44.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.286+0000 7f32c3400700 1 -- 172.21.15.12:0/466834515 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d00117a0 con 0x560131f30820 2024-05-27T07:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 -- 172.21.15.12:0/466834515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 msgr2=0x560131f3ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3ca10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f32d001c630 tx=0x7f32d0008fb0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 -- 172.21.15.12:0/466834515 shutdown_connections 2024-05-27T07:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131f30820 0x560131f3ca10 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131db1e40 0x560131f35d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/466834515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30410 0x560131f3af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 -- 172.21.15.12:0/466834515 >> 172.21.15.12:0/466834515 conn(0x560131e11fc0 msgr2=0x560131f45a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:44.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 -- 172.21.15.12:0/466834515 shutdown_connections 2024-05-27T07:32:44.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.287+0000 7f32ebfcab80 1 -- 172.21.15.12:0/466834515 wait complete. 2024-05-27T07:32:44.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.288+0000 7f32ebfcab80 1 -- start start 2024-05-27T07:32:44.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.288+0000 7f32ebfcab80 1 Processor -- start 2024-05-27T07:32:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131fad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131fadc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32ebfcab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560131e60d90 con 0x560131db1e40 2024-05-27T07:32:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560131e5a570 con 0x560131f30820 2024-05-27T07:32:44.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32ebfcab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560131f3a380 con 0x560131f30410 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32ebfcab80 10 client.0 _ll_register_callbacks cb 0x560131f47230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32d6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131fad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32d6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131fad6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35636/0 (socket says 172.21.15.12:35636) 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32d6800700 1 -- 172.21.15.12:0/682661801 learned_addr learned my addr 172.21.15.12:0/682661801 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32d5e00700 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131fadc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.289+0000 7f32d7200700 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32d6800700 1 -- 172.21.15.12:0/682661801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 msgr2=0x560131fadc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32d6800700 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131fadc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32d6800700 1 -- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 msgr2=0x560131f3e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32d6800700 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32d6800700 1 -- 172.21.15.12:0/682661801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560131f3e860 con 0x560131db1e40 2024-05-27T07:32:44.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32d6800700 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131fad6b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x560131f39e20 tx=0x7f32c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314e92b6f5cac3c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:44.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32c0c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:44.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32c0c00700 1 -- 172.21.15.12:0/682661801 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c801be30 con 0x560131db1e40 2024-05-27T07:32:44.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32c0c00700 1 -- 172.21.15.12:0/682661801 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f32c8027430 con 0x560131db1e40 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32d0011560 con 0x560131db1e40 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.290+0000 7f32c0c00700 1 -- 172.21.15.12:0/682661801 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f32c8038050 con 0x560131db1e40 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 10 client.16689 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x560131fb4e50 con 0x560131db1e40 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32c0c00700 1 -- 172.21.15.12:0/682661801 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32d0011560 con 0x560131db1e40 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32c0c00700 1 -- 172.21.15.12:0/682661801 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f32c8033820 con 0x560131db1e40 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 10 client.16689 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 20 client.16689 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131fb55e0 0x560131fb7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 4 client.16689 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:32:44.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.291+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560131cd1250 con 0x560131fb55e0 2024-05-27T07:32:44.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.292+0000 7f32d5e00700 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131fb55e0 0x560131fb7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:44.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.293+0000 7f32d5e00700 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131fb55e0 0x560131fb7a00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c455cf4ca9fb37a server_cookie=618d25f3ed3a3f31 in_seq=0 out_seq=0 2024-05-27T07:32:44.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.293+0000 7f32c0c00700 10 client.16689 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:44.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32c0c00700 1 -- 172.21.15.12:0/682661801 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560131cd1250 con 0x560131fb55e0 2024-05-27T07:32:44.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32c0c00700 10 client.16689 handle_command_reply: tid=0 2024-05-27T07:32:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 client.16689 shutdown 2024-05-27T07:32:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 msgr2=0x560131fad6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131fad6b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x560131f39e20 tx=0x7f32c80047c0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 shutdown_connections 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560131fb55e0 0x560131fb7a00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560131f30820 0x560131f3e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560131db1e40 0x560131fad6b0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 --2- 172.21.15.12:0/682661801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560131f30410 0x560131fadc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 >> 172.21.15.12:0/682661801 conn(0x560131e11fc0 msgr2=0x560131f3acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32c0c00700 0 client.16689 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 shutdown_connections 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.296+0000 7f32ebfcab80 1 -- 172.21.15.12:0/682661801 wait complete. 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32ebfcab80 20 client.16689 trim_cache size 0 max 16384 2024-05-27T07:32:44.291 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:32:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:32:44.292 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:32:44.292 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:32:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f32bc0b2a70 msgr2=0x7f32bc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f32bc0b2a70 0x7f32bc0b4f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x560131f31430 tx=0x7f32c800a380 comp rx=0 tx=0).stop 2024-05-27T07:32:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d8107c60 msgr2=0x7f32d81a2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d8107c60 0x7f32d81a2ef0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f32c400d950 tx=0x7f32c400dc60 comp rx=0 tx=0).stop 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 shutdown_connections 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f32bc0b2a70 0x7f32bc0b4f80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d81039b0 0x7f32d81a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32d81085b0 0x7f32d819cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 --2- 172.21.15.12:0/866873973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32d8107c60 0x7f32d81a2ef0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.297+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 >> 172.21.15.12:0/866873973 conn(0x7f32d80783a0 msgr2=0x7f32d8101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.298+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 shutdown_connections 2024-05-27T07:32:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:44.298+0000 7f32dca00700 1 -- 172.21.15.12:0/866873973 wait complete. 2024-05-27T07:32:44.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:44 smithi012 ceph-mon[31473]: pgmap v360: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:44.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:44 smithi162 ceph-mon[35676]: pgmap v360: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:44.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:44 smithi062 ceph-mon[35304]: pgmap v360: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-27T07:32:45.133 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:32:45.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:32:45.597 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:45.620 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0x9ff7e4 2024-05-27T07:32:46.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:46 smithi012 ceph-mon[31473]: pgmap v361: 129 pgs: 129 active+clean; 161 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-05-27T07:32:46.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:46 smithi162 ceph-mon[35676]: pgmap v361: 129 pgs: 129 active+clean; 161 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-05-27T07:32:46.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:46 smithi062 ceph-mon[35304]: pgmap v361: 129 pgs: 129 active+clean; 161 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-05-27T07:32:47.826 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:47 smithi012 ceph-mon[31473]: pgmap v362: 129 pgs: 129 active+clean; 161 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 975 KiB/s wr, 122 op/s 2024-05-27T07:32:47.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:47 smithi162 ceph-mon[35676]: pgmap v362: 129 pgs: 129 active+clean; 161 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 975 KiB/s wr, 122 op/s 2024-05-27T07:32:47.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:47 smithi062 ceph-mon[35304]: pgmap v362: 129 pgs: 129 active+clean; 161 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 975 KiB/s wr, 122 op/s 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 -- 172.21.15.12:0/3354508471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7760106560 msgr2=0x7f7760106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 --2- 172.21.15.12:0/3354508471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7760106560 0x7f7760106970 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f774c009990 tx=0x7f774c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 -- 172.21.15.12:0/3354508471 shutdown_connections 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 --2- 172.21.15.12:0/3354508471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7760106eb0 0x7f7760113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 --2- 172.21.15.12:0/3354508471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f776010c5a0 0x7f776010c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 --2- 172.21.15.12:0/3354508471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7760106560 0x7f7760106970 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 -- 172.21.15.12:0/3354508471 >> 172.21.15.12:0/3354508471 conn(0x7f7760102040 msgr2=0x7f7760104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:48.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 -- 172.21.15.12:0/3354508471 shutdown_connections 2024-05-27T07:32:48.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.118+0000 7f7765400700 1 -- 172.21.15.12:0/3354508471 wait complete. 2024-05-27T07:32:48.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.120+0000 7f7765400700 1 Processor -- start 2024-05-27T07:32:48.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.120+0000 7f7765400700 1 -- start start 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f7765400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7760106560 0x7f77600753c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f7765400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7760106eb0 0x7f776006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f7765400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f776010c5a0 0x7f776006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f7765400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7760116580 con 0x7f776010c5a0 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f7765400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7760116400 con 0x7f7760106eb0 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f7765400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7760116700 con 0x7f7760106560 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f775f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f776010c5a0 0x7f776006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f775f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f776010c5a0 0x7f776006eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35652/0 (socket says 172.21.15.12:35652) 2024-05-27T07:32:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f775f200700 1 -- 172.21.15.12:0/2102131392 learned_addr learned my addr 172.21.15.12:0/2102131392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f775de00700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7760106eb0 0x7f776006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.121+0000 7f775e800700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7760106560 0x7f77600753c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f775f200700 1 -- 172.21.15.12:0/2102131392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7760106560 msgr2=0x7f77600753c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f775f200700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7760106560 0x7f77600753c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f775f200700 1 -- 172.21.15.12:0/2102131392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7760106eb0 msgr2=0x7f776006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f775f200700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7760106eb0 0x7f776006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f775f200700 1 -- 172.21.15.12:0/2102131392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776006f310 con 0x7f776010c5a0 2024-05-27T07:32:48.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f775de00700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7760106eb0 0x7f776006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:48.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f775f200700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f776010c5a0 0x7f776006eb40 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f775800ada0 tx=0x7f775800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:48.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.122+0000 7f7757400700 1 -- 172.21.15.12:0/2102131392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758017020 con 0x7f776010c5a0 2024-05-27T07:32:48.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.123+0000 7f7757400700 1 -- 172.21.15.12:0/2102131392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f775800ce10 con 0x7f776010c5a0 2024-05-27T07:32:48.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.123+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7760073010 con 0x7f776010c5a0 2024-05-27T07:32:48.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.123+0000 7f7757400700 1 -- 172.21.15.12:0/2102131392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758005440 con 0x7f776010c5a0 2024-05-27T07:32:48.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.123+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7760073500 con 0x7f776010c5a0 2024-05-27T07:32:48.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.124+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f776010e960 con 0x7f776010c5a0 2024-05-27T07:32:48.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.129+0000 7f7757400700 1 -- 172.21.15.12:0/2102131392 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7758004ad0 con 0x7f776010c5a0 2024-05-27T07:32:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.131+0000 7f7757400700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77440b29f0 0x7f77440b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.131+0000 7f7757400700 1 -- 172.21.15.12:0/2102131392 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f77580e2800 con 0x7f776010c5a0 2024-05-27T07:32:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.132+0000 7f775e800700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77440b29f0 0x7f77440b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:48.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.133+0000 7f775e800700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77440b29f0 0x7f77440b4f00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7750005fd0 tx=0x7f775000af00 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:48.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.135+0000 7f7757400700 1 -- 172.21.15.12:0/2102131392 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7758021020 con 0x7f776010c5a0 2024-05-27T07:32:48.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.484+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7760107c80 con 0x7f776010c5a0 2024-05-27T07:32:48.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.485+0000 7f7757400700 1 -- 172.21.15.12:0/2102131392 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f77580a9e60 con 0x7f776010c5a0 2024-05-27T07:32:48.479 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:32:48.479 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:32:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.488+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77440b29f0 msgr2=0x7f77440b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:48.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.488+0000 7f7765400700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77440b29f0 0x7f77440b4f00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7750005fd0 tx=0x7f775000af00 comp rx=0 tx=0).stop 2024-05-27T07:32:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.488+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f776010c5a0 msgr2=0x7f776006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.488+0000 7f7765400700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f776010c5a0 0x7f776006eb40 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f775800ada0 tx=0x7f775800c330 comp rx=0 tx=0).stop 2024-05-27T07:32:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.489+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 shutdown_connections 2024-05-27T07:32:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.489+0000 7f7765400700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f77440b29f0 0x7f77440b4f00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.489+0000 7f7765400700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f776010c5a0 0x7f776006eb40 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.489+0000 7f7765400700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7760106560 0x7f77600753c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.489+0000 7f7765400700 1 --2- 172.21.15.12:0/2102131392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7760106eb0 0x7f776006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.489+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 >> 172.21.15.12:0/2102131392 conn(0x7f7760102040 msgr2=0x7f7760103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.490+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 shutdown_connections 2024-05-27T07:32:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:48.490+0000 7f7765400700 1 -- 172.21.15.12:0/2102131392 wait complete. 2024-05-27T07:32:48.485 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:32:48.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:48 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2102131392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:32:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:48 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2102131392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:32:48.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:48 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2102131392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:32:49.485 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:32:49.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:49 smithi012 ceph-mon[31473]: pgmap v363: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:32:49.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:49 smithi162 ceph-mon[35676]: pgmap v363: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:32:49.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:49 smithi062 ceph-mon[35304]: pgmap v363: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-27T07:32:49.992 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:51.402 INFO:tasks.workunit.client.0.smithi012.stdout:10000 write 0x723fbd thru 0x733aaa (0xfaee bytes) 2024-05-27T07:32:51.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:51 smithi012 ceph-mon[31473]: pgmap v364: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:32:51.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:51 smithi162 ceph-mon[35676]: pgmap v364: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:32:51.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:51 smithi062 ceph-mon[35304]: pgmap v364: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:32:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 -- 172.21.15.12:0/3111707512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106eb0 msgr2=0x7fd830113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 --2- 172.21.15.12:0/3111707512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106eb0 0x7fd830113340 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd82800b0a0 tx=0x7fd82800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 -- 172.21.15.12:0/3111707512 shutdown_connections 2024-05-27T07:32:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 --2- 172.21.15.12:0/3111707512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106eb0 0x7fd830113340 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 --2- 172.21.15.12:0/3111707512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd83010c5a0 0x7fd83010c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 --2- 172.21.15.12:0/3111707512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd830106560 0x7fd830106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 -- 172.21.15.12:0/3111707512 >> 172.21.15.12:0/3111707512 conn(0x7fd830102040 msgr2=0x7fd830104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 -- 172.21.15.12:0/3111707512 shutdown_connections 2024-05-27T07:32:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.521+0000 7fd834c00700 1 -- 172.21.15.12:0/3111707512 wait complete. 2024-05-27T07:32:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 Processor -- start 2024-05-27T07:32:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 -- start start 2024-05-27T07:32:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106560 0x7fd83019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd830106eb0 0x7fd83019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83010c5a0 0x7fd8301a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd830116620 con 0x7fd830106560 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301164a0 con 0x7fd83010c5a0 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.523+0000 7fd834c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301167a0 con 0x7fd830106eb0 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106560 0x7fd83019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106560 0x7fd83019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47432/0 (socket says 172.21.15.12:47432) 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 -- 172.21.15.12:0/1964194334 learned_addr learned my addr 172.21.15.12:0/1964194334 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82e800700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83010c5a0 0x7fd8301a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82d400700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd830106eb0 0x7fd83019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 -- 172.21.15.12:0/1964194334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd830106eb0 msgr2=0x7fd83019d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd830106eb0 0x7fd83019d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 -- 172.21.15.12:0/1964194334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83010c5a0 msgr2=0x7fd8301a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83010c5a0 0x7fd8301a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82de00700 1 -- 172.21.15.12:0/1964194334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8301a1e20 con 0x7fd830106560 2024-05-27T07:32:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.524+0000 7fd82d400700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd830106eb0 0x7fd83019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.525+0000 7fd82de00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106560 0x7fd83019cf50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd82000d980 tx=0x7fd82000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.525+0000 7fd826a00700 1 -- 172.21.15.12:0/1964194334 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820017070 con 0x7fd830106560 2024-05-27T07:32:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.525+0000 7fd826a00700 1 -- 172.21.15.12:0/1964194334 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd820010460 con 0x7fd830106560 2024-05-27T07:32:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.525+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8301a2110 con 0x7fd830106560 2024-05-27T07:32:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.525+0000 7fd826a00700 1 -- 172.21.15.12:0/1964194334 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82000de60 con 0x7fd830106560 2024-05-27T07:32:52.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.525+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8301a9a30 con 0x7fd830106560 2024-05-27T07:32:52.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.531+0000 7fd826a00700 1 -- 172.21.15.12:0/1964194334 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd820010610 con 0x7fd830106560 2024-05-27T07:32:52.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.532+0000 7fd843723b80 1 Processor -- start 2024-05-27T07:32:52.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.532+0000 7fd843723b80 1 -- start start 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd843723b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65b869f0 0x563b65c19330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd843723b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65a9ae40 0x563b65c19870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd843723b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c26110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd843723b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563b65b47e40 con 0x563b65a9ae40 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd843723b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563b65b379e0 con 0x563b65b869f0 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd843723b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563b65b3ba20 con 0x563b65c19e70 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd826a00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8140b2a70 0x7fd8140b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd82de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65a9ae40 0x563b65c19870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd82de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65a9ae40 0x563b65c19870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47446/0 (socket says 172.21.15.12:47446) 2024-05-27T07:32:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd82de00700 1 -- 172.21.15.12:0/1119526200 learned_addr learned my addr 172.21.15.12:0/1119526200 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd82e800700 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65b869f0 0x563b65c19330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.533+0000 7fd826a00700 1 -- 172.21.15.12:0/1964194334 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fd8200e25a0 con 0x7fd830106560 2024-05-27T07:32:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.534+0000 7fd82d400700 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c26110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.534+0000 7fd82e800700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8140b2a70 0x7fd8140b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.534+0000 7fd82de00700 1 -- 172.21.15.12:0/1119526200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 msgr2=0x563b65c26110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.534+0000 7fd82de00700 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c26110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.534+0000 7fd82de00700 1 -- 172.21.15.12:0/1119526200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65b869f0 msgr2=0x563b65c19330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.534+0000 7fd82de00700 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65b869f0 0x563b65c19330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.534+0000 7fd82de00700 1 -- 172.21.15.12:0/1119526200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b65b00420 con 0x563b65a9ae40 2024-05-27T07:32:52.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.535+0000 7fd82de00700 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65a9ae40 0x563b65c19870 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd8200a93a0 tx=0x7fd8200a8b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d781ec6bca5da1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:52.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.535+0000 7fd82e800700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8140b2a70 0x7fd8140b4f80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd828009610 tx=0x7fd8280099e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:52.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.535+0000 7fd81b400700 1 -- 172.21.15.12:0/1119526200 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820020070 con 0x563b65a9ae40 2024-05-27T07:32:52.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.535+0000 7fd81b400700 1 -- 172.21.15.12:0/1119526200 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8200dfd00 con 0x563b65a9ae40 2024-05-27T07:32:52.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.535+0000 7fd81b400700 1 -- 172.21.15.12:0/1119526200 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8200b2040 con 0x563b65a9ae40 2024-05-27T07:32:52.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.536+0000 7fd843723b80 1 -- 172.21.15.12:0/1119526200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65a9ae40 msgr2=0x563b65c19870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.536+0000 7fd843723b80 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65a9ae40 0x563b65c19870 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd8200a93a0 tx=0x7fd8200a8b80 comp rx=0 tx=0).stop 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.537+0000 7fd843723b80 1 -- 172.21.15.12:0/1119526200 shutdown_connections 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.537+0000 7fd843723b80 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65b869f0 0x563b65c19330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.537+0000 7fd843723b80 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c26110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.537+0000 7fd843723b80 1 --2- 172.21.15.12:0/1119526200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65a9ae40 0x563b65c19870 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.537+0000 7fd843723b80 1 -- 172.21.15.12:0/1119526200 >> 172.21.15.12:0/1119526200 conn(0x563b65b34390 msgr2=0x563b65c154c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.538+0000 7fd843723b80 1 -- 172.21.15.12:0/1119526200 shutdown_connections 2024-05-27T07:32:52.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.538+0000 7fd843723b80 1 -- 172.21.15.12:0/1119526200 wait complete. 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.538+0000 7fd843723b80 1 -- start start 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.539+0000 7fd843723b80 1 Processor -- start 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.540+0000 7fd843723b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65a9ae40 0x563b65c236f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.540+0000 7fd843723b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c23c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.540+0000 7fd843723b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65c19e70 0x563b65c24170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.540+0000 7fd843723b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563b65a3dfa0 con 0x563b65b869f0 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.540+0000 7fd843723b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563b65ad8760 con 0x563b65c19e70 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.540+0000 7fd843723b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563b65c26ac0 con 0x563b65a9ae40 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.540+0000 7fd843723b80 10 client.0 _ll_register_callbacks cb 0x563b65ae1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.541+0000 7fd82e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65c19e70 0x563b65c24170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.541+0000 7fd82e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65c19e70 0x563b65c24170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:42392/0 (socket says 172.21.15.12:42392) 2024-05-27T07:32:52.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.541+0000 7fd82e800700 1 -- 172.21.15.12:0/118483080 learned_addr learned my addr 172.21.15.12:0/118483080 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:52.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.541+0000 7fd82d400700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c23c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82de00700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65a9ae40 0x563b65c236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82d400700 1 -- 172.21.15.12:0/118483080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65a9ae40 msgr2=0x563b65c236f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82d400700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65a9ae40 0x563b65c236f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82d400700 1 -- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65c19e70 msgr2=0x563b65c24170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82d400700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65c19e70 0x563b65c24170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82d400700 1 -- 172.21.15.12:0/118483080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b65a878d0 con 0x563b65b869f0 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82e800700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65c19e70 0x563b65c24170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd82d400700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c23c30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd83006c610 tx=0x7fd81c00f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c5dbafa2a0195d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.542+0000 7fd80fe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.543+0000 7fd80fe00700 1 -- 172.21.15.12:0/118483080 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c018070 con 0x563b65b869f0 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.543+0000 7fd80fe00700 1 -- 172.21.15.12:0/118483080 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fd81c014480 con 0x563b65b869f0 2024-05-27T07:32:52.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.543+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8200dfd00 con 0x563b65b869f0 2024-05-27T07:32:52.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.543+0000 7fd80fe00700 1 -- 172.21.15.12:0/118483080 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fd81c024050 con 0x563b65b869f0 2024-05-27T07:32:52.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.543+0000 7fd843723b80 10 client.16707 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:52.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.543+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x563b65b08060 con 0x563b65b869f0 2024-05-27T07:32:52.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.543+0000 7fd80fe00700 1 -- 172.21.15.12:0/118483080 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8200dfd00 con 0x563b65b869f0 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.544+0000 7fd80fe00700 1 -- 172.21.15.12:0/118483080 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fd81c0137b0 con 0x563b65b869f0 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.544+0000 7fd843723b80 10 client.16707 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.544+0000 7fd843723b80 10 client.16707 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.544+0000 7fd843723b80 20 client.16707 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.544+0000 7fd843723b80 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65c30680 0x563b65c32ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.544+0000 7fd843723b80 4 client.16707 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.544+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b65b49d90 con 0x563b65c30680 2024-05-27T07:32:52.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.545+0000 7fd82de00700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65c30680 0x563b65c32ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.546+0000 7fd82de00700 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65c30680 0x563b65c32ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=becfaaa96c5ee269 server_cookie=e20c27dd8fed9888 in_seq=0 out_seq=0 2024-05-27T07:32:52.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.546+0000 7fd80fe00700 10 client.16707 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:32:52.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.549+0000 7fd80fe00700 1 -- 172.21.15.12:0/118483080 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563b65b49d90 con 0x563b65c30680 2024-05-27T07:32:52.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.549+0000 7fd80fe00700 10 client.16707 handle_command_reply: tid=0 2024-05-27T07:32:52.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.549+0000 7fd843723b80 1 client.16707 shutdown 2024-05-27T07:32:52.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.550+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 msgr2=0x563b65c23c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.550+0000 7fd843723b80 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c23c30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd83006c610 tx=0x7fd81c00f4c0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 shutdown_connections 2024-05-27T07:32:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65c30680 0x563b65c32ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65c19e70 0x563b65c24170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65a9ae40 0x563b65c236f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 --2- 172.21.15.12:0/118483080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c23c30 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd80fe00700 0 client.16707 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:32:52.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 >> 172.21.15.12:0/118483080 conn(0x563b65b34390 msgr2=0x563b65c195c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:52.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 shutdown_connections 2024-05-27T07:32:52.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.551+0000 7fd843723b80 1 -- 172.21.15.12:0/118483080 wait complete. 2024-05-27T07:32:52.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.552+0000 7fd843723b80 20 client.16707 trim_cache size 0 max 16384 2024-05-27T07:32:52.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.573+0000 7fd843723b80 1 Processor -- start 2024-05-27T07:32:52.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.574+0000 7fd843723b80 1 -- start start 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.574+0000 7fd843723b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c382d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.574+0000 7fd843723b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c1d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.574+0000 7fd843723b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c4fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.574+0000 7fd843723b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563b65c19930 con 0x563b65b869f0 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.574+0000 7fd843723b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301adb50 con 0x563b65a9ae40 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.574+0000 7fd843723b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301acdb0 con 0x563b65c19e70 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c1d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c1d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47474/0 (socket says 172.21.15.12:47474) 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 -- 172.21.15.12:0/80771163 learned_addr learned my addr 172.21.15.12:0/80771163 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:52.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82de00700 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82e800700 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c4fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 -- 172.21.15.12:0/80771163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 msgr2=0x563b65c4fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c4fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 -- 172.21.15.12:0/80771163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 msgr2=0x563b65c382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c382d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.575+0000 7fd82d400700 1 -- 172.21.15.12:0/80771163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b65c21c60 con 0x563b65b869f0 2024-05-27T07:32:52.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.576+0000 7fd82d400700 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c1d5f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd81c014040 tx=0x7fd81c00d420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46da01fa8a09112f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:52.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.576+0000 7fd81aa00700 1 -- 172.21.15.12:0/80771163 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c02c020 con 0x563b65b869f0 2024-05-27T07:32:52.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.576+0000 7fd81aa00700 1 -- 172.21.15.12:0/80771163 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd81c013870 con 0x563b65b869f0 2024-05-27T07:32:52.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.576+0000 7fd81aa00700 1 -- 172.21.15.12:0/80771163 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c018040 con 0x563b65b869f0 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.577+0000 7fd843723b80 1 -- 172.21.15.12:0/80771163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 msgr2=0x563b65c1d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.577+0000 7fd843723b80 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c1d5f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd81c014040 tx=0x7fd81c00d420 comp rx=0 tx=0).stop 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.577+0000 7fd843723b80 1 -- 172.21.15.12:0/80771163 shutdown_connections 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.577+0000 7fd843723b80 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c382d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.577+0000 7fd843723b80 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c4fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.577+0000 7fd843723b80 1 --2- 172.21.15.12:0/80771163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c1d5f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.577+0000 7fd843723b80 1 -- 172.21.15.12:0/80771163 >> 172.21.15.12:0/80771163 conn(0x563b65b34390 msgr2=0x563b65c1cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.578+0000 7fd843723b80 1 -- 172.21.15.12:0/80771163 shutdown_connections 2024-05-27T07:32:52.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.578+0000 7fd843723b80 1 -- 172.21.15.12:0/80771163 wait complete. 2024-05-27T07:32:52.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.578+0000 7fd843723b80 1 -- start start 2024-05-27T07:32:52.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 1 Processor -- start 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c27b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c2bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x563b65c14150 con 0x563b65b869f0 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301ad480 con 0x563b65a9ae40 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563b65c2c060 con 0x563b65c19e70 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.579+0000 7fd843723b80 10 client.0 _ll_register_callbacks cb 0x563b65ae1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:52.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c27b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c27b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47486/0 (socket says 172.21.15.12:47486) 2024-05-27T07:32:52.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 -- 172.21.15.12:0/3908019434 learned_addr learned my addr 172.21.15.12:0/3908019434 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82de00700 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82e800700 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c2bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 -- 172.21.15.12:0/3908019434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 msgr2=0x563b65c2bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c2bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 -- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 msgr2=0x563b65c27600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c27600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.580+0000 7fd82d400700 1 -- 172.21.15.12:0/3908019434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b65c2c260 con 0x563b65b869f0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.581+0000 7fd82d400700 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c27b40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd81c000c00 tx=0x7fd81c00ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9710dfdf4dad76c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.581+0000 7fd80fe00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.581+0000 7fd80fe00700 1 -- 172.21.15.12:0/3908019434 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c02c070 con 0x563b65b869f0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.581+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd81c013870 con 0x563b65b869f0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.581+0000 7fd80fe00700 1 -- 172.21.15.12:0/3908019434 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fd81c0135e0 con 0x563b65b869f0 2024-05-27T07:32:52.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd80fe00700 1 -- 172.21.15.12:0/3908019434 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fd81c012360 con 0x563b65b869f0 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 10 client.16713 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x563b65caf5b0 con 0x563b65b869f0 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd80fe00700 1 -- 172.21.15.12:0/3908019434 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd81c013870 con 0x563b65b869f0 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd80fe00700 1 -- 172.21.15.12:0/3908019434 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fd81c014920 con 0x563b65b869f0 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 10 client.16713 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 10 client.16713 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 20 client.16713 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:52.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65cafbc0 0x563b65cb1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:52.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 4 client.16713 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:32:52.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.582+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b65c14b10 con 0x563b65cafbc0 2024-05-27T07:32:52.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.583+0000 7fd82de00700 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65cafbc0 0x563b65cb1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.584+0000 7fd82de00700 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65cafbc0 0x563b65cb1fe0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=423a8983194b1366 server_cookie=e190c82aca4b554b in_seq=0 out_seq=0 2024-05-27T07:32:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.584+0000 7fd80fe00700 10 client.16713 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:32:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd80fe00700 1 -- 172.21.15.12:0/3908019434 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b65c14b10 con 0x563b65cafbc0 2024-05-27T07:32:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd80fe00700 10 client.16713 handle_command_reply: tid=0 2024-05-27T07:32:52.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 client.16713 shutdown 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 msgr2=0x563b65c27b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c27b40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd81c000c00 tx=0x7fd81c00ffa0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 shutdown_connections 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x563b65cafbc0 0x563b65cb1fe0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563b65a9ae40 0x563b65c27600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563b65c19e70 0x563b65c2bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 --2- 172.21.15.12:0/3908019434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x563b65b869f0 0x563b65c27b40 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd80fe00700 0 client.16713 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 >> 172.21.15.12:0/3908019434 conn(0x563b65b34390 msgr2=0x563b65c234d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 shutdown_connections 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 1 -- 172.21.15.12:0/3908019434 wait complete. 2024-05-27T07:32:52.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.585+0000 7fd843723b80 20 client.16713 trim_cache size 0 max 16384 2024-05-27T07:32:52.581 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8140b2a70 msgr2=0x7fd8140b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8140b2a70 0x7fd8140b4f80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd828009610 tx=0x7fd8280099e0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106560 msgr2=0x7fd83019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106560 0x7fd83019cf50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd82000d980 tx=0x7fd82000dd40 comp rx=0 tx=0).stop 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 shutdown_connections 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd8140b2a70 0x7fd8140b4f80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd83010c5a0 0x7fd8301a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd830106560 0x7fd83019cf50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 --2- 172.21.15.12:0/1964194334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd830106eb0 0x7fd83019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:52.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.586+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 >> 172.21.15.12:0/1964194334 conn(0x7fd830102040 msgr2=0x7fd8301035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.587+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 shutdown_connections 2024-05-27T07:32:52.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:52.587+0000 7fd834c00700 1 -- 172.21.15.12:0/1964194334 wait complete. 2024-05-27T07:32:53.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:32:53.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:53 smithi012 ceph-mon[31473]: pgmap v365: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:32:53.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:53 smithi162 ceph-mon[35676]: pgmap v365: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:32:53.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:53 smithi062 ceph-mon[35304]: pgmap v365: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:32:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:54.773 INFO:tasks.workunit.client.0.smithi012.stdout:fallocating to largest ever: 0xa00000 2024-05-27T07:32:55.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:55 smithi162 ceph-mon[35676]: pgmap v366: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:32:55.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:55 smithi062 ceph-mon[35304]: pgmap v366: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:32:56.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:55 smithi012 ceph-mon[31473]: pgmap v366: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:32:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 -- 172.21.15.12:0/681730056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8076670 msgr2=0x7fa2d8110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 --2- 172.21.15.12:0/681730056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8076670 0x7fa2d8110650 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009990 tx=0x7fa2c0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 -- 172.21.15.12:0/681730056 shutdown_connections 2024-05-27T07:32:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 --2- 172.21.15.12:0/681730056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2d8110b90 0x7fa2d8112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 --2- 172.21.15.12:0/681730056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d8075d20 0x7fa2d8076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 --2- 172.21.15.12:0/681730056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8076670 0x7fa2d8110650 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 -- 172.21.15.12:0/681730056 >> 172.21.15.12:0/681730056 conn(0x7fa2d8100030 msgr2=0x7fa2d8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 -- 172.21.15.12:0/681730056 shutdown_connections 2024-05-27T07:32:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.682+0000 7fa2dec00700 1 -- 172.21.15.12:0/681730056 wait complete. 2024-05-27T07:32:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.683+0000 7fa2dec00700 1 Processor -- start 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.684+0000 7fa2dec00700 1 -- start start 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.684+0000 7fa2dec00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8075d20 0x7fa2d819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.684+0000 7fa2dec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2d8076670 0x7fa2d819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.684+0000 7fa2dec00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d8110b90 0x7fa2d81a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.684+0000 7fa2dec00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8116590 con 0x7fa2d8075d20 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.684+0000 7fa2dec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8116410 con 0x7fa2d8076670 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.684+0000 7fa2dec00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8116710 con 0x7fa2d8110b90 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8075d20 0x7fa2d819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8075d20 0x7fa2d819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47506/0 (socket says 172.21.15.12:47506) 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 -- 172.21.15.12:0/576150643 learned_addr learned my addr 172.21.15.12:0/576150643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2c7400700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2d8076670 0x7fa2d819d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2dcc00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d8110b90 0x7fa2d81a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 -- 172.21.15.12:0/576150643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d8110b90 msgr2=0x7fa2d81a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d8110b90 0x7fa2d81a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 -- 172.21.15.12:0/576150643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2d8076670 msgr2=0x7fa2d819d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2d8076670 0x7fa2d819d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.685+0000 7fa2d7e00700 1 -- 172.21.15.12:0/576150643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d81a1dc0 con 0x7fa2d8075d20 2024-05-27T07:32:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.686+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8075d20 0x7fa2d819cfa0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8007e20 tx=0x7fa2c800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.686+0000 7fa2d5600700 1 -- 172.21.15.12:0/576150643 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8016020 con 0x7fa2d8075d20 2024-05-27T07:32:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.686+0000 7fa2d5600700 1 -- 172.21.15.12:0/576150643 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2c800ae00 con 0x7fa2d8075d20 2024-05-27T07:32:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.686+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d81a20b0 con 0x7fa2d8075d20 2024-05-27T07:32:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.686+0000 7fa2d5600700 1 -- 172.21.15.12:0/576150643 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8011670 con 0x7fa2d8075d20 2024-05-27T07:32:56.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.686+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d81a25f0 con 0x7fa2d8075d20 2024-05-27T07:32:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.690+0000 7fa2d5600700 1 -- 172.21.15.12:0/576150643 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa2c8019020 con 0x7fa2d8075d20 2024-05-27T07:32:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.691+0000 7fa2ed698b80 1 Processor -- start 2024-05-27T07:32:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.691+0000 7fa2ed698b80 1 -- start start 2024-05-27T07:32:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f2799f0 0x556c5f30c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f30c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f319110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556c5f23ae40 con 0x556c5f30ce70 2024-05-27T07:32:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556c5f22a9e0 con 0x556c5f18de40 2024-05-27T07:32:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556c5f22ea20 con 0x556c5f2799f0 2024-05-27T07:32:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2d5600700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.692+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f319110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f319110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47516/0 (socket says 172.21.15.12:47516) 2024-05-27T07:32:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2c7400700 1 -- 172.21.15.12:0/2067804736 learned_addr learned my addr 172.21.15.12:0/2067804736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2d5600700 1 -- 172.21.15.12:0/576150643 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fa2c80e2760 con 0x7fa2d8075d20 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f30c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2dcc00700 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f2799f0 0x556c5f30c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2dcc00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2c7400700 1 -- 172.21.15.12:0/2067804736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f2799f0 msgr2=0x556c5f30c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2c7400700 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f2799f0 0x556c5f30c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2c7400700 1 -- 172.21.15.12:0/2067804736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 msgr2=0x556c5f30c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2c7400700 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f30c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.693+0000 7fa2c7400700 1 -- 172.21.15.12:0/2067804736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c5f1f3420 con 0x556c5f30ce70 2024-05-27T07:32:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.694+0000 7fa2c7400700 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f319110 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0015000 tx=0x7fa2c0003720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b8ffb8e057a469 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.694+0000 7fa2dcc00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x556c5f30eff0 tx=0x7fa2d00058e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.694+0000 7fa2ce000700 1 -- 172.21.15.12:0/2067804736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0026070 con 0x556c5f30ce70 2024-05-27T07:32:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.694+0000 7fa2ce000700 1 -- 172.21.15.12:0/2067804736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2c0003d10 con 0x556c5f30ce70 2024-05-27T07:32:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.694+0000 7fa2ce000700 1 -- 172.21.15.12:0/2067804736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c000ba50 con 0x556c5f30ce70 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.695+0000 7fa2ed698b80 1 -- 172.21.15.12:0/2067804736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 msgr2=0x556c5f319110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.695+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f319110 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0015000 tx=0x7fa2c0003720 comp rx=0 tx=0).stop 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 -- 172.21.15.12:0/2067804736 shutdown_connections 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f319110 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f2799f0 0x556c5f30c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/2067804736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f30c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 -- 172.21.15.12:0/2067804736 >> 172.21.15.12:0/2067804736 conn(0x556c5f227390 msgr2=0x556c5f3084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 -- 172.21.15.12:0/2067804736 shutdown_connections 2024-05-27T07:32:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 -- 172.21.15.12:0/2067804736 wait complete. 2024-05-27T07:32:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.696+0000 7fa2ed698b80 1 -- start start 2024-05-27T07:32:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 1 Processor -- start 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f18de40 0x556c5f316660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f2799f0 0x556c5f316ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f30ce70 0x556c5f3170e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556c5f130fa0 con 0x556c5f2799f0 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556c5f1cb760 con 0x556c5f30ce70 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556c5f319b20 con 0x556c5f18de40 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.697+0000 7fa2ed698b80 10 client.0 _ll_register_callbacks cb 0x556c5f1d4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f2799f0 0x556c5f316ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f2799f0 0x556c5f316ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47526/0 (socket says 172.21.15.12:47526) 2024-05-27T07:32:56.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 -- 172.21.15.12:0/532712732 learned_addr learned my addr 172.21.15.12:0/532712732 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:56.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f18de40 0x556c5f316660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 -- 172.21.15.12:0/532712732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f18de40 msgr2=0x556c5f316660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f18de40 0x556c5f316660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 -- 172.21.15.12:0/532712732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f30ce70 msgr2=0x556c5f3170e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f30ce70 0x556c5f3170e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.698+0000 7fa2c7400700 1 -- 172.21.15.12:0/532712732 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c5f17a8d0 con 0x556c5f2799f0 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.699+0000 7fa2c7400700 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f2799f0 0x556c5f316ba0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00052f0 tx=0x7fa2c0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e53f8139c3a4f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.699+0000 7fa2c6000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.699+0000 7fa2c6000700 1 -- 172.21.15.12:0/532712732 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0026070 con 0x556c5f2799f0 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.699+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c0003d10 con 0x556c5f2799f0 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.699+0000 7fa2c6000700 1 -- 172.21.15.12:0/532712732 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fa2c000b9e0 con 0x556c5f2799f0 2024-05-27T07:32:56.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2c6000700 1 -- 172.21.15.12:0/532712732 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa2c0032050 con 0x556c5f2799f0 2024-05-27T07:32:56.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2ed698b80 10 client.16725 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:56.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x556c5f1fb060 con 0x556c5f2799f0 2024-05-27T07:32:56.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2c6000700 1 -- 172.21.15.12:0/532712732 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2c0003d10 con 0x556c5f2799f0 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2c6000700 1 -- 172.21.15.12:0/532712732 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa2c000fb20 con 0x556c5f2799f0 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2ed698b80 10 client.16725 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2ed698b80 10 client.16725 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.700+0000 7fa2ed698b80 20 client.16725 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.701+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f326ab0 0x556c5f328f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.701+0000 7fa2ed698b80 4 client.16725 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.701+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c5f23cd90 con 0x556c5f326ab0 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.701+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f326ab0 0x556c5f328f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.703+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f326ab0 0x556c5f328f20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a3030bd9ae061f9 server_cookie=fa34fd42f0ee5127 in_seq=0 out_seq=0 2024-05-27T07:32:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.703+0000 7fa2c6000700 10 client.16725 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:32:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.705+0000 7fa2c6000700 1 -- 172.21.15.12:0/532712732 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556c5f23cd90 con 0x556c5f326ab0 2024-05-27T07:32:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.705+0000 7fa2c6000700 10 client.16725 handle_command_reply: tid=0 2024-05-27T07:32:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.705+0000 7fa2ed698b80 1 client.16725 shutdown 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.705+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f2799f0 msgr2=0x556c5f316ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.705+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f2799f0 0x556c5f316ba0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00052f0 tx=0x7fa2c0003f90 comp rx=0 tx=0).stop 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 shutdown_connections 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f326ab0 0x556c5f328f20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f30ce70 0x556c5f3170e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f18de40 0x556c5f316660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2c6000700 0 client.16725 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/532712732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f2799f0 0x556c5f316ba0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 >> 172.21.15.12:0/532712732 conn(0x556c5f227390 msgr2=0x556c5f30c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 shutdown_connections 2024-05-27T07:32:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.706+0000 7fa2ed698b80 1 -- 172.21.15.12:0/532712732 wait complete. 2024-05-27T07:32:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.707+0000 7fa2ed698b80 20 client.16725 trim_cache size 0 max 16384 2024-05-27T07:32:56.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 Processor -- start 2024-05-27T07:32:56.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 -- start start 2024-05-27T07:32:56.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f311e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f3084f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f318420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b1df0 con 0x556c5f30ce70 2024-05-27T07:32:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b1050 con 0x556c5f18de40 2024-05-27T07:32:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.727+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b1720 con 0x556c5f322bc0 2024-05-27T07:32:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.728+0000 7fa2dcc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f318420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.728+0000 7fa2d7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f311e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.728+0000 7fa2dcc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f318420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:52144/0 (socket says 172.21.15.12:52144) 2024-05-27T07:32:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.728+0000 7fa2dcc00700 1 -- 172.21.15.12:0/356861443 learned_addr learned my addr 172.21.15.12:0/356861443 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.728+0000 7fa2c7400700 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f3084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.729+0000 7fa2d7e00700 1 -- 172.21.15.12:0/356861443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 msgr2=0x556c5f318420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.729+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f318420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.729+0000 7fa2d7e00700 1 -- 172.21.15.12:0/356861443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 msgr2=0x556c5f3084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.729+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f3084f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.729+0000 7fa2d7e00700 1 -- 172.21.15.12:0/356861443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c5f309000 con 0x556c5f18de40 2024-05-27T07:32:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.729+0000 7fa2dcc00700 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f318420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.729+0000 7fa2c7400700 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f3084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:56.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.730+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f311e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80dfb50 tx=0x7fa2c80e4bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3031ba4260b09375 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:56.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.736+0000 7fa2cd600700 1 -- 172.21.15.12:0/356861443 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c80ada90 con 0x556c5f18de40 2024-05-27T07:32:56.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.736+0000 7fa2cd600700 1 -- 172.21.15.12:0/356861443 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2c80b2540 con 0x556c5f18de40 2024-05-27T07:32:56.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.736+0000 7fa2cd600700 1 -- 172.21.15.12:0/356861443 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c80dfdb0 con 0x556c5f18de40 2024-05-27T07:32:56.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.737+0000 7fa2ed698b80 1 -- 172.21.15.12:0/356861443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 msgr2=0x556c5f311e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.737+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f311e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80dfb50 tx=0x7fa2c80e4bb0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.738+0000 7fa2ed698b80 1 -- 172.21.15.12:0/356861443 shutdown_connections 2024-05-27T07:32:56.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.738+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f311e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.738+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f318420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.738+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/356861443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f3084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.738+0000 7fa2ed698b80 1 -- 172.21.15.12:0/356861443 >> 172.21.15.12:0/356861443 conn(0x556c5f227390 msgr2=0x556c5f317d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:56.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.738+0000 7fa2ed698b80 1 -- 172.21.15.12:0/356861443 shutdown_connections 2024-05-27T07:32:56.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.738+0000 7fa2ed698b80 1 -- 172.21.15.12:0/356861443 wait complete. 2024-05-27T07:32:56.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.739+0000 7fa2ed698b80 1 -- start start 2024-05-27T07:32:56.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.739+0000 7fa2ed698b80 1 Processor -- start 2024-05-27T07:32:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f31f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f311080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2ed698b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f311740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556c5f307150 con 0x556c5f30ce70 2024-05-27T07:32:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556c5f308a30 con 0x556c5f18de40 2024-05-27T07:32:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2ed698b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556c5f311e60 con 0x556c5f322bc0 2024-05-27T07:32:56.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2ed698b80 10 client.0 _ll_register_callbacks cb 0x556c5f1d4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f311080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f311080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47538/0 (socket says 172.21.15.12:47538) 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2c7400700 1 -- 172.21.15.12:0/130881105 learned_addr learned my addr 172.21.15.12:0/130881105 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f31f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.740+0000 7fa2dcc00700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f311740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2c7400700 1 -- 172.21.15.12:0/130881105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 msgr2=0x556c5f311740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2c7400700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f311740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2c7400700 1 -- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 msgr2=0x556c5f31f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2c7400700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f31f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2c7400700 1 -- 172.21.15.12:0/130881105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c5f31d110 con 0x556c5f30ce70 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f31f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2c7400700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f311080 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000fec0 tx=0x7fa2c0009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ddad929de650d7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.741+0000 7fa2c6000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2c6000700 1 -- 172.21.15.12:0/130881105 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0026040 con 0x556c5f30ce70 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2c6000700 1 -- 172.21.15.12:0/130881105 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fa2c000bbf0 con 0x556c5f30ce70 2024-05-27T07:32:56.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c80b2540 con 0x556c5f30ce70 2024-05-27T07:32:56.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2c6000700 1 -- 172.21.15.12:0/130881105 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa2c003f050 con 0x556c5f30ce70 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2ed698b80 10 client.16731 fetch_fsmap learned FSMap version 119 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x556c5f39a010 con 0x556c5f30ce70 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2c6000700 1 -- 172.21.15.12:0/130881105 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2c80b2540 con 0x556c5f30ce70 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.742+0000 7fa2c6000700 1 -- 172.21.15.12:0/130881105 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa2c0029c40 con 0x556c5f30ce70 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.743+0000 7fa2ed698b80 10 client.16731 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.743+0000 7fa2ed698b80 10 client.16731 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.743+0000 7fa2ed698b80 20 client.16731 populate_metadata read hostname 'smithi012' 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.743+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f39a630 0x556c5f39ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.743+0000 7fa2ed698b80 4 client.16731 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.743+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c5f236570 con 0x556c5f39a630 2024-05-27T07:32:56.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.743+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f39a630 0x556c5f39ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:56.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.745+0000 7fa2d7e00700 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f39a630 0x556c5f39ca50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5df7311427931469 server_cookie=ecf9b006c543fa97 in_seq=0 out_seq=0 2024-05-27T07:32:56.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.745+0000 7fa2c6000700 10 client.16731 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.745+0000 7fa2c6000700 1 -- 172.21.15.12:0/130881105 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c5f236570 con 0x556c5f39a630 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.745+0000 7fa2c6000700 10 client.16731 handle_command_reply: tid=0 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.745+0000 7fa2ed698b80 1 client.16731 shutdown 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.745+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 msgr2=0x556c5f311080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.745+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f311080 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000fec0 tx=0x7fa2c0009710 comp rx=0 tx=0).stop 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 shutdown_connections 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x556c5f39a630 0x556c5f39ca50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c5f18de40 0x556c5f31f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c5f322bc0 0x556c5f311740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 --2- 172.21.15.12:0/130881105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c5f30ce70 0x556c5f311080 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 >> 172.21.15.12:0/130881105 conn(0x556c5f227390 msgr2=0x556c5f317c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:56.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2c6000700 0 client.16731 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:32:56.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 shutdown_connections 2024-05-27T07:32:56.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 1 -- 172.21.15.12:0/130881105 wait complete. 2024-05-27T07:32:56.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.746+0000 7fa2ed698b80 20 client.16731 trim_cache size 0 max 16384 2024-05-27T07:32:56.741 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa2b80b29f0 msgr2=0x7fa2b80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x556c5f30eff0 tx=0x7fa2d00058e0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8075d20 msgr2=0x7fa2d819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8075d20 0x7fa2d819cfa0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8007e20 tx=0x7fa2c800a340 comp rx=0 tx=0).stop 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 shutdown_connections 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa2b80b29f0 0x7fa2b80b4f00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d8110b90 0x7fa2d81a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2d8075d20 0x7fa2d819cfa0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 --2- 172.21.15.12:0/576150643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2d8076670 0x7fa2d819d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:56.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 >> 172.21.15.12:0/576150643 conn(0x7fa2d8100030 msgr2=0x7fa2d8100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:56.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 shutdown_connections 2024-05-27T07:32:56.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:32:56.747+0000 7fa2dec00700 1 -- 172.21.15.12:0/576150643 wait complete. 2024-05-27T07:32:57.620 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:32:57.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:57 smithi012 ceph-mon[31473]: pgmap v367: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-05-27T07:32:57.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:57 smithi162 ceph-mon[35676]: pgmap v367: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-05-27T07:32:57.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:57 smithi062 ceph-mon[35304]: pgmap v367: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-05-27T07:32:58.107 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:59.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:59 smithi062 ceph-mon[35304]: pgmap v368: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-05-27T07:32:59.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:32:59 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:00.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:59 smithi012 ceph-mon[31473]: pgmap v368: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-05-27T07:33:00.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:32:59 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:00.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:59 smithi162 ceph-mon[35676]: pgmap v368: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-05-27T07:33:00.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:32:59 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 -- 172.21.15.12:0/2928360043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c1045e0 msgr2=0x7f429c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 --2- 172.21.15.12:0/2928360043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c1045e0 0x7f429c1049f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f428c0099c0 tx=0x7f428c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 -- 172.21.15.12:0/2928360043 shutdown_connections 2024-05-27T07:33:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 --2- 172.21.15.12:0/2928360043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c109860 0x7f429c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 --2- 172.21.15.12:0/2928360043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c1045e0 0x7f429c1049f0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 --2- 172.21.15.12:0/2928360043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f429c108f10 0x7f429c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 -- 172.21.15.12:0/2928360043 >> 172.21.15.12:0/2928360043 conn(0x7f429c100030 msgr2=0x7f429c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:01.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 -- 172.21.15.12:0/2928360043 shutdown_connections 2024-05-27T07:33:01.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.531+0000 7f42a2000700 1 -- 172.21.15.12:0/2928360043 wait complete. 2024-05-27T07:33:01.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.533+0000 7f42a2000700 1 Processor -- start 2024-05-27T07:33:01.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.534+0000 7f42a2000700 1 -- start start 2024-05-27T07:33:01.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.534+0000 7f42a2000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c1045e0 0x7f429c077650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.534+0000 7f42a2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c108f10 0x7f429c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.534+0000 7f42a2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f429c109860 0x7f429c076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.534+0000 7f42a2000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1143e0 con 0x7f429c108f10 2024-05-27T07:33:01.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.534+0000 7f42a2000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f429c114260 con 0x7f429c109860 2024-05-27T07:33:01.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.534+0000 7f42a2000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f429c114560 con 0x7f429c1045e0 2024-05-27T07:33:01.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.535+0000 7f429b400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c1045e0 0x7f429c077650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.535+0000 7f429b400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c1045e0 0x7f429c077650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:52176/0 (socket says 172.21.15.12:52176) 2024-05-27T07:33:01.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.535+0000 7f429b400700 1 -- 172.21.15.12:0/1084133714 learned_addr learned my addr 172.21.15.12:0/1084133714 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:01.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.535+0000 7f4293e00700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c108f10 0x7f429c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.536+0000 7f429be00700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f429c109860 0x7f429c076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.536+0000 7f429b400700 1 -- 172.21.15.12:0/1084133714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f429c109860 msgr2=0x7f429c076090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.536+0000 7f429b400700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f429c109860 0x7f429c076090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.536+0000 7f429b400700 1 -- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c108f10 msgr2=0x7f429c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.536+0000 7f429b400700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c108f10 0x7f429c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.536+0000 7f429b400700 1 -- 172.21.15.12:0/1084133714 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c076830 con 0x7f429c1045e0 2024-05-27T07:33:01.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.537+0000 7f429b400700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c1045e0 0x7f429c077650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f428c005910 tx=0x7f428c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:01.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.538+0000 7f4298c00700 1 -- 172.21.15.12:0/1084133714 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c009e60 con 0x7f429c1045e0 2024-05-27T07:33:01.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.538+0000 7f4298c00700 1 -- 172.21.15.12:0/1084133714 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f428c00f460 con 0x7f429c1045e0 2024-05-27T07:33:01.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.538+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429c19ef40 con 0x7f429c1045e0 2024-05-27T07:33:01.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.538+0000 7f4298c00700 1 -- 172.21.15.12:0/1084133714 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c003e60 con 0x7f429c1045e0 2024-05-27T07:33:01.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.538+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f429c19f470 con 0x7f429c1045e0 2024-05-27T07:33:01.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.544+0000 7f4298c00700 1 -- 172.21.15.12:0/1084133714 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f428c00f910 con 0x7f429c1045e0 2024-05-27T07:33:01.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 Processor -- start 2024-05-27T07:33:01.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 -- start start 2024-05-27T07:33:01.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b049f0 0x558a43b97330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43a18e40 0x558a43b97870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43ba4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558a43ac5e40 con 0x558a43b049f0 2024-05-27T07:33:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558a43ab59e0 con 0x558a43a18e40 2024-05-27T07:33:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.545+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a43ab9a20 con 0x558a43b97e70 2024-05-27T07:33:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.546+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b049f0 0x558a43b97330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.546+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b049f0 0x558a43b97330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47570/0 (socket says 172.21.15.12:47570) 2024-05-27T07:33:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.546+0000 7f429be00700 1 -- 172.21.15.12:0/2521897621 learned_addr learned my addr 172.21.15.12:0/2521897621 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.546+0000 7f429b400700 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43a18e40 0x558a43b97870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.546+0000 7f4293e00700 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43ba4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f429be00700 1 -- 172.21.15.12:0/2521897621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 msgr2=0x558a43ba4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f429be00700 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43ba4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f429be00700 1 -- 172.21.15.12:0/2521897621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43a18e40 msgr2=0x558a43b97870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f429be00700 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43a18e40 0x558a43b97870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f429be00700 1 -- 172.21.15.12:0/2521897621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a43a7e420 con 0x558a43b049f0 2024-05-27T07:33:01.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f429b400700 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43a18e40 0x558a43b97870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:01.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f4298c00700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f42800b2aa0 0x7f42800b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f429be00700 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b049f0 0x558a43b97330 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4294011680 tx=0x7f4294009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b75f5160c83daa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:01.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.547+0000 7f4298c00700 1 -- 172.21.15.12:0/1084133714 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f428c0e3960 con 0x7f429c1045e0 2024-05-27T07:33:01.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.548+0000 7f4290c00700 1 -- 172.21.15.12:0/2521897621 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294019030 con 0x558a43b049f0 2024-05-27T07:33:01.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.548+0000 7f429b400700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f42800b2aa0 0x7f42800b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.548+0000 7f4290c00700 1 -- 172.21.15.12:0/2521897621 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42940093a0 con 0x558a43b049f0 2024-05-27T07:33:01.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.548+0000 7f4290c00700 1 -- 172.21.15.12:0/2521897621 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294007500 con 0x558a43b049f0 2024-05-27T07:33:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2521897621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b049f0 msgr2=0x558a43b97330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f429b400700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f42800b2aa0 0x7f42800b4fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f428c0e0d00 tx=0x7f428c009cd0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b049f0 0x558a43b97330 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4294011680 tx=0x7f4294009920 comp rx=0 tx=0).stop 2024-05-27T07:33:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2521897621 shutdown_connections 2024-05-27T07:33:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43ba4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b049f0 0x558a43b97330 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2521897621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43a18e40 0x558a43b97870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2521897621 >> 172.21.15.12:0/2521897621 conn(0x558a43ab2390 msgr2=0x558a43b934c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.549+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2521897621 shutdown_connections 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.550+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2521897621 wait complete. 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.550+0000 7f42b0c1ab80 1 -- start start 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.550+0000 7f42b0c1ab80 1 Processor -- start 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.551+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43ba1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.551+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b049f0 0x558a43ba1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.551+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b97e70 0x558a43ba2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.551+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558a439bbfa0 con 0x558a43a18e40 2024-05-27T07:33:01.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.551+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558a43a56760 con 0x558a43b97e70 2024-05-27T07:33:01.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.551+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a43ba4b80 con 0x558a43b049f0 2024-05-27T07:33:01.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.552+0000 7f42b0c1ab80 10 client.0 _ll_register_callbacks cb 0x558a43a5fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:01.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.553+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43ba1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.554+0000 7f4293e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b049f0 0x558a43ba1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.554+0000 7f4293e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b049f0 0x558a43ba1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:52194/0 (socket says 172.21.15.12:52194) 2024-05-27T07:33:01.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.554+0000 7f4293e00700 1 -- 172.21.15.12:0/2463022551 learned_addr learned my addr 172.21.15.12:0/2463022551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:01.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.554+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43ba1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47578/0 (socket says 172.21.15.12:47578) 2024-05-27T07:33:01.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.554+0000 7f429be00700 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b97e70 0x558a43ba2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.555+0000 7f4293e00700 1 -- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b97e70 msgr2=0x558a43ba2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.555+0000 7f4293e00700 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b97e70 0x558a43ba2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.555+0000 7f4293e00700 1 -- 172.21.15.12:0/2463022551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 msgr2=0x558a43ba1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.555+0000 7f4293e00700 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43ba1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.555+0000 7f4293e00700 1 -- 172.21.15.12:0/2463022551 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a43a058d0 con 0x558a43b049f0 2024-05-27T07:33:01.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.556+0000 7f4293e00700 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b049f0 0x558a43ba1c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4284012830 tx=0x7f4284012b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a82929b42c3fe3b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:01.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.556+0000 7f4289600700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:33:01.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.556+0000 7f4289600700 1 -- 172.21.15.12:0/2463022551 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428401f060 con 0x558a43b049f0 2024-05-27T07:33:01.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.556+0000 7f4289600700 1 -- 172.21.15.12:0/2463022551 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f4284004030 con 0x558a43b049f0 2024-05-27T07:33:01.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.557+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558a43ac7d90 con 0x558a43b049f0 2024-05-27T07:33:01.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.557+0000 7f4289600700 1 -- 172.21.15.12:0/2463022551 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4284004230 con 0x558a43b049f0 2024-05-27T07:33:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.557+0000 7f42b0c1ab80 10 client.26278 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.557+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558a43a86060 con 0x558a43b049f0 2024-05-27T07:33:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.558+0000 7f4289600700 1 -- 172.21.15.12:0/2463022551 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x558a43ac7d90 con 0x558a43b049f0 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.558+0000 7f4289600700 1 -- 172.21.15.12:0/2463022551 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f4284003930 con 0x558a43b049f0 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.559+0000 7f42b0c1ab80 10 client.26278 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.559+0000 7f42b0c1ab80 10 client.26278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.559+0000 7f42b0c1ab80 20 client.26278 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.559+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43bb1da0 0x558a43bb4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.559+0000 7f42b0c1ab80 4 client.26278 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.559+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a43ac1570 con 0x558a43bb1da0 2024-05-27T07:33:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.559+0000 7f429be00700 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43bb1da0 0x558a43bb4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.560+0000 7f429be00700 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43bb1da0 0x558a43bb4230 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f66c0cb705c120cd server_cookie=bfdf00f493f677d3 in_seq=0 out_seq=0 2024-05-27T07:33:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.561+0000 7f4289600700 10 client.26278 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.562+0000 7f4289600700 1 -- 172.21.15.12:0/2463022551 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558a43ac1570 con 0x558a43bb1da0 2024-05-27T07:33:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.562+0000 7f4289600700 10 client.26278 handle_command_reply: tid=0 2024-05-27T07:33:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.562+0000 7f42b0c1ab80 1 client.26278 shutdown 2024-05-27T07:33:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b049f0 msgr2=0x558a43ba1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b049f0 0x558a43ba1c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4284012830 tx=0x7f4284012b40 comp rx=0 tx=0).stop 2024-05-27T07:33:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 shutdown_connections 2024-05-27T07:33:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43bb1da0 0x558a43bb4230 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b97e70 0x558a43ba2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43ba1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2463022551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b049f0 0x558a43ba1c40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f4289600700 0 client.26278 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 >> 172.21.15.12:0/2463022551 conn(0x558a43ab2390 msgr2=0x558a43b97450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 shutdown_connections 2024-05-27T07:33:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.563+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2463022551 wait complete. 2024-05-27T07:33:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.564+0000 7f42b0c1ab80 20 client.26278 trim_cache size 0 max 16384 2024-05-27T07:33:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.587+0000 7f42b0c1ab80 1 Processor -- start 2024-05-27T07:33:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.587+0000 7f42b0c1ab80 1 -- start start 2024-05-27T07:33:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.587+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43a18e40 0x558a43ba0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.587+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b97e70 0x558a43b93590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1beab0 con 0x558a43b97e70 2024-05-27T07:33:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1a6cd0 con 0x558a43b049f0 2024-05-27T07:33:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1a73a0 con 0x558a43a18e40 2024-05-27T07:33:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b97e70 0x558a43b93590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f4293e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b97e70 0x558a43b93590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47594/0 (socket says 172.21.15.12:47594) 2024-05-27T07:33:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f429be00700 1 -- 172.21.15.12:0/2112136608 learned_addr learned my addr 172.21.15.12:0/2112136608 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.588+0000 7f4293e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:42494/0 (socket says 172.21.15.12:42494) 2024-05-27T07:33:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.589+0000 7f429be00700 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43a18e40 0x558a43ba0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.589+0000 7f429be00700 1 -- 172.21.15.12:0/2112136608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43a18e40 msgr2=0x558a43ba0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.589+0000 7f429be00700 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43a18e40 0x558a43ba0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.589+0000 7f429be00700 1 -- 172.21.15.12:0/2112136608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 msgr2=0x558a43b9cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.589+0000 7f429be00700 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.589+0000 7f429be00700 1 -- 172.21.15.12:0/2112136608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a43b9fc70 con 0x558a43b97e70 2024-05-27T07:33:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.590+0000 7f429be00700 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b97e70 0x558a43b93590 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4294011680 tx=0x7f4294011fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a24d4a0279ee88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.590+0000 7f428be00700 1 -- 172.21.15.12:0/2112136608 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294031070 con 0x558a43b97e70 2024-05-27T07:33:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.590+0000 7f428be00700 1 -- 172.21.15.12:0/2112136608 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4294003e90 con 0x558a43b97e70 2024-05-27T07:33:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.590+0000 7f428be00700 1 -- 172.21.15.12:0/2112136608 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429402b7d0 con 0x558a43b97e70 2024-05-27T07:33:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.591+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2112136608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b97e70 msgr2=0x558a43b93590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.591+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b97e70 0x558a43b93590 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4294011680 tx=0x7f4294011fd0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.591+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2112136608 shutdown_connections 2024-05-27T07:33:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.591+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43b97e70 0x558a43b93590 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.591+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43a18e40 0x558a43ba0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.591+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/2112136608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.591+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2112136608 >> 172.21.15.12:0/2112136608 conn(0x558a43ab2390 msgr2=0x558a43b92e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.592+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2112136608 shutdown_connections 2024-05-27T07:33:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.592+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/2112136608 wait complete. 2024-05-27T07:33:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.592+0000 7f42b0c1ab80 1 -- start start 2024-05-27T07:33:01.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.593+0000 7f42b0c1ab80 1 Processor -- start 2024-05-27T07:33:01.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.593+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43baa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.593+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.593+0000 7f42b0c1ab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43b9c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.593+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558a43ba14e0 con 0x558a43a18e40 2024-05-27T07:33:01.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.593+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558a43b8f950 con 0x558a43b049f0 2024-05-27T07:33:01.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.593+0000 7f42b0c1ab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a43b97c50 con 0x558a43b97e70 2024-05-27T07:33:01.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.594+0000 7f42b0c1ab80 10 client.0 _ll_register_callbacks cb 0x558a43a5fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:01.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.594+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43baa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.594+0000 7f429be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43baa340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47610/0 (socket says 172.21.15.12:47610) 2024-05-27T07:33:01.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.594+0000 7f429be00700 1 -- 172.21.15.12:0/3547023174 learned_addr learned my addr 172.21.15.12:0/3547023174 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:01.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.594+0000 7f4293e00700 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.594+0000 7f429be00700 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43b9c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.595+0000 7f429be00700 1 -- 172.21.15.12:0/3547023174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 msgr2=0x558a43b9c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.595+0000 7f429be00700 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43b9c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.595+0000 7f429be00700 1 -- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 msgr2=0x558a43b9bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:01.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.595+0000 7f429be00700 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.595+0000 7f429be00700 1 -- 172.21.15.12:0/3547023174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a43b9c7b0 con 0x558a43a18e40 2024-05-27T07:33:01.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.595+0000 7f429be00700 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43baa340 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f429400c010 tx=0x7f429402b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7295155be907fe5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:01.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.595+0000 7f4289600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:01.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.596+0000 7f4289600700 1 -- 172.21.15.12:0/3547023174 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429403b060 con 0x558a43a18e40 2024-05-27T07:33:01.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.596+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4294003e90 con 0x558a43a18e40 2024-05-27T07:33:01.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.596+0000 7f4289600700 1 -- 172.21.15.12:0/3547023174 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f4294003a00 con 0x558a43a18e40 2024-05-27T07:33:01.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.596+0000 7f4289600700 1 -- 172.21.15.12:0/3547023174 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4294042050 con 0x558a43a18e40 2024-05-27T07:33:01.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.596+0000 7f42b0c1ab80 10 client.16749 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:01.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.596+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558a43ba8b80 con 0x558a43a18e40 2024-05-27T07:33:01.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f4289600700 1 -- 172.21.15.12:0/3547023174 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4294003e90 con 0x558a43a18e40 2024-05-27T07:33:01.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f4289600700 1 -- 172.21.15.12:0/3547023174 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f4294007500 con 0x558a43a18e40 2024-05-27T07:33:01.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f42b0c1ab80 10 client.16749 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:01.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f42b0c1ab80 10 client.16749 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:01.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f42b0c1ab80 20 client.16749 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:01.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43c1c900 0x558a43c1ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:01.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f42b0c1ab80 4 client.16749 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:01.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.597+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a43938250 con 0x558a43c1c900 2024-05-27T07:33:01.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.599+0000 7f4293e00700 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43c1c900 0x558a43c1ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:01.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.600+0000 7f4293e00700 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43c1c900 0x558a43c1ed20 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad51d7677ae4f964 server_cookie=9ba8b3b452bbbae3 in_seq=0 out_seq=0 2024-05-27T07:33:01.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.600+0000 7f4289600700 10 client.16749 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:01.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.600+0000 7f4289600700 1 -- 172.21.15.12:0/3547023174 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a43938250 con 0x558a43c1c900 2024-05-27T07:33:01.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.600+0000 7f4289600700 10 client.16749 handle_command_reply: tid=0 2024-05-27T07:33:01.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.600+0000 7f42b0c1ab80 1 client.16749 shutdown 2024-05-27T07:33:01.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.600+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 msgr2=0x558a43baa340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43baa340 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f429400c010 tx=0x7f429402b3e0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 shutdown_connections 2024-05-27T07:33:01.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a43b97e70 0x558a43b9c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558a43a18e40 0x558a43baa340 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x558a43c1c900 0x558a43c1ed20 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 --2- 172.21.15.12:0/3547023174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a43b049f0 0x558a43b9bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 >> 172.21.15.12:0/3547023174 conn(0x558a43ab2390 msgr2=0x558a43ba2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f4289600700 0 client.16749 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.601+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 shutdown_connections 2024-05-27T07:33:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.602+0000 7f42b0c1ab80 1 -- 172.21.15.12:0/3547023174 wait complete. 2024-05-27T07:33:01.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.602+0000 7f42b0c1ab80 20 client.16749 trim_cache size 0 max 16384 2024-05-27T07:33:01.604 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:33:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.603+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f42800b2aa0 msgr2=0x7f42800b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.603+0000 7f42a2000700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f42800b2aa0 0x7f42800b4fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f428c0e0d00 tx=0x7f428c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.603+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c1045e0 msgr2=0x7f429c077650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.603+0000 7f42a2000700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c1045e0 0x7f429c077650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f428c005910 tx=0x7f428c0045e0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.604+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 shutdown_connections 2024-05-27T07:33:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.604+0000 7f42a2000700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f42800b2aa0 0x7f42800b4fb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.604+0000 7f42a2000700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f429c109860 0x7f429c076090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.604+0000 7f42a2000700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f429c1045e0 0x7f429c077650 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.604+0000 7f42a2000700 1 --2- 172.21.15.12:0/1084133714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f429c108f10 0x7f429c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.604+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 >> 172.21.15.12:0/1084133714 conn(0x7f429c100030 msgr2=0x7f429c102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.604+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 shutdown_connections 2024-05-27T07:33:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:01.605+0000 7f42a2000700 1 -- 172.21.15.12:0/1084133714 wait complete. 2024-05-27T07:33:01.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:01 smithi012 ceph-mon[31473]: pgmap v369: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:33:01.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:01 smithi012 ceph-mon[31473]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:33:02.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:01 smithi162 ceph-mon[35676]: pgmap v369: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:33:02.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:01 smithi162 ceph-mon[35676]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:33:02.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:01 smithi062 ceph-mon[35304]: pgmap v369: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-05-27T07:33:02.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:01 smithi062 ceph-mon[35304]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:33:03.777 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:33:03.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:33:04.290 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:04.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:04 smithi012 ceph-mon[31473]: pgmap v371: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:33:04.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:04 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:04 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:04 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:04 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:04 smithi162 ceph-mon[35676]: pgmap v371: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:33:04.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:04 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:04 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:04 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:04 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:04 smithi062 ceph-mon[35304]: pgmap v371: 129 pgs: 129 active+clean; 167 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:33:04.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:04 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:04 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:04 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:04.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:04 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:05.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:05 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:05 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:05 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:05 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:05 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:33:05.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:05 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:33:05.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:05 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:05 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:05 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:05 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:05 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:33:05.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:05 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:33:05.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:05 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:05 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:05 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:05 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:05.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:05 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:33:05.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:05 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:33:06.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:06 smithi012 ceph-mon[31473]: pgmap v372: 129 pgs: 129 active+clean; 178 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:33:06.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:06 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:06 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:06 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:06 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:06 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:33:06.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:06 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:33:06.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:06 smithi162 ceph-mon[35676]: pgmap v372: 129 pgs: 129 active+clean; 178 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:33:06.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:06 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:06 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:06 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:06 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:06 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:33:06.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:06 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:33:06.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:06 smithi062 ceph-mon[35304]: pgmap v372: 129 pgs: 129 active+clean; 178 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:33:06.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:06 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:06 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.489 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:06 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:06 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:06.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:06 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:33:06.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:06 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:33:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.187+0000 7f09c5600700 1 -- 172.21.15.12:0/1056680118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c01034b0 msgr2=0x7f09c01038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.187+0000 7f09c5600700 1 --2- 172.21.15.12:0/1056680118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c01034b0 0x7f09c01038c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f09b00099c0 tx=0x7f09b0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.187+0000 7f09c5600700 1 -- 172.21.15.12:0/1056680118 shutdown_connections 2024-05-27T07:33:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.187+0000 7f09c5600700 1 --2- 172.21.15.12:0/1056680118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c0108730 0x7f09c0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.187+0000 7f09c5600700 1 --2- 172.21.15.12:0/1056680118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c01034b0 0x7f09c01038c0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.187+0000 7f09c5600700 1 --2- 172.21.15.12:0/1056680118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0107de0 0x7f09c01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.187+0000 7f09c5600700 1 -- 172.21.15.12:0/1056680118 >> 172.21.15.12:0/1056680118 conn(0x7f09c00fedb0 msgr2=0x7f09c01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.188+0000 7f09c5600700 1 -- 172.21.15.12:0/1056680118 shutdown_connections 2024-05-27T07:33:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.188+0000 7f09c5600700 1 -- 172.21.15.12:0/1056680118 wait complete. 2024-05-27T07:33:07.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.190+0000 7f09c5600700 1 Processor -- start 2024-05-27T07:33:07.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.190+0000 7f09c5600700 1 -- start start 2024-05-27T07:33:07.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.190+0000 7f09c5600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c01034b0 0x7f09c010a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09c5600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c0107de0 0x7f09c010a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09c5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0108730 0x7f09c010abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09c5600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0113290 con 0x7f09c0107de0 2024-05-27T07:33:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09c5600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0113110 con 0x7f09c0108730 2024-05-27T07:33:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09c5600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0113410 con 0x7f09c01034b0 2024-05-27T07:33:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c0107de0 0x7f09c010a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09be000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c0107de0 0x7f09c010a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41446/0 (socket says 172.21.15.12:41446) 2024-05-27T07:33:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09be000700 1 -- 172.21.15.12:0/944895428 learned_addr learned my addr 172.21.15.12:0/944895428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.191+0000 7f09bf400700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0108730 0x7f09c010abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.192+0000 7f09bea00700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c01034b0 0x7f09c010a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.192+0000 7f09be000700 1 -- 172.21.15.12:0/944895428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c01034b0 msgr2=0x7f09c010a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.192+0000 7f09be000700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c01034b0 0x7f09c010a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.192+0000 7f09be000700 1 -- 172.21.15.12:0/944895428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0108730 msgr2=0x7f09c010abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.192+0000 7f09be000700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0108730 0x7f09c010abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.192+0000 7f09be000700 1 -- 172.21.15.12:0/944895428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c01a9210 con 0x7f09c0107de0 2024-05-27T07:33:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.192+0000 7f09be000700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c0107de0 0x7f09c010a670 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c8f0 tx=0x7f09ac00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.193+0000 7f09b7400700 1 -- 172.21.15.12:0/944895428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac007940 con 0x7f09c0107de0 2024-05-27T07:33:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.193+0000 7f09b7400700 1 -- 172.21.15.12:0/944895428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09ac00f450 con 0x7f09c0107de0 2024-05-27T07:33:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.193+0000 7f09b7400700 1 -- 172.21.15.12:0/944895428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac018660 con 0x7f09c0107de0 2024-05-27T07:33:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.194+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c01a9500 con 0x7f09c0107de0 2024-05-27T07:33:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.194+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c01a9a90 con 0x7f09c0107de0 2024-05-27T07:33:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.201+0000 7f09b7400700 1 -- 172.21.15.12:0/944895428 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f09ac00f900 con 0x7f09c0107de0 2024-05-27T07:33:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 Processor -- start 2024-05-27T07:33:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 -- start start 2024-05-27T07:33:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e26dec0 0x56089e3f2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3f2db0 0x56089e3ec8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3ece30 0x56089e3ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56089e31aec0 con 0x56089e3f2db0 2024-05-27T07:33:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56089e30aa60 con 0x56089e3ece30 2024-05-27T07:33:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.202+0000 7f09d4101b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56089e30eaa0 con 0x56089e26dec0 2024-05-27T07:33:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.203+0000 7f09bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3f2db0 0x56089e3ec8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.203+0000 7f09bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3f2db0 0x56089e3ec8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41458/0 (socket says 172.21.15.12:41458) 2024-05-27T07:33:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.203+0000 7f09bea00700 1 -- 172.21.15.12:0/3537830851 learned_addr learned my addr 172.21.15.12:0/3537830851 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.203+0000 7f09be000700 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3ece30 0x56089e3ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.203+0000 7f09bf400700 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e26dec0 0x56089e3f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09b7400700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b2a70 0x7f09a40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09bea00700 1 -- 172.21.15.12:0/3537830851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e26dec0 msgr2=0x56089e3f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09bea00700 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e26dec0 0x56089e3f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09bea00700 1 -- 172.21.15.12:0/3537830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3ece30 msgr2=0x56089e3ed240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09bea00700 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3ece30 0x56089e3ed240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09bea00700 1 -- 172.21.15.12:0/3537830851 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56089e2d34a0 con 0x56089e3f2db0 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09b7400700 1 -- 172.21.15.12:0/944895428 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f09ac0e28f0 con 0x7f09c0107de0 2024-05-27T07:33:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09bf400700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b2a70 0x7f09a40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.204+0000 7f09bea00700 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3f2db0 0x56089e3ec8f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009fd0 tx=0x7f09b0009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97ea138cb1d3ee1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:07.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.205+0000 7f09abe00700 1 -- 172.21.15.12:0/3537830851 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0027070 con 0x56089e3f2db0 2024-05-27T07:33:07.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.205+0000 7f09abe00700 1 -- 172.21.15.12:0/3537830851 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09b000bc20 con 0x56089e3f2db0 2024-05-27T07:33:07.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.205+0000 7f09abe00700 1 -- 172.21.15.12:0/3537830851 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b000fd20 con 0x56089e3f2db0 2024-05-27T07:33:07.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.205+0000 7f09bf400700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b2a70 0x7f09a40b4f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x56089e3edbb0 tx=0x7f09b8004be0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.206+0000 7f09d4101b80 1 -- 172.21.15.12:0/3537830851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3f2db0 msgr2=0x56089e3ec8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.206+0000 7f09d4101b80 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3f2db0 0x56089e3ec8f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009fd0 tx=0x7f09b0009530 comp rx=0 tx=0).stop 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.206+0000 7f09d4101b80 1 -- 172.21.15.12:0/3537830851 shutdown_connections 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.206+0000 7f09d4101b80 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3ece30 0x56089e3ed240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.206+0000 7f09d4101b80 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e26dec0 0x56089e3f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.206+0000 7f09d4101b80 1 --2- 172.21.15.12:0/3537830851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3f2db0 0x56089e3ec8f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.206+0000 7f09d4101b80 1 -- 172.21.15.12:0/3537830851 >> 172.21.15.12:0/3537830851 conn(0x56089e2ce040 msgr2=0x56089e3e7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.207+0000 7f09d4101b80 1 -- 172.21.15.12:0/3537830851 shutdown_connections 2024-05-27T07:33:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.207+0000 7f09d4101b80 1 -- 172.21.15.12:0/3537830851 wait complete. 2024-05-27T07:33:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.207+0000 7f09d4101b80 1 -- start start 2024-05-27T07:33:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.207+0000 7f09d4101b80 1 Processor -- start 2024-05-27T07:33:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.208+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e26dec0 0x56089e3f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.208+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3ece30 0x56089e3f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.208+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.208+0000 7f09d4101b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56089e211020 con 0x56089e26dec0 2024-05-27T07:33:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.208+0000 7f09d4101b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56089e2ab7e0 con 0x56089e3f2db0 2024-05-27T07:33:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.208+0000 7f09d4101b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56089e3f9820 con 0x56089e3ece30 2024-05-27T07:33:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.208+0000 7f09d4101b80 10 client.0 _ll_register_callbacks cb 0x56089e2b4cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e26dec0 0x56089e3f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e26dec0 0x56089e3f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41470/0 (socket says 172.21.15.12:41470) 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bea00700 1 -- 172.21.15.12:0/1615221956 learned_addr learned my addr 172.21.15.12:0/1615221956 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bf400700 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09be000700 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3ece30 0x56089e3f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bea00700 1 -- 172.21.15.12:0/1615221956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3ece30 msgr2=0x56089e3f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bea00700 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3ece30 0x56089e3f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bea00700 1 -- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 msgr2=0x56089e3f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.209+0000 7f09bea00700 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.210+0000 7f09bea00700 1 -- 172.21.15.12:0/1615221956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56089e25a950 con 0x56089e26dec0 2024-05-27T07:33:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.210+0000 7f09bea00700 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e26dec0 0x56089e3f65f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009750 tx=0x7f09b000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac3432907d1bebb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:07.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.210+0000 7f09a8c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.210+0000 7f09a8c00700 1 -- 172.21.15.12:0/1615221956 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0032060 con 0x56089e26dec0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.210+0000 7f09a8c00700 1 -- 172.21.15.12:0/1615221956 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f09b000f770 con 0x56089e26dec0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.210+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09b000bc20 con 0x56089e26dec0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09a8c00700 1 -- 172.21.15.12:0/1615221956 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f09b0027070 con 0x56089e26dec0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09d4101b80 10 client.16761 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56089e2db0e0 con 0x56089e26dec0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09a8c00700 1 -- 172.21.15.12:0/1615221956 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09b000bc20 con 0x56089e26dec0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09a8c00700 1 -- 172.21.15.12:0/1615221956 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f09b0022a70 con 0x56089e26dec0 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09d4101b80 10 client.16761 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09d4101b80 10 client.16761 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:07.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.211+0000 7f09d4101b80 20 client.16761 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:07.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.212+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e40ef20 0x56089e4113e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.212+0000 7f09d4101b80 4 client.16761 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.212+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56089e31ce10 con 0x56089e40ef20 2024-05-27T07:33:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.212+0000 7f09be000700 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e40ef20 0x56089e4113e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.213+0000 7f09be000700 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e40ef20 0x56089e4113e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3188be7acb00b6aa server_cookie=78e5db231196055f in_seq=0 out_seq=0 2024-05-27T07:33:07.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.213+0000 7f09a8c00700 10 client.16761 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:07.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.216+0000 7f09a8c00700 1 -- 172.21.15.12:0/1615221956 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56089e31ce10 con 0x56089e40ef20 2024-05-27T07:33:07.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.216+0000 7f09a8c00700 10 client.16761 handle_command_reply: tid=0 2024-05-27T07:33:07.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.217+0000 7f09d4101b80 1 client.16761 shutdown 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.217+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e26dec0 msgr2=0x56089e3f65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.217+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e26dec0 0x56089e3f65f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009750 tx=0x7f09b000be50 comp rx=0 tx=0).stop 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 shutdown_connections 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e40ef20 0x56089e4113e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e26dec0 0x56089e3f65f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1615221956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3ece30 0x56089e3f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 >> 172.21.15.12:0/1615221956 conn(0x56089e2ce040 msgr2=0x56089e3e7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:07.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09a8c00700 0 client.16761 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:07.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 shutdown_connections 2024-05-27T07:33:07.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.218+0000 7f09d4101b80 1 -- 172.21.15.12:0/1615221956 wait complete. 2024-05-27T07:33:07.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.219+0000 7f09d4101b80 20 client.16761 trim_cache size 0 max 16384 2024-05-27T07:33:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.263+0000 7f09d4101b80 1 Processor -- start 2024-05-27T07:33:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.263+0000 7f09d4101b80 1 -- start start 2024-05-27T07:33:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3ec8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e4028c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09d4101b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56089e3f58f0 con 0x56089e3ece30 2024-05-27T07:33:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09d4101b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01ad9f0 con 0x56089e3f2db0 2024-05-27T07:33:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09d4101b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01acc50 con 0x56089e3e7be0 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3ec8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3ec8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41482/0 (socket says 172.21.15.12:41482) 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.264+0000 7f09bea00700 1 -- 172.21.15.12:0/515537536 learned_addr learned my addr 172.21.15.12:0/515537536 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09be000700 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e4028c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bf400700 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bea00700 1 -- 172.21.15.12:0/515537536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 msgr2=0x56089e3f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bea00700 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bea00700 1 -- 172.21.15.12:0/515537536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 msgr2=0x56089e4028c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bea00700 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e4028c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bea00700 1 -- 172.21.15.12:0/515537536 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56089e3f1e40 con 0x56089e3ece30 2024-05-27T07:33:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09be000700 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e4028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bf400700 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.265+0000 7f09bea00700 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3ec8f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f09b00098e0 tx=0x7f09b000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2d84c88e1c0135 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.266+0000 7f09ab400700 1 -- 172.21.15.12:0/515537536 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0032040 con 0x56089e3ece30 2024-05-27T07:33:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.266+0000 7f09ab400700 1 -- 172.21.15.12:0/515537536 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09b002ccd0 con 0x56089e3ece30 2024-05-27T07:33:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.266+0000 7f09ab400700 1 -- 172.21.15.12:0/515537536 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0039030 con 0x56089e3ece30 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 -- 172.21.15.12:0/515537536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 msgr2=0x56089e3ec8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3ec8f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f09b00098e0 tx=0x7f09b000bdc0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 -- 172.21.15.12:0/515537536 shutdown_connections 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3ec8f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 --2- 172.21.15.12:0/515537536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e4028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 -- 172.21.15.12:0/515537536 >> 172.21.15.12:0/515537536 conn(0x56089e2ce040 msgr2=0x56089e3f63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 -- 172.21.15.12:0/515537536 shutdown_connections 2024-05-27T07:33:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.267+0000 7f09d4101b80 1 -- 172.21.15.12:0/515537536 wait complete. 2024-05-27T07:33:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 -- start start 2024-05-27T07:33:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 Processor -- start 2024-05-27T07:33:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3feff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56089e211020 con 0x56089e3ece30 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01ad320 con 0x56089e3f2db0 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.268+0000 7f09d4101b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56089e3fbfc0 con 0x56089e3e7be0 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.269+0000 7f09d4101b80 10 client.0 _ll_register_callbacks cb 0x56089e2b4cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.269+0000 7f09bea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.269+0000 7f09bea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:53346/0 (socket says 172.21.15.12:53346) 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.269+0000 7f09bea00700 1 -- 172.21.15.12:0/1034449108 learned_addr learned my addr 172.21.15.12:0/1034449108 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.269+0000 7f09bf400700 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.270+0000 7f09be000700 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3feff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.270+0000 7f09bea00700 1 -- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 msgr2=0x56089e3fba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.270+0000 7f09bea00700 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3fba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.270+0000 7f09bea00700 1 -- 172.21.15.12:0/1034449108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 msgr2=0x56089e3feff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.270+0000 7f09bea00700 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3feff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.270+0000 7f09bea00700 1 -- 172.21.15.12:0/1034449108 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56089e3fc170 con 0x56089e3e7be0 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.271+0000 7f09bea00700 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3feab0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09b00058e0 tx=0x7f09b002c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=613518f7c52189ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.271+0000 7f09a8c00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:33:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.272+0000 7f09a8c00700 1 -- 172.21.15.12:0/1034449108 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0039070 con 0x56089e3e7be0 2024-05-27T07:33:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.272+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09b002ccd0 con 0x56089e3e7be0 2024-05-27T07:33:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.272+0000 7f09a8c00700 1 -- 172.21.15.12:0/1034449108 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f09b0022420 con 0x56089e3e7be0 2024-05-27T07:33:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.273+0000 7f09a8c00700 1 -- 172.21.15.12:0/1034449108 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f09b004b050 con 0x56089e3e7be0 2024-05-27T07:33:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.273+0000 7f09d4101b80 10 client.26293 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.273+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56089e47a370 con 0x56089e3e7be0 2024-05-27T07:33:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.273+0000 7f09a8c00700 1 -- 172.21.15.12:0/1034449108 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09b002ccd0 con 0x56089e3e7be0 2024-05-27T07:33:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.273+0000 7f09a8c00700 1 -- 172.21.15.12:0/1034449108 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f09b000e3e0 con 0x56089e3e7be0 2024-05-27T07:33:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.274+0000 7f09d4101b80 10 client.26293 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.274+0000 7f09d4101b80 10 client.26293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.274+0000 7f09d4101b80 20 client.26293 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.274+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e47a6d0 0x56089e47caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.274+0000 7f09d4101b80 4 client.26293 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:33:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.274+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56089e3165f0 con 0x56089e47a6d0 2024-05-27T07:33:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.275+0000 7f09be000700 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e47a6d0 0x56089e47caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.276+0000 7f09be000700 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e47a6d0 0x56089e47caf0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6caf996b46839ef server_cookie=1cf7a9479b37262e in_seq=0 out_seq=0 2024-05-27T07:33:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.276+0000 7f09a8c00700 10 client.26293 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:07.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.281+0000 7f09a8c00700 1 -- 172.21.15.12:0/1034449108 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56089e3165f0 con 0x56089e47a6d0 2024-05-27T07:33:07.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.281+0000 7f09a8c00700 10 client.26293 handle_command_reply: tid=0 2024-05-27T07:33:07.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.281+0000 7f09d4101b80 1 client.26293 shutdown 2024-05-27T07:33:07.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 msgr2=0x56089e3feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3feab0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09b00058e0 tx=0x7f09b002c700 comp rx=0 tx=0).stop 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 shutdown_connections 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56089e47a6d0 0x56089e47caf0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56089e3f2db0 0x56089e3fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56089e3e7be0 0x56089e3feab0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 --2- 172.21.15.12:0/1034449108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56089e3ece30 0x56089e3feff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 >> 172.21.15.12:0/1034449108 conn(0x56089e2ce040 msgr2=0x56089e3f63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09a8c00700 0 client.26293 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 shutdown_connections 2024-05-27T07:33:07.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 1 -- 172.21.15.12:0/1034449108 wait complete. 2024-05-27T07:33:07.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.282+0000 7f09d4101b80 20 client.26293 trim_cache size 0 max 16384 2024-05-27T07:33:07.278 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:33:07.278 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:33:07.278 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "8929fe18-c3aa-409f-ac51-09022475f871", 2024-05-27T07:33:07.278 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:33:07.278 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b2a70 msgr2=0x7f09a40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b2a70 0x7f09a40b4f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x56089e3edbb0 tx=0x7f09b8004be0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c0107de0 msgr2=0x7f09c010a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c0107de0 0x7f09c010a670 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c8f0 tx=0x7f09ac00cc00 comp rx=0 tx=0).stop 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 shutdown_connections 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b2a70 0x7f09a40b4f80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0108730 0x7f09c010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c01034b0 0x7f09c010a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 --2- 172.21.15.12:0/944895428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c0107de0 0x7f09c010a670 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 >> 172.21.15.12:0/944895428 conn(0x7f09c00fedb0 msgr2=0x7f09c01069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.284+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 shutdown_connections 2024-05-27T07:33:07.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:07.285+0000 7f09c5600700 1 -- 172.21.15.12:0/944895428 wait complete. 2024-05-27T07:33:08.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:33:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:08.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: pgmap v373: 129 pgs: 129 active+clean; 178 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:33:08.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:33:08.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:33:08.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:33:08.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:33:08.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:08 smithi162 ceph-mon[35676]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: pgmap v373: 129 pgs: 129 active+clean; 178 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:08 smithi062 ceph-mon[35304]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:33:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: pgmap v373: 129 pgs: 129 active+clean; 178 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:33:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:33:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:33:08.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:33:08.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:33:08.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:08.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:08 smithi012 ceph-mon[31473]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:33:09.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:09 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:09.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:09 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:09.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:09 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:09.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:09 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:09.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:09 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:09.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:09 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:09.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:09 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:09.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:09 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:09.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:09 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:10.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: pgmap v375: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.0 MiB/s wr, 263 op/s 2024-05-27T07:33:10.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.726 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:33:10.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:33:10.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:10.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:10.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: pgmap v376: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-05-27T07:33:10.727 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:10 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: pgmap v375: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.0 MiB/s wr, 263 op/s 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: pgmap v376: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-05-27T07:33:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:10 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:10.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: pgmap v375: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.0 MiB/s wr, 263 op/s 2024-05-27T07:33:10.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: pgmap v376: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-05-27T07:33:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:10 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 -- 172.21.15.12:0/2457116735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 msgr2=0x7f4c84109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 --2- 172.21.15.12:0/2457116735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84109320 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4c700099c0 tx=0x7f4c70009cd0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 -- 172.21.15.12:0/2457116735 shutdown_connections 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 --2- 172.21.15.12:0/2457116735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c841045e0 0x7f4c841049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 --2- 172.21.15.12:0/2457116735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c84109860 0x7f4c841111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 --2- 172.21.15.12:0/2457116735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84109320 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 -- 172.21.15.12:0/2457116735 >> 172.21.15.12:0/2457116735 conn(0x7f4c84100030 msgr2=0x7f4c84102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 -- 172.21.15.12:0/2457116735 shutdown_connections 2024-05-27T07:33:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.021+0000 7f4c8be00700 1 -- 172.21.15.12:0/2457116735 wait complete. 2024-05-27T07:33:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 Processor -- start 2024-05-27T07:33:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 -- start start 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c841045e0 0x7f4c84075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84109860 0x7f4c84076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84114370 con 0x7f4c84108f10 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841141f0 con 0x7f4c84109860 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.023+0000 7f4c8be00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841144f0 con 0x7f4c841045e0 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41494/0 (socket says 172.21.15.12:41494) 2024-05-27T07:33:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 -- 172.21.15.12:0/3646807295 learned_addr learned my addr 172.21.15.12:0/3646807295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c89400700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c841045e0 0x7f4c84075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c89e00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84109860 0x7f4c84076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 -- 172.21.15.12:0/3646807295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c841045e0 msgr2=0x7f4c84075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c841045e0 0x7f4c84075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 -- 172.21.15.12:0/3646807295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84109860 msgr2=0x7f4c84076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84109860 0x7f4c84076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.024+0000 7f4c88a00700 1 -- 172.21.15.12:0/3646807295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c841a9120 con 0x7f4c84108f10 2024-05-27T07:33:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.025+0000 7f4c88a00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84075ff0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4c700098e0 tx=0x7f4c7000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.025+0000 7f4c82000700 1 -- 172.21.15.12:0/3646807295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70027070 con 0x7f4c84108f10 2024-05-27T07:33:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.025+0000 7f4c82000700 1 -- 172.21.15.12:0/3646807295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c7000fdd0 con 0x7f4c84108f10 2024-05-27T07:33:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.025+0000 7f4c82000700 1 -- 172.21.15.12:0/3646807295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70022e30 con 0x7f4c84108f10 2024-05-27T07:33:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.025+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c841a93b0 con 0x7f4c84108f10 2024-05-27T07:33:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.025+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c841a97b0 con 0x7f4c84108f10 2024-05-27T07:33:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.030+0000 7f4c82000700 1 -- 172.21.15.12:0/3646807295 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4c7002a070 con 0x7f4c84108f10 2024-05-27T07:33:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.032+0000 7f4c9a94db80 1 Processor -- start 2024-05-27T07:33:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.032+0000 7f4c9a94db80 1 -- start start 2024-05-27T07:33:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd83330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd83870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dd83e70 0x55fa2dd90110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dcb1e40 con 0x55fa2dcf09f0 2024-05-27T07:33:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dca19e0 con 0x55fa2dd83e70 2024-05-27T07:33:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dca5a20 con 0x55fa2dc04e40 2024-05-27T07:33:11.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c82000700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4c6c0b2a70 0x7f4c6c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c82000700 1 -- 172.21.15.12:0/3646807295 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f4c700e3720 con 0x7f4c84108f10 2024-05-27T07:33:11.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c89e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd83330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c89e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd83330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41498/0 (socket says 172.21.15.12:41498) 2024-05-27T07:33:11.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c89e00700 1 -- 172.21.15.12:0/2482541915 learned_addr learned my addr 172.21.15.12:0/2482541915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c89400700 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd83870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.033+0000 7f4c89e00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4c6c0b2a70 0x7f4c6c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.034+0000 7f4c88a00700 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dd83e70 0x55fa2dd90110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.034+0000 7f4c89400700 1 -- 172.21.15.12:0/2482541915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dd83e70 msgr2=0x55fa2dd90110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.034+0000 7f4c89400700 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dd83e70 0x55fa2dd90110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.034+0000 7f4c89400700 1 -- 172.21.15.12:0/2482541915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dcf09f0 msgr2=0x55fa2dd83330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.034+0000 7f4c89400700 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd83330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.034+0000 7f4c89400700 1 -- 172.21.15.12:0/2482541915 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2dc6a420 con 0x55fa2dc04e40 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.035+0000 7f4c89e00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4c6c0b2a70 0x7f4c6c0b4f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0060b0 tx=0x7f4c7c0098c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.035+0000 7f4c89400700 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd83870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800bc70 tx=0x7f4c7800bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f88855bd9b4f22e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.035+0000 7f4c76a00700 1 -- 172.21.15.12:0/2482541915 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78015070 con 0x55fa2dc04e40 2024-05-27T07:33:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.035+0000 7f4c76a00700 1 -- 172.21.15.12:0/2482541915 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c7801a460 con 0x55fa2dc04e40 2024-05-27T07:33:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.035+0000 7f4c76a00700 1 -- 172.21.15.12:0/2482541915 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c780116a0 con 0x55fa2dc04e40 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 -- 172.21.15.12:0/2482541915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 msgr2=0x55fa2dd83870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd83870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800bc70 tx=0x7f4c7800bf80 comp rx=0 tx=0).stop 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 -- 172.21.15.12:0/2482541915 shutdown_connections 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dd83e70 0x55fa2dd90110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd83330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/2482541915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd83870 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 -- 172.21.15.12:0/2482541915 >> 172.21.15.12:0/2482541915 conn(0x55fa2dc9e390 msgr2=0x55fa2dd7f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.036+0000 7f4c9a94db80 1 -- 172.21.15.12:0/2482541915 shutdown_connections 2024-05-27T07:33:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.037+0000 7f4c9a94db80 1 -- 172.21.15.12:0/2482541915 wait complete. 2024-05-27T07:33:11.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.037+0000 7f4c9a94db80 1 -- start start 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.037+0000 7f4c9a94db80 1 Processor -- start 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd8d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd8db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd8e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dba7fa0 con 0x55fa2dc04e40 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dc42760 con 0x55fa2dcf09f0 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dd90b80 con 0x55fa2dd83e70 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c9a94db80 10 client.0 _ll_register_callbacks cb 0x55fa2dc4bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.038+0000 7f4c89400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd8d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd8d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41502/0 (socket says 172.21.15.12:41502) 2024-05-27T07:33:11.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89400700 1 -- 172.21.15.12:0/455631521 learned_addr learned my addr 172.21.15.12:0/455631521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:11.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c88a00700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd8db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89e00700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd8e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89400700 1 -- 172.21.15.12:0/455631521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 msgr2=0x55fa2dd8e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89400700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd8e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89400700 1 -- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 msgr2=0x55fa2dd8db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89400700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd8db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89400700 1 -- 172.21.15.12:0/455631521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2dbf18d0 con 0x55fa2dc04e40 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.039+0000 7f4c89e00700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd8e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c89400700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd8d650 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4c7801a870 tx=0x7f4c7801a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5caa75e203c326c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c6b400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c6b400700 1 -- 172.21.15.12:0/455631521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78028060 con 0x55fa2dc04e40 2024-05-27T07:33:11.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c6b400700 1 -- 172.21.15.12:0/455631521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f4c7800fc40 con 0x55fa2dc04e40 2024-05-27T07:33:11.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c7801a460 con 0x55fa2dc04e40 2024-05-27T07:33:11.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c6b400700 1 -- 172.21.15.12:0/455631521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4c7802f050 con 0x55fa2dc04e40 2024-05-27T07:33:11.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c9a94db80 10 client.16779 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:11.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.040+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55fa2dc72060 con 0x55fa2dc04e40 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.041+0000 7f4c6b400700 1 -- 172.21.15.12:0/455631521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c7801a460 con 0x55fa2dc04e40 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.041+0000 7f4c6b400700 1 -- 172.21.15.12:0/455631521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f4c780048e0 con 0x55fa2dc04e40 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.042+0000 7f4c9a94db80 10 client.16779 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.042+0000 7f4c9a94db80 10 client.16779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.042+0000 7f4c9a94db80 20 client.16779 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.042+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2dd9dd10 0x55fa2dda0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.042+0000 7f4c9a94db80 4 client.16779 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.042+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa2dcb3d90 con 0x55fa2dd9dd10 2024-05-27T07:33:11.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.042+0000 7f4c88a00700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2dd9dd10 0x55fa2dda0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.044+0000 7f4c88a00700 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2dd9dd10 0x55fa2dda0200 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82810b2142016bf2 server_cookie=fc6204951cb1a8bc in_seq=0 out_seq=0 2024-05-27T07:33:11.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.044+0000 7f4c6b400700 10 client.16779 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:11.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c6b400700 1 -- 172.21.15.12:0/455631521 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fa2dcb3d90 con 0x55fa2dd9dd10 2024-05-27T07:33:11.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c6b400700 10 client.16779 handle_command_reply: tid=0 2024-05-27T07:33:11.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 client.16779 shutdown 2024-05-27T07:33:11.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 msgr2=0x55fa2dd8d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd8d650 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4c7801a870 tx=0x7f4c7801a950 comp rx=0 tx=0).stop 2024-05-27T07:33:11.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 shutdown_connections 2024-05-27T07:33:11.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd8e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd8d650 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2dd9dd10 0x55fa2dda0200 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/455631521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd8db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 >> 172.21.15.12:0/455631521 conn(0x55fa2dc9e390 msgr2=0x55fa2dd83310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.049+0000 7f4c6b400700 0 client.16779 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.050+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 shutdown_connections 2024-05-27T07:33:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.050+0000 7f4c9a94db80 1 -- 172.21.15.12:0/455631521 wait complete. 2024-05-27T07:33:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.050+0000 7f4c9a94db80 20 client.16779 trim_cache size 0 max 16384 2024-05-27T07:33:11.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.074+0000 7f4c9a94db80 1 Processor -- start 2024-05-27T07:33:11.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.074+0000 7f4c9a94db80 1 -- start start 2024-05-27T07:33:11.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd7f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd85900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd99c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dd8c7d0 con 0x55fa2dc04e40 2024-05-27T07:33:11.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dd7eba0 con 0x55fa2dcf09f0 2024-05-27T07:33:11.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841b1d90 con 0x55fa2dd83e70 2024-05-27T07:33:11.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c89400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd7f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c89400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd7f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41510/0 (socket says 172.21.15.12:41510) 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.075+0000 7f4c89400700 1 -- 172.21.15.12:0/3112544411 learned_addr learned my addr 172.21.15.12:0/3112544411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c88a00700 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd85900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89e00700 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd99c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89400700 1 -- 172.21.15.12:0/3112544411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 msgr2=0x55fa2dd99c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89400700 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd99c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89400700 1 -- 172.21.15.12:0/3112544411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 msgr2=0x55fa2dd85900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89400700 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd85900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89400700 1 -- 172.21.15.12:0/3112544411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2dd84a50 con 0x55fa2dc04e40 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c88a00700 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd85900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:11.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89e00700 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd99c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:11.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.076+0000 7f4c89400700 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd7f620 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4c7801a840 tx=0x7f4c78010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62b572dd6268559 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:11.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.077+0000 7f4c76000700 1 -- 172.21.15.12:0/3112544411 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78028070 con 0x55fa2dc04e40 2024-05-27T07:33:11.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.077+0000 7f4c76000700 1 -- 172.21.15.12:0/3112544411 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c7801a460 con 0x55fa2dc04e40 2024-05-27T07:33:11.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.077+0000 7f4c76000700 1 -- 172.21.15.12:0/3112544411 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78015070 con 0x55fa2dc04e40 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 -- 172.21.15.12:0/3112544411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 msgr2=0x55fa2dd7f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd7f620 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4c7801a840 tx=0x7f4c78010fd0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 -- 172.21.15.12:0/3112544411 shutdown_connections 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dd83e70 0x55fa2dd99c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dc04e40 0x55fa2dd7f620 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/3112544411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd85900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 -- 172.21.15.12:0/3112544411 >> 172.21.15.12:0/3112544411 conn(0x55fa2dc9e390 msgr2=0x55fa2dd8d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 -- 172.21.15.12:0/3112544411 shutdown_connections 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.078+0000 7f4c9a94db80 1 -- 172.21.15.12:0/3112544411 wait complete. 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 -- start start 2024-05-27T07:33:11.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 Processor -- start 2024-05-27T07:33:11.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd88120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd88660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dd83e70 0x55fa2dd88ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841b16c0 con 0x55fa2dd83e70 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841b0ff0 con 0x55fa2dcf09f0 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.079+0000 7f4c9a94db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fa2dda07e0 con 0x55fa2dc04e40 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c9a94db80 10 client.0 _ll_register_callbacks cb 0x55fa2dc4bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dd83e70 0x55fa2dd88ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dd83e70 0x55fa2dd88ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41518/0 (socket says 172.21.15.12:41518) 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 -- 172.21.15.12:0/18885580 learned_addr learned my addr 172.21.15.12:0/18885580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:11.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89400700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd88120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c88a00700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd88660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 -- 172.21.15.12:0/18885580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 msgr2=0x55fa2dd88120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd88120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 -- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 msgr2=0x55fa2dd88660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd88660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c89e00700 1 -- 172.21.15.12:0/18885580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2dd94290 con 0x55fa2dd83e70 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.080+0000 7f4c88a00700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd88660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.081+0000 7f4c89e00700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dd83e70 0x55fa2dd88ba0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x55fa2dd85ff0 tx=0x7f4c7c009320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9d15b56785ef08 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.081+0000 7f4c6b400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.081+0000 7f4c6b400700 1 -- 172.21.15.12:0/18885580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c028070 con 0x55fa2dd83e70 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.081+0000 7f4c6b400700 1 -- 172.21.15.12:0/18885580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f4c7c012040 con 0x55fa2dd83e70 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.081+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c7801a460 con 0x55fa2dd83e70 2024-05-27T07:33:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c6b400700 1 -- 172.21.15.12:0/18885580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4c7c031050 con 0x55fa2dd83e70 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 10 client.16785 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55fa2de111a0 con 0x55fa2dd83e70 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c6b400700 1 -- 172.21.15.12:0/18885580 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4c7801a460 con 0x55fa2dd83e70 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c6b400700 1 -- 172.21.15.12:0/18885580 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f4c7c02b020 con 0x55fa2dd83e70 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 10 client.16785 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 20 client.16785 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2de11900 0x55fa2de13d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 4 client.16785 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:33:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.082+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fa2dcad570 con 0x55fa2de11900 2024-05-27T07:33:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.083+0000 7f4c89400700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2de11900 0x55fa2de13d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.085+0000 7f4c89400700 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2de11900 0x55fa2de13d20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=652bef86ba46c066 server_cookie=4f323d811a00221f in_seq=0 out_seq=0 2024-05-27T07:33:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.085+0000 7f4c6b400700 10 client.16785 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.085+0000 7f4c6b400700 1 -- 172.21.15.12:0/18885580 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fa2dcad570 con 0x55fa2de11900 2024-05-27T07:33:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.085+0000 7f4c6b400700 10 client.16785 handle_command_reply: tid=0 2024-05-27T07:33:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.085+0000 7f4c9a94db80 1 client.16785 shutdown 2024-05-27T07:33:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dd83e70 msgr2=0x55fa2dd88ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dd83e70 0x55fa2dd88ba0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x55fa2dd85ff0 tx=0x7f4c7c009320 comp rx=0 tx=0).stop 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 shutdown_connections 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fa2dd83e70 0x55fa2dd88ba0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa2dc04e40 0x55fa2dd88120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55fa2de11900 0x55fa2de13d20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 --2- 172.21.15.12:0/18885580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa2dcf09f0 0x55fa2dd88660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 >> 172.21.15.12:0/18885580 conn(0x55fa2dc9e390 msgr2=0x55fa2dd85660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c6b400700 0 client.16785 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 shutdown_connections 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 1 -- 172.21.15.12:0/18885580 wait complete. 2024-05-27T07:33:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.086+0000 7f4c9a94db80 20 client.16785 trim_cache size 0 max 16384 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: "8929fe18-c3aa-409f-ac51-09022475f871": { 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "8929fe18-c3aa-409f-ac51-09022475f871", 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:33:11.082 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:33:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4c6c0b2a70 msgr2=0x7f4c6c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4c6c0b2a70 0x7f4c6c0b4f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0060b0 tx=0x7f4c7c0098c0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 msgr2=0x7f4c84075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84075ff0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4c700098e0 tx=0x7f4c7000f510 comp rx=0 tx=0).stop 2024-05-27T07:33:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 shutdown_connections 2024-05-27T07:33:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f4c6c0b2a70 0x7f4c6c0b4f80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84109860 0x7f4c84076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c841045e0 0x7f4c84075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 --2- 172.21.15.12:0/3646807295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c84108f10 0x7f4c84075ff0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.087+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 >> 172.21.15.12:0/3646807295 conn(0x7f4c84100030 msgr2=0x7f4c8410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.088+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 shutdown_connections 2024-05-27T07:33:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:11.088+0000 7f4c8be00700 1 -- 172.21.15.12:0/3646807295 wait complete. 2024-05-27T07:33:11.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:11 smithi012 ceph-mon[31473]: Health check failed: 9 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:33:11.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:11 smithi162 ceph-mon[35676]: Health check failed: 9 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:33:11.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:11 smithi062 ceph-mon[35304]: Health check failed: 9 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:33:12.038 INFO:tasks.cephfs.filesystem:scrub status for tag:8929fe18-c3aa-409f-ac51-09022475f871 - {'path': '/', 'tag': '8929fe18-c3aa-409f-ac51-09022475f871', 'options': 'recursive,force'} 2024-05-27T07:33:12.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:12 smithi012 ceph-mon[31473]: pgmap v377: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-05-27T07:33:12.878 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:12 smithi162 ceph-mon[35676]: pgmap v377: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-05-27T07:33:12.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:12 smithi062 ceph-mon[35304]: pgmap v377: 129 pgs: 129 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-05-27T07:33:15.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:15 smithi012 ceph-mon[31473]: pgmap v378: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-05-27T07:33:15.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:15 smithi162 ceph-mon[35676]: pgmap v378: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-05-27T07:33:15.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:15 smithi062 ceph-mon[35304]: pgmap v378: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-05-27T07:33:17.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:17 smithi012 ceph-mon[31473]: pgmap v379: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-05-27T07:33:17.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:17 smithi162 ceph-mon[35676]: pgmap v379: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-05-27T07:33:17.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:17 smithi062 ceph-mon[35304]: pgmap v379: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-05-27T07:33:19.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:19 smithi012 ceph-mon[31473]: pgmap v380: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 163 op/s 2024-05-27T07:33:19.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:19 smithi162 ceph-mon[35676]: pgmap v380: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 163 op/s 2024-05-27T07:33:19.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:19 smithi062 ceph-mon[35304]: pgmap v380: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 163 op/s 2024-05-27T07:33:21.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:21 smithi012 ceph-mon[31473]: pgmap v381: 129 pgs: 129 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-05-27T07:33:21.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:33:21.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:33:21.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:21 smithi162 ceph-mon[35676]: pgmap v381: 129 pgs: 129 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-05-27T07:33:21.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:33:21.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:33:21.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:21 smithi062 ceph-mon[35304]: pgmap v381: 129 pgs: 129 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-05-27T07:33:21.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:33:21.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:33:22.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:22 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:33:22.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:22 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:33:22.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:22 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:33:23.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:23 smithi162 ceph-mon[35676]: pgmap v382: 129 pgs: 129 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 125 op/s 2024-05-27T07:33:23.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:23 smithi062 ceph-mon[35304]: pgmap v382: 129 pgs: 129 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 125 op/s 2024-05-27T07:33:23.843 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:23 smithi012 ceph-mon[31473]: pgmap v382: 129 pgs: 129 active+clean; 179 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 125 op/s 2024-05-27T07:33:25.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:25 smithi162 ceph-mon[35676]: pgmap v383: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-05-27T07:33:25.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:25 smithi062 ceph-mon[35304]: pgmap v383: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-05-27T07:33:25.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:25 smithi012 ceph-mon[31473]: pgmap v383: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-05-27T07:33:27.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:27 smithi162 ceph-mon[35676]: pgmap v384: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:33:27.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:27 smithi062 ceph-mon[35304]: pgmap v384: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:33:27.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:27 smithi012 ceph-mon[31473]: pgmap v384: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:33:29.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:29 smithi162 ceph-mon[35676]: pgmap v385: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:33:29.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:29 smithi062 ceph-mon[35304]: pgmap v385: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:33:29.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:29 smithi012 ceph-mon[31473]: pgmap v385: 129 pgs: 129 active+clean; 185 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:33:31.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:31 smithi162 ceph-mon[35676]: pgmap v386: 129 pgs: 129 active+clean; 180 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-27T07:33:31.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:31 smithi062 ceph-mon[35304]: pgmap v386: 129 pgs: 129 active+clean; 180 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-27T07:33:31.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:31 smithi012 ceph-mon[31473]: pgmap v386: 129 pgs: 129 active+clean; 180 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-27T07:33:33.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:33 smithi162 ceph-mon[35676]: pgmap v387: 129 pgs: 129 active+clean; 180 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1000 KiB/s wr, 116 op/s 2024-05-27T07:33:33.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:33 smithi062 ceph-mon[35304]: pgmap v387: 129 pgs: 129 active+clean; 180 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1000 KiB/s wr, 116 op/s 2024-05-27T07:33:33.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:33 smithi012 ceph-mon[31473]: pgmap v387: 129 pgs: 129 active+clean; 180 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1000 KiB/s wr, 116 op/s 2024-05-27T07:33:35.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:35 smithi062 ceph-mon[35304]: pgmap v388: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-05-27T07:33:35.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:35 smithi012 ceph-mon[31473]: pgmap v388: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-05-27T07:33:35.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:35 smithi162 ceph-mon[35676]: pgmap v388: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-05-27T07:33:36.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:36 smithi012 ceph-mon[31473]: pgmap v389: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 KiB/s wr, 112 op/s 2024-05-27T07:33:36.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:36 smithi162 ceph-mon[35676]: pgmap v389: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 KiB/s wr, 112 op/s 2024-05-27T07:33:36.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:36 smithi062 ceph-mon[35304]: pgmap v389: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 KiB/s wr, 112 op/s 2024-05-27T07:33:39.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:39 smithi012 ceph-mon[31473]: pgmap v390: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 KiB/s wr, 112 op/s 2024-05-27T07:33:39.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:39 smithi162 ceph-mon[35676]: pgmap v390: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 KiB/s wr, 112 op/s 2024-05-27T07:33:39.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:39 smithi062 ceph-mon[35304]: pgmap v390: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 907 KiB/s wr, 112 op/s 2024-05-27T07:33:41.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:41 smithi012 ceph-mon[31473]: pgmap v391: 129 pgs: 129 active+clean; 184 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-05-27T07:33:41.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:41 smithi162 ceph-mon[35676]: pgmap v391: 129 pgs: 129 active+clean; 184 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-05-27T07:33:41.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:41 smithi062 ceph-mon[35304]: pgmap v391: 129 pgs: 129 active+clean; 184 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-05-27T07:33:42.038 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:33:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:43.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:43 smithi012 ceph-mon[31473]: pgmap v392: 129 pgs: 129 active+clean; 184 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 979 KiB/s wr, 111 op/s 2024-05-27T07:33:43.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:43 smithi162 ceph-mon[35676]: pgmap v392: 129 pgs: 129 active+clean; 184 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 979 KiB/s wr, 111 op/s 2024-05-27T07:33:43.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:43 smithi062 ceph-mon[35304]: pgmap v392: 129 pgs: 129 active+clean; 184 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 979 KiB/s wr, 111 op/s 2024-05-27T07:33:45.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.205+0000 7f5fe4400700 1 -- 172.21.15.12:0/1207506053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc10c5a0 msgr2=0x7f5fdc10c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.205+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1207506053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc10c5a0 0x7f5fdc10c9b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000b0a0 tx=0x7f5fd000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.206+0000 7f5fe4400700 1 -- 172.21.15.12:0/1207506053 shutdown_connections 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.206+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1207506053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fdc106eb0 0x7f5fdc113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.206+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1207506053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc10c5a0 0x7f5fdc10c9b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.206+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1207506053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fdc106560 0x7f5fdc106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.206+0000 7f5fe4400700 1 -- 172.21.15.12:0/1207506053 >> 172.21.15.12:0/1207506053 conn(0x7f5fdc102040 msgr2=0x7f5fdc104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.206+0000 7f5fe4400700 1 -- 172.21.15.12:0/1207506053 shutdown_connections 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.206+0000 7f5fe4400700 1 -- 172.21.15.12:0/1207506053 wait complete. 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.207+0000 7f5fe4400700 1 Processor -- start 2024-05-27T07:33:45.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.208+0000 7f5fe4400700 1 -- start start 2024-05-27T07:33:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.208+0000 7f5fe4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fdc106560 0x7f5fdc19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.208+0000 7f5fe4400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc106eb0 0x7f5fdc19d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.208+0000 7f5fe4400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fdc10c5a0 0x7f5fdc1a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.208+0000 7f5fe4400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1165b0 con 0x7f5fdc106eb0 2024-05-27T07:33:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.208+0000 7f5fe4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc116430 con 0x7f5fdc106560 2024-05-27T07:33:45.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.208+0000 7f5fe4400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc116730 con 0x7f5fdc10c5a0 2024-05-27T07:33:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.209+0000 7f5fe1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fdc106560 0x7f5fdc19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.209+0000 7f5fe1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc106eb0 0x7f5fdc19d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.209+0000 7f5fe2400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fdc10c5a0 0x7f5fdc1a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.209+0000 7f5fe1000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc106eb0 0x7f5fdc19d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45288/0 (socket says 172.21.15.12:45288) 2024-05-27T07:33:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.209+0000 7f5fe1000700 1 -- 172.21.15.12:0/1446254322 learned_addr learned my addr 172.21.15.12:0/1446254322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:45.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.210+0000 7f5fe1000700 1 -- 172.21.15.12:0/1446254322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fdc10c5a0 msgr2=0x7f5fdc1a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.210+0000 7f5fe1000700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fdc10c5a0 0x7f5fdc1a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.210+0000 7f5fe1000700 1 -- 172.21.15.12:0/1446254322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fdc106560 msgr2=0x7f5fdc19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.210+0000 7f5fe1000700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fdc106560 0x7f5fdc19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.210+0000 7f5fe1000700 1 -- 172.21.15.12:0/1446254322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fdc1a1e30 con 0x7f5fdc106eb0 2024-05-27T07:33:45.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.210+0000 7f5fe1000700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc106eb0 0x7f5fdc19d500 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800d950 tx=0x7f5fc800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:45.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.211+0000 7f5fda000700 1 -- 172.21.15.12:0/1446254322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8016070 con 0x7f5fdc106eb0 2024-05-27T07:33:45.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.211+0000 7f5fda000700 1 -- 172.21.15.12:0/1446254322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fc8004830 con 0x7f5fdc106eb0 2024-05-27T07:33:45.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.211+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fdc1a20c0 con 0x7f5fdc106eb0 2024-05-27T07:33:45.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.211+0000 7f5fda000700 1 -- 172.21.15.12:0/1446254322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8005520 con 0x7f5fdc106eb0 2024-05-27T07:33:45.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.211+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fdc1a9a40 con 0x7f5fdc106eb0 2024-05-27T07:33:45.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.216+0000 7f5fda000700 1 -- 172.21.15.12:0/1446254322 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5fc800f690 con 0x7f5fdc106eb0 2024-05-27T07:33:45.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.217+0000 7f5ff2fb7b80 1 Processor -- start 2024-05-27T07:33:45.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5ff2fb7b80 1 -- start start 2024-05-27T07:33:45.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55625512ce40 0x5562552b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562552b1d30 0x5562552ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5fda000700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fc40b2a70 0x7f5fc40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5562551d9e40 con 0x5562552abdb0 2024-05-27T07:33:45.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5562551c99e0 con 0x55625512ce40 2024-05-27T07:33:45.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.218+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5562551cda20 con 0x5562552b1d30 2024-05-27T07:33:45.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.219+0000 7f5fda000700 1 -- 172.21.15.12:0/1446254322 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f5fc80e2800 con 0x7f5fdc106eb0 2024-05-27T07:33:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.219+0000 7f5fe2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.219+0000 7f5fe2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45292/0 (socket says 172.21.15.12:45292) 2024-05-27T07:33:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.219+0000 7f5fe2400700 1 -- 172.21.15.12:0/1918808758 learned_addr learned my addr 172.21.15.12:0/1918808758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.219+0000 7f5fe1000700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fc40b2a70 0x7f5fc40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.219+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562552b1d30 0x5562552ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.219+0000 7f5fe2400700 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55625512ce40 0x5562552b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fe2400700 1 -- 172.21.15.12:0/1918808758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562552b1d30 msgr2=0x5562552ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fe2400700 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562552b1d30 0x5562552ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fe2400700 1 -- 172.21.15.12:0/1918808758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55625512ce40 msgr2=0x5562552b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fe2400700 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55625512ce40 0x5562552b17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fe2400700 1 -- 172.21.15.12:0/1918808758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556255192420 con 0x5562552abdb0 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fe2400700 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552ac1c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40046e0 tx=0x7f5fd400b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7713898d6b4251e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fe1000700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fc40b2a70 0x7f5fc40b4f80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80a56f0 tx=0x7f5fc80047c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fcea00700 1 -- 172.21.15.12:0/1918808758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4012d60 con 0x5562552abdb0 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fcea00700 1 -- 172.21.15.12:0/1918808758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fd4004cc0 con 0x5562552abdb0 2024-05-27T07:33:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.220+0000 7f5fcea00700 1 -- 172.21.15.12:0/1918808758 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400f3f0 con 0x5562552abdb0 2024-05-27T07:33:45.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.221+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/1918808758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 msgr2=0x5562552ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.221+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552ac1c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40046e0 tx=0x7f5fd400b740 comp rx=0 tx=0).stop 2024-05-27T07:33:45.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/1918808758 shutdown_connections 2024-05-27T07:33:45.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55625512ce40 0x5562552b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552ac1c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/1918808758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5562552b1d30 0x5562552ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/1918808758 >> 172.21.15.12:0/1918808758 conn(0x55625518cfc0 msgr2=0x5562552a68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:45.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/1918808758 shutdown_connections 2024-05-27T07:33:45.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/1918808758 wait complete. 2024-05-27T07:33:45.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.222+0000 7f5ff2fb7b80 1 -- start start 2024-05-27T07:33:45.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 1 Processor -- start 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5562550cffa0 con 0x5562552b1d30 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55625516a760 con 0x5562552abdb0 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5562552b8800 con 0x55625512ce40 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.223+0000 7f5ff2fb7b80 10 client.0 _ll_register_callbacks cb 0x556255173c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45304/0 (socket says 172.21.15.12:45304) 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 -- 172.21.15.12:0/3285659197 learned_addr learned my addr 172.21.15.12:0/3285659197 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:45.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 -- 172.21.15.12:0/3285659197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 msgr2=0x5562552b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 -- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 msgr2=0x5562552b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.224+0000 7f5fe2400700 1 -- 172.21.15.12:0/3285659197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562551198d0 con 0x5562552b1d30 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.225+0000 7f5fe2400700 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552b5f30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40046b0 tx=0x7f5fd4012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bf0b7b9a1c679 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.225+0000 7f5fc3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.225+0000 7f5fc3400700 1 -- 172.21.15.12:0/3285659197 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4005000 con 0x5562552b1d30 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.225+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fd4004cc0 con 0x5562552b1d30 2024-05-27T07:33:45.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.225+0000 7f5fc3400700 1 -- 172.21.15.12:0/3285659197 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f5fd4004750 con 0x5562552b1d30 2024-05-27T07:33:45.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.226+0000 7f5fc3400700 1 -- 172.21.15.12:0/3285659197 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5fd4004950 con 0x5562552b1d30 2024-05-27T07:33:45.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.226+0000 7f5ff2fb7b80 10 client.16797 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:45.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.226+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55625519a060 con 0x5562552b1d30 2024-05-27T07:33:45.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.226+0000 7f5fc3400700 1 -- 172.21.15.12:0/3285659197 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fd4004cc0 con 0x5562552b1d30 2024-05-27T07:33:45.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.226+0000 7f5fc3400700 1 -- 172.21.15.12:0/3285659197 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5fd401fc60 con 0x5562552b1d30 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.226+0000 7f5ff2fb7b80 10 client.16797 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.227+0000 7f5ff2fb7b80 10 client.16797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.227+0000 7f5ff2fb7b80 20 client.16797 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.227+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562552c58f0 0x5562552c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.227+0000 7f5ff2fb7b80 4 client.16797 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.227+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562551dbd90 con 0x5562552c58f0 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.227+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562552c58f0 0x5562552c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.229+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562552c58f0 0x5562552c7da0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad2766b0b6f2bf52 server_cookie=77667aeff48a1c18 in_seq=0 out_seq=0 2024-05-27T07:33:45.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.229+0000 7f5fc3400700 10 client.16797 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:45.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5fc3400700 1 -- 172.21.15.12:0/3285659197 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5562551dbd90 con 0x5562552c58f0 2024-05-27T07:33:45.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5fc3400700 10 client.16797 handle_command_reply: tid=0 2024-05-27T07:33:45.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5ff2fb7b80 1 client.16797 shutdown 2024-05-27T07:33:45.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 msgr2=0x5562552b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552b5f30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40046b0 tx=0x7f5fd4012f90 comp rx=0 tx=0).stop 2024-05-27T07:33:45.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 shutdown_connections 2024-05-27T07:33:45.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552b5f30 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.231+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562552c58f0 0x5562552c7da0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.232+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3285659197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.232+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 >> 172.21.15.12:0/3285659197 conn(0x55625518cfc0 msgr2=0x5562552a6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:45.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.232+0000 7f5fc3400700 0 client.16797 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:45.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.232+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 shutdown_connections 2024-05-27T07:33:45.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.232+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3285659197 wait complete. 2024-05-27T07:33:45.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.232+0000 7f5ff2fb7b80 20 client.16797 trim_cache size 0 max 16384 2024-05-27T07:33:45.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.262+0000 7f5ff2fb7b80 1 Processor -- start 2024-05-27T07:33:45.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.262+0000 7f5ff2fb7b80 1 -- start start 2024-05-27T07:33:45.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552afc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552b1d30 0x5562552b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5562552b0d90 con 0x5562552abdb0 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1b1e60 con 0x5562552b1d30 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1b10c0 con 0x55625512ce40 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5fe1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5fe1a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552afc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45314/0 (socket says 172.21.15.12:45314) 2024-05-27T07:33:45.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5fe1a00700 1 -- 172.21.15.12:0/3947460041 learned_addr learned my addr 172.21.15.12:0/3947460041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:45.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5fe2400700 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552b1d30 0x5562552b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.263+0000 7f5fe2400700 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fe1a00700 1 -- 172.21.15.12:0/3947460041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 msgr2=0x5562552ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fe1a00700 1 -- 172.21.15.12:0/3947460041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552b1d30 msgr2=0x5562552b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552b1d30 0x5562552b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fe1a00700 1 -- 172.21.15.12:0/3947460041 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562552b3a20 con 0x5562552abdb0 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552afc40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000d010 tx=0x7f5fd0006210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6062d6477039f0a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fce000700 1 -- 172.21.15.12:0/3947460041 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd002b070 con 0x5562552abdb0 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.264+0000 7f5fce000700 1 -- 172.21.15.12:0/3947460041 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fd00075c0 con 0x5562552abdb0 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.265+0000 7f5fce000700 1 -- 172.21.15.12:0/3947460041 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0026500 con 0x5562552abdb0 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3947460041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 msgr2=0x5562552afc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552afc40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000d010 tx=0x7f5fd0006210 comp rx=0 tx=0).stop 2024-05-27T07:33:45.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3947460041 shutdown_connections 2024-05-27T07:33:45.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552b1d30 0x5562552b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/3947460041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552abdb0 0x5562552afc40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3947460041 >> 172.21.15.12:0/3947460041 conn(0x55625518cfc0 msgr2=0x5562552b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:45.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3947460041 shutdown_connections 2024-05-27T07:33:45.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.266+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/3947460041 wait complete. 2024-05-27T07:33:45.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.267+0000 7f5ff2fb7b80 1 -- start start 2024-05-27T07:33:45.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.267+0000 7f5ff2fb7b80 1 Processor -- start 2024-05-27T07:33:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.268+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552bdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.268+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552be000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.268+0000 7f5ff2fb7b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552baa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.268+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5562552b5290 con 0x5562552b1d30 2024-05-27T07:33:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.268+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc1b1790 con 0x5562552abdb0 2024-05-27T07:33:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.268+0000 7f5ff2fb7b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5562552a8150 con 0x55625512ce40 2024-05-27T07:33:45.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.268+0000 7f5ff2fb7b80 10 client.0 _ll_register_callbacks cb 0x556255173c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552baa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552baa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45328/0 (socket says 172.21.15.12:45328) 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 -- 172.21.15.12:0/2237976287 learned_addr learned my addr 172.21.15.12:0/2237976287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552bdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 -- 172.21.15.12:0/2237976287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 msgr2=0x5562552bdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552bdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 -- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 msgr2=0x5562552be000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:45.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552be000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.269+0000 7f5fe2400700 1 -- 172.21.15.12:0/2237976287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562552bafd0 con 0x5562552b1d30 2024-05-27T07:33:45.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.270+0000 7f5fe2400700 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552baa90 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4004f90 tx=0x7f5fd4012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14c5835b1389933 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:45.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.270+0000 7f5fc3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:45.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.270+0000 7f5fc3400700 1 -- 172.21.15.12:0/2237976287 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4005000 con 0x5562552b1d30 2024-05-27T07:33:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.270+0000 7f5fc3400700 1 -- 172.21.15.12:0/2237976287 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f5fd40051a0 con 0x5562552b1d30 2024-05-27T07:33:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.270+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fd00075c0 con 0x5562552b1d30 2024-05-27T07:33:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5fc3400700 1 -- 172.21.15.12:0/2237976287 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5fd403e050 con 0x5562552b1d30 2024-05-27T07:33:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 10 client.16803 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:45.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x556255330bb0 con 0x5562552b1d30 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5fc3400700 1 -- 172.21.15.12:0/2237976287 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fd00075c0 con 0x5562552b1d30 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5fc3400700 1 -- 172.21.15.12:0/2237976287 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5fd401e8b0 con 0x5562552b1d30 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 10 client.16803 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 20 client.16803 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562553310f0 0x556255333510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 4 client.16803 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.271+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5562551d5570 con 0x5562553310f0 2024-05-27T07:33:45.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.272+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562553310f0 0x556255333510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:45.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.273+0000 7f5fe1a00700 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562553310f0 0x556255333510 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e31ff674825e9649 server_cookie=be0ad77f7b6b597b in_seq=0 out_seq=0 2024-05-27T07:33:45.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.273+0000 7f5fc3400700 10 client.16803 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:45.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.273+0000 7f5fc3400700 1 -- 172.21.15.12:0/2237976287 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5562551d5570 con 0x5562553310f0 2024-05-27T07:33:45.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.273+0000 7f5fc3400700 10 client.16803 handle_command_reply: tid=0 2024-05-27T07:33:45.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.273+0000 7f5ff2fb7b80 1 client.16803 shutdown 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 msgr2=0x5562552baa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552baa90 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4004f90 tx=0x7f5fd4012f90 comp rx=0 tx=0).stop 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 shutdown_connections 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5562552b1d30 0x5562552baa90 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55625512ce40 0x5562552bdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5562553310f0 0x556255333510 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 --2- 172.21.15.12:0/2237976287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562552abdb0 0x5562552be000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 >> 172.21.15.12:0/2237976287 conn(0x55625518cfc0 msgr2=0x5562552af9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:45.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5fc3400700 0 client.16803 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:33:45.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 shutdown_connections 2024-05-27T07:33:45.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 1 -- 172.21.15.12:0/2237976287 wait complete. 2024-05-27T07:33:45.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.274+0000 7f5ff2fb7b80 20 client.16803 trim_cache size 0 max 16384 2024-05-27T07:33:45.270 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:33:45.270 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:33:45.270 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:33:45.270 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:33:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fc40b2a70 msgr2=0x7f5fc40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fc40b2a70 0x7f5fc40b4f80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80a56f0 tx=0x7f5fc80047c0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc106eb0 msgr2=0x7f5fdc19d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc106eb0 0x7f5fdc19d500 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800d950 tx=0x7f5fc800dc60 comp rx=0 tx=0).stop 2024-05-27T07:33:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 shutdown_connections 2024-05-27T07:33:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fc40b2a70 0x7f5fc40b4f80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fdc106560 0x7f5fdc19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fdc10c5a0 0x7f5fdc1a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 --2- 172.21.15.12:0/1446254322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fdc106eb0 0x7f5fdc19d500 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.275+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 >> 172.21.15.12:0/1446254322 conn(0x7f5fdc102040 msgr2=0x7f5fdc103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.276+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 shutdown_connections 2024-05-27T07:33:45.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:45.276+0000 7f5fe4400700 1 -- 172.21.15.12:0/1446254322 wait complete. 2024-05-27T07:33:45.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:45 smithi012 ceph-mon[31473]: pgmap v393: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-05-27T07:33:45.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:45 smithi162 ceph-mon[35676]: pgmap v393: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-05-27T07:33:45.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:45 smithi062 ceph-mon[35304]: pgmap v393: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-05-27T07:33:46.385 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:33:46.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:33:46.894 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:47.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:47 smithi012 ceph-mon[31473]: pgmap v394: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 963 KiB/s wr, 113 op/s 2024-05-27T07:33:47.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:47 smithi162 ceph-mon[35676]: pgmap v394: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 963 KiB/s wr, 113 op/s 2024-05-27T07:33:47.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:47 smithi062 ceph-mon[35304]: pgmap v394: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 963 KiB/s wr, 113 op/s 2024-05-27T07:33:48.200 INFO:tasks.workunit.client.0.smithi012.stdout:20000 mapread 0x1c4208 thru 0x1c56d4 (0x14cd bytes) 2024-05-27T07:33:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 -- 172.21.15.12:0/4139719732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c1054d0 msgr2=0x7f1f8c1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 --2- 172.21.15.12:0/4139719732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c1054d0 0x7f1f8c1058e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009990 tx=0x7f1f7c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 -- 172.21.15.12:0/4139719732 shutdown_connections 2024-05-27T07:33:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 --2- 172.21.15.12:0/4139719732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f8c10e900 0x7f1f8c110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 --2- 172.21.15.12:0/4139719732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c1054d0 0x7f1f8c1058e0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 --2- 172.21.15.12:0/4139719732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f8c105e20 0x7f1f8c10e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 -- 172.21.15.12:0/4139719732 >> 172.21.15.12:0/4139719732 conn(0x7f1f8c0fedb0 msgr2=0x7f1f8c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 -- 172.21.15.12:0/4139719732 shutdown_connections 2024-05-27T07:33:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.458+0000 7f1f92800700 1 -- 172.21.15.12:0/4139719732 wait complete. 2024-05-27T07:33:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.459+0000 7f1f92800700 1 Processor -- start 2024-05-27T07:33:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f92800700 1 -- start start 2024-05-27T07:33:49.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f92800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f8c1054d0 0x7f1f8c19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:49.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f92800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f8c105e20 0x7f1f8c19d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:49.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f92800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c10e900 0x7f1f8c1a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:49.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f92800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c1142f0 con 0x7f1f8c10e900 2024-05-27T07:33:49.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f92800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c114170 con 0x7f1f8c105e20 2024-05-27T07:33:49.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f92800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c114470 con 0x7f1f8c1054d0 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f8be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c10e900 0x7f1f8c1a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f8be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c10e900 0x7f1f8c1a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45344/0 (socket says 172.21.15.12:45344) 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.460+0000 7f1f8be00700 1 -- 172.21.15.12:0/4021324869 learned_addr learned my addr 172.21.15.12:0/4021324869 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8b400700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f8c1054d0 0x7f1f8c19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8aa00700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f8c105e20 0x7f1f8c19d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8be00700 1 -- 172.21.15.12:0/4021324869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f8c1054d0 msgr2=0x7f1f8c19d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8be00700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f8c1054d0 0x7f1f8c19d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8be00700 1 -- 172.21.15.12:0/4021324869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f8c105e20 msgr2=0x7f1f8c19d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8be00700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f8c105e20 0x7f1f8c19d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8be00700 1 -- 172.21.15.12:0/4021324869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c1a1ed0 con 0x7f1f8c10e900 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.461+0000 7f1f8be00700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c10e900 0x7f1f8c1a1830 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1f84007ee0 tx=0x7f1f8400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.462+0000 7f1f83e00700 1 -- 172.21.15.12:0/4021324869 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400dbd0 con 0x7f1f8c10e900 2024-05-27T07:33:49.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.462+0000 7f1f83e00700 1 -- 172.21.15.12:0/4021324869 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f8400dd70 con 0x7f1f8c10e900 2024-05-27T07:33:49.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.462+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c1a21c0 con 0x7f1f8c10e900 2024-05-27T07:33:49.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.462+0000 7f1f83e00700 1 -- 172.21.15.12:0/4021324869 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f840058d0 con 0x7f1f8c10e900 2024-05-27T07:33:49.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.462+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8c1a9ae0 con 0x7f1f8c10e900 2024-05-27T07:33:49.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.463+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f8c0512b0 con 0x7f1f8c10e900 2024-05-27T07:33:49.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.466+0000 7f1f83e00700 1 -- 172.21.15.12:0/4021324869 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1f8401a020 con 0x7f1f8c10e900 2024-05-27T07:33:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.467+0000 7f1f83e00700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f700b2a70 0x7f1f700b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.468+0000 7f1f83e00700 1 -- 172.21.15.12:0/4021324869 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f1f840e2160 con 0x7f1f8c10e900 2024-05-27T07:33:49.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.468+0000 7f1f8b400700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f700b2a70 0x7f1f700b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:49.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.469+0000 7f1f8b400700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f700b2a70 0x7f1f700b4f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0098e0 tx=0x7f1f7c005960 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:49.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.473+0000 7f1f83e00700 1 -- 172.21.15.12:0/4021324869 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1f84010070 con 0x7f1f8c10e900 2024-05-27T07:33:49.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:49 smithi162 ceph-mon[35676]: pgmap v395: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 963 KiB/s wr, 113 op/s 2024-05-27T07:33:49.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:49 smithi062 ceph-mon[35304]: pgmap v395: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 963 KiB/s wr, 113 op/s 2024-05-27T07:33:49.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.837+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f8c02e5e0 con 0x7f1f8c10e900 2024-05-27T07:33:49.833 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:49 smithi012 ceph-mon[31473]: pgmap v395: 129 pgs: 129 active+clean; 190 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 963 KiB/s wr, 113 op/s 2024-05-27T07:33:49.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.839+0000 7f1f83e00700 1 -- 172.21.15.12:0/4021324869 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f1f840a96d0 con 0x7f1f8c10e900 2024-05-27T07:33:49.834 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:33:49.834 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:33:49.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f700b2a70 msgr2=0x7f1f700b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:49.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f700b2a70 0x7f1f700b4f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0098e0 tx=0x7f1f7c005960 comp rx=0 tx=0).stop 2024-05-27T07:33:49.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c10e900 msgr2=0x7f1f8c1a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:49.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c10e900 0x7f1f8c1a1830 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1f84007ee0 tx=0x7f1f8400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 shutdown_connections 2024-05-27T07:33:49.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1f700b2a70 0x7f1f700b4f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f8c10e900 0x7f1f8c1a1830 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f8c1054d0 0x7f1f8c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 --2- 172.21.15.12:0/4021324869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f8c105e20 0x7f1f8c19d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:49.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.843+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 >> 172.21.15.12:0/4021324869 conn(0x7f1f8c0fedb0 msgr2=0x7f1f8c103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:49.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.844+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 shutdown_connections 2024-05-27T07:33:49.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:49.844+0000 7f1f92800700 1 -- 172.21.15.12:0/4021324869 wait complete. 2024-05-27T07:33:49.840 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:33:50.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:50 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/4021324869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:50.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:50 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/4021324869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:50.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:50 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/4021324869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:50.830 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:33:51.324 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:51.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:51 smithi012 ceph-mon[31473]: pgmap v396: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-05-27T07:33:51.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:51 smithi162 ceph-mon[35676]: pgmap v396: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-05-27T07:33:51.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:51 smithi062 ceph-mon[35304]: pgmap v396: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-05-27T07:33:53.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:53 smithi162 ceph-mon[35676]: pgmap v397: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 960 KiB/s wr, 125 op/s 2024-05-27T07:33:53.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:53 smithi062 ceph-mon[35304]: pgmap v397: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 960 KiB/s wr, 125 op/s 2024-05-27T07:33:53.764 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:53 smithi012 ceph-mon[31473]: pgmap v397: 129 pgs: 129 active+clean; 193 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 960 KiB/s wr, 125 op/s 2024-05-27T07:33:54.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.413+0000 7f6c03c00700 1 -- 172.21.15.12:0/2066329099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc110bf0 msgr2=0x7f6bfc113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.413+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2066329099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc110bf0 0x7f6bfc113010 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400b0a0 tx=0x7f6bf400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.414+0000 7f6c03c00700 1 -- 172.21.15.12:0/2066329099 shutdown_connections 2024-05-27T07:33:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.414+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2066329099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bfc075af0 0x7f6bfc075f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.414+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2066329099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc110bf0 0x7f6bfc113010 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.414+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2066329099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc076440 0x7f6bfc1106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.414+0000 7f6c03c00700 1 -- 172.21.15.12:0/2066329099 >> 172.21.15.12:0/2066329099 conn(0x7f6bfc100030 msgr2=0x7f6bfc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.414+0000 7f6c03c00700 1 -- 172.21.15.12:0/2066329099 shutdown_connections 2024-05-27T07:33:54.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.414+0000 7f6c03c00700 1 -- 172.21.15.12:0/2066329099 wait complete. 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.416+0000 7f6c03c00700 1 Processor -- start 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.416+0000 7f6c03c00700 1 -- start start 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6c03c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bfc075af0 0x7f6bfc19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6c03c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc076440 0x7f6bfc19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6c03c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc110bf0 0x7f6bfc1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6c03c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc116560 con 0x7f6bfc076440 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6c03c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1163e0 con 0x7f6bfc075af0 2024-05-27T07:33:54.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6c03c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1166e0 con 0x7f6bfc110bf0 2024-05-27T07:33:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6bfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc076440 0x7f6bfc19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6bfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc076440 0x7f6bfc19d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33878/0 (socket says 172.21.15.12:33878) 2024-05-27T07:33:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.417+0000 7f6bfbe00700 1 -- 172.21.15.12:0/2741881091 learned_addr learned my addr 172.21.15.12:0/2741881091 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.418+0000 7f6c01200700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bfc075af0 0x7f6bfc19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.418+0000 7f6c01c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc110bf0 0x7f6bfc1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.418+0000 7f6bfbe00700 1 -- 172.21.15.12:0/2741881091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc110bf0 msgr2=0x7f6bfc1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.418+0000 7f6bfbe00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc110bf0 0x7f6bfc1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.418+0000 7f6bfbe00700 1 -- 172.21.15.12:0/2741881091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bfc075af0 msgr2=0x7f6bfc19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.418+0000 7f6bfbe00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bfc075af0 0x7f6bfc19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.418+0000 7f6bfbe00700 1 -- 172.21.15.12:0/2741881091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc1a1dc0 con 0x7f6bfc076440 2024-05-27T07:33:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.419+0000 7f6bfbe00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc076440 0x7f6bfc19d490 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6be800e970 tx=0x7f6be800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.419+0000 7f6bf9600700 1 -- 172.21.15.12:0/2741881091 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8004a30 con 0x7f6bfc076440 2024-05-27T07:33:54.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.419+0000 7f6bf9600700 1 -- 172.21.15.12:0/2741881091 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6be800cc70 con 0x7f6bfc076440 2024-05-27T07:33:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.419+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc1a2050 con 0x7f6bfc076440 2024-05-27T07:33:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.419+0000 7f6bf9600700 1 -- 172.21.15.12:0/2741881091 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8005520 con 0x7f6bfc076440 2024-05-27T07:33:54.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.419+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc1a99d0 con 0x7f6bfc076440 2024-05-27T07:33:54.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.423+0000 7f6bf9600700 1 -- 172.21.15.12:0/2741881091 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6be8021440 con 0x7f6bfc076440 2024-05-27T07:33:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.425+0000 7f6c12834b80 1 Processor -- start 2024-05-27T07:33:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.425+0000 7f6c12834b80 1 -- start start 2024-05-27T07:33:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.425+0000 7f6bf9600700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6be40b2a70 0x7f6be40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.425+0000 7f6bf9600700 1 -- 172.21.15.12:0/2741881091 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f6be80e2b20 con 0x7f6bfc076440 2024-05-27T07:33:54.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc12d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc12dd30 0x55b3fc127870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc127db0 0x55b3fc1281c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c12834b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b3fc055e40 con 0x55b3fc127db0 2024-05-27T07:33:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c12834b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b3fc0459e0 con 0x55b3fbfa8e40 2024-05-27T07:33:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c12834b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3fc049a20 con 0x55b3fc12dd30 2024-05-27T07:33:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c01c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6be40b2a70 0x7f6be40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c01c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc127db0 0x55b3fc1281c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c01c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc127db0 0x55b3fc1281c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33886/0 (socket says 172.21.15.12:33886) 2024-05-27T07:33:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c01c00700 1 -- 172.21.15.12:0/1652375278 learned_addr learned my addr 172.21.15.12:0/1652375278 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6c01200700 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc12d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.426+0000 7f6bfbe00700 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc12dd30 0x55b3fc127870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.427+0000 7f6c01c00700 1 -- 172.21.15.12:0/1652375278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc12dd30 msgr2=0x55b3fc127870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.427+0000 7f6c01c00700 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc12dd30 0x55b3fc127870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.427+0000 7f6c01c00700 1 -- 172.21.15.12:0/1652375278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 msgr2=0x55b3fc12d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.427+0000 7f6c01c00700 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc12d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.427+0000 7f6c01c00700 1 -- 172.21.15.12:0/1652375278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3fc00e420 con 0x55b3fc127db0 2024-05-27T07:33:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.427+0000 7f6c01c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6be40b2a70 0x7f6be40b4f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4015000 tx=0x7f6bf4009dc0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.427+0000 7f6c01c00700 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc127db0 0x55b3fc1281c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4004c50 tx=0x7f6bf4026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfdf0a5538c9fbb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.428+0000 7f6bee000700 1 -- 172.21.15.12:0/1652375278 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4009b80 con 0x55b3fc127db0 2024-05-27T07:33:54.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.428+0000 7f6bee000700 1 -- 172.21.15.12:0/1652375278 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bf4007d60 con 0x55b3fc127db0 2024-05-27T07:33:54.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.428+0000 7f6bee000700 1 -- 172.21.15.12:0/1652375278 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4021d10 con 0x55b3fc127db0 2024-05-27T07:33:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 -- 172.21.15.12:0/1652375278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc127db0 msgr2=0x55b3fc1281c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc127db0 0x55b3fc1281c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4004c50 tx=0x7f6bf4026000 comp rx=0 tx=0).stop 2024-05-27T07:33:54.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 -- 172.21.15.12:0/1652375278 shutdown_connections 2024-05-27T07:33:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc12d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc127db0 0x55b3fc1281c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 --2- 172.21.15.12:0/1652375278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc12dd30 0x55b3fc127870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 -- 172.21.15.12:0/1652375278 >> 172.21.15.12:0/1652375278 conn(0x55b3fc008fc0 msgr2=0x55b3fc1228d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 -- 172.21.15.12:0/1652375278 shutdown_connections 2024-05-27T07:33:54.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.429+0000 7f6c12834b80 1 -- 172.21.15.12:0/1652375278 wait complete. 2024-05-27T07:33:54.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.430+0000 7f6c12834b80 1 -- start start 2024-05-27T07:33:54.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.430+0000 7f6c12834b80 1 Processor -- start 2024-05-27T07:33:54.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc1314b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc1319f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c12834b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b3fbf4bfa0 con 0x55b3fc12dd30 2024-05-27T07:33:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c12834b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b3fbfe6760 con 0x55b3fbfa8e40 2024-05-27T07:33:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c12834b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3fc1347a0 con 0x55b3fc127db0 2024-05-27T07:33:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c12834b80 10 client.0 _ll_register_callbacks cb 0x55b3fbfefc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c01c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c01200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc1314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c01c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33892/0 (socket says 172.21.15.12:33892) 2024-05-27T07:33:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.431+0000 7f6c01c00700 1 -- 172.21.15.12:0/3884984654 learned_addr learned my addr 172.21.15.12:0/3884984654 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.432+0000 7f6bfbe00700 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc1319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.432+0000 7f6c01c00700 1 -- 172.21.15.12:0/3884984654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 msgr2=0x55b3fc1319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.432+0000 7f6c01c00700 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc1319f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.432+0000 7f6c01c00700 1 -- 172.21.15.12:0/3884984654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 msgr2=0x55b3fc1314b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.432+0000 7f6c01c00700 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc1314b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.432+0000 7f6c01c00700 1 -- 172.21.15.12:0/3884984654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3fbf958d0 con 0x55b3fc12dd30 2024-05-27T07:33:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.433+0000 7f6c01c00700 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc131f30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400d010 tx=0x7f6bf4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9718f13f73ec09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.433+0000 7f6be2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.433+0000 7f6be2a00700 1 -- 172.21.15.12:0/3884984654 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4032050 con 0x55b3fc12dd30 2024-05-27T07:33:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.434+0000 7f6be2a00700 1 -- 172.21.15.12:0/3884984654 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f6bf4021970 con 0x55b3fc12dd30 2024-05-27T07:33:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.434+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bf4007d60 con 0x55b3fc12dd30 2024-05-27T07:33:54.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.434+0000 7f6be2a00700 1 -- 172.21.15.12:0/3884984654 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f6bf4009b80 con 0x55b3fc12dd30 2024-05-27T07:33:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.434+0000 7f6c12834b80 10 client.16821 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.434+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55b3fc016060 con 0x55b3fc12dd30 2024-05-27T07:33:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6be2a00700 1 -- 172.21.15.12:0/3884984654 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bf4007d60 con 0x55b3fc12dd30 2024-05-27T07:33:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6be2a00700 1 -- 172.21.15.12:0/3884984654 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f6bf4011070 con 0x55b3fc12dd30 2024-05-27T07:33:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6c12834b80 10 client.16821 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6c12834b80 10 client.16821 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:33:54.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6c12834b80 20 client.16821 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:54.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6c12834b80 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1415c0 0x55b3fc143a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6c12834b80 4 client.16821 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:54.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.435+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3fc057d90 con 0x55b3fc1415c0 2024-05-27T07:33:54.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.436+0000 7f6c01200700 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1415c0 0x55b3fc143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.437+0000 7f6be2a00700 10 client.16821 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:33:54.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.437+0000 7f6c01200700 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1415c0 0x55b3fc143a20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de1ed8c07dd89d79 server_cookie=befa3466bd26abdc in_seq=0 out_seq=0 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.439+0000 7f6be2a00700 1 -- 172.21.15.12:0/3884984654 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b3fc057d90 con 0x55b3fc1415c0 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.439+0000 7f6be2a00700 10 client.16821 handle_command_reply: tid=0 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.439+0000 7f6c12834b80 1 client.16821 shutdown 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 msgr2=0x55b3fc131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc131f30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400d010 tx=0x7f6bf4006000 comp rx=0 tx=0).stop 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 shutdown_connections 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1415c0 0x55b3fc143a20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc1314b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc131f30 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 --2- 172.21.15.12:0/3884984654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc1319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 >> 172.21.15.12:0/3884984654 conn(0x55b3fc008fc0 msgr2=0x55b3fc122580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:54.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6be2a00700 0 client.16821 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:33:54.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 shutdown_connections 2024-05-27T07:33:54.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 1 -- 172.21.15.12:0/3884984654 wait complete. 2024-05-27T07:33:54.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.440+0000 7f6c12834b80 20 client.16821 trim_cache size 0 max 16384 2024-05-27T07:33:54.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.459+0000 7f6c12834b80 1 Processor -- start 2024-05-27T07:33:54.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.459+0000 7f6c12834b80 1 -- start start 2024-05-27T07:33:54.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.459+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fbfa8e40 0x55b3fc127870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.459+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.459+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fc12dd30 0x55b3fc13d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.459+0000 7f6c12834b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1b1d90 con 0x55b3fbfa8e40 2024-05-27T07:33:54.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c12834b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1b0ff0 con 0x55b3fc12dd30 2024-05-27T07:33:54.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c12834b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1b16c0 con 0x55b3fc127db0 2024-05-27T07:33:54.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fbfa8e40 0x55b3fc127870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fbfa8e40 0x55b3fc127870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33904/0 (socket says 172.21.15.12:33904) 2024-05-27T07:33:54.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 -- 172.21.15.12:0/2022100068 learned_addr learned my addr 172.21.15.12:0/2022100068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:54.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6bfbe00700 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01c00700 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fc12dd30 0x55b3fc13d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 -- 172.21.15.12:0/2022100068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 msgr2=0x55b3fc133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 -- 172.21.15.12:0/2022100068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fc12dd30 msgr2=0x55b3fc13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fc12dd30 0x55b3fc13d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.460+0000 7f6c01200700 1 -- 172.21.15.12:0/2022100068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3fc1235f0 con 0x55b3fbfa8e40 2024-05-27T07:33:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.461+0000 7f6c01200700 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fbfa8e40 0x55b3fc127870 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x55b3fc144180 tx=0x7f6bf0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf52dd02b515e83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.461+0000 7f6bed600700 1 -- 172.21.15.12:0/2022100068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0004830 con 0x55b3fbfa8e40 2024-05-27T07:33:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.461+0000 7f6bed600700 1 -- 172.21.15.12:0/2022100068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bf0005020 con 0x55b3fbfa8e40 2024-05-27T07:33:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.461+0000 7f6bed600700 1 -- 172.21.15.12:0/2022100068 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf001e680 con 0x55b3fbfa8e40 2024-05-27T07:33:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.462+0000 7f6c12834b80 1 -- 172.21.15.12:0/2022100068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fbfa8e40 msgr2=0x55b3fc127870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.462+0000 7f6c12834b80 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fbfa8e40 0x55b3fc127870 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x55b3fc144180 tx=0x7f6bf0009260 comp rx=0 tx=0).stop 2024-05-27T07:33:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.462+0000 7f6c12834b80 1 -- 172.21.15.12:0/2022100068 shutdown_connections 2024-05-27T07:33:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.462+0000 7f6c12834b80 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fc12dd30 0x55b3fc13d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.462+0000 7f6c12834b80 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fbfa8e40 0x55b3fc127870 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.462+0000 7f6c12834b80 1 --2- 172.21.15.12:0/2022100068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.462+0000 7f6c12834b80 1 -- 172.21.15.12:0/2022100068 >> 172.21.15.12:0/2022100068 conn(0x55b3fc008fc0 msgr2=0x55b3fc12b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.463+0000 7f6c12834b80 1 -- 172.21.15.12:0/2022100068 shutdown_connections 2024-05-27T07:33:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.463+0000 7f6c12834b80 1 -- 172.21.15.12:0/2022100068 wait complete. 2024-05-27T07:33:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.463+0000 7f6c12834b80 1 -- start start 2024-05-27T07:33:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 1 Processor -- start 2024-05-27T07:33:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc139c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc13a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc148810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b3fc1346e0 con 0x55b3fc12dd30 2024-05-27T07:33:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b3fc123e00 con 0x55b3fbfa8e40 2024-05-27T07:33:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3fc12f440 con 0x55b3fc127db0 2024-05-27T07:33:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.464+0000 7f6c12834b80 10 client.0 _ll_register_callbacks cb 0x55b3fbfefc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.465+0000 7f6c01200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.465+0000 7f6bfbe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc13a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.465+0000 7f6c01200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc139c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:53858/0 (socket says 172.21.15.12:53858) 2024-05-27T07:33:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.465+0000 7f6c01c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.465+0000 7f6c01200700 1 -- 172.21.15.12:0/171970616 learned_addr learned my addr 172.21.15.12:0/171970616 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c01c00700 1 -- 172.21.15.12:0/171970616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 msgr2=0x55b3fc13a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c01c00700 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc13a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c01c00700 1 -- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 msgr2=0x55b3fc139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c01c00700 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c01c00700 1 -- 172.21.15.12:0/171970616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3fc148fb0 con 0x55b3fc12dd30 2024-05-27T07:33:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c01200700 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:54.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c01c00700 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc148810 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6bf401e5f0 tx=0x7f6bf4012600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6acf371cea62f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:54.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6be2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:54.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6be2a00700 1 -- 172.21.15.12:0/171970616 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4045060 con 0x55b3fc12dd30 2024-05-27T07:33:54.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6be2a00700 1 -- 172.21.15.12:0/171970616 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f6bf400f070 con 0x55b3fc12dd30 2024-05-27T07:33:54.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.466+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bf0005020 con 0x55b3fc12dd30 2024-05-27T07:33:54.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6be2a00700 1 -- 172.21.15.12:0/171970616 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f6bf404c050 con 0x55b3fc12dd30 2024-05-27T07:33:54.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 10 client.16827 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:54.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55b3fc1b4320 con 0x55b3fc12dd30 2024-05-27T07:33:54.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6be2a00700 1 -- 172.21.15.12:0/171970616 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bf0005020 con 0x55b3fc12dd30 2024-05-27T07:33:54.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6be2a00700 1 -- 172.21.15.12:0/171970616 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f6bf4011040 con 0x55b3fc12dd30 2024-05-27T07:33:54.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 10 client.16827 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:54.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 10 client.16827 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:33:54.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 20 client.16827 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:54.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1b49c0 0x55b3fc1b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:54.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 4 client.16827 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:54.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.467+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3fc051570 con 0x55b3fc1b49c0 2024-05-27T07:33:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.468+0000 7f6c01200700 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1b49c0 0x55b3fc1b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.469+0000 7f6c01200700 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1b49c0 0x55b3fc1b6de0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8bac31347cf6717d server_cookie=6619059c511c2410 in_seq=0 out_seq=0 2024-05-27T07:33:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.469+0000 7f6be2a00700 10 client.16827 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:33:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6be2a00700 1 -- 172.21.15.12:0/171970616 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3fc051570 con 0x55b3fc1b49c0 2024-05-27T07:33:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6be2a00700 10 client.16827 handle_command_reply: tid=0 2024-05-27T07:33:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 client.16827 shutdown 2024-05-27T07:33:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 msgr2=0x55b3fc148810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc148810 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6bf401e5f0 tx=0x7f6bf4012600 comp rx=0 tx=0).stop 2024-05-27T07:33:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 shutdown_connections 2024-05-27T07:33:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55b3fc1b49c0 0x55b3fc1b6de0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b3fbfa8e40 0x55b3fc139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b3fc12dd30 0x55b3fc148810 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 --2- 172.21.15.12:0/171970616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3fc127db0 0x55b3fc13a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 >> 172.21.15.12:0/171970616 conn(0x55b3fc008fc0 msgr2=0x55b3fc12cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6be2a00700 0 client.16827 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:33:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 shutdown_connections 2024-05-27T07:33:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.470+0000 7f6c12834b80 1 -- 172.21.15.12:0/171970616 wait complete. 2024-05-27T07:33:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.471+0000 7f6c12834b80 20 client.16827 trim_cache size 0 max 16384 2024-05-27T07:33:54.475 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:33:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.471+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6be40b2a70 msgr2=0x7f6be40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.471+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6be40b2a70 0x7f6be40b4f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4015000 tx=0x7f6bf4009dc0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.471+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc076440 msgr2=0x7f6bfc19d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.471+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc076440 0x7f6bfc19d490 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6be800e970 tx=0x7f6be800ec80 comp rx=0 tx=0).stop 2024-05-27T07:33:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 shutdown_connections 2024-05-27T07:33:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6be40b2a70 0x7f6be40b4f80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bfc075af0 0x7f6bfc19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc110bf0 0x7f6bfc1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 --2- 172.21.15.12:0/2741881091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bfc076440 0x7f6bfc19d490 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 >> 172.21.15.12:0/2741881091 conn(0x7f6bfc100030 msgr2=0x7f6bfc100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 shutdown_connections 2024-05-27T07:33:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:54.472+0000 7f6c03c00700 1 -- 172.21.15.12:0/2741881091 wait complete. 2024-05-27T07:33:55.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:33:55.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:55 smithi012 ceph-mon[31473]: pgmap v398: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:33:55.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:55 smithi162 ceph-mon[35676]: pgmap v398: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:33:55.739 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:55 smithi062 ceph-mon[35304]: pgmap v398: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2024-05-27T07:33:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:57.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:57 smithi012 ceph-mon[31473]: pgmap v399: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 933 KiB/s wr, 124 op/s 2024-05-27T07:33:57.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:57 smithi162 ceph-mon[35676]: pgmap v399: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 933 KiB/s wr, 124 op/s 2024-05-27T07:33:57.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:57 smithi062 ceph-mon[35304]: pgmap v399: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 933 KiB/s wr, 124 op/s 2024-05-27T07:33:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2078556096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f441034b0 msgr2=0x7f5f441038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2078556096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f441034b0 0x7f5f441038c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5f340099c0 tx=0x7f5f34009cd0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2078556096 shutdown_connections 2024-05-27T07:33:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2078556096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f44108730 0x7f5f44110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2078556096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f441034b0 0x7f5f441038c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2078556096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f44107de0 0x7f5f441081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2078556096 >> 172.21.15.12:0/2078556096 conn(0x7f5f440fedb0 msgr2=0x7f5f441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.251+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2078556096 shutdown_connections 2024-05-27T07:33:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.252+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2078556096 wait complete. 2024-05-27T07:33:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.253+0000 7f5f4ac00700 1 Processor -- start 2024-05-27T07:33:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.253+0000 7f5f4ac00700 1 -- start start 2024-05-27T07:33:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.254+0000 7f5f4ac00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f441034b0 0x7f5f440ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.254+0000 7f5f4ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f44107de0 0x7f5f440ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.254+0000 7f5f4ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f44108730 0x7f5f440ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.254+0000 7f5f4ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5f44113260 con 0x7f5f44107de0 2024-05-27T07:33:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.254+0000 7f5f4ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441130e0 con 0x7f5f44108730 2024-05-27T07:33:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.254+0000 7f5f4ac00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441133e0 con 0x7f5f441034b0 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.254+0000 7f5f43400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f44107de0 0x7f5f440ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f44107de0 0x7f5f440ff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33936/0 (socket says 172.21.15.12:33936) 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43400700 1 -- 172.21.15.12:0/2377951110 learned_addr learned my addr 172.21.15.12:0/2377951110 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f48c00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f44108730 0x7f5f440ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43e00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f441034b0 0x7f5f440ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43400700 1 -- 172.21.15.12:0/2377951110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f441034b0 msgr2=0x7f5f440ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43400700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f441034b0 0x7f5f440ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43400700 1 -- 172.21.15.12:0/2377951110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f44108730 msgr2=0x7f5f440ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43400700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f44108730 0x7f5f440ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.255+0000 7f5f43400700 1 -- 172.21.15.12:0/2377951110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f441a9100 con 0x7f5f44107de0 2024-05-27T07:33:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.256+0000 7f5f43400700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f44107de0 0x7f5f440ff7d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00d950 tx=0x7f5f2c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.256+0000 7f5f40c00700 1 -- 172.21.15.12:0/2377951110 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c016070 con 0x7f5f44107de0 2024-05-27T07:33:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.256+0000 7f5f40c00700 1 -- 172.21.15.12:0/2377951110 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f2c004830 con 0x7f5f44107de0 2024-05-27T07:33:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.256+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f441a93f0 con 0x7f5f44107de0 2024-05-27T07:33:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.256+0000 7f5f40c00700 1 -- 172.21.15.12:0/2377951110 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c005420 con 0x7f5f44107de0 2024-05-27T07:33:58.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.256+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f441a9930 con 0x7f5f44107de0 2024-05-27T07:33:58.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.262+0000 7f5f40c00700 1 -- 172.21.15.12:0/2377951110 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5f2c004220 con 0x7f5f44107de0 2024-05-27T07:33:58.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.263+0000 7f5f59713b80 1 Processor -- start 2024-05-27T07:33:58.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.263+0000 7f5f59713b80 1 -- start start 2024-05-27T07:33:58.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564ade32e40 0x5564adfb77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfb1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f59713b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5564adedfe40 con 0x5564adfb7d30 2024-05-27T07:33:58.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f59713b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5564adecf9e0 con 0x5564adfb1db0 2024-05-27T07:33:58.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f59713b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5564aded3a20 con 0x5564ade32e40 2024-05-27T07:33:58.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f43e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfb1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f43e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfb1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33938/0 (socket says 172.21.15.12:33938) 2024-05-27T07:33:58.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.264+0000 7f5f43e00700 1 -- 172.21.15.12:0/1260733918 learned_addr learned my addr 172.21.15.12:0/1260733918 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43400700 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43e00700 1 -- 172.21.15.12:0/1260733918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564ade32e40 msgr2=0x5564adfb77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43e00700 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564ade32e40 0x5564adfb77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43e00700 1 -- 172.21.15.12:0/1260733918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 msgr2=0x5564adfb21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f40c00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f280b29f0 0x7f5f280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43e00700 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43e00700 1 -- 172.21.15.12:0/1260733918 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564ade98420 con 0x5564adfb7d30 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43400700 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f40c00700 1 -- 172.21.15.12:0/2377951110 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f5f2c0e2780 con 0x7f5f44107de0 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.265+0000 7f5f43e00700 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfb1870 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400b580 tx=0x7f5f340096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25134d9398479077 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.266+0000 7f5f48c00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f280b29f0 0x7f5f280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.266+0000 7f5f39600700 1 -- 172.21.15.12:0/1260733918 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34027070 con 0x5564adfb7d30 2024-05-27T07:33:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.266+0000 7f5f39600700 1 -- 172.21.15.12:0/1260733918 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f3400bc20 con 0x5564adfb7d30 2024-05-27T07:33:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.266+0000 7f5f39600700 1 -- 172.21.15.12:0/1260733918 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34003b10 con 0x5564adfb7d30 2024-05-27T07:33:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 -- 172.21.15.12:0/1260733918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 msgr2=0x5564adfb1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfb1870 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400b580 tx=0x7f5f340096e0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f48c00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f280b29f0 0x7f5f280b4f00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0093f0 tx=0x7f5f3c009380 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 -- 172.21.15.12:0/1260733918 shutdown_connections 2024-05-27T07:33:58.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564ade32e40 0x5564adfb77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1260733918 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfb1870 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 -- 172.21.15.12:0/1260733918 >> 172.21.15.12:0/1260733918 conn(0x5564ade92fc0 msgr2=0x5564adfac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.268+0000 7f5f59713b80 1 -- 172.21.15.12:0/1260733918 shutdown_connections 2024-05-27T07:33:58.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.269+0000 7f5f59713b80 1 -- 172.21.15.12:0/1260733918 wait complete. 2024-05-27T07:33:58.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.269+0000 7f5f59713b80 1 -- start start 2024-05-27T07:33:58.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 1 Processor -- start 2024-05-27T07:33:58.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564ade32e40 0x5564adfbb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfbbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfbbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5564addd5fa0 con 0x5564ade32e40 2024-05-27T07:33:58.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5564ade70760 con 0x5564adfb1db0 2024-05-27T07:33:58.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5564adfbe7a0 con 0x5564adfb7d30 2024-05-27T07:33:58.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.270+0000 7f5f59713b80 10 client.0 _ll_register_callbacks cb 0x5564ade79c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:58.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.271+0000 7f5f43e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564ade32e40 0x5564adfbb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.271+0000 7f5f43e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564ade32e40 0x5564adfbb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33950/0 (socket says 172.21.15.12:33950) 2024-05-27T07:33:58.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.271+0000 7f5f43e00700 1 -- 172.21.15.12:0/3247314614 learned_addr learned my addr 172.21.15.12:0/3247314614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:58.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.271+0000 7f5f43400700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfbbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.272+0000 7f5f48c00700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfbbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f43e00700 1 -- 172.21.15.12:0/3247314614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 msgr2=0x5564adfbbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f43e00700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfbbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f43e00700 1 -- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 msgr2=0x5564adfbbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f43e00700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfbbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f43e00700 1 -- 172.21.15.12:0/3247314614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564ade1f8d0 con 0x5564ade32e40 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f43400700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfbbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f48c00700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfbbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f43e00700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564ade32e40 0x5564adfbb560 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f5f340058e0 tx=0x7f5f3400ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914e1bc078c9e7bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.273+0000 7f5f32000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.274+0000 7f5f32000700 1 -- 172.21.15.12:0/3247314614 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34030020 con 0x5564ade32e40 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.274+0000 7f5f32000700 1 -- 172.21.15.12:0/3247314614 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f5f34003790 con 0x5564ade32e40 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.274+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f3400bc20 con 0x5564ade32e40 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.274+0000 7f5f32000700 1 -- 172.21.15.12:0/3247314614 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5f3403b050 con 0x5564ade32e40 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.274+0000 7f5f59713b80 10 client.16839 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.274+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5564adea0060 con 0x5564ade32e40 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.274+0000 7f5f32000700 1 -- 172.21.15.12:0/3247314614 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f3400bc20 con 0x5564ade32e40 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f32000700 1 -- 172.21.15.12:0/3247314614 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5f34022910 con 0x5564ade32e40 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f59713b80 10 client.16839 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f59713b80 10 client.16839 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f59713b80 20 client.16839 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f59713b80 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564adfd3ee0 0x5564adfd6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f59713b80 4 client.16839 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:58.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564adee1d90 con 0x5564adfd3ee0 2024-05-27T07:33:58.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.275+0000 7f5f43400700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564adfd3ee0 0x5564adfd6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.276+0000 7f5f43400700 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564adfd3ee0 0x5564adfd6380 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a961e68c0f544bf server_cookie=ed919b2a2e811df6 in_seq=0 out_seq=0 2024-05-27T07:33:58.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.276+0000 7f5f32000700 10 client.16839 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:33:58.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f32000700 1 -- 172.21.15.12:0/3247314614 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5564adee1d90 con 0x5564adfd3ee0 2024-05-27T07:33:58.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f32000700 10 client.16839 handle_command_reply: tid=0 2024-05-27T07:33:58.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 client.16839 shutdown 2024-05-27T07:33:58.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564ade32e40 msgr2=0x5564adfbb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564ade32e40 0x5564adfbb560 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f5f340058e0 tx=0x7f5f3400ba70 comp rx=0 tx=0).stop 2024-05-27T07:33:58.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 shutdown_connections 2024-05-27T07:33:58.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564adfd3ee0 0x5564adfd6380 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfbbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564ade32e40 0x5564adfbb560 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 --2- 172.21.15.12:0/3247314614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfbbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 >> 172.21.15.12:0/3247314614 conn(0x5564ade92fc0 msgr2=0x5564adfac580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.279+0000 7f5f32000700 0 client.16839 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:33:58.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.280+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 shutdown_connections 2024-05-27T07:33:58.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.280+0000 7f5f59713b80 1 -- 172.21.15.12:0/3247314614 wait complete. 2024-05-27T07:33:58.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.280+0000 7f5f59713b80 20 client.16839 trim_cache size 0 max 16384 2024-05-27T07:33:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.301+0000 7f5f59713b80 1 Processor -- start 2024-05-27T07:33:58.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.301+0000 7f5f59713b80 1 -- start start 2024-05-27T07:33:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfba4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb5b70 0x5564adfbd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f59713b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5564adfb6de0 con 0x5564adfb5b70 2024-05-27T07:33:58.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f59713b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441ad8d0 con 0x5564adfb1db0 2024-05-27T07:33:58.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f59713b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441acb30 con 0x5564adfb7d30 2024-05-27T07:33:58.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f48c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb5b70 0x5564adfbd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f48c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb5b70 0x5564adfbd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33954/0 (socket says 172.21.15.12:33954) 2024-05-27T07:33:58.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f48c00700 1 -- 172.21.15.12:0/1913134561 learned_addr learned my addr 172.21.15.12:0/1913134561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:58.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.302+0000 7f5f43e00700 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.303+0000 7f5f43400700 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfba4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.303+0000 7f5f48c00700 1 -- 172.21.15.12:0/1913134561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 msgr2=0x5564adfba4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.303+0000 7f5f48c00700 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfba4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.303+0000 7f5f48c00700 1 -- 172.21.15.12:0/1913134561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 msgr2=0x5564adfb1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.303+0000 7f5f48c00700 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.303+0000 7f5f48c00700 1 -- 172.21.15.12:0/1913134561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564adfbde00 con 0x5564adfb5b70 2024-05-27T07:33:58.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.303+0000 7f5f48c00700 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb5b70 0x5564adfbd8c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c004680 tx=0x7f5f3c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6531dd51c4e5cf6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.304+0000 7f5f38c00700 1 -- 172.21.15.12:0/1913134561 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c01c030 con 0x5564adfb5b70 2024-05-27T07:33:58.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.304+0000 7f5f38c00700 1 -- 172.21.15.12:0/1913134561 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f3c027070 con 0x5564adfb5b70 2024-05-27T07:33:58.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.304+0000 7f5f38c00700 1 -- 172.21.15.12:0/1913134561 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c022680 con 0x5564adfb5b70 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 -- 172.21.15.12:0/1913134561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb5b70 msgr2=0x5564adfbd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb5b70 0x5564adfbd8c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c004680 tx=0x7f5f3c013000 comp rx=0 tx=0).stop 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 -- 172.21.15.12:0/1913134561 shutdown_connections 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfb1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb5b70 0x5564adfbd8c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 --2- 172.21.15.12:0/1913134561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb7d30 0x5564adfba4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 -- 172.21.15.12:0/1913134561 >> 172.21.15.12:0/1913134561 conn(0x5564ade92fc0 msgr2=0x5564adfbb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 -- 172.21.15.12:0/1913134561 shutdown_connections 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.305+0000 7f5f59713b80 1 -- 172.21.15.12:0/1913134561 wait complete. 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.306+0000 7f5f59713b80 1 -- start start 2024-05-27T07:33:58.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.306+0000 7f5f59713b80 1 Processor -- start 2024-05-27T07:33:58.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfc3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb5b70 0x5564adfdb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f59713b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfdb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f59713b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5564adee1d90 con 0x5564adfb7d30 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f59713b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441ad200 con 0x5564adfb1db0 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f59713b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5564adfc85b0 con 0x5564adfb5b70 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f59713b80 10 client.0 _ll_register_callbacks cb 0x5564ade79c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f48c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfdb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f48c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfdb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33962/0 (socket says 172.21.15.12:33962) 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f48c00700 1 -- 172.21.15.12:0/214839602 learned_addr learned my addr 172.21.15.12:0/214839602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f43e00700 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfc3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.307+0000 7f5f43400700 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb5b70 0x5564adfdb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.308+0000 7f5f48c00700 1 -- 172.21.15.12:0/214839602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb5b70 msgr2=0x5564adfdb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.308+0000 7f5f48c00700 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb5b70 0x5564adfdb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.308+0000 7f5f48c00700 1 -- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 msgr2=0x5564adfc3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.308+0000 7f5f48c00700 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfc3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.308+0000 7f5f48c00700 1 -- 172.21.15.12:0/214839602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564adfdbcb0 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.308+0000 7f5f48c00700 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfdb6f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c015f30 tx=0x7f5f3c015f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e8e7cacff1f2e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.308+0000 7f5f32000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.309+0000 7f5f32000700 1 -- 172.21.15.12:0/214839602 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c02f070 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.309+0000 7f5f32000700 1 -- 172.21.15.12:0/214839602 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f5f3c004cd0 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.309+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f3c027070 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.309+0000 7f5f32000700 1 -- 172.21.15.12:0/214839602 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5f3c01c030 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.309+0000 7f5f59713b80 10 client.16845 fetch_fsmap learned FSMap version 119 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.309+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5564ae03e6e0 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.309+0000 7f5f32000700 1 -- 172.21.15.12:0/214839602 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5f3c027070 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.310+0000 7f5f32000700 1 -- 172.21.15.12:0/214839602 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5f3c0057e0 con 0x5564adfb7d30 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.310+0000 7f5f59713b80 10 client.16845 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.310+0000 7f5f59713b80 10 client.16845 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:33:58.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.310+0000 7f5f59713b80 20 client.16845 populate_metadata read hostname 'smithi012' 2024-05-27T07:33:58.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.310+0000 7f5f59713b80 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564ae03ed60 0x5564ae041180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.310+0000 7f5f59713b80 4 client.16845 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:58.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.310+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564adedb570 con 0x5564ae03ed60 2024-05-27T07:33:58.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.311+0000 7f5f43e00700 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564ae03ed60 0x5564ae041180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.312+0000 7f5f43e00700 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564ae03ed60 0x5564ae041180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=856e69a311fb4687 server_cookie=a709ed23dd3a6f4a in_seq=0 out_seq=0 2024-05-27T07:33:58.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.312+0000 7f5f32000700 10 client.16845 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:33:58.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.313+0000 7f5f32000700 1 -- 172.21.15.12:0/214839602 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564adedb570 con 0x5564ae03ed60 2024-05-27T07:33:58.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.313+0000 7f5f32000700 10 client.16845 handle_command_reply: tid=0 2024-05-27T07:33:58.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.313+0000 7f5f59713b80 1 client.16845 shutdown 2024-05-27T07:33:58.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.313+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 msgr2=0x5564adfdb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.313+0000 7f5f59713b80 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfdb6f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c015f30 tx=0x7f5f3c015f60 comp rx=0 tx=0).stop 2024-05-27T07:33:58.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 shutdown_connections 2024-05-27T07:33:58.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5564ae03ed60 0x5564ae041180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5564adfb1db0 0x5564adfc3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5564adfb7d30 0x5564adfdb6f0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 --2- 172.21.15.12:0/214839602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564adfb5b70 0x5564adfdb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 >> 172.21.15.12:0/214839602 conn(0x5564ade92fc0 msgr2=0x5564adfbbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f32000700 0 client.16845 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:33:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 shutdown_connections 2024-05-27T07:33:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 1 -- 172.21.15.12:0/214839602 wait complete. 2024-05-27T07:33:58.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.314+0000 7f5f59713b80 20 client.16845 trim_cache size 0 max 16384 2024-05-27T07:33:58.311 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:33:58.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.315+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f280b29f0 msgr2=0x7f5f280b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.315+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f280b29f0 0x7f5f280b4f00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0093f0 tx=0x7f5f3c009380 comp rx=0 tx=0).stop 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.315+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f44107de0 msgr2=0x7f5f440ff7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.315+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f44107de0 0x7f5f440ff7d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00d950 tx=0x7f5f2c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.315+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 shutdown_connections 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.316+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f280b29f0 0x7f5f280b4f00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.316+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f44108730 0x7f5f440ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.316+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f441034b0 0x7f5f440ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.316+0000 7f5f4ac00700 1 --2- 172.21.15.12:0/2377951110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f44107de0 0x7f5f440ff7d0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.316+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 >> 172.21.15.12:0/2377951110 conn(0x7f5f440fedb0 msgr2=0x7f5f4406b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.316+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 shutdown_connections 2024-05-27T07:33:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:33:58.316+0000 7f5f4ac00700 1 -- 172.21.15.12:0/2377951110 wait complete. 2024-05-27T07:33:59.664 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:33:59 smithi012 ceph-mon[31473]: pgmap v400: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 933 KiB/s wr, 124 op/s 2024-05-27T07:33:59.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:33:59.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:33:59 smithi162 ceph-mon[35676]: pgmap v400: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 933 KiB/s wr, 124 op/s 2024-05-27T07:33:59.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:33:59 smithi062 ceph-mon[35304]: pgmap v400: 129 pgs: 129 active+clean; 186 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 933 KiB/s wr, 124 op/s 2024-05-27T07:34:00.170 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:01.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:01 smithi162 ceph-mon[35676]: pgmap v401: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:34:01.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:01 smithi062 ceph-mon[35304]: pgmap v401: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:34:01.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:01 smithi012 ceph-mon[31473]: pgmap v401: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:34:02.455 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:02 smithi012 ceph-mon[31473]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:34:02.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:02 smithi162 ceph-mon[35676]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:34:02.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:02 smithi062 ceph-mon[35304]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:34:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 -- 172.21.15.12:0/2284072783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 msgr2=0x7fb78010e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 --2- 172.21.15.12:0/2284072783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb78010e3b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009990 tx=0x7fb76c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 -- 172.21.15.12:0/2284072783 shutdown_connections 2024-05-27T07:34:02.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 --2- 172.21.15.12:0/2284072783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb78010e8f0 0x7fb780110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 --2- 172.21.15.12:0/2284072783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb780105510 0x7fb780105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 --2- 172.21.15.12:0/2284072783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb78010e3b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 -- 172.21.15.12:0/2284072783 >> 172.21.15.12:0/2284072783 conn(0x7fb7800fedb0 msgr2=0x7fb7801011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 -- 172.21.15.12:0/2284072783 shutdown_connections 2024-05-27T07:34:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.884+0000 7fb787a00700 1 -- 172.21.15.12:0/2284072783 wait complete. 2024-05-27T07:34:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.886+0000 7fb787a00700 1 Processor -- start 2024-05-27T07:34:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.886+0000 7fb787a00700 1 -- start start 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.887+0000 7fb787a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb780105510 0x7fb780103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.887+0000 7fb787a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb780103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.887+0000 7fb787a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb78010e8f0 0x7fb780103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.887+0000 7fb787a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb780114330 con 0x7fb780105e60 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.887+0000 7fb787a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801141b0 con 0x7fb78010e8f0 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.887+0000 7fb787a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801144b0 con 0x7fb780105510 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.888+0000 7fb785000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb780105510 0x7fb780103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.888+0000 7fb777e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb780103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.888+0000 7fb777e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb780103740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47190/0 (socket says 172.21.15.12:47190) 2024-05-27T07:34:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.888+0000 7fb777e00700 1 -- 172.21.15.12:0/1678347333 learned_addr learned my addr 172.21.15.12:0/1678347333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.889+0000 7fb777e00700 1 -- 172.21.15.12:0/1678347333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb780105510 msgr2=0x7fb780103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.889+0000 7fb777e00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb780105510 0x7fb780103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.889+0000 7fb777e00700 1 -- 172.21.15.12:0/1678347333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb78010e8f0 msgr2=0x7fb780103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.889+0000 7fb785a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb78010e8f0 0x7fb780103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.889+0000 7fb777e00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb78010e8f0 0x7fb780103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.889+0000 7fb777e00700 1 -- 172.21.15.12:0/1678347333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7801a9100 con 0x7fb780105e60 2024-05-27T07:34:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.890+0000 7fb785000700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb780105510 0x7fb780103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.890+0000 7fb785a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb78010e8f0 0x7fb780103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.890+0000 7fb777e00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb780103740 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb76c000c00 tx=0x7fb76c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.891+0000 7fb77e000700 1 -- 172.21.15.12:0/1678347333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c027070 con 0x7fb780105e60 2024-05-27T07:34:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.891+0000 7fb77e000700 1 -- 172.21.15.12:0/1678347333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb76c00fb80 con 0x7fb780105e60 2024-05-27T07:34:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.891+0000 7fb77e000700 1 -- 172.21.15.12:0/1678347333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c022c30 con 0x7fb780105e60 2024-05-27T07:34:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.891+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78019aac0 con 0x7fb780105e60 2024-05-27T07:34:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.891+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78019af50 con 0x7fb780105e60 2024-05-27T07:34:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.896+0000 7fb77e000700 1 -- 172.21.15.12:0/1678347333 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb76c02a070 con 0x7fb780105e60 2024-05-27T07:34:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.898+0000 7fb796481b80 1 Processor -- start 2024-05-27T07:34:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb796481b80 1 -- start start 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb796481b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d59b9f0 0x55726d62e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb77e000700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb7640b2a70 0x7fb7640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb796481b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d4afe40 0x55726d62e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb796481b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d63b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb796481b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55726d55ce40 con 0x55726d62ee70 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb796481b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55726d54c9e0 con 0x55726d59b9f0 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.899+0000 7fb796481b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55726d550a20 con 0x55726d4afe40 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.900+0000 7fb77e000700 1 -- 172.21.15.12:0/1678347333 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 9642+0+0 (secure 0 0 0) 0x7fb76c0e3a30 con 0x7fb780105e60 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.900+0000 7fb785000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d59b9f0 0x55726d62e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.900+0000 7fb777e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d4afe40 0x55726d62e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.900+0000 7fb785a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb7640b2a70 0x7fb7640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.900+0000 7fb785000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d59b9f0 0x55726d62e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:37126/0 (socket says 172.21.15.12:37126) 2024-05-27T07:34:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.900+0000 7fb785000700 1 -- 172.21.15.12:0/4072177359 learned_addr learned my addr 172.21.15.12:0/4072177359 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.900+0000 7fb785a00700 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d63b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.901+0000 7fb785000700 1 -- 172.21.15.12:0/4072177359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d4afe40 msgr2=0x55726d62e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.901+0000 7fb785000700 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d4afe40 0x55726d62e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.901+0000 7fb785000700 1 -- 172.21.15.12:0/4072177359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 msgr2=0x55726d63b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.901+0000 7fb785000700 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d63b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.901+0000 7fb785000700 1 -- 172.21.15.12:0/4072177359 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55726d515420 con 0x55726d59b9f0 2024-05-27T07:34:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.901+0000 7fb785a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb7640b2a70 0x7fb7640b4f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb778005950 tx=0x7fb77800b260 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.901+0000 7fb785000700 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d59b9f0 0x55726d62e330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb770009ee0 tx=0x7fb7700077e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8fcd542fa3a4d05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.902+0000 7fb776000700 1 -- 172.21.15.12:0/4072177359 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770013580 con 0x55726d59b9f0 2024-05-27T07:34:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.902+0000 7fb776000700 1 -- 172.21.15.12:0/4072177359 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb770019070 con 0x55726d59b9f0 2024-05-27T07:34:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.902+0000 7fb776000700 1 -- 172.21.15.12:0/4072177359 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770014990 con 0x55726d59b9f0 2024-05-27T07:34:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.903+0000 7fb796481b80 1 -- 172.21.15.12:0/4072177359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d59b9f0 msgr2=0x55726d62e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.903+0000 7fb796481b80 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d59b9f0 0x55726d62e330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb770009ee0 tx=0x7fb7700077e0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.903+0000 7fb796481b80 1 -- 172.21.15.12:0/4072177359 shutdown_connections 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.903+0000 7fb796481b80 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d59b9f0 0x55726d62e330 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.903+0000 7fb796481b80 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d63b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.903+0000 7fb796481b80 1 --2- 172.21.15.12:0/4072177359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d4afe40 0x55726d62e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.903+0000 7fb796481b80 1 -- 172.21.15.12:0/4072177359 >> 172.21.15.12:0/4072177359 conn(0x55726d549390 msgr2=0x55726d62a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.904+0000 7fb796481b80 1 -- 172.21.15.12:0/4072177359 shutdown_connections 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.904+0000 7fb796481b80 1 -- 172.21.15.12:0/4072177359 wait complete. 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.904+0000 7fb796481b80 1 -- start start 2024-05-27T07:34:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.905+0000 7fb796481b80 1 Processor -- start 2024-05-27T07:34:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.905+0000 7fb796481b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d6385d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb796481b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d59b9f0 0x55726d638b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb796481b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d62ee70 0x55726d639050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb796481b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55726d452fa0 con 0x55726d59b9f0 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb796481b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55726d4ed760 con 0x55726d4afe40 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb796481b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55726d63bb80 con 0x55726d62ee70 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb796481b80 10 client.0 _ll_register_callbacks cb 0x55726d4f6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb777e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d59b9f0 0x55726d638b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb777e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d59b9f0 0x55726d638b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47208/0 (socket says 172.21.15.12:47208) 2024-05-27T07:34:02.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb777e00700 1 -- 172.21.15.12:0/3317165061 learned_addr learned my addr 172.21.15.12:0/3317165061 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:02.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.906+0000 7fb785000700 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d6385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb785a00700 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d62ee70 0x55726d639050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb777e00700 1 -- 172.21.15.12:0/3317165061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d62ee70 msgr2=0x55726d639050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb777e00700 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d62ee70 0x55726d639050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb777e00700 1 -- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 msgr2=0x55726d6385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb777e00700 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d6385d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb777e00700 1 -- 172.21.15.12:0/3317165061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55726d49c8d0 con 0x55726d59b9f0 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb777e00700 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d59b9f0 0x55726d638b10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0e0c60 tx=0x7fb76c0aa2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc9f56bd4fa98cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb76aa00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.907+0000 7fb76aa00700 1 -- 172.21.15.12:0/3317165061 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0aafe0 con 0x55726d59b9f0 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.908+0000 7fb76aa00700 1 -- 172.21.15.12:0/3317165061 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9642+0+0 (secure 0 0 0) 0x7fb76c0a6ed0 con 0x55726d59b9f0 2024-05-27T07:34:02.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.908+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb770019070 con 0x55726d59b9f0 2024-05-27T07:34:02.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.908+0000 7fb76aa00700 1 -- 172.21.15.12:0/3317165061 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fb76c0aa8e0 con 0x55726d59b9f0 2024-05-27T07:34:02.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.908+0000 7fb796481b80 10 client.16857 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.908+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55726d51d060 con 0x55726d59b9f0 2024-05-27T07:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.908+0000 7fb76aa00700 1 -- 172.21.15.12:0/3317165061 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb770019070 con 0x55726d59b9f0 2024-05-27T07:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.908+0000 7fb76aa00700 1 -- 172.21.15.12:0/3317165061 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fb76c0e0f40 con 0x55726d59b9f0 2024-05-27T07:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.909+0000 7fb796481b80 10 client.16857 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:02.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.909+0000 7fb796481b80 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.909+0000 7fb796481b80 20 client.16857 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.909+0000 7fb796481b80 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d648b90 0x55726d64b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.909+0000 7fb796481b80 4 client.16857 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.909+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55726d55ed90 con 0x55726d648b90 2024-05-27T07:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.909+0000 7fb785000700 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d648b90 0x55726d64b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.910+0000 7fb785000700 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d648b90 0x55726d64b000 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ded80243f39ce8ba server_cookie=4e965e39a9b00054 in_seq=0 out_seq=0 2024-05-27T07:34:02.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.910+0000 7fb76aa00700 10 client.16857 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.912+0000 7fb76aa00700 1 -- 172.21.15.12:0/3317165061 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55726d55ed90 con 0x55726d648b90 2024-05-27T07:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.912+0000 7fb76aa00700 10 client.16857 handle_command_reply: tid=0 2024-05-27T07:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.912+0000 7fb796481b80 1 client.16857 shutdown 2024-05-27T07:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d59b9f0 msgr2=0x55726d638b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d59b9f0 0x55726d638b10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0e0c60 tx=0x7fb76c0aa2b0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 shutdown_connections 2024-05-27T07:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d648b90 0x55726d64b000 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d6385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d62ee70 0x55726d639050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 --2- 172.21.15.12:0/3317165061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d59b9f0 0x55726d638b10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb76aa00700 0 client.16857 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 >> 172.21.15.12:0/3317165061 conn(0x55726d549390 msgr2=0x55726d62e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:02.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 shutdown_connections 2024-05-27T07:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 1 -- 172.21.15.12:0/3317165061 wait complete. 2024-05-27T07:34:02.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.913+0000 7fb796481b80 20 client.16857 trim_cache size 0 max 16384 2024-05-27T07:34:02.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.933+0000 7fb796481b80 1 Processor -- start 2024-05-27T07:34:02.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb796481b80 1 -- start start 2024-05-27T07:34:02.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb796481b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb796481b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d644c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb796481b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d632030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb796481b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801b6390 con 0x55726d62ee70 2024-05-27T07:34:02.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb796481b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801b55f0 con 0x55726d4afe40 2024-05-27T07:34:02.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb796481b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801b5cc0 con 0x55726d59b9f0 2024-05-27T07:34:02.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.934+0000 7fb785a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d632030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d632030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47222/0 (socket says 172.21.15.12:47222) 2024-05-27T07:34:02.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785a00700 1 -- 172.21.15.12:0/910954152 learned_addr learned my addr 172.21.15.12:0/910954152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:02.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785000700 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb777e00700 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d644c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785a00700 1 -- 172.21.15.12:0/910954152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 msgr2=0x55726d644c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785a00700 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d644c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785a00700 1 -- 172.21.15.12:0/910954152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 msgr2=0x55726d633df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785a00700 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785a00700 1 -- 172.21.15.12:0/910954152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55726d6393c0 con 0x55726d62ee70 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb777e00700 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d644c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.935+0000 7fb785000700 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.936+0000 7fb785a00700 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d632030 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb778013fc0 tx=0x7fb778015050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2ee46d447592e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.936+0000 7fb775600700 1 -- 172.21.15.12:0/910954152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77802c070 con 0x55726d62ee70 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.936+0000 7fb775600700 1 -- 172.21.15.12:0/910954152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb77800ee20 con 0x55726d62ee70 2024-05-27T07:34:02.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.936+0000 7fb775600700 1 -- 172.21.15.12:0/910954152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778027720 con 0x55726d62ee70 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 -- 172.21.15.12:0/910954152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 msgr2=0x55726d632030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d632030 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb778013fc0 tx=0x7fb778015050 comp rx=0 tx=0).stop 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 -- 172.21.15.12:0/910954152 shutdown_connections 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d632030 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 --2- 172.21.15.12:0/910954152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d644c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 -- 172.21.15.12:0/910954152 >> 172.21.15.12:0/910954152 conn(0x55726d549390 msgr2=0x55726d62e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 -- 172.21.15.12:0/910954152 shutdown_connections 2024-05-27T07:34:02.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.937+0000 7fb796481b80 1 -- 172.21.15.12:0/910954152 wait complete. 2024-05-27T07:34:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.938+0000 7fb796481b80 1 -- start start 2024-05-27T07:34:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.938+0000 7fb796481b80 1 Processor -- start 2024-05-27T07:34:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.938+0000 7fb796481b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb796481b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d633740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb796481b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d633c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb796481b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55726d636b40 con 0x55726d62ee70 2024-05-27T07:34:02.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb796481b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55726d62b000 con 0x55726d4afe40 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb796481b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55726d62f800 con 0x55726d59b9f0 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb796481b80 10 client.0 _ll_register_callbacks cb 0x55726d4f6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb785000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb785000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:37154/0 (socket says 172.21.15.12:37154) 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb785000700 1 -- 172.21.15.12:0/2181097411 learned_addr learned my addr 172.21.15.12:0/2181097411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb777e00700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d633740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.939+0000 7fb785a00700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d633c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.940+0000 7fb785000700 1 -- 172.21.15.12:0/2181097411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 msgr2=0x55726d633740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.940+0000 7fb785000700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d633740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.940+0000 7fb785000700 1 -- 172.21.15.12:0/2181097411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 msgr2=0x55726d633c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.940+0000 7fb785000700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d633c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.940+0000 7fb785000700 1 -- 172.21.15.12:0/2181097411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55726d63f4a0 con 0x55726d4afe40 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.940+0000 7fb777e00700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d633740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.940+0000 7fb785a00700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d633c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.941+0000 7fb785000700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e00 tx=0x7fb7700093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=853522883209e77e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.941+0000 7fb76aa00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.941+0000 7fb76aa00700 1 -- 172.21.15.12:0/2181097411 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770004d60 con 0x55726d4afe40 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.941+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb77800ee20 con 0x55726d4afe40 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.941+0000 7fb76aa00700 1 -- 172.21.15.12:0/2181097411 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9642+0+0 (secure 0 0 0) 0x7fb770027330 con 0x55726d4afe40 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.942+0000 7fb76aa00700 1 -- 172.21.15.12:0/2181097411 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fb770027530 con 0x55726d4afe40 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.942+0000 7fb796481b80 10 client.36329 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:02.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.942+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55726d6b3a00 con 0x55726d4afe40 2024-05-27T07:34:02.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.942+0000 7fb76aa00700 1 -- 172.21.15.12:0/2181097411 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb77800ee20 con 0x55726d4afe40 2024-05-27T07:34:02.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.942+0000 7fb76aa00700 1 -- 172.21.15.12:0/2181097411 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fb770014980 con 0x55726d4afe40 2024-05-27T07:34:02.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.942+0000 7fb796481b80 10 client.36329 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:02.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.942+0000 7fb796481b80 10 client.36329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:02.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.943+0000 7fb796481b80 20 client.36329 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:02.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.943+0000 7fb796481b80 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d6b4110 0x55726d6b6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:02.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.943+0000 7fb796481b80 4 client.36329 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:02.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.943+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55726d629120 con 0x55726d6b4110 2024-05-27T07:34:02.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.943+0000 7fb777e00700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d6b4110 0x55726d6b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:02.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.944+0000 7fb777e00700 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d6b4110 0x55726d6b6530 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7f2d2d9ce1a412 server_cookie=4cafe6342da67c4a in_seq=0 out_seq=0 2024-05-27T07:34:02.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.944+0000 7fb76aa00700 10 client.36329 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.946+0000 7fb76aa00700 1 -- 172.21.15.12:0/2181097411 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55726d629120 con 0x55726d6b4110 2024-05-27T07:34:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.946+0000 7fb76aa00700 10 client.36329 handle_command_reply: tid=0 2024-05-27T07:34:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.946+0000 7fb796481b80 1 client.36329 shutdown 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.946+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 msgr2=0x55726d633200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.946+0000 7fb796481b80 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e00 tx=0x7fb7700093d0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 shutdown_connections 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55726d6b4110 0x55726d6b6530 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55726d4afe40 0x55726d633200 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55726d62ee70 0x55726d633c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 --2- 172.21.15.12:0/2181097411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55726d59b9f0 0x55726d633740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 >> 172.21.15.12:0/2181097411 conn(0x55726d549390 msgr2=0x55726d62e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb76aa00700 0 client.36329 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 shutdown_connections 2024-05-27T07:34:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 1 -- 172.21.15.12:0/2181097411 wait complete. 2024-05-27T07:34:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.947+0000 7fb796481b80 20 client.36329 trim_cache size 0 max 16384 2024-05-27T07:34:02.952 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb7640b2a70 msgr2=0x7fb7640b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb7640b2a70 0x7fb7640b4f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb778005950 tx=0x7fb77800b260 comp rx=0 tx=0).stop 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 msgr2=0x7fb780103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb780103740 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb76c000c00 tx=0x7fb76c005ef0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 shutdown_connections 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb7640b2a70 0x7fb7640b4f80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb78010e8f0 0x7fb780103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb780105510 0x7fb780103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 --2- 172.21.15.12:0/1678347333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb780105e60 0x7fb780103740 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.948+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 >> 172.21.15.12:0/1678347333 conn(0x7fb7800fedb0 msgr2=0x7fb7800ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.949+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 shutdown_connections 2024-05-27T07:34:02.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:02.949+0000 7fb787a00700 1 -- 172.21.15.12:0/1678347333 wait complete. 2024-05-27T07:34:03.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:03 smithi162 ceph-mon[35676]: pgmap v403: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:34:03.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:03 smithi062 ceph-mon[35304]: pgmap v403: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:34:03.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:03 smithi012 ceph-mon[31473]: pgmap v403: 129 pgs: 129 active+clean; 190 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-05-27T07:34:05.081 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:34:05.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:34:05.584 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:05.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:05 smithi162 ceph-mon[35676]: pgmap v404: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:34:05.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:05 smithi062 ceph-mon[35304]: pgmap v404: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:34:05.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:05 smithi012 ceph-mon[31473]: pgmap v404: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:34:07.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:07 smithi012 ceph-mon[31473]: pgmap v405: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:34:07.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:07 smithi162 ceph-mon[35676]: pgmap v405: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:34:07.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:07 smithi062 ceph-mon[35304]: pgmap v405: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-27T07:34:07.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:07.998+0000 7f75d1200700 1 -- 172.21.15.12:0/889042382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 msgr2=0x7f75cc104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:07.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:07.998+0000 7f75d1200700 1 --2- 172.21.15.12:0/889042382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc104800 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00b0d0 tx=0x7f75bc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.002+0000 7f75d1200700 1 -- 172.21.15.12:0/889042382 shutdown_connections 2024-05-27T07:34:07.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.002+0000 7f75d1200700 1 --2- 172.21.15.12:0/889042382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75cc109670 0x7f75cc110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.002+0000 7f75d1200700 1 --2- 172.21.15.12:0/889042382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc104800 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.002+0000 7f75d1200700 1 --2- 172.21.15.12:0/889042382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75cc108d20 0x7f75cc109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:07.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.002+0000 7f75d1200700 1 -- 172.21.15.12:0/889042382 >> 172.21.15.12:0/889042382 conn(0x7f75cc079d00 msgr2=0x7f75cc07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:07.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.002+0000 7f75d1200700 1 -- 172.21.15.12:0/889042382 shutdown_connections 2024-05-27T07:34:07.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.002+0000 7f75d1200700 1 -- 172.21.15.12:0/889042382 wait complete. 2024-05-27T07:34:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.004+0000 7f75d1200700 1 Processor -- start 2024-05-27T07:34:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.004+0000 7f75d1200700 1 -- start start 2024-05-27T07:34:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75d1200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75d1200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75cc108d20 0x7f75cc19d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75d1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75cc109670 0x7f75cc1a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75d1200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc114150 con 0x7f75cc1043f0 2024-05-27T07:34:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75d1200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc113fd0 con 0x7f75cc109670 2024-05-27T07:34:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75d1200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc1142d0 con 0x7f75cc108d20 2024-05-27T07:34:08.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75cb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47250/0 (socket says 172.21.15.12:47250) 2024-05-27T07:34:08.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.005+0000 7f75cb400700 1 -- 172.21.15.12:0/4262624400 learned_addr learned my addr 172.21.15.12:0/4262624400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:08.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.006+0000 7f75cbe00700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75cc109670 0x7f75cc1a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.006+0000 7f75c2a00700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75cc108d20 0x7f75cc19d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.006+0000 7f75cb400700 1 -- 172.21.15.12:0/4262624400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75cc108d20 msgr2=0x7f75cc19d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.006+0000 7f75cb400700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75cc108d20 0x7f75cc19d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.006+0000 7f75cb400700 1 -- 172.21.15.12:0/4262624400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75cc109670 msgr2=0x7f75cc1a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.006+0000 7f75cb400700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75cc109670 0x7f75cc1a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.006+0000 7f75cb400700 1 -- 172.21.15.12:0/4262624400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75cc1a1cd0 con 0x7f75cc1043f0 2024-05-27T07:34:08.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.007+0000 7f75cb400700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc19ce00 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f75bc007d90 tx=0x7f75bc007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:08.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.007+0000 7f75c8c00700 1 -- 172.21.15.12:0/4262624400 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc01f070 con 0x7f75cc1043f0 2024-05-27T07:34:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.007+0000 7f75c8c00700 1 -- 172.21.15.12:0/4262624400 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75bc005500 con 0x7f75cc1043f0 2024-05-27T07:34:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.007+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75cc1a1f60 con 0x7f75cc1043f0 2024-05-27T07:34:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.007+0000 7f75c8c00700 1 -- 172.21.15.12:0/4262624400 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc01b860 con 0x7f75cc1043f0 2024-05-27T07:34:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.007+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75cc1a2440 con 0x7f75cc1043f0 2024-05-27T07:34:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.014+0000 7f75c8c00700 1 -- 172.21.15.12:0/4262624400 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f75bc022070 con 0x7f75cc1043f0 2024-05-27T07:34:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.014+0000 7f75e13f0b80 1 Processor -- start 2024-05-27T07:34:08.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.014+0000 7f75e13f0b80 1 -- start start 2024-05-27T07:34:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0e59ec0 0x564de0fd7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fd8490 0x564de0fe2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 0x564de0fe50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564de0f06ec0 con 0x564de0e59ec0 2024-05-27T07:34:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564de0ef6a60 con 0x564de0fd88a0 2024-05-27T07:34:08.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564de0efaaa0 con 0x564de0fd8490 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0e59ec0 0x564de0fd7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0e59ec0 0x564de0fd7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47256/0 (socket says 172.21.15.12:47256) 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.015+0000 7f75cbe00700 1 -- 172.21.15.12:0/3104991495 learned_addr learned my addr 172.21.15.12:0/3104991495 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75cb400700 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fd8490 0x564de0fe2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75cbe00700 1 -- 172.21.15.12:0/3104991495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fd8490 msgr2=0x564de0fe2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fd8490 0x564de0fe2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75cbe00700 1 -- 172.21.15.12:0/3104991495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 msgr2=0x564de0fe50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 0x564de0fe50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75cbe00700 1 -- 172.21.15.12:0/3104991495 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564de0ebf4a0 con 0x564de0e59ec0 2024-05-27T07:34:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75c8c00700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f75b00b29f0 0x7f75b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.016+0000 7f75c8c00700 1 -- 172.21.15.12:0/4262624400 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f75bc00f020 con 0x7f75cc1043f0 2024-05-27T07:34:08.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.017+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0e59ec0 0x564de0fd7f50 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f75c40046a0 tx=0x7f75c400b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7386753b32dd1472 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:08.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.017+0000 7f75cb400700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f75b00b29f0 0x7f75b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.017+0000 7f75c0c00700 1 -- 172.21.15.12:0/3104991495 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4004a30 con 0x564de0e59ec0 2024-05-27T07:34:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.017+0000 7f75c0c00700 1 -- 172.21.15.12:0/3104991495 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75c4015070 con 0x564de0e59ec0 2024-05-27T07:34:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.017+0000 7f75c0c00700 1 -- 172.21.15.12:0/3104991495 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c40039d0 con 0x564de0e59ec0 2024-05-27T07:34:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.018+0000 7f75cb400700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f75b00b29f0 0x7f75b00b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0e1310 tx=0x7f75bc006080 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.018+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3104991495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0e59ec0 msgr2=0x564de0fd7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.018+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0e59ec0 0x564de0fd7f50 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f75c40046a0 tx=0x7f75c400b970 comp rx=0 tx=0).stop 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.018+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3104991495 shutdown_connections 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.018+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 0x564de0fe50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.018+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0e59ec0 0x564de0fd7f50 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.019+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3104991495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fd8490 0x564de0fe2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.019+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3104991495 >> 172.21.15.12:0/3104991495 conn(0x564de0eba040 msgr2=0x564de0fd3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.019+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3104991495 shutdown_connections 2024-05-27T07:34:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.019+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3104991495 wait complete. 2024-05-27T07:34:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.019+0000 7f75e13f0b80 1 -- start start 2024-05-27T07:34:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.020+0000 7f75e13f0b80 1 Processor -- start 2024-05-27T07:34:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.020+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0e59ec0 0x564de0fe2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.020+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd8490 0x564de0fe2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.020+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 0x564de0fe30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.020+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564de0dfd020 con 0x564de0fd8490 2024-05-27T07:34:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.020+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564de0e977e0 con 0x564de0fd88a0 2024-05-27T07:34:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.020+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564de0fe5a10 con 0x564de0e59ec0 2024-05-27T07:34:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.021+0000 7f75e13f0b80 10 client.0 _ll_register_callbacks cb 0x564de0ea0cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.021+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd8490 0x564de0fe2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.021+0000 7f75cbe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0e59ec0 0x564de0fe2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.021+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd8490 0x564de0fe2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47268/0 (socket says 172.21.15.12:47268) 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.021+0000 7f75c2a00700 1 -- 172.21.15.12:0/3657327108 learned_addr learned my addr 172.21.15.12:0/3657327108 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.021+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 0x564de0fe30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.022+0000 7f75c2a00700 1 -- 172.21.15.12:0/3657327108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0e59ec0 msgr2=0x564de0fe2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.022+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0e59ec0 0x564de0fe2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.022+0000 7f75c2a00700 1 -- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 msgr2=0x564de0fe30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.022+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 0x564de0fe30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.022+0000 7f75c2a00700 1 -- 172.21.15.12:0/3657327108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564de0e46950 con 0x564de0fd8490 2024-05-27T07:34:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.022+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd8490 0x564de0fe2bb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f75b400dda0 tx=0x7f75b400f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f81a17e711ed939 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.022+0000 7f75b9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.023+0000 7f75b9600700 1 -- 172.21.15.12:0/3657327108 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4011440 con 0x564de0fd8490 2024-05-27T07:34:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.023+0000 7f75b9600700 1 -- 172.21.15.12:0/3657327108 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f75b40115e0 con 0x564de0fd8490 2024-05-27T07:34:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.023+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75c4015070 con 0x564de0fd8490 2024-05-27T07:34:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.023+0000 7f75b9600700 1 -- 172.21.15.12:0/3657327108 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f75b4023050 con 0x564de0fd8490 2024-05-27T07:34:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.023+0000 7f75e13f0b80 10 client.16875 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x564de0ec70e0 con 0x564de0fd8490 2024-05-27T07:34:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75b9600700 1 -- 172.21.15.12:0/3657327108 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75c4015070 con 0x564de0fd8490 2024-05-27T07:34:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75b9600700 1 -- 172.21.15.12:0/3657327108 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f75b40107f0 con 0x564de0fd8490 2024-05-27T07:34:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75e13f0b80 10 client.16875 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75e13f0b80 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75e13f0b80 20 client.16875 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de0ff2de0 0x564de0ff5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75e13f0b80 4 client.16875 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.024+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564de0f08e10 con 0x564de0ff2de0 2024-05-27T07:34:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.025+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de0ff2de0 0x564de0ff5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.026+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de0ff2de0 0x564de0ff5230 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=980ae3e605c6f9fc server_cookie=fa52637104b76491 in_seq=0 out_seq=0 2024-05-27T07:34:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.026+0000 7f75b9600700 10 client.16875 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.028+0000 7f75b9600700 1 -- 172.21.15.12:0/3657327108 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564de0f08e10 con 0x564de0ff2de0 2024-05-27T07:34:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.028+0000 7f75b9600700 10 client.16875 handle_command_reply: tid=0 2024-05-27T07:34:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.028+0000 7f75e13f0b80 1 client.16875 shutdown 2024-05-27T07:34:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.028+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd8490 msgr2=0x564de0fe2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.028+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd8490 0x564de0fe2bb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f75b400dda0 tx=0x7f75b400f640 comp rx=0 tx=0).stop 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 shutdown_connections 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de0ff2de0 0x564de0ff5230 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0fd88a0 0x564de0fe30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0e59ec0 0x564de0fe2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3657327108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd8490 0x564de0fe2bb0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 >> 172.21.15.12:0/3657327108 conn(0x564de0eba040 msgr2=0x564de0fd3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75b9600700 0 client.16875 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 shutdown_connections 2024-05-27T07:34:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3657327108 wait complete. 2024-05-27T07:34:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.029+0000 7f75e13f0b80 20 client.16875 trim_cache size 0 max 16384 2024-05-27T07:34:08.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.054+0000 7f75e13f0b80 1 Processor -- start 2024-05-27T07:34:08.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.054+0000 7f75e13f0b80 1 -- start start 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fdde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd88a0 0x564de0fd3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fda3a0 0x564de0fe2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc1b1c00 con 0x564de0fd88a0 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc1b0e60 con 0x564de0e59ec0 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc1b1530 con 0x564de0fda3a0 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd88a0 0x564de0fd3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd88a0 0x564de0fd3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47284/0 (socket says 172.21.15.12:47284) 2024-05-27T07:34:08.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75c2a00700 1 -- 172.21.15.12:0/3844724856 learned_addr learned my addr 172.21.15.12:0/3844724856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:08.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.055+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fdde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fda3a0 0x564de0fe2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75c2a00700 1 -- 172.21.15.12:0/3844724856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fda3a0 msgr2=0x564de0fe2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fda3a0 0x564de0fe2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75c2a00700 1 -- 172.21.15.12:0/3844724856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 msgr2=0x564de0fdde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fdde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75c2a00700 1 -- 172.21.15.12:0/3844724856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564de0fd53d0 con 0x564de0fd88a0 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fda3a0 0x564de0fe2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd88a0 0x564de0fd3eb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f75b4010c40 tx=0x7f75b4005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae283ad6edfac22 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75bbe00700 1 -- 172.21.15.12:0/3844724856 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4025cb0 con 0x564de0fd88a0 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75bbe00700 1 -- 172.21.15.12:0/3844724856 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75b4017070 con 0x564de0fd88a0 2024-05-27T07:34:08.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.056+0000 7f75bbe00700 1 -- 172.21.15.12:0/3844724856 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4010750 con 0x564de0fd88a0 2024-05-27T07:34:08.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.057+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3844724856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd88a0 msgr2=0x564de0fd3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.057+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd88a0 0x564de0fd3eb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f75b4010c40 tx=0x7f75b4005720 comp rx=0 tx=0).stop 2024-05-27T07:34:08.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3844724856 shutdown_connections 2024-05-27T07:34:08.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fdde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fda3a0 0x564de0fe2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3844724856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fd88a0 0x564de0fd3eb0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3844724856 >> 172.21.15.12:0/3844724856 conn(0x564de0eba040 msgr2=0x564de0fe2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3844724856 shutdown_connections 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3844724856 wait complete. 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.058+0000 7f75e13f0b80 1 -- start start 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.059+0000 7f75e13f0b80 1 Processor -- start 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.059+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fe7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.059+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fda3a0 0x564de0fe8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.059+0000 7f75e13f0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fe87a0 0x564de10660e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.059+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564de0f08e10 con 0x564de0fda3a0 2024-05-27T07:34:08.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.059+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564de0fd4990 con 0x564de0e59ec0 2024-05-27T07:34:08.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.059+0000 7f75e13f0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564de0fe39c0 con 0x564de0fe87a0 2024-05-27T07:34:08.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.060+0000 7f75e13f0b80 10 client.0 _ll_register_callbacks cb 0x564de0ea0cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:08.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.060+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fda3a0 0x564de0fe8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.060+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fda3a0 0x564de0fe8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47300/0 (socket says 172.21.15.12:47300) 2024-05-27T07:34:08.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.060+0000 7f75c2a00700 1 -- 172.21.15.12:0/3225238823 learned_addr learned my addr 172.21.15.12:0/3225238823 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.060+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fe7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.060+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fe87a0 0x564de10660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75c2a00700 1 -- 172.21.15.12:0/3225238823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fe87a0 msgr2=0x564de10660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fe87a0 0x564de10660e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75c2a00700 1 -- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 msgr2=0x564de0fe7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fe7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75c2a00700 1 -- 172.21.15.12:0/3225238823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564de10667c0 con 0x564de0fda3a0 2024-05-27T07:34:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75c2a00700 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fda3a0 0x564de0fe8260 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f75b400e010 tx=0x7f75b400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863e4e132ff89fbc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75b9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.061+0000 7f75b9600700 1 -- 172.21.15.12:0/3225238823 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4034070 con 0x564de0fda3a0 2024-05-27T07:34:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.062+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b4017070 con 0x564de0fda3a0 2024-05-27T07:34:08.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.062+0000 7f75b9600700 1 -- 172.21.15.12:0/3225238823 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f75b4004240 con 0x564de0fda3a0 2024-05-27T07:34:08.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.062+0000 7f75b9600700 1 -- 172.21.15.12:0/3225238823 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f75b403e050 con 0x564de0fda3a0 2024-05-27T07:34:08.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.062+0000 7f75e13f0b80 10 client.16881 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:08.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.062+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x564de10679b0 con 0x564de0fda3a0 2024-05-27T07:34:08.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75b9600700 1 -- 172.21.15.12:0/3225238823 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f75b4017070 con 0x564de0fda3a0 2024-05-27T07:34:08.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75b9600700 1 -- 172.21.15.12:0/3225238823 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f75b40252e0 con 0x564de0fda3a0 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75e13f0b80 10 client.16881 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75e13f0b80 10 client.16881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75e13f0b80 20 client.16881 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de1067f40 0x564de106a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75e13f0b80 4 client.16881 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.063+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564de0f025f0 con 0x564de1067f40 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.064+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de1067f40 0x564de106a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:08.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.065+0000 7f75cbe00700 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de1067f40 0x564de106a360 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fa9f902e18f4878 server_cookie=11846bdd5379da26 in_seq=0 out_seq=0 2024-05-27T07:34:08.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.065+0000 7f75b9600700 10 client.16881 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:08.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75b9600700 1 -- 172.21.15.12:0/3225238823 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564de0f025f0 con 0x564de1067f40 2024-05-27T07:34:08.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75b9600700 10 client.16881 handle_command_reply: tid=0 2024-05-27T07:34:08.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 client.16881 shutdown 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fda3a0 msgr2=0x564de0fe8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fda3a0 0x564de0fe8260 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f75b400e010 tx=0x7f75b400d4a0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 shutdown_connections 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564de1067f40 0x564de106a360 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564de0e59ec0 0x564de0fe7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564de0fe87a0 0x564de10660e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 --2- 172.21.15.12:0/3225238823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564de0fda3a0 0x564de0fe8260 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75b9600700 0 client.16881 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.069+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 >> 172.21.15.12:0/3225238823 conn(0x564de0eba040 msgr2=0x564de0fe2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.070+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 shutdown_connections 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.070+0000 7f75e13f0b80 1 -- 172.21.15.12:0/3225238823 wait complete. 2024-05-27T07:34:08.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.070+0000 7f75e13f0b80 20 client.16881 trim_cache size 0 max 16384 2024-05-27T07:34:08.066 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:34:08.066 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:34:08.066 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "7981795c-8a84-45a0-a083-64a9f0cca5b9", 2024-05-27T07:34:08.066 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:34:08.066 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f75b00b29f0 msgr2=0x7f75b00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f75b00b29f0 0x7f75b00b4f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0e1310 tx=0x7f75bc006080 comp rx=0 tx=0).stop 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 msgr2=0x7f75cc19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc19ce00 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f75bc007d90 tx=0x7f75bc007dc0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 shutdown_connections 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f75b00b29f0 0x7f75b00b4f00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75cc109670 0x7f75cc1a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75cc1043f0 0x7f75cc19ce00 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 --2- 172.21.15.12:0/4262624400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75cc108d20 0x7f75cc19d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:08.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 >> 172.21.15.12:0/4262624400 conn(0x7f75cc079d00 msgr2=0x7f75cc101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:08.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 shutdown_connections 2024-05-27T07:34:08.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:08.071+0000 7f75d1200700 1 -- 172.21.15.12:0/4262624400 wait complete. 2024-05-27T07:34:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:08 smithi012 ceph-mon[31473]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:34:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:08 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:34:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:08 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:08.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:08 smithi012 ceph-mon[31473]: pgmap v407: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 940 KiB/s wr, 122 op/s 2024-05-27T07:34:08.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:34:08.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:08 smithi162 ceph-mon[35676]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:34:08.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:08 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:34:08.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:08 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:08.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:08 smithi162 ceph-mon[35676]: pgmap v407: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 940 KiB/s wr, 122 op/s 2024-05-27T07:34:08.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:08 smithi062 ceph-mon[35304]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:34:08.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:08 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:34:08.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:08 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:08.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:08 smithi062 ceph-mon[35304]: pgmap v407: 129 pgs: 129 active+clean; 200 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 940 KiB/s wr, 122 op/s 2024-05-27T07:34:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:09.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:09 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:34:09.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:09 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:09.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:09 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:34:09.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:09 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:34:09.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:09 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:09.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:09 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:34:09.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:09 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:09.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:09 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:34:09.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:09 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:34:09.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:09 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:09.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:09 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:34:09.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:09 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:09.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:09 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:34:09.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:09 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:34:09.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:09 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:10.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:10 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:10.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:10 smithi012 ceph-mon[31473]: pgmap v408: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-05-27T07:34:10.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:10 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:10.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:10 smithi162 ceph-mon[35676]: pgmap v408: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-05-27T07:34:10.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:10 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:10.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:10 smithi062 ceph-mon[35304]: pgmap v408: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-05-27T07:34:11.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:11 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:11.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:11 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 -- 172.21.15.12:0/1880286229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34104600 msgr2=0x7f9c34104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/1880286229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34104600 0x7f9c34104a10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9c280099c0 tx=0x7f9c28009cd0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 -- 172.21.15.12:0/1880286229 shutdown_connections 2024-05-27T07:34:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/1880286229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c34109880 0x7f9c341111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/1880286229 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34104600 0x7f9c34104a10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/1880286229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c34108f30 0x7f9c34109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 -- 172.21.15.12:0/1880286229 >> 172.21.15.12:0/1880286229 conn(0x7f9c3407af60 msgr2=0x7f9c3407b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:11.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.988+0000 7f9c3bc00700 1 -- 172.21.15.12:0/1880286229 shutdown_connections 2024-05-27T07:34:11.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.989+0000 7f9c3bc00700 1 -- 172.21.15.12:0/1880286229 wait complete. 2024-05-27T07:34:11.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.990+0000 7f9c3bc00700 1 Processor -- start 2024-05-27T07:34:11.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.990+0000 7f9c3bc00700 1 -- start start 2024-05-27T07:34:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.991+0000 7f9c3bc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c34104600 0x7f9c3419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.991+0000 7f9c3bc00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c34108f30 0x7f9c3419d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.991+0000 7f9c3bc00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34109880 0x7f9c341a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.991+0000 7f9c3bc00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9c341143b0 con 0x7f9c34109880 2024-05-27T07:34:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.991+0000 7f9c3bc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c34114230 con 0x7f9c34104600 2024-05-27T07:34:11.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.991+0000 7f9c3bc00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c34114530 con 0x7f9c34108f30 2024-05-27T07:34:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34109880 0x7f9c341a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34109880 0x7f9c341a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37020/0 (socket says 172.21.15.12:37020) 2024-05-27T07:34:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 -- 172.21.15.12:0/665148831 learned_addr learned my addr 172.21.15.12:0/665148831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39200700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c34104600 0x7f9c3419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:11.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c33e00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c34108f30 0x7f9c3419d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 -- 172.21.15.12:0/665148831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c34108f30 msgr2=0x7f9c3419d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c34108f30 0x7f9c3419d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 -- 172.21.15.12:0/665148831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c34104600 msgr2=0x7f9c3419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c34104600 0x7f9c3419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.992+0000 7f9c39c00700 1 -- 172.21.15.12:0/665148831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c341a1e30 con 0x7f9c34109880 2024-05-27T07:34:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.993+0000 7f9c39c00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34109880 0x7f9c341a1790 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00ef30 tx=0x7f9c2c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:11.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.993+0000 7f9c31600700 1 -- 172.21.15.12:0/665148831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0092e0 con 0x7f9c34109880 2024-05-27T07:34:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.994+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c341a20c0 con 0x7f9c34109880 2024-05-27T07:34:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.994+0000 7f9c31600700 1 -- 172.21.15.12:0/665148831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c2c004bb0 con 0x7f9c34109880 2024-05-27T07:34:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.994+0000 7f9c31600700 1 -- 172.21.15.12:0/665148831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c004020 con 0x7f9c34109880 2024-05-27T07:34:11.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.994+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c341a9a40 con 0x7f9c34109880 2024-05-27T07:34:11.989 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:11 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:11.998+0000 7f9c31600700 1 -- 172.21.15.12:0/665148831 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9c2c01b030 con 0x7f9c34109880 2024-05-27T07:34:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.000+0000 7f9c31600700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9c1c0b2a70 0x7f9c1c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.000+0000 7f9c31600700 1 -- 172.21.15.12:0/665148831 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f9c2c0e2470 con 0x7f9c34109880 2024-05-27T07:34:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.001+0000 7f9c39200700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9c1c0b2a70 0x7f9c1c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.001+0000 7f9c4a801b80 1 Processor -- start 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.001+0000 7f9c4a801b80 1 -- start start 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.002+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91059f0 0x5638a91a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.002+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a9019e40 0x5638a91a1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.002+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91ad900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.002+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5638a90c6e40 con 0x5638a91059f0 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.002+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638a90b69e0 con 0x5638a91a1670 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.002+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5638a90baa20 con 0x5638a9019e40 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c33e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a9019e40 0x5638a91a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91059f0 0x5638a91a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91059f0 0x5638a91a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37026/0 (socket says 172.21.15.12:37026) 2024-05-27T07:34:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 -- 172.21.15.12:0/961794307 learned_addr learned my addr 172.21.15.12:0/961794307 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9c1c0b2a70 0x7f9c1c0b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c28003760 tx=0x7f9c2801a040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 -- 172.21.15.12:0/961794307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a9019e40 msgr2=0x5638a91a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 --2- 172.21.15.12:0/961794307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a9019e40 0x5638a91a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 -- 172.21.15.12:0/961794307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 msgr2=0x5638a91ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:11.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 --2- 172.21.15.12:0/961794307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:11.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.003+0000 7f9c39200700 1 -- 172.21.15.12:0/961794307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a907f420 con 0x5638a91059f0 2024-05-27T07:34:11.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.004+0000 7f9c39200700 1 --2- 172.21.15.12:0/961794307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91059f0 0x5638a91a0b30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9c280099c0 tx=0x7f9c28009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adab2563479d36f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:11.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.004+0000 7f9c26000700 1 -- 172.21.15.12:0/961794307 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28004ad0 con 0x5638a91059f0 2024-05-27T07:34:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.005+0000 7f9c26000700 1 -- 172.21.15.12:0/961794307 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c28004c70 con 0x5638a91059f0 2024-05-27T07:34:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.005+0000 7f9c26000700 1 -- 172.21.15.12:0/961794307 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c280267e0 con 0x5638a91059f0 2024-05-27T07:34:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.006+0000 7f9c4a801b80 1 -- 172.21.15.12:0/961794307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91059f0 msgr2=0x5638a91a0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.006+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/961794307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91059f0 0x5638a91a0b30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9c280099c0 tx=0x7f9c28009cd0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.006+0000 7f9c4a801b80 1 -- 172.21.15.12:0/961794307 shutdown_connections 2024-05-27T07:34:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.006+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/961794307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.006+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/961794307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91059f0 0x5638a91a0b30 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.006+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/961794307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a9019e40 0x5638a91a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.007+0000 7f9c4a801b80 1 -- 172.21.15.12:0/961794307 >> 172.21.15.12:0/961794307 conn(0x5638a90b3390 msgr2=0x5638a9194380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.007+0000 7f9c4a801b80 1 -- 172.21.15.12:0/961794307 shutdown_connections 2024-05-27T07:34:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.007+0000 7f9c4a801b80 1 -- 172.21.15.12:0/961794307 wait complete. 2024-05-27T07:34:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.008+0000 7f9c4a801b80 1 -- start start 2024-05-27T07:34:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.008+0000 7f9c4a801b80 1 Processor -- start 2024-05-27T07:34:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.009+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.009+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91ab430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.009+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91ab970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.009+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5638a8fbcfa0 con 0x5638a91a1670 2024-05-27T07:34:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.009+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638a9057760 con 0x5638a9019e40 2024-05-27T07:34:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.009+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5638a91ae340 con 0x5638a91059f0 2024-05-27T07:34:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.010+0000 7f9c4a801b80 10 client.0 _ll_register_callbacks cb 0x5638a9060c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.010+0000 7f9c39c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.010+0000 7f9c39c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91ab970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37028/0 (socket says 172.21.15.12:37028) 2024-05-27T07:34:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.010+0000 7f9c39200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.010+0000 7f9c39c00700 1 -- 172.21.15.12:0/3013380222 learned_addr learned my addr 172.21.15.12:0/3013380222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.010+0000 7f9c33e00700 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.011+0000 7f9c39200700 1 -- 172.21.15.12:0/3013380222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 msgr2=0x5638a91ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.011+0000 7f9c39200700 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.011+0000 7f9c39200700 1 -- 172.21.15.12:0/3013380222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 msgr2=0x5638a91ab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.011+0000 7f9c39200700 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91ab970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.011+0000 7f9c39200700 1 -- 172.21.15.12:0/3013380222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a90068d0 con 0x5638a9019e40 2024-05-27T07:34:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.011+0000 7f9c39c00700 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91ab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.012+0000 7f9c39200700 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91aaef0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c28000c00 tx=0x7f9c28026d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ffab7bf0abd450 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.012+0000 7f9c1aa00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:34:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.013+0000 7f9c1aa00700 1 -- 172.21.15.12:0/3013380222 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2800b790 con 0x5638a9019e40 2024-05-27T07:34:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.013+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c28004c70 con 0x5638a9019e40 2024-05-27T07:34:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.013+0000 7f9c1aa00700 1 -- 172.21.15.12:0/3013380222 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f9c2800b930 con 0x5638a9019e40 2024-05-27T07:34:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.013+0000 7f9c1aa00700 1 -- 172.21.15.12:0/3013380222 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f9c2800bb30 con 0x5638a9019e40 2024-05-27T07:34:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.013+0000 7f9c4a801b80 10 client.36356 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.013+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5638a9087060 con 0x5638a9019e40 2024-05-27T07:34:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c1aa00700 1 -- 172.21.15.12:0/3013380222 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c28004c70 con 0x5638a9019e40 2024-05-27T07:34:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c1aa00700 1 -- 172.21.15.12:0/3013380222 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f9c2802d070 con 0x5638a9019e40 2024-05-27T07:34:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c4a801b80 10 client.36356 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c4a801b80 10 client.36356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c4a801b80 20 client.36356 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a9199ac0 0x5638a919bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c4a801b80 4 client.36356 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.014+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638a90c8d90 con 0x5638a9199ac0 2024-05-27T07:34:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.015+0000 7f9c33e00700 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a9199ac0 0x5638a919bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.016+0000 7f9c1aa00700 10 client.36356 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.016+0000 7f9c33e00700 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a9199ac0 0x5638a919bee0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1052a2cf813390e0 server_cookie=3aa70b0ceef19838 in_seq=0 out_seq=0 2024-05-27T07:34:12.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c1aa00700 1 -- 172.21.15.12:0/3013380222 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5638a90c8d90 con 0x5638a9199ac0 2024-05-27T07:34:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c1aa00700 10 client.36356 handle_command_reply: tid=0 2024-05-27T07:34:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 client.36356 shutdown 2024-05-27T07:34:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 msgr2=0x5638a91aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91aaef0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c28000c00 tx=0x7f9c28026d90 comp rx=0 tx=0).stop 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 shutdown_connections 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a9199ac0 0x5638a919bee0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91aaef0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91ab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/3013380222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.019+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 >> 172.21.15.12:0/3013380222 conn(0x5638a90b3390 msgr2=0x5638a91a0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.020+0000 7f9c1aa00700 0 client.36356 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.020+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 shutdown_connections 2024-05-27T07:34:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.020+0000 7f9c4a801b80 1 -- 172.21.15.12:0/3013380222 wait complete. 2024-05-27T07:34:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.020+0000 7f9c4a801b80 20 client.36356 trim_cache size 0 max 16384 2024-05-27T07:34:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 Processor -- start 2024-05-27T07:34:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 -- start start 2024-05-27T07:34:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a9019e40 0x5638a91a6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5638a919c080 con 0x5638a9019e40 2024-05-27T07:34:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c341ad900 con 0x5638a91a1670 2024-05-27T07:34:12.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.049+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c341acb60 con 0x5638a91059f0 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.050+0000 7f9c39200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a9019e40 0x5638a91a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.050+0000 7f9c39200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a9019e40 0x5638a91a6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37030/0 (socket says 172.21.15.12:37030) 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.050+0000 7f9c39200700 1 -- 172.21.15.12:0/62792661 learned_addr learned my addr 172.21.15.12:0/62792661 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.050+0000 7f9c33e00700 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.050+0000 7f9c39c00700 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c39200700 1 -- 172.21.15.12:0/62792661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 msgr2=0x5638a91a30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c39200700 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c39200700 1 -- 172.21.15.12:0/62792661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 msgr2=0x5638a91aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:12.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c39200700 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c39200700 1 -- 172.21.15.12:0/62792661 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a9190240 con 0x5638a9019e40 2024-05-27T07:34:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c39c00700 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c39200700 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a9019e40 0x5638a91a6700 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9c2802fce0 tx=0x7f9c28009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb05e1deb27f11f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.051+0000 7f9c25600700 1 -- 172.21.15.12:0/62792661 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28004ed0 con 0x5638a9019e40 2024-05-27T07:34:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.052+0000 7f9c25600700 1 -- 172.21.15.12:0/62792661 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c2802f910 con 0x5638a9019e40 2024-05-27T07:34:12.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.052+0000 7f9c25600700 1 -- 172.21.15.12:0/62792661 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2800be90 con 0x5638a9019e40 2024-05-27T07:34:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 -- 172.21.15.12:0/62792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a9019e40 msgr2=0x5638a91a6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a9019e40 0x5638a91a6700 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9c2802fce0 tx=0x7f9c28009cd0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 -- 172.21.15.12:0/62792661 shutdown_connections 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a91a1670 0x5638a91aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a9019e40 0x5638a91a6700 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/62792661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 -- 172.21.15.12:0/62792661 >> 172.21.15.12:0/62792661 conn(0x5638a90b3390 msgr2=0x5638a91ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 -- 172.21.15.12:0/62792661 shutdown_connections 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.053+0000 7f9c4a801b80 1 -- 172.21.15.12:0/62792661 wait complete. 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.054+0000 7f9c4a801b80 1 -- start start 2024-05-27T07:34:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.054+0000 7f9c4a801b80 1 Processor -- start 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.054+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.054+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.054+0000 7f9c4a801b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5638a9194f60 con 0x5638a91a1670 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c341ad230 con 0x5638a9019e40 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c4a801b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5638a919c080 con 0x5638a91059f0 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c4a801b80 10 client.0 _ll_register_callbacks cb 0x5638a9060c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c39c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c39c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91a5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37040/0 (socket says 172.21.15.12:37040) 2024-05-27T07:34:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c39c00700 1 -- 172.21.15.12:0/146806034 learned_addr learned my addr 172.21.15.12:0/146806034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c33e00700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c39200700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c39c00700 1 -- 172.21.15.12:0/146806034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 msgr2=0x5638a91a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.055+0000 7f9c39c00700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c39c00700 1 -- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 msgr2=0x5638a91b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c39c00700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c39c00700 1 -- 172.21.15.12:0/146806034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a91abce0 con 0x5638a91a1670 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c39200700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c39c00700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91a5ac0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0a5450 tx=0x7f9c2c0a9910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d529340db3a8c2c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c1aa00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c1aa00700 1 -- 172.21.15.12:0/146806034 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c013ea0 con 0x5638a91a1670 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c1aa00700 1 -- 172.21.15.12:0/146806034 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f9c2c0a8630 con 0x5638a91a1670 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.056+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c2802f910 con 0x5638a91a1670 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c1aa00700 1 -- 172.21.15.12:0/146806034 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f9c2c0a8ee0 con 0x5638a91a1670 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 10 client.16899 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5638a91b21e0 con 0x5638a91a1670 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c1aa00700 1 -- 172.21.15.12:0/146806034 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c2802f910 con 0x5638a91a1670 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c1aa00700 1 -- 172.21.15.12:0/146806034 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f9c2c0a4d40 con 0x5638a91a1670 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 10 client.16899 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 10 client.16899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 20 client.16899 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a922e8f0 0x5638a9230d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 4 client.16899 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:34:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.057+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638a90c2570 con 0x5638a922e8f0 2024-05-27T07:34:12.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.058+0000 7f9c39200700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a922e8f0 0x5638a9230d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:12.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.060+0000 7f9c39200700 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a922e8f0 0x5638a9230d10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=866e4ed0ce39cf34 server_cookie=26a82503378be6d1 in_seq=0 out_seq=0 2024-05-27T07:34:12.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.060+0000 7f9c1aa00700 10 client.16899 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:12.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.060+0000 7f9c1aa00700 1 -- 172.21.15.12:0/146806034 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5638a90c2570 con 0x5638a922e8f0 2024-05-27T07:34:12.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.060+0000 7f9c1aa00700 10 client.16899 handle_command_reply: tid=0 2024-05-27T07:34:12.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.060+0000 7f9c4a801b80 1 client.16899 shutdown 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 msgr2=0x5638a91a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91a5ac0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0a5450 tx=0x7f9c2c0a9910 comp rx=0 tx=0).stop 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 shutdown_connections 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5638a922e8f0 0x5638a9230d10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a9019e40 0x5638a91b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5638a91a1670 0x5638a91a5ac0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 --2- 172.21.15.12:0/146806034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638a91059f0 0x5638a91a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c1aa00700 0 client.16899 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 >> 172.21.15.12:0/146806034 conn(0x5638a90b3390 msgr2=0x5638a91aa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 shutdown_connections 2024-05-27T07:34:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 1 -- 172.21.15.12:0/146806034 wait complete. 2024-05-27T07:34:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.061+0000 7f9c4a801b80 20 client.16899 trim_cache size 0 max 16384 2024-05-27T07:34:12.056 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:34:12.056 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:34:12.056 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:34:12.056 INFO:teuthology.orchestra.run.smithi012.stdout: "7981795c-8a84-45a0-a083-64a9f0cca5b9": { 2024-05-27T07:34:12.057 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:34:12.057 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "7981795c-8a84-45a0-a083-64a9f0cca5b9", 2024-05-27T07:34:12.057 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:34:12.057 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:34:12.057 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:34:12.057 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:34:12.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9c1c0b2a70 msgr2=0x7f9c1c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9c1c0b2a70 0x7f9c1c0b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c28003760 tx=0x7f9c2801a040 comp rx=0 tx=0).stop 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34109880 msgr2=0x7f9c341a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34109880 0x7f9c341a1790 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00ef30 tx=0x7f9c2c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 shutdown_connections 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f9c1c0b2a70 0x7f9c1c0b4f80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c34104600 0x7f9c3419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c34109880 0x7f9c341a1790 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 --2- 172.21.15.12:0/665148831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c34108f30 0x7f9c3419d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.062+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 >> 172.21.15.12:0/665148831 conn(0x7f9c3407af60 msgr2=0x7f9c34110e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.063+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 shutdown_connections 2024-05-27T07:34:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:12.063+0000 7f9c3bc00700 1 -- 172.21.15.12:0/665148831 wait complete. 2024-05-27T07:34:12.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:12 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:12.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:12 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:12.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:12 smithi012 ceph-mon[31473]: pgmap v409: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-05-27T07:34:12.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:12 smithi012 ceph-mon[31473]: pgmap v410: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 902 KiB/s wr, 100 op/s 2024-05-27T07:34:12.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:12 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:34:12.939 INFO:tasks.cephfs.filesystem:scrub status for tag:7981795c-8a84-45a0-a083-64a9f0cca5b9 - {'path': '/', 'tag': '7981795c-8a84-45a0-a083-64a9f0cca5b9', 'options': 'recursive,force'} 2024-05-27T07:34:12.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:12 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:12.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:12 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:12.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:12 smithi162 ceph-mon[35676]: pgmap v409: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-05-27T07:34:12.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:12 smithi162 ceph-mon[35676]: pgmap v410: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 902 KiB/s wr, 100 op/s 2024-05-27T07:34:12.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:12 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:34:12.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:12 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:12.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:12 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:12.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:12 smithi062 ceph-mon[35304]: pgmap v409: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-05-27T07:34:12.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:12 smithi062 ceph-mon[35304]: pgmap v410: 129 pgs: 129 active+clean; 203 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 902 KiB/s wr, 100 op/s 2024-05-27T07:34:12.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:12 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:34:15.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:15 smithi162 ceph-mon[35676]: pgmap v411: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-05-27T07:34:15.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:15 smithi062 ceph-mon[35304]: pgmap v411: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-05-27T07:34:15.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:15 smithi012 ceph-mon[31473]: pgmap v411: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-05-27T07:34:17.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:17 smithi062 ceph-mon[35304]: pgmap v412: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-05-27T07:34:17.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:17 smithi012 ceph-mon[31473]: pgmap v412: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-05-27T07:34:17.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:17 smithi162 ceph-mon[35676]: pgmap v412: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-05-27T07:34:19.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:19 smithi062 ceph-mon[35304]: pgmap v413: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:34:19.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:19 smithi012 ceph-mon[31473]: pgmap v413: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:34:19.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:19 smithi162 ceph-mon[35676]: pgmap v413: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:34:21.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:21 smithi012 ceph-mon[31473]: pgmap v414: 129 pgs: 129 active+clean; 211 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-05-27T07:34:21.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:34:21.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:34:21.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:21 smithi162 ceph-mon[35676]: pgmap v414: 129 pgs: 129 active+clean; 211 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-05-27T07:34:21.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:34:21.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:34:21.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:21 smithi062 ceph-mon[35304]: pgmap v414: 129 pgs: 129 active+clean; 211 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-05-27T07:34:21.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:34:21.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:34:22.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:22 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:34:22.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:22 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:34:22.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:22 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:34:23.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:23 smithi012 ceph-mon[31473]: pgmap v415: 129 pgs: 129 active+clean; 211 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:34:23.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:23 smithi162 ceph-mon[35676]: pgmap v415: 129 pgs: 129 active+clean; 211 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:34:23.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:23 smithi062 ceph-mon[35304]: pgmap v415: 129 pgs: 129 active+clean; 211 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-27T07:34:25.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:25 smithi012 ceph-mon[31473]: pgmap v416: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-05-27T07:34:25.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:25 smithi162 ceph-mon[35676]: pgmap v416: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-05-27T07:34:25.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:25 smithi062 ceph-mon[35304]: pgmap v416: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-05-27T07:34:27.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:27 smithi012 ceph-mon[31473]: pgmap v417: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 917 KiB/s wr, 107 op/s 2024-05-27T07:34:27.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:27 smithi162 ceph-mon[35676]: pgmap v417: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 917 KiB/s wr, 107 op/s 2024-05-27T07:34:27.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:27 smithi062 ceph-mon[35304]: pgmap v417: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 917 KiB/s wr, 107 op/s 2024-05-27T07:34:29.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:29 smithi012 ceph-mon[31473]: pgmap v418: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 917 KiB/s wr, 107 op/s 2024-05-27T07:34:29.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:29 smithi162 ceph-mon[35676]: pgmap v418: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 917 KiB/s wr, 107 op/s 2024-05-27T07:34:29.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:29 smithi062 ceph-mon[35304]: pgmap v418: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 917 KiB/s wr, 107 op/s 2024-05-27T07:34:31.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:31 smithi012 ceph-mon[31473]: pgmap v419: 129 pgs: 129 active+clean; 209 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-05-27T07:34:31.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:31 smithi162 ceph-mon[35676]: pgmap v419: 129 pgs: 129 active+clean; 209 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-05-27T07:34:31.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:31 smithi062 ceph-mon[35304]: pgmap v419: 129 pgs: 129 active+clean; 209 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-05-27T07:34:33.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:33 smithi012 ceph-mon[31473]: pgmap v420: 129 pgs: 129 active+clean; 209 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 912 KiB/s wr, 107 op/s 2024-05-27T07:34:33.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:33 smithi162 ceph-mon[35676]: pgmap v420: 129 pgs: 129 active+clean; 209 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 912 KiB/s wr, 107 op/s 2024-05-27T07:34:33.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:33 smithi062 ceph-mon[35304]: pgmap v420: 129 pgs: 129 active+clean; 209 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 912 KiB/s wr, 107 op/s 2024-05-27T07:34:35.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:35 smithi012 ceph-mon[31473]: pgmap v421: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-05-27T07:34:35.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:35 smithi162 ceph-mon[35676]: pgmap v421: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-05-27T07:34:35.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:35 smithi062 ceph-mon[35304]: pgmap v421: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.3 MiB/s wr, 159 op/s 2024-05-27T07:34:37.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:37 smithi162 ceph-mon[35676]: pgmap v422: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 105 op/s 2024-05-27T07:34:37.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:37 smithi062 ceph-mon[35304]: pgmap v422: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 105 op/s 2024-05-27T07:34:37.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:37 smithi012 ceph-mon[31473]: pgmap v422: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 105 op/s 2024-05-27T07:34:39.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:39 smithi162 ceph-mon[35676]: pgmap v423: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 105 op/s 2024-05-27T07:34:39.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:39 smithi062 ceph-mon[35304]: pgmap v423: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 105 op/s 2024-05-27T07:34:39.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:39 smithi012 ceph-mon[31473]: pgmap v423: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 105 op/s 2024-05-27T07:34:41.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:41 smithi162 ceph-mon[35676]: pgmap v424: 129 pgs: 129 active+clean; 212 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-05-27T07:34:41.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:41 smithi062 ceph-mon[35304]: pgmap v424: 129 pgs: 129 active+clean; 212 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-05-27T07:34:41.844 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:41 smithi012 ceph-mon[31473]: pgmap v424: 129 pgs: 129 active+clean; 212 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-05-27T07:34:42.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:34:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:43.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:43 smithi162 ceph-mon[35676]: pgmap v425: 129 pgs: 129 active+clean; 212 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 900 KiB/s wr, 106 op/s 2024-05-27T07:34:43.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:43 smithi062 ceph-mon[35304]: pgmap v425: 129 pgs: 129 active+clean; 212 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 900 KiB/s wr, 106 op/s 2024-05-27T07:34:43.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:43 smithi012 ceph-mon[31473]: pgmap v425: 129 pgs: 129 active+clean; 212 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 900 KiB/s wr, 106 op/s 2024-05-27T07:34:45.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:45 smithi012 ceph-mon[31473]: pgmap v426: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-05-27T07:34:45.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:45 smithi162 ceph-mon[35676]: pgmap v426: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-05-27T07:34:45.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:45 smithi062 ceph-mon[35304]: pgmap v426: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-05-27T07:34:46.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 -- 172.21.15.12:0/405720336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1052a0 msgr2=0x7f371c1056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 --2- 172.21.15.12:0/405720336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1052a0 0x7f371c1056b0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f370c009990 tx=0x7f370c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 -- 172.21.15.12:0/405720336 shutdown_connections 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 --2- 172.21.15.12:0/405720336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371c10e710 0x7f371c110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 --2- 172.21.15.12:0/405720336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1052a0 0x7f371c1056b0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 --2- 172.21.15.12:0/405720336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f371c105bf0 0x7f371c10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 -- 172.21.15.12:0/405720336 >> 172.21.15.12:0/405720336 conn(0x7f371c078c30 msgr2=0x7f371c079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 -- 172.21.15.12:0/405720336 shutdown_connections 2024-05-27T07:34:46.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.108+0000 7f3721200700 1 -- 172.21.15.12:0/405720336 wait complete. 2024-05-27T07:34:46.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.109+0000 7f3721200700 1 Processor -- start 2024-05-27T07:34:46.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.109+0000 7f3721200700 1 -- start start 2024-05-27T07:34:46.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.110+0000 7f3721200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f371c1052a0 0x7f371c1a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.110+0000 7f3721200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c105bf0 0x7f371c1a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.110+0000 7f3721200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371c10e710 0x7f371c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.110+0000 7f3721200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f371c114110 con 0x7f371c105bf0 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.110+0000 7f3721200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f371c113f90 con 0x7f371c10e710 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.110+0000 7f3721200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f371c114290 con 0x7f371c1052a0 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.110+0000 7f371aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c105bf0 0x7f371c1a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c105bf0 0x7f371c1a2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45282/0 (socket says 172.21.15.12:45282) 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371aa00700 1 -- 172.21.15.12:0/1670209561 learned_addr learned my addr 172.21.15.12:0/1670209561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371b400700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f371c1052a0 0x7f371c1a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371be00700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371c10e710 0x7f371c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371aa00700 1 -- 172.21.15.12:0/1670209561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f371c1052a0 msgr2=0x7f371c1a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371aa00700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f371c1052a0 0x7f371c1a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371aa00700 1 -- 172.21.15.12:0/1670209561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371c10e710 msgr2=0x7f371c19cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371aa00700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371c10e710 0x7f371c19cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.111+0000 7f371aa00700 1 -- 172.21.15.12:0/1670209561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c19d280 con 0x7f371c105bf0 2024-05-27T07:34:46.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.112+0000 7f371aa00700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c105bf0 0x7f371c1a2f00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f370800c920 tx=0x7f370800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.112+0000 7f3713e00700 1 -- 172.21.15.12:0/1670209561 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3708007b00 con 0x7f371c105bf0 2024-05-27T07:34:46.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.112+0000 7f3713e00700 1 -- 172.21.15.12:0/1670209561 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3708004cc0 con 0x7f371c105bf0 2024-05-27T07:34:46.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.112+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c19d570 con 0x7f371c105bf0 2024-05-27T07:34:46.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.112+0000 7f3713e00700 1 -- 172.21.15.12:0/1670209561 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37080054f0 con 0x7f371c105bf0 2024-05-27T07:34:46.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.112+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c1a9770 con 0x7f371c105bf0 2024-05-27T07:34:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.117+0000 7f3713e00700 1 -- 172.21.15.12:0/1670209561 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3708007ca0 con 0x7f371c105bf0 2024-05-27T07:34:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.117+0000 7f373129bb80 1 Processor -- start 2024-05-27T07:34:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.117+0000 7f373129bb80 1 -- start start 2024-05-27T07:34:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.118+0000 7f373129bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb55750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.118+0000 7f373129bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb55c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.118+0000 7f373129bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb616e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.118+0000 7f373129bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f6fea82e40 con 0x55f6feac19f0 2024-05-27T07:34:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.118+0000 7f373129bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f6fea729e0 con 0x55f6fe9d5e40 2024-05-27T07:34:46.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.118+0000 7f373129bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f6fea76a20 con 0x55f6feb5f2c0 2024-05-27T07:34:46.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb55750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb55750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45286/0 (socket says 172.21.15.12:45286) 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 -- 172.21.15.12:0/3094666252 learned_addr learned my addr 172.21.15.12:0/3094666252 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371b400700 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb55c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371aa00700 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 -- 172.21.15.12:0/3094666252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 msgr2=0x55f6feb616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb616e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 -- 172.21.15.12:0/3094666252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 msgr2=0x55f6feb55c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb55c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f371be00700 1 -- 172.21.15.12:0/3094666252 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6fea3b420 con 0x55f6feac19f0 2024-05-27T07:34:46.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.119+0000 7f3713e00700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f37000b2a70 0x7f37000b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.120+0000 7f371be00700 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb55750 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3714009d20 tx=0x7f37140077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b6b9222f90f729 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.120+0000 7f3713e00700 1 -- 172.21.15.12:0/1670209561 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f37080e2600 con 0x7f371c105bf0 2024-05-27T07:34:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.120+0000 7f371b400700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f37000b2a70 0x7f37000b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.120+0000 7f3710c00700 1 -- 172.21.15.12:0/3094666252 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714004bb0 con 0x55f6feac19f0 2024-05-27T07:34:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.120+0000 7f3710c00700 1 -- 172.21.15.12:0/3094666252 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3714004d50 con 0x55f6feac19f0 2024-05-27T07:34:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.120+0000 7f3710c00700 1 -- 172.21.15.12:0/3094666252 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714014610 con 0x55f6feac19f0 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f373129bb80 1 -- 172.21.15.12:0/3094666252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 msgr2=0x55f6feb55750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f373129bb80 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb55750 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3714009d20 tx=0x7f37140077e0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f371b400700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f37000b2a70 0x7f37000b4f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f370c00b5c0 tx=0x7f370c005940 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f373129bb80 1 -- 172.21.15.12:0/3094666252 shutdown_connections 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f373129bb80 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f373129bb80 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb55750 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f373129bb80 1 --2- 172.21.15.12:0/3094666252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb55c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.121+0000 7f373129bb80 1 -- 172.21.15.12:0/3094666252 >> 172.21.15.12:0/3094666252 conn(0x55f6fea6f390 msgr2=0x55f6feb50710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.122+0000 7f373129bb80 1 -- 172.21.15.12:0/3094666252 shutdown_connections 2024-05-27T07:34:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.122+0000 7f373129bb80 1 -- 172.21.15.12:0/3094666252 wait complete. 2024-05-27T07:34:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.122+0000 7f373129bb80 1 -- start start 2024-05-27T07:34:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.122+0000 7f373129bb80 1 Processor -- start 2024-05-27T07:34:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.123+0000 7f373129bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.123+0000 7f373129bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb5c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.123+0000 7f373129bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feb5f2c0 0x55f6feb5cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.123+0000 7f373129bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f6fe978fa0 con 0x55f6feac19f0 2024-05-27T07:34:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.123+0000 7f373129bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f6fea13760 con 0x55f6feb5f2c0 2024-05-27T07:34:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.123+0000 7f373129bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f6feb61e00 con 0x55f6fe9d5e40 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.123+0000 7f373129bb80 10 client.0 _ll_register_callbacks cb 0x55f6fea1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb5c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb5c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45298/0 (socket says 172.21.15.12:45298) 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 -- 172.21.15.12:0/813208222 learned_addr learned my addr 172.21.15.12:0/813208222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371b400700 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371be00700 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feb5f2c0 0x55f6feb5cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 -- 172.21.15.12:0/813208222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6fe9d5e40 msgr2=0x55f6feb5c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 -- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feb5f2c0 msgr2=0x55f6feb5cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feb5f2c0 0x55f6feb5cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.124+0000 7f371aa00700 1 -- 172.21.15.12:0/813208222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6fe9c28d0 con 0x55f6feac19f0 2024-05-27T07:34:46.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.125+0000 7f371aa00700 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb5c740 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f37080df9a0 tx=0x7f37080a92a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5ab1a151206ead server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.125+0000 7f3705600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:46.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.125+0000 7f3705600700 1 -- 172.21.15.12:0/813208222 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370801d040 con 0x55f6feac19f0 2024-05-27T07:34:46.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.125+0000 7f3705600700 1 -- 172.21.15.12:0/813208222 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f37080b20b0 con 0x55f6feac19f0 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.125+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3714004d50 con 0x55f6feac19f0 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.126+0000 7f3705600700 1 -- 172.21.15.12:0/813208222 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f37080a8c40 con 0x55f6feac19f0 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.126+0000 7f373129bb80 10 client.16911 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.126+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55f6fea43060 con 0x55f6feac19f0 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.126+0000 7f3705600700 1 -- 172.21.15.12:0/813208222 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3714004d50 con 0x55f6feac19f0 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.127+0000 7f3705600700 1 -- 172.21.15.12:0/813208222 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f37080e2190 con 0x55f6feac19f0 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.127+0000 7f373129bb80 10 client.16911 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.127+0000 7f373129bb80 10 client.16911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:46.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.127+0000 7f373129bb80 20 client.16911 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:46.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.127+0000 7f373129bb80 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6feb6eee0 0x55f6feb71380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.127+0000 7f373129bb80 4 client.16911 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:46.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.127+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6fea84d90 con 0x55f6feb6eee0 2024-05-27T07:34:46.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.128+0000 7f371b400700 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6feb6eee0 0x55f6feb71380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.130+0000 7f371b400700 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6feb6eee0 0x55f6feb71380 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aae21dfa1ab654bd server_cookie=d077a4b8fbe3aecd in_seq=0 out_seq=0 2024-05-27T07:34:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.130+0000 7f3705600700 10 client.16911 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.132+0000 7f3705600700 1 -- 172.21.15.12:0/813208222 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f6fea84d90 con 0x55f6feb6eee0 2024-05-27T07:34:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.132+0000 7f3705600700 10 client.16911 handle_command_reply: tid=0 2024-05-27T07:34:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.132+0000 7f373129bb80 1 client.16911 shutdown 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 msgr2=0x55f6feb5c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb5c740 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f37080df9a0 tx=0x7f37080a92a0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 shutdown_connections 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6feb6eee0 0x55f6feb71380 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feb5f2c0 0x55f6feb5cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 --2- 172.21.15.12:0/813208222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb5c740 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 >> 172.21.15.12:0/813208222 conn(0x55f6fea6f390 msgr2=0x55f6feb50300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f3705600700 0 client.16911 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 shutdown_connections 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.133+0000 7f373129bb80 1 -- 172.21.15.12:0/813208222 wait complete. 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.134+0000 7f373129bb80 20 client.16911 trim_cache size 0 max 16384 2024-05-27T07:34:46.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 Processor -- start 2024-05-27T07:34:46.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 -- start start 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb6aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb54830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f6feb71780 con 0x55f6feac19f0 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1b1ca0 con 0x55f6fe9d5e40 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.163+0000 7f373129bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1b0f00 con 0x55f6feb5f2c0 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb6aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb6aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45312/0 (socket says 172.21.15.12:45312) 2024-05-27T07:34:46.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 -- 172.21.15.12:0/2180474163 learned_addr learned my addr 172.21.15.12:0/2180474163 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:46.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371be00700 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb54830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371b400700 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 -- 172.21.15.12:0/2180474163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 msgr2=0x55f6feb54830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb54830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 -- 172.21.15.12:0/2180474163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 msgr2=0x55f6feb5b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.164+0000 7f371aa00700 1 -- 172.21.15.12:0/2180474163 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6feb5a770 con 0x55f6feac19f0 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.165+0000 7f371aa00700 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb6aea0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f37080dff40 tx=0x7f37080e8240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea112916f10067e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.165+0000 7f3707e00700 1 -- 172.21.15.12:0/2180474163 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370801d040 con 0x55f6feac19f0 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.165+0000 7f3707e00700 1 -- 172.21.15.12:0/2180474163 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37080a8fe0 con 0x55f6feac19f0 2024-05-27T07:34:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.165+0000 7f3707e00700 1 -- 172.21.15.12:0/2180474163 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37080a4f80 con 0x55f6feac19f0 2024-05-27T07:34:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 -- 172.21.15.12:0/2180474163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 msgr2=0x55f6feb6aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb6aea0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f37080dff40 tx=0x7f37080e8240 comp rx=0 tx=0).stop 2024-05-27T07:34:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 -- 172.21.15.12:0/2180474163 shutdown_connections 2024-05-27T07:34:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6fe9d5e40 0x55f6feb5b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb54830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 --2- 172.21.15.12:0/2180474163 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6feac19f0 0x55f6feb6aea0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 -- 172.21.15.12:0/2180474163 >> 172.21.15.12:0/2180474163 conn(0x55f6fea6f390 msgr2=0x55f6feb54120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.166+0000 7f373129bb80 1 -- 172.21.15.12:0/2180474163 shutdown_connections 2024-05-27T07:34:46.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.167+0000 7f373129bb80 1 -- 172.21.15.12:0/2180474163 wait complete. 2024-05-27T07:34:46.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.167+0000 7f373129bb80 1 -- start start 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.167+0000 7f373129bb80 1 Processor -- start 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f373129bb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6fe9d5e40 0x55f6feb673a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f373129bb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feac19f0 0x55f6feb678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f373129bb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb628b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f373129bb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f6feb4cba0 con 0x55f6fe9d5e40 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f373129bb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1b15d0 con 0x55f6feac19f0 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f373129bb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f6feb6ec70 con 0x55f6feb5f2c0 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f373129bb80 10 client.0 _ll_register_callbacks cb 0x55f6fea1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f371b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6fe9d5e40 0x55f6feb673a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f371b400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6fe9d5e40 0x55f6feb673a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45330/0 (socket says 172.21.15.12:45330) 2024-05-27T07:34:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.168+0000 7f371b400700 1 -- 172.21.15.12:0/3636745740 learned_addr learned my addr 172.21.15.12:0/3636745740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371aa00700 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feac19f0 0x55f6feb678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371be00700 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371b400700 1 -- 172.21.15.12:0/3636745740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 msgr2=0x55f6feb628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371b400700 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb628b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371b400700 1 -- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feac19f0 msgr2=0x55f6feb678e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371b400700 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feac19f0 0x55f6feb678e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371b400700 1 -- 172.21.15.12:0/3636745740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6feb63030 con 0x55f6fe9d5e40 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f371b400700 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6fe9d5e40 0x55f6feb673a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f370c000c00 tx=0x7f370c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9f62058b3e97db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.169+0000 7f3705600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.170+0000 7f3705600700 1 -- 172.21.15.12:0/3636745740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c00f460 con 0x55f6fe9d5e40 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.170+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37080a8fe0 con 0x55f6fe9d5e40 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.170+0000 7f3705600700 1 -- 172.21.15.12:0/3636745740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f370c00f600 con 0x55f6fe9d5e40 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.170+0000 7f3705600700 1 -- 172.21.15.12:0/3636745740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f370c043050 con 0x55f6fe9d5e40 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.170+0000 7f373129bb80 10 client.16917 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:46.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.170+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55f6febd95e0 con 0x55f6fe9d5e40 2024-05-27T07:34:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f3705600700 1 -- 172.21.15.12:0/3636745740 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f37080a8fe0 con 0x55f6fe9d5e40 2024-05-27T07:34:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f3705600700 1 -- 172.21.15.12:0/3636745740 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f370c031520 con 0x55f6fe9d5e40 2024-05-27T07:34:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f373129bb80 10 client.16917 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f373129bb80 10 client.16917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f373129bb80 20 client.16917 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:46.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f373129bb80 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6febd9c50 0x55f6febdc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f373129bb80 4 client.16917 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:34:46.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.171+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f6fea7e570 con 0x55f6febd9c50 2024-05-27T07:34:46.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.172+0000 7f371aa00700 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6febd9c50 0x55f6febdc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.173+0000 7f371aa00700 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6febd9c50 0x55f6febdc070 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34673e9a7ae1e136 server_cookie=79712d2a04d550f4 in_seq=0 out_seq=0 2024-05-27T07:34:46.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.173+0000 7f3705600700 10 client.16917 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:46.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.173+0000 7f3705600700 1 -- 172.21.15.12:0/3636745740 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f6fea7e570 con 0x55f6febd9c50 2024-05-27T07:34:46.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.173+0000 7f3705600700 10 client.16917 handle_command_reply: tid=0 2024-05-27T07:34:46.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.173+0000 7f373129bb80 1 client.16917 shutdown 2024-05-27T07:34:46.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6fe9d5e40 msgr2=0x55f6feb673a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6fe9d5e40 0x55f6feb673a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f370c000c00 tx=0x7f370c009310 comp rx=0 tx=0).stop 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 shutdown_connections 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6feb5f2c0 0x55f6feb628b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f6fe9d5e40 0x55f6feb673a0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f6febd9c50 0x55f6febdc070 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 --2- 172.21.15.12:0/3636745740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f6feac19f0 0x55f6feb678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 >> 172.21.15.12:0/3636745740 conn(0x55f6fea6f390 msgr2=0x55f6feb57800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f3705600700 0 client.16917 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 shutdown_connections 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 1 -- 172.21.15.12:0/3636745740 wait complete. 2024-05-27T07:34:46.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.174+0000 7f373129bb80 20 client.16917 trim_cache size 0 max 16384 2024-05-27T07:34:46.171 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:34:46.171 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:34:46.171 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:34:46.171 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:34:46.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.175+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f37000b2a70 msgr2=0x7f37000b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.175+0000 7f3721200700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f37000b2a70 0x7f37000b4f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f370c00b5c0 tx=0x7f370c005940 comp rx=0 tx=0).stop 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.175+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c105bf0 msgr2=0x7f371c1a2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.175+0000 7f3721200700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c105bf0 0x7f371c1a2f00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f370800c920 tx=0x7f370800cce0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 shutdown_connections 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f37000b2a70 0x7f37000b4f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371c10e710 0x7f371c19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f371c1052a0 0x7f371c1a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 --2- 172.21.15.12:0/1670209561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c105bf0 0x7f371c1a2f00 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 >> 172.21.15.12:0/1670209561 conn(0x7f371c078c30 msgr2=0x7f371c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 shutdown_connections 2024-05-27T07:34:46.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:46.176+0000 7f3721200700 1 -- 172.21.15.12:0/1670209561 wait complete. 2024-05-27T07:34:47.182 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:34:47.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:34:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:47.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:47 smithi162 ceph-mon[35676]: pgmap v427: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 937 KiB/s wr, 110 op/s 2024-05-27T07:34:47.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:47 smithi062 ceph-mon[35304]: pgmap v427: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 937 KiB/s wr, 110 op/s 2024-05-27T07:34:47.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:47 smithi012 ceph-mon[31473]: pgmap v427: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 937 KiB/s wr, 110 op/s 2024-05-27T07:34:48.379 INFO:tasks.workunit.client.0.smithi012.stdout:30000 mapread 0x3fc9cb thru 0x4052a3 (0x88d9 bytes) 2024-05-27T07:34:49.594 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:49 smithi012 ceph-mon[31473]: pgmap v428: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 937 KiB/s wr, 110 op/s 2024-05-27T07:34:49.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:49 smithi162 ceph-mon[35676]: pgmap v428: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 937 KiB/s wr, 110 op/s 2024-05-27T07:34:49.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:49 smithi062 ceph-mon[35304]: pgmap v428: 129 pgs: 129 active+clean; 216 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 937 KiB/s wr, 110 op/s 2024-05-27T07:34:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 -- 172.21.15.12:0/2144993736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f691010c5a0 msgr2=0x7f691010c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 --2- 172.21.15.12:0/2144993736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f691010c5a0 0x7f691010c9b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f690800b0a0 tx=0x7f690800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 -- 172.21.15.12:0/2144993736 shutdown_connections 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 --2- 172.21.15.12:0/2144993736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6910106eb0 0x7f6910113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 --2- 172.21.15.12:0/2144993736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f691010c5a0 0x7f691010c9b0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 --2- 172.21.15.12:0/2144993736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6910106560 0x7f6910106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 -- 172.21.15.12:0/2144993736 >> 172.21.15.12:0/2144993736 conn(0x7f6910102040 msgr2=0x7f6910104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 -- 172.21.15.12:0/2144993736 shutdown_connections 2024-05-27T07:34:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.513+0000 7f6916c00700 1 -- 172.21.15.12:0/2144993736 wait complete. 2024-05-27T07:34:50.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.514+0000 7f6916c00700 1 Processor -- start 2024-05-27T07:34:50.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.515+0000 7f6916c00700 1 -- start start 2024-05-27T07:34:50.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.515+0000 7f6916c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6910106560 0x7f691019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.515+0000 7f6916c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6910106eb0 0x7f691019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.515+0000 7f6916c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f691010c5a0 0x7f69101a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.515+0000 7f6916c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f69101165b0 con 0x7f6910106eb0 2024-05-27T07:34:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.515+0000 7f6916c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6910116430 con 0x7f691010c5a0 2024-05-27T07:34:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.515+0000 7f6916c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6910116730 con 0x7f6910106560 2024-05-27T07:34:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6910106eb0 0x7f691019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6910106eb0 0x7f691019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45338/0 (socket says 172.21.15.12:45338) 2024-05-27T07:34:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 -- 172.21.15.12:0/1042382982 learned_addr learned my addr 172.21.15.12:0/1042382982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f6914c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f691010c5a0 0x7f69101a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690fe00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6910106560 0x7f691019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 -- 172.21.15.12:0/1042382982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6910106560 msgr2=0x7f691019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6910106560 0x7f691019d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 -- 172.21.15.12:0/1042382982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f691010c5a0 msgr2=0x7f69101a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f691010c5a0 0x7f69101a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.516+0000 7f690f400700 1 -- 172.21.15.12:0/1042382982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69101a1e80 con 0x7f6910106eb0 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.517+0000 7f6914c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f691010c5a0 0x7f69101a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.517+0000 7f690f400700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6910106eb0 0x7f691019d5a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f68f800d950 tx=0x7f68f800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.517+0000 7f690cc00700 1 -- 172.21.15.12:0/1042382982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8017070 con 0x7f6910106eb0 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.517+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69101a2170 con 0x7f6910106eb0 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.517+0000 7f690cc00700 1 -- 172.21.15.12:0/1042382982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68f800de30 con 0x7f6910106eb0 2024-05-27T07:34:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.518+0000 7f690cc00700 1 -- 172.21.15.12:0/1042382982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f800f3c0 con 0x7f6910106eb0 2024-05-27T07:34:50.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.518+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69101a9a90 con 0x7f6910106eb0 2024-05-27T07:34:50.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.519+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f691010e960 con 0x7f6910106eb0 2024-05-27T07:34:50.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.522+0000 7f690cc00700 1 -- 172.21.15.12:0/1042382982 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f68f801a070 con 0x7f6910106eb0 2024-05-27T07:34:50.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.524+0000 7f690cc00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68f40b2a70 0x7f68f40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.524+0000 7f690cc00700 1 -- 172.21.15.12:0/1042382982 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f68f80e2310 con 0x7f6910106eb0 2024-05-27T07:34:50.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.524+0000 7f690fe00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68f40b2a70 0x7f68f40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.526+0000 7f690fe00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68f40b2a70 0x7f68f40b4f80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6908009fd0 tx=0x7f690800b7b0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.528+0000 7f690cc00700 1 -- 172.21.15.12:0/1042382982 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f68f80a9900 con 0x7f6910106eb0 2024-05-27T07:34:50.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.890+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f691019e310 con 0x7f6910106eb0 2024-05-27T07:34:50.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.892+0000 7f690cc00700 1 -- 172.21.15.12:0/1042382982 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f68f80a9200 con 0x7f6910106eb0 2024-05-27T07:34:50.887 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:34:50.887 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.895+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68f40b2a70 msgr2=0x7f68f40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.895+0000 7f6916c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68f40b2a70 0x7f68f40b4f80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6908009fd0 tx=0x7f690800b7b0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.895+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6910106eb0 msgr2=0x7f691019d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.895+0000 7f6916c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6910106eb0 0x7f691019d5a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f68f800d950 tx=0x7f68f800dd10 comp rx=0 tx=0).stop 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.895+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 shutdown_connections 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.896+0000 7f6916c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f68f40b2a70 0x7f68f40b4f80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.896+0000 7f6916c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f691010c5a0 0x7f69101a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.896+0000 7f6916c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6910106560 0x7f691019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.896+0000 7f6916c00700 1 --2- 172.21.15.12:0/1042382982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6910106eb0 0x7f691019d5a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.896+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 >> 172.21.15.12:0/1042382982 conn(0x7f6910102040 msgr2=0x7f6910103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.896+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 shutdown_connections 2024-05-27T07:34:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:50.896+0000 7f6916c00700 1 -- 172.21.15.12:0/1042382982 wait complete. 2024-05-27T07:34:50.893 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:34:51.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:51 smithi012 ceph-mon[31473]: pgmap v429: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-27T07:34:51.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:51 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1042382982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:34:51.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:51 smithi162 ceph-mon[35676]: pgmap v429: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-27T07:34:51.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:51 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1042382982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:34:51.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:51 smithi062 ceph-mon[35304]: pgmap v429: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-27T07:34:51.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:51 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1042382982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:34:52.193 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:34:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:53.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:53 smithi012 ceph-mon[31473]: pgmap v430: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:34:53.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:53 smithi162 ceph-mon[35676]: pgmap v430: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:34:53.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:53 smithi062 ceph-mon[35304]: pgmap v430: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-05-27T07:34:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.153+0000 7f616fe00700 1 -- 172.21.15.12:0/270962240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170104400 msgr2=0x7f6170104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.153+0000 7f616fe00700 1 --2- 172.21.15.12:0/270962240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170104400 0x7f6170104810 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f61600099c0 tx=0x7f6160009cd0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.154+0000 7f616fe00700 1 -- 172.21.15.12:0/270962240 shutdown_connections 2024-05-27T07:34:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.154+0000 7f616fe00700 1 --2- 172.21.15.12:0/270962240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170109680 0x7f6170110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.154+0000 7f616fe00700 1 --2- 172.21.15.12:0/270962240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170104400 0x7f6170104810 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.154+0000 7f616fe00700 1 --2- 172.21.15.12:0/270962240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6170108d30 0x7f6170109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.154+0000 7f616fe00700 1 -- 172.21.15.12:0/270962240 >> 172.21.15.12:0/270962240 conn(0x7f6170079c10 msgr2=0x7f617007a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.154+0000 7f616fe00700 1 -- 172.21.15.12:0/270962240 shutdown_connections 2024-05-27T07:34:55.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.155+0000 7f616fe00700 1 -- 172.21.15.12:0/270962240 wait complete. 2024-05-27T07:34:55.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.156+0000 7f616fe00700 1 Processor -- start 2024-05-27T07:34:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.157+0000 7f616fe00700 1 -- start start 2024-05-27T07:34:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.157+0000 7f616fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170104400 0x7f6170109f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.157+0000 7f616fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170108d30 0x7f617010a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6170109680 0x7f617010a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170108d30 0x7f617010a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170108d30 0x7f617010a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33378/0 (socket says 172.21.15.12:33378) 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170104400 0x7f6170109f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6170114280 con 0x7f6170108d30 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6170114100 con 0x7f6170109680 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616fe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6170114400 con 0x7f6170104400 2024-05-27T07:34:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.158+0000 7f616e000700 1 -- 172.21.15.12:0/4085120778 learned_addr learned my addr 172.21.15.12:0/4085120778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.159+0000 7f616f400700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6170109680 0x7f617010a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.159+0000 7f616e000700 1 -- 172.21.15.12:0/4085120778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170104400 msgr2=0x7f6170109f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.159+0000 7f616e000700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170104400 0x7f6170109f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.159+0000 7f616e000700 1 -- 172.21.15.12:0/4085120778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6170109680 msgr2=0x7f617010a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.159+0000 7f616e000700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6170109680 0x7f617010a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.159+0000 7f616e000700 1 -- 172.21.15.12:0/4085120778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61701ad3b0 con 0x7f6170108d30 2024-05-27T07:34:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.160+0000 7f616e000700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170108d30 0x7f617010a4a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f615c00d980 tx=0x7f615c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.160+0000 7f6167400700 1 -- 172.21.15.12:0/4085120778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c017070 con 0x7f6170108d30 2024-05-27T07:34:55.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.160+0000 7f6167400700 1 -- 172.21.15.12:0/4085120778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f615c009dc0 con 0x7f6170108d30 2024-05-27T07:34:55.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.160+0000 7f6167400700 1 -- 172.21.15.12:0/4085120778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c00f3c0 con 0x7f6170108d30 2024-05-27T07:34:55.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.160+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61701ad640 con 0x7f6170108d30 2024-05-27T07:34:55.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.160+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61701adbd0 con 0x7f6170108d30 2024-05-27T07:34:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.166+0000 7f61846a1b80 1 Processor -- start 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f61846a1b80 1 -- start start 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f6167400700 1 -- 172.21.15.12:0/4085120778 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f615c00f560 con 0x7f6170108d30 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05529f0 0x55e2a05e6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f61846a1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e2a0513e40 con 0x55e2a05f02c0 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f61846a1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e2a05039e0 con 0x55e2a05529f0 2024-05-27T07:34:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.167+0000 7f61846a1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e2a0507a20 con 0x55e2a0466e40 2024-05-27T07:34:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.168+0000 7f616ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.168+0000 7f616ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05e6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:53560/0 (socket says 172.21.15.12:53560) 2024-05-27T07:34:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.168+0000 7f616ea00700 1 -- 172.21.15.12:0/3856510076 learned_addr learned my addr 172.21.15.12:0/3856510076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616f400700 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05529f0 0x55e2a05e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616ea00700 1 -- 172.21.15.12:0/3856510076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05529f0 msgr2=0x55e2a05e6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616ea00700 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05529f0 0x55e2a05e6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616e000700 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616ea00700 1 -- 172.21.15.12:0/3856510076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 msgr2=0x55e2a05f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616ea00700 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616ea00700 1 -- 172.21.15.12:0/3856510076 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2a04cc420 con 0x55e2a0466e40 2024-05-27T07:34:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616e000700 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f6167400700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f61540b2b20 0x7f61540b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f6167400700 1 -- 172.21.15.12:0/4085120778 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f615c0e25b0 con 0x7f6170108d30 2024-05-27T07:34:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.169+0000 7f616ea00700 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05e6c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6160015000 tx=0x7f61600036c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ec13dbdc674741f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.170+0000 7f616f400700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f61540b2b20 0x7f61540b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.170+0000 7f615be00700 1 -- 172.21.15.12:0/3856510076 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160003b50 con 0x55e2a0466e40 2024-05-27T07:34:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.170+0000 7f615be00700 1 -- 172.21.15.12:0/3856510076 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6160003cf0 con 0x55e2a0466e40 2024-05-27T07:34:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.170+0000 7f615be00700 1 -- 172.21.15.12:0/3856510076 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616000be30 con 0x55e2a0466e40 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f61846a1b80 1 -- 172.21.15.12:0/3856510076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 msgr2=0x55e2a05e6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f616f400700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f61540b2b20 0x7f61540b5030 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55e2a05e2a90 tx=0x7f61680082c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05e6c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6160015000 tx=0x7f61600036c0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f61846a1b80 1 -- 172.21.15.12:0/3856510076 shutdown_connections 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05529f0 0x55e2a05e6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3856510076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05e6c90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.171+0000 7f61846a1b80 1 -- 172.21.15.12:0/3856510076 >> 172.21.15.12:0/3856510076 conn(0x55e2a0500390 msgr2=0x55e2a05e1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.172+0000 7f61846a1b80 1 -- 172.21.15.12:0/3856510076 shutdown_connections 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.172+0000 7f61846a1b80 1 -- 172.21.15.12:0/3856510076 wait complete. 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.172+0000 7f61846a1b80 1 -- start start 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.173+0000 7f61846a1b80 1 Processor -- start 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.173+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a0466e40 0x55e2a05ed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.173+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05529f0 0x55e2a05ed7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.173+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a05f02c0 0x55e2a05edd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.173+0000 7f61846a1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e2a0409fa0 con 0x55e2a05529f0 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.173+0000 7f61846a1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e2a04a4760 con 0x55e2a0466e40 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.173+0000 7f61846a1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e2a05f2da0 con 0x55e2a05f02c0 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f61846a1b80 10 client.0 _ll_register_callbacks cb 0x55e2a04adc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05529f0 0x55e2a05ed7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05529f0 0x55e2a05ed7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33398/0 (socket says 172.21.15.12:33398) 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f616e000700 1 -- 172.21.15.12:0/408649486 learned_addr learned my addr 172.21.15.12:0/408649486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f616ea00700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a0466e40 0x55e2a05ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f616f400700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a05f02c0 0x55e2a05edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f616e000700 1 -- 172.21.15.12:0/408649486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a05f02c0 msgr2=0x55e2a05edd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.174+0000 7f616e000700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a05f02c0 0x55e2a05edd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f616e000700 1 -- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a0466e40 msgr2=0x55e2a05ed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f616e000700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a0466e40 0x55e2a05ed280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f616e000700 1 -- 172.21.15.12:0/408649486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2a04538d0 con 0x55e2a05529f0 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f616ea00700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a0466e40 0x55e2a05ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f616e000700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05529f0 0x55e2a05ed7c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f615c0a9120 tx=0x7f615c0a8f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b479cd8a831caf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f6158c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f6158c00700 1 -- 172.21.15.12:0/408649486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c020070 con 0x55e2a05529f0 2024-05-27T07:34:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.175+0000 7f6158c00700 1 -- 172.21.15.12:0/408649486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f615c010460 con 0x55e2a05529f0 2024-05-27T07:34:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6160003cf0 con 0x55e2a05529f0 2024-05-27T07:34:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f6158c00700 1 -- 172.21.15.12:0/408649486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f615c0ad7d0 con 0x55e2a05529f0 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f61846a1b80 10 client.16932 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55e2a04d4060 con 0x55e2a05529f0 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f6158c00700 1 -- 172.21.15.12:0/408649486 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6160003cf0 con 0x55e2a05529f0 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f6158c00700 1 -- 172.21.15.12:0/408649486 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f615c0e2120 con 0x55e2a05529f0 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f61846a1b80 10 client.16932 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f61846a1b80 10 client.16932 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.176+0000 7f61846a1b80 20 client.16932 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.177+0000 7f61846a1b80 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a0600020 0x55e2a06024f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.177+0000 7f61846a1b80 4 client.16932 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.177+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2a0515d90 con 0x55e2a0600020 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.177+0000 7f616ea00700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a0600020 0x55e2a06024f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.179+0000 7f6158c00700 10 client.16932 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:34:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.179+0000 7f616ea00700 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a0600020 0x55e2a06024f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=11b3e4972bf9ac75 server_cookie=dc6f073008c91679 in_seq=0 out_seq=0 2024-05-27T07:34:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f6158c00700 1 -- 172.21.15.12:0/408649486 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e2a0515d90 con 0x55e2a0600020 2024-05-27T07:34:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f6158c00700 10 client.16932 handle_command_reply: tid=0 2024-05-27T07:34:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 client.16932 shutdown 2024-05-27T07:34:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05529f0 msgr2=0x55e2a05ed7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05529f0 0x55e2a05ed7c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f615c0a9120 tx=0x7f615c0a8f30 comp rx=0 tx=0).stop 2024-05-27T07:34:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 shutdown_connections 2024-05-27T07:34:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a0600020 0x55e2a06024f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a0466e40 0x55e2a05ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a05f02c0 0x55e2a05edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 --2- 172.21.15.12:0/408649486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05529f0 0x55e2a05ed7c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 >> 172.21.15.12:0/408649486 conn(0x55e2a0500390 msgr2=0x55e2a05e67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f6158c00700 0 client.16932 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:34:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.181+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 shutdown_connections 2024-05-27T07:34:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.182+0000 7f61846a1b80 1 -- 172.21.15.12:0/408649486 wait complete. 2024-05-27T07:34:55.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.182+0000 7f61846a1b80 20 client.16932 trim_cache size 0 max 16384 2024-05-27T07:34:55.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 Processor -- start 2024-05-27T07:34:55.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 -- start start 2024-05-27T07:34:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a06026a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05e16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05e0730 0x55e2a05fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e2a05e0130 con 0x55e2a05f02c0 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f61701b6080 con 0x55e2a05e0730 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.202+0000 7f61846a1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61701b52e0 con 0x55e2a0466e40 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05e16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05e16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33414/0 (socket says 172.21.15.12:33414) 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 -- 172.21.15.12:0/2196132911 learned_addr learned my addr 172.21.15.12:0/2196132911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616f400700 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05e0730 0x55e2a05fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616ea00700 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a06026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 -- 172.21.15.12:0/2196132911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 msgr2=0x55e2a06026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a06026a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 -- 172.21.15.12:0/2196132911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05e0730 msgr2=0x55e2a05fbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05e0730 0x55e2a05fbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.203+0000 7f616e000700 1 -- 172.21.15.12:0/2196132911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2a05ee070 con 0x55e2a05f02c0 2024-05-27T07:34:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.204+0000 7f616e000700 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05e16a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f615c0a93b0 tx=0x7f615c0dfdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b65d13ec5aed6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.204+0000 7f615b400700 1 -- 172.21.15.12:0/2196132911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c020020 con 0x55e2a05f02c0 2024-05-27T07:34:55.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.204+0000 7f615b400700 1 -- 172.21.15.12:0/2196132911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f615c0a9680 con 0x55e2a05f02c0 2024-05-27T07:34:55.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.204+0000 7f615b400700 1 -- 172.21.15.12:0/2196132911 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c0e2210 con 0x55e2a05f02c0 2024-05-27T07:34:55.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 -- 172.21.15.12:0/2196132911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 msgr2=0x55e2a05e16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05e16a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f615c0a93b0 tx=0x7f615c0dfdf0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 -- 172.21.15.12:0/2196132911 shutdown_connections 2024-05-27T07:34:55.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05e0730 0x55e2a05fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a06026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 --2- 172.21.15.12:0/2196132911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05f02c0 0x55e2a05e16a0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 -- 172.21.15.12:0/2196132911 >> 172.21.15.12:0/2196132911 conn(0x55e2a0500390 msgr2=0x55e2a05e0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 -- 172.21.15.12:0/2196132911 shutdown_connections 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.205+0000 7f61846a1b80 1 -- 172.21.15.12:0/2196132911 wait complete. 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.206+0000 7f61846a1b80 1 -- start start 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.206+0000 7f61846a1b80 1 Processor -- start 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05e0730 0x55e2a05f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f61846a1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05f02c0 0x55e2a05f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f61846a1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e2a05f0080 con 0x55e2a05e0730 2024-05-27T07:34:55.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f61846a1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f61701b59b0 con 0x55e2a05f02c0 2024-05-27T07:34:55.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f61846a1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e2a05ec5a0 con 0x55e2a0466e40 2024-05-27T07:34:55.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f61846a1b80 10 client.0 _ll_register_callbacks cb 0x55e2a04adc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:55.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05e0730 0x55e2a05f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f616e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05e0730 0x55e2a05f3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33428/0 (socket says 172.21.15.12:33428) 2024-05-27T07:34:55.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f616e000700 1 -- 172.21.15.12:0/3242773283 learned_addr learned my addr 172.21.15.12:0/3242773283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:55.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f616ea00700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.207+0000 7f616f400700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05f02c0 0x55e2a05f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f616e000700 1 -- 172.21.15.12:0/3242773283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 msgr2=0x55e2a05f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f616e000700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f616e000700 1 -- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05f02c0 msgr2=0x55e2a05f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f616e000700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05f02c0 0x55e2a05f40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f616e000700 1 -- 172.21.15.12:0/3242773283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2a05f4840 con 0x55e2a05e0730 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f616ea00700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f616e000700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05e0730 0x55e2a05f3b20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f615c0a9120 tx=0x7f615c0aede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436576f4e734cbe7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f6158c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f6158c00700 1 -- 172.21.15.12:0/3242773283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c0c8060 con 0x55e2a05e0730 2024-05-27T07:34:55.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f6158c00700 1 -- 172.21.15.12:0/3242773283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f615c010460 con 0x55e2a05e0730 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.208+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f615c0a9680 con 0x55e2a05e0730 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f6158c00700 1 -- 172.21.15.12:0/3242773283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f615c020030 con 0x55e2a05e0730 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 10 client.16938 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55e2a05f4840 con 0x55e2a05e0730 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f6158c00700 1 -- 172.21.15.12:0/3242773283 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f615c0a9680 con 0x55e2a05e0730 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f6158c00700 1 -- 172.21.15.12:0/3242773283 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f615c0ca300 con 0x55e2a05e0730 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 10 client.16938 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 10 client.16938 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 20 client.16938 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a066ac20 0x55e2a066d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 4 client.16938 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:55.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.209+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2a050f570 con 0x55e2a066ac20 2024-05-27T07:34:55.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.210+0000 7f616ea00700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a066ac20 0x55e2a066d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.211+0000 7f616ea00700 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a066ac20 0x55e2a066d040 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50c80d293891224f server_cookie=a0674db83aa450e9 in_seq=0 out_seq=0 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.211+0000 7f6158c00700 10 client.16938 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.211+0000 7f6158c00700 1 -- 172.21.15.12:0/3242773283 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2a050f570 con 0x55e2a066ac20 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.211+0000 7f6158c00700 10 client.16938 handle_command_reply: tid=0 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.211+0000 7f61846a1b80 1 client.16938 shutdown 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05e0730 msgr2=0x55e2a05f3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05e0730 0x55e2a05f3b20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f615c0a9120 tx=0x7f615c0aede0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 shutdown_connections 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x55e2a066ac20 0x55e2a066d040 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e2a05f02c0 0x55e2a05f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2a0466e40 0x55e2a05f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 --2- 172.21.15.12:0/3242773283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e2a05e0730 0x55e2a05f3b20 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f6158c00700 0 client.16938 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 >> 172.21.15.12:0/3242773283 conn(0x55e2a0500390 msgr2=0x55e2a05e0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 shutdown_connections 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 1 -- 172.21.15.12:0/3242773283 wait complete. 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.212+0000 7f61846a1b80 20 client.16938 trim_cache size 0 max 16384 2024-05-27T07:34:55.210 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:34:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f61540b2b20 msgr2=0x7f61540b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f61540b2b20 0x7f61540b5030 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55e2a05e2a90 tx=0x7f61680082c0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170108d30 msgr2=0x7f617010a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170108d30 0x7f617010a4a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f615c00d980 tx=0x7f615c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:34:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 shutdown_connections 2024-05-27T07:34:55.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f61540b2b20 0x7f61540b5030 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6170109680 0x7f617010a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170104400 0x7f6170109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 --2- 172.21.15.12:0/4085120778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6170108d30 0x7f617010a4a0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 >> 172.21.15.12:0/4085120778 conn(0x7f6170079c10 msgr2=0x7f6170102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 shutdown_connections 2024-05-27T07:34:55.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:55.213+0000 7f616fe00700 1 -- 172.21.15.12:0/4085120778 wait complete. 2024-05-27T07:34:55.674 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:55 smithi012 ceph-mon[31473]: pgmap v431: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-05-27T07:34:55.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:55 smithi162 ceph-mon[35676]: pgmap v431: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-05-27T07:34:55.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:55 smithi062 ceph-mon[35304]: pgmap v431: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-05-27T07:34:56.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:34:57.128 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:57.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:57 smithi012 ceph-mon[31473]: pgmap v432: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1024 KiB/s wr, 121 op/s 2024-05-27T07:34:57.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:57 smithi162 ceph-mon[35676]: pgmap v432: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1024 KiB/s wr, 121 op/s 2024-05-27T07:34:57.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:57 smithi062 ceph-mon[35304]: pgmap v432: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1024 KiB/s wr, 121 op/s 2024-05-27T07:34:59.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:34:59 smithi012 ceph-mon[31473]: pgmap v433: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1024 KiB/s wr, 121 op/s 2024-05-27T07:34:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.634+0000 7f578fe00700 1 -- 172.21.15.12:0/4014172194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57901028e0 msgr2=0x7f5790102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.634+0000 7f578fe00700 1 --2- 172.21.15.12:0/4014172194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57901028e0 0x7f5790102cf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f57800099c0 tx=0x7f5780009cd0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.635+0000 7f578fe00700 1 -- 172.21.15.12:0/4014172194 shutdown_connections 2024-05-27T07:34:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.635+0000 7f578fe00700 1 --2- 172.21.15.12:0/4014172194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5790109650 0x7f5790110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.635+0000 7f578fe00700 1 --2- 172.21.15.12:0/4014172194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57901028e0 0x7f5790102cf0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.635+0000 7f578fe00700 1 --2- 172.21.15.12:0/4014172194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790108d00 0x7f5790109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.635+0000 7f578fe00700 1 -- 172.21.15.12:0/4014172194 >> 172.21.15.12:0/4014172194 conn(0x7f57900783a0 msgr2=0x7f57900787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.635+0000 7f578fe00700 1 -- 172.21.15.12:0/4014172194 shutdown_connections 2024-05-27T07:34:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.635+0000 7f578fe00700 1 -- 172.21.15.12:0/4014172194 wait complete. 2024-05-27T07:34:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.636+0000 7f578fe00700 1 Processor -- start 2024-05-27T07:34:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.636+0000 7f578fe00700 1 -- start start 2024-05-27T07:34:59.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57901028e0 0x7f5790103830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5790108d00 0x7f5790103d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790109650 0x7f57901042b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578fe00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5790114130 con 0x7f5790108d00 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5790113fb0 con 0x7f5790109650 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578fe00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57901142b0 con 0x7f57901028e0 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5790108d00 0x7f5790103d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5790108d00 0x7f5790103d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33450/0 (socket says 172.21.15.12:33450) 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578e000700 1 -- 172.21.15.12:0/4088869590 learned_addr learned my addr 172.21.15.12:0/4088869590 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578f400700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790109650 0x7f57901042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.637+0000 7f578ea00700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57901028e0 0x7f5790103830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f578e000700 1 -- 172.21.15.12:0/4088869590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57901028e0 msgr2=0x7f5790103830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f578e000700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57901028e0 0x7f5790103830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f578e000700 1 -- 172.21.15.12:0/4088869590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790109650 msgr2=0x7f57901042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f578e000700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790109650 0x7f57901042b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f578e000700 1 -- 172.21.15.12:0/4088869590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57901a8eb0 con 0x7f5790108d00 2024-05-27T07:34:59.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f578e000700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5790108d00 0x7f5790103d70 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f577c00d980 tx=0x7f577c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f5787400700 1 -- 172.21.15.12:0/4088869590 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c017070 con 0x7f5790108d00 2024-05-27T07:34:59.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57901a91a0 con 0x7f5790108d00 2024-05-27T07:34:59.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f5787400700 1 -- 172.21.15.12:0/4088869590 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f577c010460 con 0x7f5790108d00 2024-05-27T07:34:59.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.638+0000 7f5787400700 1 -- 172.21.15.12:0/4088869590 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c00de60 con 0x7f5790108d00 2024-05-27T07:34:59.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.639+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57901a96e0 con 0x7f5790108d00 2024-05-27T07:34:59.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.643+0000 7f5787400700 1 -- 172.21.15.12:0/4088869590 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f577c010610 con 0x7f5790108d00 2024-05-27T07:34:59.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.645+0000 7f57a46b4b80 1 Processor -- start 2024-05-27T07:34:59.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.645+0000 7f57a46b4b80 1 -- start start 2024-05-27T07:34:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.645+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fdbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc410 0x55f321fe6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc820 0x55f321fe9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f321f0ae40 con 0x55f321fdc820 2024-05-27T07:34:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f321efa9e0 con 0x55f321fdc410 2024-05-27T07:34:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f321efea20 con 0x55f321e5de40 2024-05-27T07:34:59.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f5787400700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57740b29f0 0x7f57740b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc820 0x55f321fe9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc820 0x55f321fe9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33452/0 (socket says 172.21.15.12:33452) 2024-05-27T07:34:59.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f578e000700 1 -- 172.21.15.12:0/392300280 learned_addr learned my addr 172.21.15.12:0/392300280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f578ea00700 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc410 0x55f321fe6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f578f400700 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fdbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.646+0000 7f5787400700 1 -- 172.21.15.12:0/4088869590 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f577c0e24f0 con 0x7f5790108d00 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578f400700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57740b29f0 0x7f57740b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578e000700 1 -- 172.21.15.12:0/392300280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 msgr2=0x55f321fdbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578e000700 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fdbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578e000700 1 -- 172.21.15.12:0/392300280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc410 msgr2=0x55f321fe6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578e000700 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc410 0x55f321fe6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578e000700 1 -- 172.21.15.12:0/392300280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f321ec3420 con 0x55f321fdc820 2024-05-27T07:34:59.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578f400700 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fdbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:59.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.647+0000 7f578e000700 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc820 0x55f321fe9020 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f577c0a5c20 tx=0x7f577c0a8c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a76373f147168f8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.648+0000 7f578f400700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57740b29f0 0x7f57740b4f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55f321fd8690 tx=0x7f5788004ad0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.649+0000 7f577be00700 1 -- 172.21.15.12:0/392300280 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c020070 con 0x55f321fdc820 2024-05-27T07:34:59.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.649+0000 7f577be00700 1 -- 172.21.15.12:0/392300280 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f577c0a9b60 con 0x55f321fdc820 2024-05-27T07:34:59.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.649+0000 7f577be00700 1 -- 172.21.15.12:0/392300280 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c0b2090 con 0x55f321fdc820 2024-05-27T07:34:59.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 -- 172.21.15.12:0/392300280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc820 msgr2=0x55f321fe9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc820 0x55f321fe9020 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f577c0a5c20 tx=0x7f577c0a8c50 comp rx=0 tx=0).stop 2024-05-27T07:34:59.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 -- 172.21.15.12:0/392300280 shutdown_connections 2024-05-27T07:34:59.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc820 0x55f321fe9020 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fdbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/392300280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc410 0x55f321fe6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 -- 172.21.15.12:0/392300280 >> 172.21.15.12:0/392300280 conn(0x55f321ebdfc0 msgr2=0x55f321fd7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 -- 172.21.15.12:0/392300280 shutdown_connections 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.650+0000 7f57a46b4b80 1 -- 172.21.15.12:0/392300280 wait complete. 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.651+0000 7f57a46b4b80 1 -- start start 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.651+0000 7f57a46b4b80 1 Processor -- start 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fe65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321fe6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f321e00fa0 con 0x55f321fdc410 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f321e9b760 con 0x55f321fdc820 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f321fe98d0 con 0x55f321e5de40 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f57a46b4b80 10 client.0 _ll_register_callbacks cb 0x55f321ea4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321fe6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321fe6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33462/0 (socket says 172.21.15.12:33462) 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.652+0000 7f578e000700 1 -- 172.21.15.12:0/2051686782 learned_addr learned my addr 172.21.15.12:0/2051686782 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578f400700 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578ea00700 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fe65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578e000700 1 -- 172.21.15.12:0/2051686782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 msgr2=0x55f321fe65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578e000700 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fe65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578e000700 1 -- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 msgr2=0x55f321fe7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578e000700 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578e000700 1 -- 172.21.15.12:0/2051686782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f321e4a8d0 con 0x55f321fdc410 2024-05-27T07:34:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f578e000700 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321fe6b30 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f577c0a9160 tx=0x7f577c0a98d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8c00f78efa515f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.653+0000 7f5778c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.654+0000 7f5778c00700 1 -- 172.21.15.12:0/2051686782 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c020070 con 0x55f321fdc410 2024-05-27T07:34:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.654+0000 7f5778c00700 1 -- 172.21.15.12:0/2051686782 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f577c0e2170 con 0x55f321fdc410 2024-05-27T07:34:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.654+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f577c0a9b60 con 0x55f321fdc410 2024-05-27T07:34:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.654+0000 7f5778c00700 1 -- 172.21.15.12:0/2051686782 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f577c0ad810 con 0x55f321fdc410 2024-05-27T07:34:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.654+0000 7f57a46b4b80 10 client.16950 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.654+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55f321ecb060 con 0x55f321fdc410 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f5778c00700 1 -- 172.21.15.12:0/2051686782 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f577c0a9b60 con 0x55f321fdc410 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f5778c00700 1 -- 172.21.15.12:0/2051686782 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f577c0a3b80 con 0x55f321fdc410 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f57a46b4b80 10 client.16950 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f57a46b4b80 10 client.16950 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f57a46b4b80 20 client.16950 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f321ff8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f57a46b4b80 4 client.16950 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.655+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f321f0cd90 con 0x55f321ff67a0 2024-05-27T07:34:59.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.656+0000 7f578ea00700 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f321ff8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.657+0000 7f578ea00700 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f321ff8c00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be3c7a20effd00dd server_cookie=58343418b5c13b4d in_seq=0 out_seq=0 2024-05-27T07:34:59.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.657+0000 7f5778c00700 10 client.16950 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:34:59.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.660+0000 7f5778c00700 1 -- 172.21.15.12:0/2051686782 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f321f0cd90 con 0x55f321ff67a0 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.660+0000 7f5778c00700 10 client.16950 handle_command_reply: tid=0 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 client.16950 shutdown 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 msgr2=0x55f321fe6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321fe6b30 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f577c0a9160 tx=0x7f577c0a98d0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 shutdown_connections 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f321ff8c00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f5778c00700 0 client.16950 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:34:59.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321e5de40 0x55f321fe65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/2051686782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321fe6b30 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.661+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 >> 172.21.15.12:0/2051686782 conn(0x55f321ebdfc0 msgr2=0x55f321fd6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.662+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 shutdown_connections 2024-05-27T07:34:59.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.662+0000 7f57a46b4b80 1 -- 172.21.15.12:0/2051686782 wait complete. 2024-05-27T07:34:59.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.662+0000 7f57a46b4b80 20 client.16950 trim_cache size 0 max 16384 2024-05-27T07:34:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.683+0000 7f57a46b4b80 1 Processor -- start 2024-05-27T07:34:59.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.683+0000 7f57a46b4b80 1 -- start start 2024-05-27T07:34:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321e5de40 0x55f321fe1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc410 0x55f321fe6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f57901b1bf0 con 0x55f321e5de40 2024-05-27T07:34:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f57901b0e50 con 0x55f321fdc820 2024-05-27T07:34:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57901b1520 con 0x55f321fdc410 2024-05-27T07:34:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f578ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321e5de40 0x55f321fe1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f578ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321e5de40 0x55f321fe1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33468/0 (socket says 172.21.15.12:33468) 2024-05-27T07:34:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.684+0000 7f578ea00700 1 -- 172.21.15.12:0/1432723027 learned_addr learned my addr 172.21.15.12:0/1432723027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.685+0000 7f578e000700 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc410 0x55f321fe6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.685+0000 7f578f400700 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.685+0000 7f578ea00700 1 -- 172.21.15.12:0/1432723027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc410 msgr2=0x55f321fe6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.685+0000 7f578ea00700 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc410 0x55f321fe6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.685+0000 7f578ea00700 1 -- 172.21.15.12:0/1432723027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 msgr2=0x55f321fe8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.685+0000 7f578ea00700 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.685+0000 7f578ea00700 1 -- 172.21.15.12:0/1432723027 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f321fd9350 con 0x55f321e5de40 2024-05-27T07:34:59.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.686+0000 7f578ea00700 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321e5de40 0x55f321fe1e70 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f57800099c0 tx=0x7f5780009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c85996e9c5f4cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.686+0000 7f577b400700 1 -- 172.21.15.12:0/1432723027 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578000b990 con 0x55f321e5de40 2024-05-27T07:34:59.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.686+0000 7f577b400700 1 -- 172.21.15.12:0/1432723027 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5780003680 con 0x55f321e5de40 2024-05-27T07:34:59.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.686+0000 7f577b400700 1 -- 172.21.15.12:0/1432723027 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57800246c0 con 0x55f321e5de40 2024-05-27T07:34:59.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.687+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1432723027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321e5de40 msgr2=0x55f321fe1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.687+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321e5de40 0x55f321fe1e70 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f57800099c0 tx=0x7f5780009310 comp rx=0 tx=0).stop 2024-05-27T07:34:59.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.687+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1432723027 shutdown_connections 2024-05-27T07:34:59.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.687+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321fdc820 0x55f321fe8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.687+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321e5de40 0x55f321fe1e70 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.687+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1432723027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc410 0x55f321fe6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.687+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1432723027 >> 172.21.15.12:0/1432723027 conn(0x55f321ebdfc0 msgr2=0x55f321ff1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.688+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1432723027 shutdown_connections 2024-05-27T07:34:59.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.688+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1432723027 wait complete. 2024-05-27T07:34:59.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.688+0000 7f57a46b4b80 1 -- start start 2024-05-27T07:34:59.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.688+0000 7f57a46b4b80 1 Processor -- start 2024-05-27T07:34:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.689+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321e5de40 0x55f321fea980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.689+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321feaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.689+0000 7f57a46b4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc820 0x55f321feb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.689+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f321fe63d0 con 0x55f321fdc410 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.689+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f321f0cd90 con 0x55f321e5de40 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.689+0000 7f57a46b4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f321fd21d0 con 0x55f321fdc820 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.689+0000 7f57a46b4b80 10 client.0 _ll_register_callbacks cb 0x55f321ff31d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321feaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321feaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33476/0 (socket says 172.21.15.12:33476) 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 -- 172.21.15.12:0/1834173908 learned_addr learned my addr 172.21.15.12:0/1834173908 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578ea00700 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321e5de40 0x55f321fea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 -- 172.21.15.12:0/1834173908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc820 msgr2=0x55f321feb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc820 0x55f321feb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 -- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321e5de40 msgr2=0x55f321fea980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321e5de40 0x55f321fea980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 -- 172.21.15.12:0/1834173908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f322060310 con 0x55f321fdc410 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f578e000700 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321feaec0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f577c000c00 tx=0x7f577c0e28d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f246c5a23fc1072 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.690+0000 7f5778c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f5778c00700 1 -- 172.21.15.12:0/1834173908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c020070 con 0x55f321fdc410 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f5778c00700 1 -- 172.21.15.12:0/1834173908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f577c0b0b90 con 0x55f321fdc410 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5780003680 con 0x55f321fdc410 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f5778c00700 1 -- 172.21.15.12:0/1834173908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f577c0a87f0 con 0x55f321fdc410 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f57a46b4b80 10 client.16956 fetch_fsmap learned FSMap version 119 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55f322061550 con 0x55f321fdc410 2024-05-27T07:34:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f5778c00700 1 -- 172.21.15.12:0/1834173908 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5780003680 con 0x55f321fdc410 2024-05-27T07:34:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f5778c00700 1 -- 172.21.15.12:0/1834173908 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f577c0a4dd0 con 0x55f321fdc410 2024-05-27T07:34:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.691+0000 7f57a46b4b80 10 client.16956 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:34:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.692+0000 7f57a46b4b80 10 client.16956 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:34:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.692+0000 7f57a46b4b80 20 client.16956 populate_metadata read hostname 'smithi012' 2024-05-27T07:34:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.692+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f3220639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.692+0000 7f57a46b4b80 4 client.16956 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.692+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f321f06570 con 0x55f321ff67a0 2024-05-27T07:34:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.692+0000 7f578ea00700 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f3220639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.694+0000 7f578ea00700 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f3220639a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5011294aed18a968 server_cookie=f93507ae2c4655b4 in_seq=0 out_seq=0 2024-05-27T07:34:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.694+0000 7f5778c00700 10 client.16956 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.694+0000 7f5778c00700 1 -- 172.21.15.12:0/1834173908 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f321f06570 con 0x55f321ff67a0 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.694+0000 7f5778c00700 10 client.16956 handle_command_reply: tid=0 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.694+0000 7f57a46b4b80 1 client.16956 shutdown 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 msgr2=0x55f321feaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321feaec0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f577c000c00 tx=0x7f577c0e28d0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 shutdown_connections 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55f321ff67a0 0x55f3220639a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f321e5de40 0x55f321fea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f321fdc820 0x55f321feb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 --2- 172.21.15.12:0/1834173908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f321fdc410 0x55f321feaec0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 >> 172.21.15.12:0/1834173908 conn(0x55f321ebdfc0 msgr2=0x55f321fe6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f5778c00700 0 client.16956 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:34:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.695+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 shutdown_connections 2024-05-27T07:34:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.696+0000 7f57a46b4b80 1 -- 172.21.15.12:0/1834173908 wait complete. 2024-05-27T07:34:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.696+0000 7f57a46b4b80 20 client.16956 trim_cache size 0 max 16384 2024-05-27T07:34:59.694 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:34:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.697+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57740b29f0 msgr2=0x7f57740b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.697+0000 7f578fe00700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57740b29f0 0x7f57740b4f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55f321fd8690 tx=0x7f5788004ad0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.697+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5790108d00 msgr2=0x7f5790103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.697+0000 7f578fe00700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5790108d00 0x7f5790103d70 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f577c00d980 tx=0x7f577c00dd40 comp rx=0 tx=0).stop 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.697+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 shutdown_connections 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.698+0000 7f578fe00700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f57740b29f0 0x7f57740b4f00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.698+0000 7f578fe00700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790109650 0x7f57901042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.698+0000 7f578fe00700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57901028e0 0x7f5790103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.698+0000 7f578fe00700 1 --2- 172.21.15.12:0/4088869590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5790108d00 0x7f5790103d70 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.698+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 >> 172.21.15.12:0/4088869590 conn(0x7f57900783a0 msgr2=0x7f5790100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.698+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 shutdown_connections 2024-05-27T07:34:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:34:59.698+0000 7f578fe00700 1 -- 172.21.15.12:0/4088869590 wait complete. 2024-05-27T07:34:59.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:34:59 smithi162 ceph-mon[35676]: pgmap v433: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1024 KiB/s wr, 121 op/s 2024-05-27T07:34:59.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:34:59 smithi062 ceph-mon[35304]: pgmap v433: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1024 KiB/s wr, 121 op/s 2024-05-27T07:35:00.671 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:35:01.150 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:01.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:01 smithi012 ceph-mon[31473]: pgmap v434: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-27T07:35:01.718 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:01 smithi062 ceph-mon[35304]: pgmap v434: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-27T07:35:01.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:01 smithi162 ceph-mon[35676]: pgmap v434: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-27T07:35:03.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:03 smithi012 ceph-mon[31473]: pgmap v435: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-05-27T07:35:03.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:03 smithi162 ceph-mon[35676]: pgmap v435: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-05-27T07:35:03.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:03 smithi062 ceph-mon[35304]: pgmap v435: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-05-27T07:35:03.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 -- 172.21.15.12:0/3467444353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7080109860 msgr2=0x7f70801111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 --2- 172.21.15.12:0/3467444353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7080109860 0x7f70801111c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f707800b0d0 tx=0x7f707800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 -- 172.21.15.12:0/3467444353 shutdown_connections 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 --2- 172.21.15.12:0/3467444353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7080109860 0x7f70801111c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 --2- 172.21.15.12:0/3467444353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70801045e0 0x7f70801049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 --2- 172.21.15.12:0/3467444353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080108f10 0x7f7080109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 -- 172.21.15.12:0/3467444353 >> 172.21.15.12:0/3467444353 conn(0x7f7080100030 msgr2=0x7f7080102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 -- 172.21.15.12:0/3467444353 shutdown_connections 2024-05-27T07:35:03.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.945+0000 7f7085400700 1 -- 172.21.15.12:0/3467444353 wait complete. 2024-05-27T07:35:03.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.946+0000 7f7085400700 1 Processor -- start 2024-05-27T07:35:03.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.946+0000 7f7085400700 1 -- start start 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f7085400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70801045e0 0x7f708019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f7085400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080108f10 0x7f708019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f7085400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7080109860 0x7f70801a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f7085400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7080114340 con 0x7f70801045e0 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f7085400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f70801141c0 con 0x7f7080108f10 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f7085400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70801144c0 con 0x7f7080109860 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f707e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70801045e0 0x7f708019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f707e800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70801045e0 0x7f708019cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51964/0 (socket says 172.21.15.12:51964) 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f707e800700 1 -- 172.21.15.12:0/971153046 learned_addr learned my addr 172.21.15.12:0/971153046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:03.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.947+0000 7f707de00700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080108f10 0x7f708019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707f200700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7080109860 0x7f70801a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707e800700 1 -- 172.21.15.12:0/971153046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7080109860 msgr2=0x7f70801a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707e800700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7080109860 0x7f70801a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707e800700 1 -- 172.21.15.12:0/971153046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080108f10 msgr2=0x7f708019d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707e800700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080108f10 0x7f708019d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707e800700 1 -- 172.21.15.12:0/971153046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70801a1d50 con 0x7f70801045e0 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707f200700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7080109860 0x7f70801a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.948+0000 7f707e800700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70801045e0 0x7f708019cf30 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f707000c920 tx=0x7f707000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.949+0000 7f7077400700 1 -- 172.21.15.12:0/971153046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070007b00 con 0x7f70801045e0 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.949+0000 7f7077400700 1 -- 172.21.15.12:0/971153046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7070004cc0 con 0x7f70801045e0 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.949+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70801a2040 con 0x7f70801045e0 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.949+0000 7f7077400700 1 -- 172.21.15.12:0/971153046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070005440 con 0x7f70801045e0 2024-05-27T07:35:03.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.949+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801a9a10 con 0x7f70801045e0 2024-05-27T07:35:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.953+0000 7f7077400700 1 -- 172.21.15.12:0/971153046 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f707000f450 con 0x7f70801045e0 2024-05-27T07:35:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.954+0000 7f7093ff5b80 1 Processor -- start 2024-05-27T07:35:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.955+0000 7f7093ff5b80 1 -- start start 2024-05-27T07:35:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.955+0000 7f7077400700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f70640b2a70 0x7f70640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.955+0000 7f7077400700 1 -- 172.21.15.12:0/971153046 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f70700e24c0 con 0x7f70801045e0 2024-05-27T07:35:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f707f200700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f70640b2a70 0x7f70640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c194553e40 0x55c1946d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946df110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c194600e40 con 0x55c194553e40 2024-05-27T07:35:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c1945f09e0 con 0x55c19463f9f0 2024-05-27T07:35:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c1945f4a20 con 0x55c1946d2e70 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f707e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f707e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946d2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:55428/0 (socket says 172.21.15.12:55428) 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f707e800700 1 -- 172.21.15.12:0/3021652882 learned_addr learned my addr 172.21.15.12:0/3021652882 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.956+0000 7f707de00700 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c194553e40 0x55c1946d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707f200700 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946df110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707f200700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f70640b2a70 0x7f70640b4f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7078009e60 tx=0x7f7078009200 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707de00700 1 -- 172.21.15.12:0/3021652882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 msgr2=0x55c1946df110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707de00700 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946df110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707de00700 1 -- 172.21.15.12:0/3021652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 msgr2=0x55c1946d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707de00700 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707de00700 1 -- 172.21.15.12:0/3021652882 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1945b9420 con 0x55c194553e40 2024-05-27T07:35:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707f200700 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946df110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.957+0000 7f707e800700 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.958+0000 7f707de00700 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c194553e40 0x55c1946d2870 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f708006c670 tx=0x7f706c00ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71a7f539cd3e55b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.958+0000 7f706be00700 1 -- 172.21.15.12:0/3021652882 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c00fd80 con 0x55c194553e40 2024-05-27T07:35:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.959+0000 7f706be00700 1 -- 172.21.15.12:0/3021652882 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f706c0044b0 con 0x55c194553e40 2024-05-27T07:35:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.959+0000 7f706be00700 1 -- 172.21.15.12:0/3021652882 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c005280 con 0x55c194553e40 2024-05-27T07:35:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 -- 172.21.15.12:0/3021652882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c194553e40 msgr2=0x55c1946d2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c194553e40 0x55c1946d2870 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f708006c670 tx=0x7f706c00ae90 comp rx=0 tx=0).stop 2024-05-27T07:35:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 -- 172.21.15.12:0/3021652882 shutdown_connections 2024-05-27T07:35:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946df110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/3021652882 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c194553e40 0x55c1946d2870 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 -- 172.21.15.12:0/3021652882 >> 172.21.15.12:0/3021652882 conn(0x55c1945ed390 msgr2=0x55c1946ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 -- 172.21.15.12:0/3021652882 shutdown_connections 2024-05-27T07:35:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.960+0000 7f7093ff5b80 1 -- 172.21.15.12:0/3021652882 wait complete. 2024-05-27T07:35:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.961+0000 7f7093ff5b80 1 -- start start 2024-05-27T07:35:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.961+0000 7f7093ff5b80 1 Processor -- start 2024-05-27T07:35:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c194553e40 0x55c1946dc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946dcc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946dd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c1944f6fa0 con 0x55c1946d2e70 2024-05-27T07:35:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c194591760 con 0x55c19463f9f0 2024-05-27T07:35:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c1946dfb80 con 0x55c194553e40 2024-05-27T07:35:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f7093ff5b80 10 client.0 _ll_register_callbacks cb 0x55c19459ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.962+0000 7f707f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946dd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946dd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51986/0 (socket says 172.21.15.12:51986) 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707f200700 1 -- 172.21.15.12:0/329643415 learned_addr learned my addr 172.21.15.12:0/329643415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707de00700 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946dcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707e800700 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c194553e40 0x55c1946dc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707f200700 1 -- 172.21.15.12:0/329643415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c194553e40 msgr2=0x55c1946dc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707f200700 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c194553e40 0x55c1946dc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707f200700 1 -- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 msgr2=0x55c1946dcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707f200700 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946dcc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.963+0000 7f707f200700 1 -- 172.21.15.12:0/329643415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1945408d0 con 0x55c1946d2e70 2024-05-27T07:35:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.964+0000 7f707f200700 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946dd140 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f707800b010 tx=0x7f707800b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beaeb70c862f618b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.964+0000 7f7068c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7068c00700 1 -- 172.21.15.12:0/329643415 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078030060 con 0x55c1946d2e70 2024-05-27T07:35:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7068c00700 1 -- 172.21.15.12:0/329643415 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f707801eda0 con 0x55c1946d2e70 2024-05-27T07:35:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f706c0044b0 con 0x55c1946d2e70 2024-05-27T07:35:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7068c00700 1 -- 172.21.15.12:0/329643415 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f7078025070 con 0x55c1946d2e70 2024-05-27T07:35:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7093ff5b80 10 client.16968 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55c1945c1060 con 0x55c1946d2e70 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7068c00700 1 -- 172.21.15.12:0/329643415 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f706c0044b0 con 0x55c1946d2e70 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.965+0000 7f7068c00700 1 -- 172.21.15.12:0/329643415 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f7078011070 con 0x55c1946d2e70 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.966+0000 7f7093ff5b80 10 client.16968 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.966+0000 7f7093ff5b80 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.966+0000 7f7093ff5b80 20 client.16968 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.966+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c1946ecfa0 0x55c1946ef430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.966+0000 7f7093ff5b80 4 client.16968 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.966+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c194602d90 con 0x55c1946ecfa0 2024-05-27T07:35:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.966+0000 7f707e800700 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c1946ecfa0 0x55c1946ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.967+0000 7f707e800700 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c1946ecfa0 0x55c1946ef430 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf68e6b02262241d server_cookie=51dbd664a842c11d in_seq=0 out_seq=0 2024-05-27T07:35:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.967+0000 7f7068c00700 10 client.16968 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.969+0000 7f7068c00700 1 -- 172.21.15.12:0/329643415 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c194602d90 con 0x55c1946ecfa0 2024-05-27T07:35:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.969+0000 7f7068c00700 10 client.16968 handle_command_reply: tid=0 2024-05-27T07:35:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.969+0000 7f7093ff5b80 1 client.16968 shutdown 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 msgr2=0x55c1946dd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946dd140 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f707800b010 tx=0x7f707800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 shutdown_connections 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946dd140 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c194553e40 0x55c1946dc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c1946ecfa0 0x55c1946ef430 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/329643415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19463f9f0 0x55c1946dcc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 >> 172.21.15.12:0/329643415 conn(0x55c1945ed390 msgr2=0x55c1946d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7068c00700 0 client.16968 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 shutdown_connections 2024-05-27T07:35:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 1 -- 172.21.15.12:0/329643415 wait complete. 2024-05-27T07:35:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.970+0000 7f7093ff5b80 20 client.16968 trim_cache size 0 max 16384 2024-05-27T07:35:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.989+0000 7f7093ff5b80 1 Processor -- start 2024-05-27T07:35:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.989+0000 7f7093ff5b80 1 -- start start 2024-05-27T07:35:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 0x55c1946d7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c19463f9f0 0x55c1946e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f706000e410 con 0x55c19463f9f0 2024-05-27T07:35:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f706000d670 con 0x55c194553e40 2024-05-27T07:35:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f706000dd40 con 0x55c1946d2e70 2024-05-27T07:35:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f707de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c19463f9f0 0x55c1946e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f707de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c19463f9f0 0x55c1946e8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51992/0 (socket says 172.21.15.12:51992) 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.990+0000 7f707de00700 1 -- 172.21.15.12:0/210776 learned_addr learned my addr 172.21.15.12:0/210776 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.991+0000 7f707f200700 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.991+0000 7f707de00700 1 -- 172.21.15.12:0/210776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 msgr2=0x55c1946cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.991+0000 7f707de00700 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946cdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.991+0000 7f707de00700 1 -- 172.21.15.12:0/210776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 msgr2=0x55c1946d7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.991+0000 7f707de00700 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 0x55c1946d7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.991+0000 7f707de00700 1 -- 172.21.15.12:0/210776 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1946d3a50 con 0x55c19463f9f0 2024-05-27T07:35:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.992+0000 7f707de00700 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c19463f9f0 0x55c1946e8c80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f706c00ab00 tx=0x7f706c00af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9227050c3a492f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.992+0000 7f706b400700 1 -- 172.21.15.12:0/210776 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c027030 con 0x55c19463f9f0 2024-05-27T07:35:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.992+0000 7f706b400700 1 -- 172.21.15.12:0/210776 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f706c01d040 con 0x55c19463f9f0 2024-05-27T07:35:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.992+0000 7f706b400700 1 -- 172.21.15.12:0/210776 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c01c070 con 0x55c19463f9f0 2024-05-27T07:35:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 -- 172.21.15.12:0/210776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c19463f9f0 msgr2=0x55c1946e8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c19463f9f0 0x55c1946e8c80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f706c00ab00 tx=0x7f706c00af60 comp rx=0 tx=0).stop 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 -- 172.21.15.12:0/210776 shutdown_connections 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 0x55c1946d7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1946d2e70 0x55c1946cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/210776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c19463f9f0 0x55c1946e8c80 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 -- 172.21.15.12:0/210776 >> 172.21.15.12:0/210776 conn(0x55c1945ed390 msgr2=0x55c1946dafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 -- 172.21.15.12:0/210776 shutdown_connections 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.994+0000 7f7093ff5b80 1 -- 172.21.15.12:0/210776 wait complete. 2024-05-27T07:35:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.995+0000 7f7093ff5b80 1 -- start start 2024-05-27T07:35:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.995+0000 7f7093ff5b80 1 Processor -- start 2024-05-27T07:35:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 0x55c1946e5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c19463f9f0 0x55c1946dfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f7093ff5b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946e0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c1946dac30 con 0x55c1946d2e70 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c1946ef9e0 con 0x55c194553e40 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f7093ff5b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c1946e91c0 con 0x55c19463f9f0 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f7093ff5b80 10 client.0 _ll_register_callbacks cb 0x55c19459ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f707f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f707f200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946e0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52002/0 (socket says 172.21.15.12:52002) 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f707f200700 1 -- 172.21.15.12:0/4200865285 learned_addr learned my addr 172.21.15.12:0/4200865285 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f707e800700 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 0x55c1946e5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.996+0000 7f707de00700 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c19463f9f0 0x55c1946dfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f707f200700 1 -- 172.21.15.12:0/4200865285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c19463f9f0 msgr2=0x55c1946dfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f707f200700 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c19463f9f0 0x55c1946dfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f707f200700 1 -- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 msgr2=0x55c1946e5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f707f200700 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 0x55c1946e5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f707f200700 1 -- 172.21.15.12:0/4200865285 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1946e0a20 con 0x55c1946d2e70 2024-05-27T07:35:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f707f200700 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946e0370 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f707800afe0 tx=0x7f7078006190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d089ed695c5f0365 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f7068c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.997+0000 7f7068c00700 1 -- 172.21.15.12:0/4200865285 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078030070 con 0x55c1946d2e70 2024-05-27T07:35:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.998+0000 7f7068c00700 1 -- 172.21.15.12:0/4200865285 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 9554+0+0 (secure 0 0 0) 0x7f707801ed30 con 0x55c1946d2e70 2024-05-27T07:35:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.998+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f706c01d040 con 0x55c1946d2e70 2024-05-27T07:35:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.998+0000 7f7068c00700 1 -- 172.21.15.12:0/4200865285 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f7078004a50 con 0x55c1946d2e70 2024-05-27T07:35:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.998+0000 7f7093ff5b80 10 client.16974 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.998+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55c194756930 con 0x55c1946d2e70 2024-05-27T07:35:03.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.998+0000 7f7068c00700 1 -- 172.21.15.12:0/4200865285 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f706c01d040 con 0x55c1946d2e70 2024-05-27T07:35:03.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f7068c00700 1 -- 172.21.15.12:0/4200865285 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f7078011040 con 0x55c1946d2e70 2024-05-27T07:35:03.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f7093ff5b80 10 client.16974 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:03.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f7093ff5b80 10 client.16974 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:03.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f7093ff5b80 20 client.16974 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:03.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c194756ef0 0x55c194759310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:03.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f7093ff5b80 4 client.16974 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:03.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1945fc570 con 0x55c194756ef0 2024-05-27T07:35:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:03.999+0000 7f707e800700 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c194756ef0 0x55c194759310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.001+0000 7f707e800700 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c194756ef0 0x55c194759310 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6f9844912d460fa server_cookie=8857286402df720 in_seq=0 out_seq=0 2024-05-27T07:35:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.001+0000 7f7068c00700 10 client.16974 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.001+0000 7f7068c00700 1 -- 172.21.15.12:0/4200865285 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1945fc570 con 0x55c194756ef0 2024-05-27T07:35:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.001+0000 7f7068c00700 10 client.16974 handle_command_reply: tid=0 2024-05-27T07:35:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.001+0000 7f7093ff5b80 1 client.16974 shutdown 2024-05-27T07:35:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 msgr2=0x55c1946e0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946e0370 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f707800afe0 tx=0x7f7078006190 comp rx=0 tx=0).stop 2024-05-27T07:35:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 shutdown_connections 2024-05-27T07:35:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c194756ef0 0x55c194759310 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c194553e40 0x55c1946e5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c1946d2e70 0x55c1946e0370 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 --2- 172.21.15.12:0/4200865285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c19463f9f0 0x55c1946dfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7068c00700 0 client.16974 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 >> 172.21.15.12:0/4200865285 conn(0x55c1945ed390 msgr2=0x55c1946daf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.002+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 shutdown_connections 2024-05-27T07:35:04.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.003+0000 7f7093ff5b80 1 -- 172.21.15.12:0/4200865285 wait complete. 2024-05-27T07:35:04.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.003+0000 7f7093ff5b80 20 client.16974 trim_cache size 0 max 16384 2024-05-27T07:35:04.003 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:35:04.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f70640b2a70 msgr2=0x7f70640b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f70640b2a70 0x7f70640b4f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7078009e60 tx=0x7f7078009200 comp rx=0 tx=0).stop 2024-05-27T07:35:04.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70801045e0 msgr2=0x7f708019cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70801045e0 0x7f708019cf30 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f707000c920 tx=0x7f707000cc30 comp rx=0 tx=0).stop 2024-05-27T07:35:04.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 shutdown_connections 2024-05-27T07:35:04.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f70640b2a70 0x7f70640b4f80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7080109860 0x7f70801a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70801045e0 0x7f708019cf30 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 --2- 172.21.15.12:0/971153046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080108f10 0x7f708019d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.004+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 >> 172.21.15.12:0/971153046 conn(0x7f7080100030 msgr2=0x7f7080110ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.005+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 shutdown_connections 2024-05-27T07:35:04.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:04.005+0000 7f7085400700 1 -- 172.21.15.12:0/971153046 wait complete. 2024-05-27T07:35:05.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:05 smithi162 ceph-mon[35676]: pgmap v436: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:35:05.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:05 smithi062 ceph-mon[35304]: pgmap v436: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:35:05.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:05 smithi012 ceph-mon[31473]: pgmap v436: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-05-27T07:35:05.984 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:35:05.984 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:35:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:07.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:07 smithi012 ceph-mon[31473]: pgmap v437: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-05-27T07:35:07.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:07 smithi012 ceph-mon[31473]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:35:07.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:07 smithi162 ceph-mon[35676]: pgmap v437: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-05-27T07:35:07.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:07 smithi162 ceph-mon[35676]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:35:07.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:07 smithi062 ceph-mon[35304]: pgmap v437: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-05-27T07:35:07.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:07 smithi062 ceph-mon[35304]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.121+0000 7f3002800700 1 -- 172.21.15.12:0/3525848051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 msgr2=0x7f2ffc102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.121+0000 7f3002800700 1 --2- 172.21.15.12:0/3525848051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc102ee0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00b0d0 tx=0x7f2fec00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.122+0000 7f3002800700 1 -- 172.21.15.12:0/3525848051 shutdown_connections 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.122+0000 7f3002800700 1 --2- 172.21.15.12:0/3525848051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ffc109840 0x7f2ffc1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.122+0000 7f3002800700 1 --2- 172.21.15.12:0/3525848051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc102ee0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.122+0000 7f3002800700 1 --2- 172.21.15.12:0/3525848051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ffc108ef0 0x7f2ffc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.122+0000 7f3002800700 1 -- 172.21.15.12:0/3525848051 >> 172.21.15.12:0/3525848051 conn(0x7f2ffc0fe510 msgr2=0x7f2ffc100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.122+0000 7f3002800700 1 -- 172.21.15.12:0/3525848051 shutdown_connections 2024-05-27T07:35:09.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.122+0000 7f3002800700 1 -- 172.21.15.12:0/3525848051 wait complete. 2024-05-27T07:35:09.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.123+0000 7f3002800700 1 Processor -- start 2024-05-27T07:35:09.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.123+0000 7f3002800700 1 -- start start 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f3002800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f3002800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ffc108ef0 0x7f2ffc19d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f3002800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ffc109840 0x7f2ffc1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f3002800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc114320 con 0x7f2ffc102ad0 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f3002800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1141a0 con 0x7f2ffc108ef0 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f3002800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1144a0 con 0x7f2ffc109840 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f2ffb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f2ffb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52030/0 (socket says 172.21.15.12:52030) 2024-05-27T07:35:09.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.124+0000 7f2ffb400700 1 -- 172.21.15.12:0/2689734263 learned_addr learned my addr 172.21.15.12:0/2689734263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ffc109840 0x7f2ffc1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ff2a00700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ffc108ef0 0x7f2ffc19d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ffb400700 1 -- 172.21.15.12:0/2689734263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ffc109840 msgr2=0x7f2ffc1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ffc109840 0x7f2ffc1a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ffb400700 1 -- 172.21.15.12:0/2689734263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ffc108ef0 msgr2=0x7f2ffc19d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ffc108ef0 0x7f2ffc19d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ffb400700 1 -- 172.21.15.12:0/2689734263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc1a1e20 con 0x7f2ffc102ad0 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ff2a00700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ffc108ef0 0x7f2ffc19d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.125+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc19cf50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2fec007d90 tx=0x7f2fec007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.126+0000 7f2ff8c00700 1 -- 172.21.15.12:0/2689734263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec01f030 con 0x7f2ffc102ad0 2024-05-27T07:35:09.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.126+0000 7f2ff8c00700 1 -- 172.21.15.12:0/2689734263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2fec01bad0 con 0x7f2ffc102ad0 2024-05-27T07:35:09.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.126+0000 7f2ff8c00700 1 -- 172.21.15.12:0/2689734263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec012b50 con 0x7f2ffc102ad0 2024-05-27T07:35:09.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.126+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ffc1a20b0 con 0x7f2ffc102ad0 2024-05-27T07:35:09.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.126+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ffc1a9a30 con 0x7f2ffc102ad0 2024-05-27T07:35:09.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.130+0000 7f2ff8c00700 1 -- 172.21.15.12:0/2689734263 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2fec022030 con 0x7f2ffc102ad0 2024-05-27T07:35:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f3011341b80 1 Processor -- start 2024-05-27T07:35:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f3011341b80 1 -- start start 2024-05-27T07:35:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f2ff8c00700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2fe00b2a70 0x7f2fe00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f2ff8c00700 1 -- 172.21.15.12:0/2689734263 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 9642+0+0 (secure 0 0 0) 0x7f2fec00f020 con 0x7f2ffc102ad0 2024-05-27T07:35:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f3011341b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fcd2ec0 0x56417fe57870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f3011341b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe57db0 0x56417fe518f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f3011341b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe52240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.132+0000 7f3011341b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56417fd7fec0 con 0x56417fcd2ec0 2024-05-27T07:35:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.133+0000 7f3011341b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56417fd6fa60 con 0x56417fe51e30 2024-05-27T07:35:09.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.133+0000 7f3011341b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56417fd73aa0 con 0x56417fe57db0 2024-05-27T07:35:09.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.133+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2fe00b2a70 0x7f2fe00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.133+0000 7f2ffb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fcd2ec0 0x56417fe57870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.133+0000 7f2ff2a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe57db0 0x56417fe518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.133+0000 7f2ffb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fcd2ec0 0x56417fe57870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52038/0 (socket says 172.21.15.12:52038) 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.133+0000 7f2ffb400700 1 -- 172.21.15.12:0/2885351276 learned_addr learned my addr 172.21.15.12:0/2885351276 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe52240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffb400700 1 -- 172.21.15.12:0/2885351276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe57db0 msgr2=0x56417fe518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe57db0 0x56417fe518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffb400700 1 -- 172.21.15.12:0/2885351276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 msgr2=0x56417fe52240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe52240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffb400700 1 -- 172.21.15.12:0/2885351276 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417fd384a0 con 0x56417fcd2ec0 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2fe00b2a70 0x7f2fe00b4f80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400bb40 tx=0x7f2ff400bf00 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ff2a00700 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe57db0 0x56417fe518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:09.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.134+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fcd2ec0 0x56417fe57870 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0e0fa0 tx=0x7f2fec005b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a54c993ffc75e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.135+0000 7f2ff0c00700 1 -- 172.21.15.12:0/2885351276 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0ab320 con 0x56417fcd2ec0 2024-05-27T07:35:09.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.135+0000 7f2ff0c00700 1 -- 172.21.15.12:0/2885351276 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2fec0aac20 con 0x56417fcd2ec0 2024-05-27T07:35:09.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.135+0000 7f2ff0c00700 1 -- 172.21.15.12:0/2885351276 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0e1280 con 0x56417fcd2ec0 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 -- 172.21.15.12:0/2885351276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fcd2ec0 msgr2=0x56417fe57870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fcd2ec0 0x56417fe57870 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0e0fa0 tx=0x7f2fec005b60 comp rx=0 tx=0).stop 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 -- 172.21.15.12:0/2885351276 shutdown_connections 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe52240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fcd2ec0 0x56417fe57870 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 --2- 172.21.15.12:0/2885351276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe57db0 0x56417fe518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 -- 172.21.15.12:0/2885351276 >> 172.21.15.12:0/2885351276 conn(0x56417fd33040 msgr2=0x56417fe4c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 -- 172.21.15.12:0/2885351276 shutdown_connections 2024-05-27T07:35:09.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.136+0000 7f3011341b80 1 -- 172.21.15.12:0/2885351276 wait complete. 2024-05-27T07:35:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.137+0000 7f3011341b80 1 -- start start 2024-05-27T07:35:09.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.137+0000 7f3011341b80 1 Processor -- start 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f3011341b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe5b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f3011341b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe5bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f3011341b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe5c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f3011341b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56417fc76020 con 0x56417fe57db0 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f3011341b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56417fd107e0 con 0x56417fe51e30 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f3011341b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56417fe5e640 con 0x56417fcd2ec0 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f3011341b80 10 client.0 _ll_register_callbacks cb 0x56417fd19cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.138+0000 7f2ffbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe5c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe5c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52040/0 (socket says 172.21.15.12:52040) 2024-05-27T07:35:09.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 -- 172.21.15.12:0/2763143248 learned_addr learned my addr 172.21.15.12:0/2763143248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ff2a00700 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe5bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe5b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 -- 172.21.15.12:0/2763143248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 msgr2=0x56417fe5b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe5b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 -- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 msgr2=0x56417fe5bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe5bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 -- 172.21.15.12:0/2763143248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417fcbf950 con 0x56417fe57db0 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe5c070 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40123f0 tx=0x7f2ff40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3a04746510a992 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.139+0000 7f2fe9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.140+0000 7f2fe9600700 1 -- 172.21.15.12:0/2763143248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff401e630 con 0x56417fe57db0 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.140+0000 7f2fe9600700 1 -- 172.21.15.12:0/2763143248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 9642+0+0 (secure 0 0 0) 0x7f2ff401e7d0 con 0x56417fe57db0 2024-05-27T07:35:09.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.140+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fec0aac20 con 0x56417fe57db0 2024-05-27T07:35:09.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.140+0000 7f2fe9600700 1 -- 172.21.15.12:0/2763143248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f2ff4031050 con 0x56417fe57db0 2024-05-27T07:35:09.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.140+0000 7f3011341b80 10 client.16986 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:09.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.140+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56417fd400e0 con 0x56417fe57db0 2024-05-27T07:35:09.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.141+0000 7f2fe9600700 1 -- 172.21.15.12:0/2763143248 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2fec0aac20 con 0x56417fe57db0 2024-05-27T07:35:09.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.142+0000 7f2fe9600700 1 -- 172.21.15.12:0/2763143248 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f2ff4026a20 con 0x56417fe57db0 2024-05-27T07:35:09.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.142+0000 7f3011341b80 10 client.16986 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:09.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.142+0000 7f3011341b80 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:09.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.142+0000 7f3011341b80 20 client.16986 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:09.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.142+0000 7f3011341b80 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fe6b940 0x56417fe6dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.142+0000 7f3011341b80 4 client.16986 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:09.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.142+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56417fd81e10 con 0x56417fe6b940 2024-05-27T07:35:09.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.143+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fe6b940 0x56417fe6dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.144+0000 7f2ffb400700 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fe6b940 0x56417fe6dd90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c0548472d848e19 server_cookie=f6a6422b5fef3236 in_seq=0 out_seq=0 2024-05-27T07:35:09.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.144+0000 7f2fe9600700 10 client.16986 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.146+0000 7f2fe9600700 1 -- 172.21.15.12:0/2763143248 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56417fd81e10 con 0x56417fe6b940 2024-05-27T07:35:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.146+0000 7f2fe9600700 10 client.16986 handle_command_reply: tid=0 2024-05-27T07:35:09.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.146+0000 7f3011341b80 1 client.16986 shutdown 2024-05-27T07:35:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.146+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 msgr2=0x56417fe5c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.146+0000 7f3011341b80 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe5c070 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40123f0 tx=0x7f2ff40047c0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 shutdown_connections 2024-05-27T07:35:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe5c070 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe5b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fe6b940 0x56417fe6dd90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 --2- 172.21.15.12:0/2763143248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe5bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f2fe9600700 0 client.16986 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 >> 172.21.15.12:0/2763143248 conn(0x56417fd33040 msgr2=0x56417fe4c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 shutdown_connections 2024-05-27T07:35:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 1 -- 172.21.15.12:0/2763143248 wait complete. 2024-05-27T07:35:09.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.147+0000 7f3011341b80 20 client.16986 trim_cache size 0 max 16384 2024-05-27T07:35:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 Processor -- start 2024-05-27T07:35:09.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 -- start start 2024-05-27T07:35:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fcd2ec0 0x56417fe518f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe51e30 0x56417fe5da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1b1ee0 con 0x56417fe57db0 2024-05-27T07:35:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1b1140 con 0x56417fcd2ec0 2024-05-27T07:35:09.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.173+0000 7f3011341b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc1b1810 con 0x56417fe51e30 2024-05-27T07:35:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe676a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52052/0 (socket says 172.21.15.12:52052) 2024-05-27T07:35:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 -- 172.21.15.12:0/4003794407 learned_addr learned my addr 172.21.15.12:0/4003794407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffb400700 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fcd2ec0 0x56417fe518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ff2a00700 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe51e30 0x56417fe5da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 -- 172.21.15.12:0/4003794407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe51e30 msgr2=0x56417fe5da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe51e30 0x56417fe5da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 -- 172.21.15.12:0/4003794407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fcd2ec0 msgr2=0x56417fe518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fcd2ec0 0x56417fe518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.174+0000 7f2ffbe00700 1 -- 172.21.15.12:0/4003794407 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417fe59b60 con 0x56417fe57db0 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.175+0000 7f2ff2a00700 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe51e30 0x56417fe5da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.175+0000 7f2ffb400700 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fcd2ec0 0x56417fe518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.175+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe676a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4026e10 tx=0x7f2ff400b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb489f556c8a3039 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.176+0000 7f2febe00700 1 -- 172.21.15.12:0/4003794407 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff403a070 con 0x56417fe57db0 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.176+0000 7f2febe00700 1 -- 172.21.15.12:0/4003794407 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ff4022070 con 0x56417fe57db0 2024-05-27T07:35:09.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.176+0000 7f2febe00700 1 -- 172.21.15.12:0/4003794407 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4010e90 con 0x56417fe57db0 2024-05-27T07:35:09.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 -- 172.21.15.12:0/4003794407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 msgr2=0x56417fe676a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe676a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4026e10 tx=0x7f2ff400b560 comp rx=0 tx=0).stop 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 -- 172.21.15.12:0/4003794407 shutdown_connections 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fcd2ec0 0x56417fe518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe676a0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 --2- 172.21.15.12:0/4003794407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fe51e30 0x56417fe5da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 -- 172.21.15.12:0/4003794407 >> 172.21.15.12:0/4003794407 conn(0x56417fd33040 msgr2=0x56417fe5be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 -- 172.21.15.12:0/4003794407 shutdown_connections 2024-05-27T07:35:09.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.177+0000 7f3011341b80 1 -- 172.21.15.12:0/4003794407 wait complete. 2024-05-27T07:35:09.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.178+0000 7f3011341b80 1 -- start start 2024-05-27T07:35:09.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.178+0000 7f3011341b80 1 Processor -- start 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.179+0000 7f3011341b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe63a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.179+0000 7f3011341b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe63fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.179+0000 7f3011341b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe60860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.179+0000 7f3011341b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56417fe4b840 con 0x56417fe57db0 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.179+0000 7f3011341b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56417fe60da0 con 0x56417fe51e30 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.179+0000 7f3011341b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56417fe60f70 con 0x56417fcd2ec0 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.179+0000 7f3011341b80 10 client.0 _ll_register_callbacks cb 0x56417fd19cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.180+0000 7f2ffbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe60860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.180+0000 7f2ffbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe60860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52056/0 (socket says 172.21.15.12:52056) 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.180+0000 7f2ffbe00700 1 -- 172.21.15.12:0/1892976160 learned_addr learned my addr 172.21.15.12:0/1892976160 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.180+0000 7f2ffb400700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe63a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.180+0000 7f2ff2a00700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe63fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2ffbe00700 1 -- 172.21.15.12:0/1892976160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 msgr2=0x56417fe63a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe63a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2ffbe00700 1 -- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 msgr2=0x56417fe63fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe63fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2ffbe00700 1 -- 172.21.15.12:0/1892976160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417fe61170 con 0x56417fe57db0 2024-05-27T07:35:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2ffb400700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe63a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2ffbe00700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe60860 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400b7c0 tx=0x7f2ff4012c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b711556749b96f64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.181+0000 7f2fe9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.182+0000 7f2fe9600700 1 -- 172.21.15.12:0/1892976160 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff403a070 con 0x56417fe57db0 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.182+0000 7f2fe9600700 1 -- 172.21.15.12:0/1892976160 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 9642+0+0 (secure 0 0 0) 0x7f2ff4005040 con 0x56417fe57db0 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.182+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff4022070 con 0x56417fe57db0 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.182+0000 7f2fe9600700 1 -- 172.21.15.12:0/1892976160 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f2ff404c050 con 0x56417fe57db0 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.182+0000 7f3011341b80 10 client.16992 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.182+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56417fed6f80 con 0x56417fe57db0 2024-05-27T07:35:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.183+0000 7f2fe9600700 1 -- 172.21.15.12:0/1892976160 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ff4022070 con 0x56417fe57db0 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.184+0000 7f2fe9600700 1 -- 172.21.15.12:0/1892976160 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f2ff4047030 con 0x56417fe57db0 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.184+0000 7f3011341b80 10 client.16992 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.184+0000 7f3011341b80 10 client.16992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.184+0000 7f3011341b80 20 client.16992 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.184+0000 7f3011341b80 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fed75c0 0x56417fed99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.184+0000 7f3011341b80 4 client.16992 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.184+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56417fd7b5f0 con 0x56417fed75c0 2024-05-27T07:35:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.185+0000 7f2ffb400700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fed75c0 0x56417fed99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:09.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.186+0000 7f2ffb400700 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fed75c0 0x56417fed99e0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ab5e4cf6caeda6 server_cookie=b2cca7ede91a1964 in_seq=0 out_seq=0 2024-05-27T07:35:09.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.186+0000 7f2fe9600700 10 client.16992 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.189+0000 7f2fe9600700 1 -- 172.21.15.12:0/1892976160 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56417fd7b5f0 con 0x56417fed75c0 2024-05-27T07:35:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.189+0000 7f2fe9600700 10 client.16992 handle_command_reply: tid=0 2024-05-27T07:35:09.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.190+0000 7f3011341b80 1 client.16992 shutdown 2024-05-27T07:35:09.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 msgr2=0x56417fe60860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe60860 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400b7c0 tx=0x7f2ff4012c30 comp rx=0 tx=0).stop 2024-05-27T07:35:09.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 shutdown_connections 2024-05-27T07:35:09.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56417fe57db0 0x56417fe60860 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56417fcd2ec0 0x56417fe63a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56417fed75c0 0x56417fed99e0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 --2- 172.21.15.12:0/1892976160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56417fe51e30 0x56417fe63fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f2fe9600700 0 client.16992 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:09.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.191+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 >> 172.21.15.12:0/1892976160 conn(0x56417fd33040 msgr2=0x56417fe58ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.192+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 shutdown_connections 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.192+0000 7f3011341b80 1 -- 172.21.15.12:0/1892976160 wait complete. 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.192+0000 7f3011341b80 20 client.16992 trim_cache size 0 max 16384 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "42ff0ffd-a2a1-4a28-a7a3-43dd3b619ce0", 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:35:09.188 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:35:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2fe00b2a70 msgr2=0x7f2fe00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2fe00b2a70 0x7f2fe00b4f80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400bb40 tx=0x7f2ff400bf00 comp rx=0 tx=0).stop 2024-05-27T07:35:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 msgr2=0x7f2ffc19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc19cf50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2fec007d90 tx=0x7f2fec007e70 comp rx=0 tx=0).stop 2024-05-27T07:35:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 shutdown_connections 2024-05-27T07:35:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2fe00b2a70 0x7f2fe00b4f80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ffc109840 0x7f2ffc1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ffc102ad0 0x7f2ffc19cf50 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 --2- 172.21.15.12:0/2689734263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ffc108ef0 0x7f2ffc19d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:09.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 >> 172.21.15.12:0/2689734263 conn(0x7f2ffc0fe510 msgr2=0x7f2ffc10a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:09.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 shutdown_connections 2024-05-27T07:35:09.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:09.193+0000 7f3002800700 1 -- 172.21.15.12:0/2689734263 wait complete. 2024-05-27T07:35:09.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:09 smithi162 ceph-mon[35676]: pgmap v439: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-05-27T07:35:09.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:09 smithi062 ceph-mon[35304]: pgmap v439: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-05-27T07:35:09.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:09 smithi012 ceph-mon[31473]: pgmap v439: 129 pgs: 129 active+clean; 224 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-05-27T07:35:10.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:35:10.582 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:10.736 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:10 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:35:10.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:10 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:10 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:35:10.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:10 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:35:10.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:10 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:35:10.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:10 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.737 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:10 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.740 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:10 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:35:10.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:10 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:10 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:35:10.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:10 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:35:10.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:10 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:35:10.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:10 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:10 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:10 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:35:10.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:10 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:10 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:35:10.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:10 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:35:10.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:10 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:35:10.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:10 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:10.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:10 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:11.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:11 smithi062 ceph-mon[35304]: pgmap v440: 129 pgs: 129 active+clean; 223 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-27T07:35:11.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:11 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:11.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:11 smithi012 ceph-mon[31473]: pgmap v440: 129 pgs: 129 active+clean; 223 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-27T07:35:11.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:11 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:11.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:11 smithi162 ceph-mon[35676]: pgmap v440: 129 pgs: 129 active+clean; 223 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-27T07:35:11.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:11 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.429+0000 7f5fbd600700 1 -- 172.21.15.12:0/3548741595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8075ad0 msgr2=0x7f5fb8075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.429+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3548741595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8075ad0 0x7f5fb8075ee0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009990 tx=0x7f5fa8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.430+0000 7f5fbd600700 1 -- 172.21.15.12:0/3548741595 shutdown_connections 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.430+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3548741595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb8110c10 0x7f5fb8113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.430+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3548741595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8075ad0 0x7f5fb8075ee0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.430+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3548741595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb8076420 0x7f5fb81106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.430+0000 7f5fbd600700 1 -- 172.21.15.12:0/3548741595 >> 172.21.15.12:0/3548741595 conn(0x7f5fb8100030 msgr2=0x7f5fb8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:13.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.430+0000 7f5fbd600700 1 -- 172.21.15.12:0/3548741595 shutdown_connections 2024-05-27T07:35:13.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.430+0000 7f5fbd600700 1 -- 172.21.15.12:0/3548741595 wait complete. 2024-05-27T07:35:13.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.431+0000 7f5fbd600700 1 Processor -- start 2024-05-27T07:35:13.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.431+0000 7f5fbd600700 1 -- start start 2024-05-27T07:35:13.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.432+0000 7f5fbd600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb8075ad0 0x7f5fb819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.432+0000 7f5fbd600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8076420 0x7f5fb819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.432+0000 7f5fbd600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb8110c10 0x7f5fb81a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.432+0000 7f5fbd600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8116590 con 0x7f5fb8075ad0 2024-05-27T07:35:13.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.432+0000 7f5fbd600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8116410 con 0x7f5fb8110c10 2024-05-27T07:35:13.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.432+0000 7f5fbd600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8116710 con 0x7f5fb8076420 2024-05-27T07:35:13.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.432+0000 7f5fb6000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8076420 0x7f5fb819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb6000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8076420 0x7f5fb819d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:34862/0 (socket says 172.21.15.12:34862) 2024-05-27T07:35:13.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb6000700 1 -- 172.21.15.12:0/3647321888 learned_addr learned my addr 172.21.15.12:0/3647321888 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:13.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb7400700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb8110c10 0x7f5fb81a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb8075ad0 0x7f5fb819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb7400700 1 -- 172.21.15.12:0/3647321888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8076420 msgr2=0x7f5fb819d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb7400700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8076420 0x7f5fb819d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb7400700 1 -- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb8075ad0 msgr2=0x7f5fb819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb7400700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb8075ad0 0x7f5fb819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb7400700 1 -- 172.21.15.12:0/3647321888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb81a1e90 con 0x7f5fb8110c10 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.433+0000 7f5fb6000700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8076420 0x7f5fb819d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.434+0000 7f5fb7400700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb8110c10 0x7f5fb81a17f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000efc0 tx=0x7f5fb000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:13.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.434+0000 7f5faf400700 1 -- 172.21.15.12:0/3647321888 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00092e0 con 0x7f5fb8110c10 2024-05-27T07:35:13.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.435+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb81a2120 con 0x7f5fb8110c10 2024-05-27T07:35:13.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.435+0000 7f5faf400700 1 -- 172.21.15.12:0/3647321888 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fb0004bb0 con 0x7f5fb8110c10 2024-05-27T07:35:13.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.435+0000 7f5faf400700 1 -- 172.21.15.12:0/3647321888 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0004020 con 0x7f5fb8110c10 2024-05-27T07:35:13.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.435+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb81a9aa0 con 0x7f5fb8110c10 2024-05-27T07:35:13.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.440+0000 7f5faf400700 1 -- 172.21.15.12:0/3647321888 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5fb001b030 con 0x7f5fb8110c10 2024-05-27T07:35:13.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.440+0000 7f5fcc1ceb80 1 Processor -- start 2024-05-27T07:35:13.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.440+0000 7f5fcc1ceb80 1 -- start start 2024-05-27T07:35:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788b9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56007873ae40 0x5600788b9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788b9e70 0x5600788c6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5600787e7e40 con 0x5600788269f0 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5600787d79e0 con 0x5600788b9e70 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5600787dba20 con 0x56007873ae40 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fb6a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56007873ae40 0x5600788b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fb6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fb6a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56007873ae40 0x5600788b9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:34866/0 (socket says 172.21.15.12:34866) 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.441+0000 7f5fb6a00700 1 -- 172.21.15.12:0/1726107415 learned_addr learned my addr 172.21.15.12:0/1726107415 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6000700 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788b9e70 0x5600788c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6a00700 1 -- 172.21.15.12:0/1726107415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788b9e70 msgr2=0x5600788c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788b9e70 0x5600788c6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6a00700 1 -- 172.21.15.12:0/1726107415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 msgr2=0x5600788b9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788b9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6a00700 1 -- 172.21.15.12:0/1726107415 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600787a0420 con 0x56007873ae40 2024-05-27T07:35:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6000700 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788b9e70 0x5600788c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:35:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56007873ae40 0x5600788b9870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800b650 tx=0x7f5fa80096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f346af1d7e0e82b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:13.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.442+0000 7f5faf400700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f9c0b29f0 0x7f5f9c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.443+0000 7f5faf400700 1 -- 172.21.15.12:0/3647321888 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f5fb00e2480 con 0x7f5fb8110c10 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.443+0000 7f5fb6000700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f9c0b29f0 0x7f5f9c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.443+0000 7f5fa3e00700 1 -- 172.21.15.12:0/1726107415 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80056b0 con 0x56007873ae40 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.443+0000 7f5fa3e00700 1 -- 172.21.15.12:0/1726107415 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fa8003b10 con 0x56007873ae40 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.443+0000 7f5fa3e00700 1 -- 172.21.15.12:0/1726107415 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa800bea0 con 0x56007873ae40 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1726107415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56007873ae40 msgr2=0x5600788b9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56007873ae40 0x5600788b9870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800b650 tx=0x7f5fa80096d0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fb6000700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f9c0b29f0 0x7f5f9c0b4f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009880 tx=0x7f5fa4012040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1726107415 shutdown_connections 2024-05-27T07:35:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788b9e70 0x5600788c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1726107415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56007873ae40 0x5600788b9870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.445+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1726107415 >> 172.21.15.12:0/1726107415 conn(0x5600787d4390 msgr2=0x5600788b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.446+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1726107415 shutdown_connections 2024-05-27T07:35:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.446+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1726107415 wait complete. 2024-05-27T07:35:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.446+0000 7f5fcc1ceb80 1 -- start start 2024-05-27T07:35:13.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.447+0000 7f5fcc1ceb80 1 Processor -- start 2024-05-27T07:35:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.447+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788c3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.447+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.447+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.447+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5600786ddfa0 con 0x5600788269f0 2024-05-27T07:35:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.447+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560078778760 con 0x56007873ae40 2024-05-27T07:35:13.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.447+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5600788c6ac0 con 0x5600788b9e70 2024-05-27T07:35:13.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.448+0000 7f5fb6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.448+0000 7f5fcc1ceb80 10 client.0 _ll_register_callbacks cb 0x560078781c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:13.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.448+0000 7f5fb7400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.448+0000 7f5fb7400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788c40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:34882/0 (socket says 172.21.15.12:34882) 2024-05-27T07:35:13.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.448+0000 7f5fb7400700 1 -- 172.21.15.12:0/1452309516 learned_addr learned my addr 172.21.15.12:0/1452309516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:13.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.448+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb7400700 1 -- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 msgr2=0x5600788c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb7400700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788c3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb7400700 1 -- 172.21.15.12:0/1452309516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 msgr2=0x5600788c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb7400700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb7400700 1 -- 172.21.15.12:0/1452309516 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600787278d0 con 0x5600788b9e70 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb6000700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.449+0000 7f5fb7400700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788c40f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00a9370 tx=0x7f5fb00a8f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=531b3ddcf68954ba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:13.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.450+0000 7f5fa0c00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:35:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.450+0000 7f5fa0c00700 1 -- 172.21.15.12:0/1452309516 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00ad790 con 0x5600788b9e70 2024-05-27T07:35:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.450+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa8003b10 con 0x5600788b9e70 2024-05-27T07:35:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.450+0000 7f5fa0c00700 1 -- 172.21.15.12:0/1452309516 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f5fb00b1d20 con 0x5600788b9e70 2024-05-27T07:35:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.451+0000 7f5fa0c00700 1 -- 172.21.15.12:0/1452309516 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5fb00b21b0 con 0x5600788b9e70 2024-05-27T07:35:13.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.451+0000 7f5fcc1ceb80 10 client.26491 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.451+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5600787a8060 con 0x5600788b9e70 2024-05-27T07:35:13.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fa0c00700 1 -- 172.21.15.12:0/1452309516 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fa8003b10 con 0x5600788b9e70 2024-05-27T07:35:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fa0c00700 1 -- 172.21.15.12:0/1452309516 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5fb00a4d50 con 0x5600788b9e70 2024-05-27T07:35:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fcc1ceb80 10 client.26491 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fcc1ceb80 10 client.26491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fcc1ceb80 20 client.26491 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5600788d3d50 0x5600788d6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fcc1ceb80 4 client.26491 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:13.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.452+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600787e9d90 con 0x5600788d3d50 2024-05-27T07:35:13.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.453+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5600788d3d50 0x5600788d6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.454+0000 7f5fa0c00700 10 client.26491 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:13.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.454+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5600788d3d50 0x5600788d6200 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e6d1396dc8eb1fb server_cookie=256904218d78ffdb in_seq=0 out_seq=0 2024-05-27T07:35:13.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fa0c00700 1 -- 172.21.15.12:0/1452309516 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5600787e9d90 con 0x5600788d3d50 2024-05-27T07:35:13.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fa0c00700 10 client.26491 handle_command_reply: tid=0 2024-05-27T07:35:13.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 client.26491 shutdown 2024-05-27T07:35:13.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 msgr2=0x5600788c40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788c40f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00a9370 tx=0x7f5fb00a8f70 comp rx=0 tx=0).stop 2024-05-27T07:35:13.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 shutdown_connections 2024-05-27T07:35:13.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5600788d3d50 0x5600788d6200 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fa0c00700 0 client.26491 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:13.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788c40f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1452309516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 >> 172.21.15.12:0/1452309516 conn(0x5600787d4390 msgr2=0x5600788b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:13.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.457+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 shutdown_connections 2024-05-27T07:35:13.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.458+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1452309516 wait complete. 2024-05-27T07:35:13.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.458+0000 7f5fcc1ceb80 20 client.26491 trim_cache size 0 max 16384 2024-05-27T07:35:13.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 Processor -- start 2024-05-27T07:35:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 -- start start 2024-05-27T07:35:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788bee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788bb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5600788c27f0 con 0x5600788269f0 2024-05-27T07:35:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb81b1eb0 con 0x56007873ae40 2024-05-27T07:35:13.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.485+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb81b1110 con 0x5600788b9e70 2024-05-27T07:35:13.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.486+0000 7f5fb6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.486+0000 7f5fb6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51536/0 (socket says 172.21.15.12:51536) 2024-05-27T07:35:13.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.486+0000 7f5fb6000700 1 -- 172.21.15.12:0/1627409245 learned_addr learned my addr 172.21.15.12:0/1627409245 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.486+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788bee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.486+0000 7f5fb7400700 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788bb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fb6000700 1 -- 172.21.15.12:0/1627409245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 msgr2=0x5600788bb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fb6000700 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788bb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fb6000700 1 -- 172.21.15.12:0/1627409245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 msgr2=0x5600788bee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fb6000700 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788bee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fb6000700 1 -- 172.21.15.12:0/1627409245 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600788b1240 con 0x5600788269f0 2024-05-27T07:35:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fb6000700 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c4060 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8103540 tx=0x7f5fa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f6f2cb41a420c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fa3400700 1 -- 172.21.15.12:0/1627409245 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4003740 con 0x5600788269f0 2024-05-27T07:35:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fa3400700 1 -- 172.21.15.12:0/1627409245 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fa4003d70 con 0x5600788269f0 2024-05-27T07:35:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.487+0000 7f5fa3400700 1 -- 172.21.15.12:0/1627409245 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa401dcf0 con 0x5600788269f0 2024-05-27T07:35:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.488+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1627409245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 msgr2=0x5600788c4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.488+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c4060 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8103540 tx=0x7f5fa4004750 comp rx=0 tx=0).stop 2024-05-27T07:35:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.488+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1627409245 shutdown_connections 2024-05-27T07:35:13.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.488+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56007873ae40 0x5600788bee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.488+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788bb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.488+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/1627409245 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5600788269f0 0x5600788c4060 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.488+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1627409245 >> 172.21.15.12:0/1627409245 conn(0x5600787d4390 msgr2=0x5600788baed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:13.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.489+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1627409245 shutdown_connections 2024-05-27T07:35:13.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.490+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/1627409245 wait complete. 2024-05-27T07:35:13.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.491+0000 7f5fcc1ceb80 1 -- start start 2024-05-27T07:35:13.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.491+0000 7f5fcc1ceb80 1 Processor -- start 2024-05-27T07:35:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.492+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56007873ae40 0x5600788cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.492+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.492+0000 7f5fcc1ceb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788c6d10 0x5600788c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.492+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5600788d67e0 con 0x56007873ae40 2024-05-27T07:35:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.492+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb81b17e0 con 0x5600788c6d10 2024-05-27T07:35:13.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.492+0000 7f5fcc1ceb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5600788b6830 con 0x5600788b9e70 2024-05-27T07:35:13.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.492+0000 7f5fcc1ceb80 10 client.0 _ll_register_callbacks cb 0x560078781c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:13.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.493+0000 7f5fb6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56007873ae40 0x5600788cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.493+0000 7f5fb6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56007873ae40 0x5600788cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51540/0 (socket says 172.21.15.12:51540) 2024-05-27T07:35:13.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.493+0000 7f5fb6a00700 1 -- 172.21.15.12:0/861292648 learned_addr learned my addr 172.21.15.12:0/861292648 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:13.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.493+0000 7f5fb7400700 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788c6d10 0x5600788c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.494+0000 7f5fb6a00700 1 -- 172.21.15.12:0/861292648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 msgr2=0x5600788cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.494+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.494+0000 7f5fb6a00700 1 -- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788c6d10 msgr2=0x5600788c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.494+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788c6d10 0x5600788c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.494+0000 7f5fb6a00700 1 -- 172.21.15.12:0/861292648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600788c77c0 con 0x56007873ae40 2024-05-27T07:35:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.495+0000 7f5fb6a00700 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56007873ae40 0x5600788cbf60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009990 tx=0x7f5fa8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d4ec111259a54d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:13.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.495+0000 7f5fa0c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.496+0000 7f5fa0c00700 1 -- 172.21.15.12:0/861292648 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa803a060 con 0x56007873ae40 2024-05-27T07:35:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.496+0000 7f5fa0c00700 1 -- 172.21.15.12:0/861292648 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f5fa800bd00 con 0x56007873ae40 2024-05-27T07:35:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.496+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa4003d70 con 0x56007873ae40 2024-05-27T07:35:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.496+0000 7f5fa0c00700 1 -- 172.21.15.12:0/861292648 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5fa8003680 con 0x56007873ae40 2024-05-27T07:35:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.497+0000 7f5fcc1ceb80 10 client.17010 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:13.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.497+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x560078958e10 con 0x56007873ae40 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fa0c00700 1 -- 172.21.15.12:0/861292648 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fa4003d70 con 0x56007873ae40 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fa0c00700 1 -- 172.21.15.12:0/861292648 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5fa8022830 con 0x56007873ae40 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fcc1ceb80 10 client.17010 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fcc1ceb80 10 client.17010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fcc1ceb80 20 client.17010 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560078959300 0x56007895b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fcc1ceb80 4 client.17010 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:35:13.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.504+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5600787e3570 con 0x560078959300 2024-05-27T07:35:13.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.505+0000 7f5fb6000700 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560078959300 0x56007895b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:13.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.506+0000 7f5fa0c00700 10 client.17010 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:13.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.506+0000 7f5fb6000700 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560078959300 0x56007895b720 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b07ad7cf5ff452b2 server_cookie=85b9a0fca50c1333 in_seq=0 out_seq=0 2024-05-27T07:35:13.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.507+0000 7f5fa0c00700 1 -- 172.21.15.12:0/861292648 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5600787e3570 con 0x560078959300 2024-05-27T07:35:13.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.507+0000 7f5fa0c00700 10 client.17010 handle_command_reply: tid=0 2024-05-27T07:35:13.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.507+0000 7f5fcc1ceb80 1 client.17010 shutdown 2024-05-27T07:35:13.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56007873ae40 msgr2=0x5600788cbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56007873ae40 0x5600788cbf60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009990 tx=0x7f5fa8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 shutdown_connections 2024-05-27T07:35:13.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x560078959300 0x56007895b720 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600788c6d10 0x5600788c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56007873ae40 0x5600788cbf60 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 --2- 172.21.15.12:0/861292648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600788b9e70 0x5600788cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fa0c00700 0 client.17010 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 >> 172.21.15.12:0/861292648 conn(0x5600787d4390 msgr2=0x5600788c3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 shutdown_connections 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.508+0000 7f5fcc1ceb80 1 -- 172.21.15.12:0/861292648 wait complete. 2024-05-27T07:35:13.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.509+0000 7f5fcc1ceb80 20 client.17010 trim_cache size 0 max 16384 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: "42ff0ffd-a2a1-4a28-a7a3-43dd3b619ce0": { 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "42ff0ffd-a2a1-4a28-a7a3-43dd3b619ce0", 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:35:13.505 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:35:13.506 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:35:13.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f9c0b29f0 msgr2=0x7f5f9c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f9c0b29f0 0x7f5f9c0b4f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009880 tx=0x7f5fa4012040 comp rx=0 tx=0).stop 2024-05-27T07:35:13.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb8110c10 msgr2=0x7f5fb81a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb8110c10 0x7f5fb81a17f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000efc0 tx=0x7f5fb000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 shutdown_connections 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5f9c0b29f0 0x7f5f9c0b4f00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb8110c10 0x7f5fb81a17f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5fb8075ad0 0x7f5fb819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 --2- 172.21.15.12:0/3647321888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb8076420 0x7f5fb819d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 >> 172.21.15.12:0/3647321888 conn(0x7f5fb8100030 msgr2=0x7f5fb8100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 shutdown_connections 2024-05-27T07:35:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:13.510+0000 7f5fbd600700 1 -- 172.21.15.12:0/3647321888 wait complete. 2024-05-27T07:35:13.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:13 smithi012 ceph-mon[31473]: pgmap v441: 129 pgs: 129 active+clean; 223 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-27T07:35:13.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:13 smithi012 ceph-mon[31473]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:35:13.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:13 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:13.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:13 smithi162 ceph-mon[35676]: pgmap v441: 129 pgs: 129 active+clean; 223 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-27T07:35:13.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:13 smithi162 ceph-mon[35676]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:35:13.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:13 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:13.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:13 smithi062 ceph-mon[35304]: pgmap v441: 129 pgs: 129 active+clean; 223 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-27T07:35:13.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:13 smithi062 ceph-mon[35304]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:35:13.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:13 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:14.368 INFO:tasks.cephfs.filesystem:scrub status for tag:42ff0ffd-a2a1-4a28-a7a3-43dd3b619ce0 - {'path': '/', 'tag': '42ff0ffd-a2a1-4a28-a7a3-43dd3b619ce0', 'options': 'recursive,force'} 2024-05-27T07:35:14.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:14 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:14 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:14.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:14 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:35:14.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:14 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:14 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:14.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:14 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:35:14.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:14 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:14.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:14 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:14.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:14 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:35:15.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:15 smithi012 ceph-mon[31473]: pgmap v443: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-05-27T07:35:15.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:15 smithi012 ceph-mon[31473]: pgmap v444: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 202 op/s 2024-05-27T07:35:15.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:15 smithi162 ceph-mon[35676]: pgmap v443: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-05-27T07:35:15.881 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:15 smithi162 ceph-mon[35676]: pgmap v444: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 202 op/s 2024-05-27T07:35:15.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:15 smithi062 ceph-mon[35304]: pgmap v443: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-05-27T07:35:15.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:15 smithi062 ceph-mon[35304]: pgmap v444: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 202 op/s 2024-05-27T07:35:16.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:16 smithi012 ceph-mon[31473]: pgmap v445: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-05-27T07:35:16.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:16 smithi162 ceph-mon[35676]: pgmap v445: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-05-27T07:35:16.990 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:16 smithi062 ceph-mon[35304]: pgmap v445: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-05-27T07:35:19.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:19 smithi012 ceph-mon[31473]: pgmap v446: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1019 KiB/s wr, 104 op/s 2024-05-27T07:35:19.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:19 smithi162 ceph-mon[35676]: pgmap v446: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1019 KiB/s wr, 104 op/s 2024-05-27T07:35:19.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:19 smithi062 ceph-mon[35304]: pgmap v446: 129 pgs: 129 active+clean; 229 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1019 KiB/s wr, 104 op/s 2024-05-27T07:35:21.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:21 smithi162 ceph-mon[35676]: pgmap v447: 129 pgs: 129 active+clean; 226 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-05-27T07:35:21.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:35:21.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:21 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:35:21.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:21 smithi062 ceph-mon[35304]: pgmap v447: 129 pgs: 129 active+clean; 226 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-05-27T07:35:21.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:35:21.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:21 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:35:21.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:21 smithi012 ceph-mon[31473]: pgmap v447: 129 pgs: 129 active+clean; 226 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-05-27T07:35:21.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:35:21.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:21 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:35:22.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:22 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:35:22.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:22 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:35:22.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:22 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:35:23.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:23 smithi012 ceph-mon[31473]: pgmap v448: 129 pgs: 129 active+clean; 226 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-05-27T07:35:23.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:23 smithi162 ceph-mon[35676]: pgmap v448: 129 pgs: 129 active+clean; 226 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-05-27T07:35:23.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:23 smithi062 ceph-mon[35304]: pgmap v448: 129 pgs: 129 active+clean; 226 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-05-27T07:35:25.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:25 smithi162 ceph-mon[35676]: pgmap v449: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-05-27T07:35:25.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:25 smithi062 ceph-mon[35304]: pgmap v449: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-05-27T07:35:25.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:25 smithi012 ceph-mon[31473]: pgmap v449: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-05-27T07:35:27.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:27 smithi162 ceph-mon[35676]: pgmap v450: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1014 KiB/s wr, 111 op/s 2024-05-27T07:35:27.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:27 smithi062 ceph-mon[35304]: pgmap v450: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1014 KiB/s wr, 111 op/s 2024-05-27T07:35:27.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:27 smithi012 ceph-mon[31473]: pgmap v450: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1014 KiB/s wr, 111 op/s 2024-05-27T07:35:29.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:29 smithi162 ceph-mon[35676]: pgmap v451: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1014 KiB/s wr, 111 op/s 2024-05-27T07:35:29.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:29 smithi062 ceph-mon[35304]: pgmap v451: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1014 KiB/s wr, 111 op/s 2024-05-27T07:35:29.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:29 smithi012 ceph-mon[31473]: pgmap v451: 129 pgs: 129 active+clean; 232 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1014 KiB/s wr, 111 op/s 2024-05-27T07:35:31.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:31 smithi162 ceph-mon[35676]: pgmap v452: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:35:31.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:31 smithi062 ceph-mon[35304]: pgmap v452: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:35:31.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:31 smithi012 ceph-mon[31473]: pgmap v452: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:35:33.433 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:33 smithi012 ceph-mon[31473]: pgmap v453: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-05-27T07:35:33.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:33 smithi162 ceph-mon[35676]: pgmap v453: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-05-27T07:35:33.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:33 smithi062 ceph-mon[35304]: pgmap v453: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-05-27T07:35:35.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:35 smithi162 ceph-mon[35676]: pgmap v454: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-05-27T07:35:35.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:35 smithi062 ceph-mon[35304]: pgmap v454: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-05-27T07:35:35.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:35 smithi012 ceph-mon[31473]: pgmap v454: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-05-27T07:35:37.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:37 smithi162 ceph-mon[35676]: pgmap v455: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-05-27T07:35:37.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:37 smithi062 ceph-mon[35304]: pgmap v455: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-05-27T07:35:37.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:37 smithi012 ceph-mon[31473]: pgmap v455: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-05-27T07:35:39.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:39 smithi162 ceph-mon[35676]: pgmap v456: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-05-27T07:35:39.490 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:39 smithi062 ceph-mon[35304]: pgmap v456: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-05-27T07:35:39.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:39 smithi012 ceph-mon[31473]: pgmap v456: 129 pgs: 129 active+clean; 236 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-05-27T07:35:41.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:41 smithi162 ceph-mon[35676]: pgmap v457: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-27T07:35:41.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:41 smithi062 ceph-mon[35304]: pgmap v457: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-27T07:35:41.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:41 smithi012 ceph-mon[31473]: pgmap v457: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-27T07:35:43.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:43 smithi162 ceph-mon[35676]: pgmap v458: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 960 KiB/s wr, 107 op/s 2024-05-27T07:35:43.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:43 smithi062 ceph-mon[35304]: pgmap v458: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 960 KiB/s wr, 107 op/s 2024-05-27T07:35:43.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:43 smithi012 ceph-mon[31473]: pgmap v458: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 960 KiB/s wr, 107 op/s 2024-05-27T07:35:44.369 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:35:44.859 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:45.266 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:45 smithi012 ceph-mon[31473]: pgmap v459: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-05-27T07:35:45.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:45 smithi162 ceph-mon[35676]: pgmap v459: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-05-27T07:35:45.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:45 smithi062 ceph-mon[35304]: pgmap v459: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-05-27T07:35:47.478 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:47 smithi162 ceph-mon[35676]: pgmap v460: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 947 KiB/s wr, 107 op/s 2024-05-27T07:35:47.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:47 smithi062 ceph-mon[35304]: pgmap v460: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 947 KiB/s wr, 107 op/s 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 -- 172.21.15.12:0/2013931134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 msgr2=0x7fde5c075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 --2- 172.21.15.12:0/2013931134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c075f00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fde50009990 tx=0x7fde50009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 -- 172.21.15.12:0/2013931134 shutdown_connections 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 --2- 172.21.15.12:0/2013931134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde5c110bf0 0x7fde5c113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 --2- 172.21.15.12:0/2013931134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c075f00 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 --2- 172.21.15.12:0/2013931134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde5c076440 0x7fde5c1106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 -- 172.21.15.12:0/2013931134 >> 172.21.15.12:0/2013931134 conn(0x7fde5c100030 msgr2=0x7fde5c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.558+0000 7fde64200700 1 -- 172.21.15.12:0/2013931134 shutdown_connections 2024-05-27T07:35:47.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.559+0000 7fde64200700 1 -- 172.21.15.12:0/2013931134 wait complete. 2024-05-27T07:35:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.560+0000 7fde64200700 1 Processor -- start 2024-05-27T07:35:47.556 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:47 smithi012 ceph-mon[31473]: pgmap v460: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 947 KiB/s wr, 107 op/s 2024-05-27T07:35:47.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.560+0000 7fde64200700 1 -- start start 2024-05-27T07:35:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde64200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde64200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde5c076440 0x7fde5c19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde64200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde5c110bf0 0x7fde5c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde64200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c116580 con 0x7fde5c075af0 2024-05-27T07:35:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde64200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c116400 con 0x7fde5c110bf0 2024-05-27T07:35:47.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde64200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c116700 con 0x7fde5c076440 2024-05-27T07:35:47.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde61800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde61800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c19cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49936/0 (socket says 172.21.15.12:49936) 2024-05-27T07:35:47.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.561+0000 7fde61800700 1 -- 172.21.15.12:0/2256143753 learned_addr learned my addr 172.21.15.12:0/2256143753 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:47.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde60e00700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde5c076440 0x7fde5c19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde62200700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde5c110bf0 0x7fde5c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde61800700 1 -- 172.21.15.12:0/2256143753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde5c076440 msgr2=0x7fde5c19d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde61800700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde5c076440 0x7fde5c19d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde61800700 1 -- 172.21.15.12:0/2256143753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde5c110bf0 msgr2=0x7fde5c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde61800700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde5c110bf0 0x7fde5c1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde61800700 1 -- 172.21.15.12:0/2256143753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5c1a1ea0 con 0x7fde5c075af0 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.562+0000 7fde61800700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c19cfd0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fde50000c00 tx=0x7fde500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.563+0000 7fde5a000700 1 -- 172.21.15.12:0/2256143753 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde50027070 con 0x7fde5c075af0 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.563+0000 7fde5a000700 1 -- 172.21.15.12:0/2256143753 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde5000fb00 con 0x7fde5c075af0 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.563+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5c1a2130 con 0x7fde5c075af0 2024-05-27T07:35:47.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.563+0000 7fde5a000700 1 -- 172.21.15.12:0/2256143753 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde50022c40 con 0x7fde5c075af0 2024-05-27T07:35:47.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.563+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5c1a9ab0 con 0x7fde5c075af0 2024-05-27T07:35:47.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.568+0000 7fde5a000700 1 -- 172.21.15.12:0/2256143753 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fde5002a070 con 0x7fde5c075af0 2024-05-27T07:35:47.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.569+0000 7fde5a000700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fde440b2a70 0x7fde440b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.570+0000 7fde5a000700 1 -- 172.21.15.12:0/2256143753 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7fde500e39f0 con 0x7fde5c075af0 2024-05-27T07:35:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.570+0000 7fde60e00700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fde440b2a70 0x7fde440b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.571+0000 7fde60e00700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fde440b2a70 0x7fde440b4f80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde5c103530 tx=0x7fde4800b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.572+0000 7fde72e05b80 1 Processor -- start 2024-05-27T07:35:47.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.572+0000 7fde72e05b80 1 -- start start 2024-05-27T07:35:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.573+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacda7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.573+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.573+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aacd51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.573+0000 7fde72e05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5643aac02e40 con 0x5643aacd4db0 2024-05-27T07:35:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.573+0000 7fde72e05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5643aabf29e0 con 0x5643aab55e40 2024-05-27T07:35:47.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.573+0000 7fde72e05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5643aabf6a20 con 0x5643aacdad30 2024-05-27T07:35:47.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde60e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde62200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aacd51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde61800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacda7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde60e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:41288/0 (socket says 172.21.15.12:41288) 2024-05-27T07:35:47.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde60e00700 1 -- 172.21.15.12:0/2822680049 learned_addr learned my addr 172.21.15.12:0/2822680049 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde62200700 1 -- 172.21.15.12:0/2822680049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 msgr2=0x5643aacd4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde62200700 1 --2- 172.21.15.12:0/2822680049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde62200700 1 -- 172.21.15.12:0/2822680049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 msgr2=0x5643aacda7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde62200700 1 --2- 172.21.15.12:0/2822680049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacda7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.574+0000 7fde62200700 1 -- 172.21.15.12:0/2822680049 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643aabbb420 con 0x5643aacd4db0 2024-05-27T07:35:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.575+0000 7fde62200700 1 --2- 172.21.15.12:0/2822680049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aacd51c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fde54003e00 tx=0x7fde54009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6b2db2c561404d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.576+0000 7fde4ea00700 1 -- 172.21.15.12:0/2822680049 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54004bb0 con 0x5643aacd4db0 2024-05-27T07:35:47.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.576+0000 7fde4ea00700 1 -- 172.21.15.12:0/2822680049 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde54004800 con 0x5643aacd4db0 2024-05-27T07:35:47.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.576+0000 7fde4ea00700 1 -- 172.21.15.12:0/2822680049 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54007ca0 con 0x5643aacd4db0 2024-05-27T07:35:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 -- 172.21.15.12:0/2822680049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 msgr2=0x5643aacd51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 --2- 172.21.15.12:0/2822680049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aacd51c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fde54003e00 tx=0x7fde54009c20 comp rx=0 tx=0).stop 2024-05-27T07:35:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 -- 172.21.15.12:0/2822680049 shutdown_connections 2024-05-27T07:35:47.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 --2- 172.21.15.12:0/2822680049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacda7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 --2- 172.21.15.12:0/2822680049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aacd51c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 --2- 172.21.15.12:0/2822680049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 -- 172.21.15.12:0/2822680049 >> 172.21.15.12:0/2822680049 conn(0x5643aabb5fc0 msgr2=0x5643aaccf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.577+0000 7fde72e05b80 1 -- 172.21.15.12:0/2822680049 shutdown_connections 2024-05-27T07:35:47.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.578+0000 7fde72e05b80 1 -- 172.21.15.12:0/2822680049 wait complete. 2024-05-27T07:35:47.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.579+0000 7fde72e05b80 1 -- start start 2024-05-27T07:35:47.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.579+0000 7fde72e05b80 1 Processor -- start 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.580+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacde5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.580+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacd4db0 0x5643aacdeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.580+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacdad30 0x5643aacdf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.580+0000 7fde72e05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5643aaaf8fa0 con 0x5643aacdad30 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.580+0000 7fde72e05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5643aab93760 con 0x5643aab55e40 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.580+0000 7fde72e05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5643aace1800 con 0x5643aacd4db0 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.581+0000 7fde60e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacd4db0 0x5643aacdeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.581+0000 7fde60e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacd4db0 0x5643aacdeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:41296/0 (socket says 172.21.15.12:41296) 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.581+0000 7fde60e00700 1 -- 172.21.15.12:0/4174143087 learned_addr learned my addr 172.21.15.12:0/4174143087 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.581+0000 7fde72e05b80 10 client.0 _ll_register_callbacks cb 0x5643aab9cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.581+0000 7fde61800700 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacde5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.582+0000 7fde62200700 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacdad30 0x5643aacdf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.582+0000 7fde62200700 1 -- 172.21.15.12:0/4174143087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacd4db0 msgr2=0x5643aacdeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.582+0000 7fde62200700 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacd4db0 0x5643aacdeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.582+0000 7fde62200700 1 -- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 msgr2=0x5643aacde5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.582+0000 7fde62200700 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacde5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.582+0000 7fde62200700 1 -- 172.21.15.12:0/4174143087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643aab428d0 con 0x5643aacdad30 2024-05-27T07:35:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.583+0000 7fde62200700 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacdad30 0x5643aacdf040 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fde54003e00 tx=0x7fde540045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930599cb5d53f9ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.583+0000 7fde43400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:47.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.583+0000 7fde43400700 1 -- 172.21.15.12:0/4174143087 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54022070 con 0x5643aacdad30 2024-05-27T07:35:47.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.583+0000 7fde43400700 1 -- 172.21.15.12:0/4174143087 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7fde54007af0 con 0x5643aacdad30 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.583+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde54004800 con 0x5643aacdad30 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.584+0000 7fde43400700 1 -- 172.21.15.12:0/4174143087 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fde5402c050 con 0x5643aacdad30 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.584+0000 7fde72e05b80 10 client.17022 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.584+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5643aabc3060 con 0x5643aacdad30 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde43400700 1 -- 172.21.15.12:0/4174143087 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde54004800 con 0x5643aacdad30 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde43400700 1 -- 172.21.15.12:0/4174143087 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fde54027440 con 0x5643aacdad30 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde72e05b80 10 client.17022 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde72e05b80 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde72e05b80 20 client.17022 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde72e05b80 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aace4230 0x5643aace6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde72e05b80 4 client.17022 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:47.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.585+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643aac04d90 con 0x5643aace4230 2024-05-27T07:35:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.586+0000 7fde61800700 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aace4230 0x5643aace6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.589+0000 7fde61800700 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aace4230 0x5643aace6700 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=557d39c54ac421e1 server_cookie=8e9095b4b2690174 in_seq=0 out_seq=0 2024-05-27T07:35:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.589+0000 7fde43400700 10 client.17022 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:47.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.591+0000 7fde43400700 1 -- 172.21.15.12:0/4174143087 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5643aac04d90 con 0x5643aace4230 2024-05-27T07:35:47.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.591+0000 7fde43400700 10 client.17022 handle_command_reply: tid=0 2024-05-27T07:35:47.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.591+0000 7fde72e05b80 1 client.17022 shutdown 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.591+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacdad30 msgr2=0x5643aacdf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.591+0000 7fde72e05b80 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacdad30 0x5643aacdf040 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fde54003e00 tx=0x7fde540045a0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 shutdown_connections 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aace4230 0x5643aace6700 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacde5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacdad30 0x5643aacdf040 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 --2- 172.21.15.12:0/4174143087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacd4db0 0x5643aacdeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 >> 172.21.15.12:0/4174143087 conn(0x5643aabb5fc0 msgr2=0x5643aaccf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde43400700 0 client.17022 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 shutdown_connections 2024-05-27T07:35:47.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 1 -- 172.21.15.12:0/4174143087 wait complete. 2024-05-27T07:35:47.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.592+0000 7fde72e05b80 20 client.17022 trim_cache size 0 max 16384 2024-05-27T07:35:47.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.619+0000 7fde72e05b80 1 Processor -- start 2024-05-27T07:35:47.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.619+0000 7fde72e05b80 1 -- start start 2024-05-27T07:35:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacd4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aace0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde72e05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c1adac0 con 0x5643aacd4db0 2024-05-27T07:35:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde72e05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c1acd20 con 0x5643aab55e40 2024-05-27T07:35:47.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde72e05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c1ad3f0 con 0x5643aacdad30 2024-05-27T07:35:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde60e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aace0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde60e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aace0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49976/0 (socket says 172.21.15.12:49976) 2024-05-27T07:35:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.620+0000 7fde60e00700 1 -- 172.21.15.12:0/1732430423 learned_addr learned my addr 172.21.15.12:0/1732430423 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.621+0000 7fde61800700 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacd4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.621+0000 7fde62200700 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.621+0000 7fde60e00700 1 -- 172.21.15.12:0/1732430423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 msgr2=0x5643aacd8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.621+0000 7fde60e00700 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.621+0000 7fde60e00700 1 -- 172.21.15.12:0/1732430423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 msgr2=0x5643aacd4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.621+0000 7fde60e00700 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacd4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.621+0000 7fde60e00700 1 -- 172.21.15.12:0/1732430423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643aacd5b40 con 0x5643aacd4db0 2024-05-27T07:35:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.622+0000 7fde60e00700 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aace0a60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fde48020570 tx=0x7fde48013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba53068658a216 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.622+0000 7fde4e000700 1 -- 172.21.15.12:0/1732430423 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde48020cf0 con 0x5643aacd4db0 2024-05-27T07:35:47.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.622+0000 7fde4e000700 1 -- 172.21.15.12:0/1732430423 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde48004cc0 con 0x5643aacd4db0 2024-05-27T07:35:47.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.622+0000 7fde4e000700 1 -- 172.21.15.12:0/1732430423 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde48005670 con 0x5643aacd4db0 2024-05-27T07:35:47.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.623+0000 7fde72e05b80 1 -- 172.21.15.12:0/1732430423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 msgr2=0x5643aace0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.623+0000 7fde72e05b80 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aace0a60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fde48020570 tx=0x7fde48013000 comp rx=0 tx=0).stop 2024-05-27T07:35:47.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.623+0000 7fde72e05b80 1 -- 172.21.15.12:0/1732430423 shutdown_connections 2024-05-27T07:35:47.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.623+0000 7fde72e05b80 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aab55e40 0x5643aacd4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.623+0000 7fde72e05b80 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacd8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.623+0000 7fde72e05b80 1 --2- 172.21.15.12:0/1732430423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aacd4db0 0x5643aace0a60 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.623+0000 7fde72e05b80 1 -- 172.21.15.12:0/1732430423 >> 172.21.15.12:0/1732430423 conn(0x5643aabb5fc0 msgr2=0x5643aacd8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:47.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.624+0000 7fde72e05b80 1 -- 172.21.15.12:0/1732430423 shutdown_connections 2024-05-27T07:35:47.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.624+0000 7fde72e05b80 1 -- 172.21.15.12:0/1732430423 wait complete. 2024-05-27T07:35:47.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.624+0000 7fde72e05b80 1 -- start start 2024-05-27T07:35:47.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.625+0000 7fde72e05b80 1 Processor -- start 2024-05-27T07:35:47.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.625+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aab55e40 0x5643aacfa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.625+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aacd4db0 0x5643aacf8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.625+0000 7fde72e05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacf8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.625+0000 7fde72e05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5643aacdf3b0 con 0x5643aab55e40 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.625+0000 7fde72e05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5643aacce710 con 0x5643aacd4db0 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.625+0000 7fde72e05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5643aacd5500 con 0x5643aacdad30 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aab55e40 0x5643aacfa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde72e05b80 10 client.0 _ll_register_callbacks cb 0x5643aab9cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aab55e40 0x5643aacfa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49992/0 (socket says 172.21.15.12:49992) 2024-05-27T07:35:47.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 -- 172.21.15.12:0/3569451651 learned_addr learned my addr 172.21.15.12:0/3569451651 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:47.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde60e00700 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aacd4db0 0x5643aacf8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde62200700 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacf8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 -- 172.21.15.12:0/3569451651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 msgr2=0x5643aacf8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacf8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 -- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aacd4db0 msgr2=0x5643aacf8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:47.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aacd4db0 0x5643aacf8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.626+0000 7fde61800700 1 -- 172.21.15.12:0/3569451651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643aacf9100 con 0x5643aab55e40 2024-05-27T07:35:47.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.627+0000 7fde61800700 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aab55e40 0x5643aacfa190 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fde500aa5f0 tx=0x7fde500b1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d2cdce2443b4cf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:47.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.627+0000 7fde43400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:47.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.627+0000 7fde43400700 1 -- 172.21.15.12:0/3569451651 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500c3020 con 0x5643aab55e40 2024-05-27T07:35:47.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.627+0000 7fde43400700 1 -- 172.21.15.12:0/3569451651 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7fde500a6b80 con 0x5643aab55e40 2024-05-27T07:35:47.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.627+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde48004cc0 con 0x5643aab55e40 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde43400700 1 -- 172.21.15.12:0/3569451651 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fde500aeca0 con 0x5643aab55e40 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 10 client.17028 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5643aad59880 con 0x5643aab55e40 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde43400700 1 -- 172.21.15.12:0/3569451651 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fde48004cc0 con 0x5643aab55e40 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde43400700 1 -- 172.21.15.12:0/3569451651 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fde500aab50 con 0x5643aab55e40 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 10 client.17028 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 10 client.17028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 20 client.17028 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aad59d30 0x5643aad5c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:47.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 4 client.17028 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:35:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.628+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5643aabfe570 con 0x5643aad59d30 2024-05-27T07:35:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.629+0000 7fde60e00700 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aad59d30 0x5643aad5c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.630+0000 7fde60e00700 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aad59d30 0x5643aad5c150 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d5816c54f1e67d0 server_cookie=ef678c75b0d2822b in_seq=0 out_seq=0 2024-05-27T07:35:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.630+0000 7fde43400700 10 client.17028 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:47.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde43400700 1 -- 172.21.15.12:0/3569451651 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5643aabfe570 con 0x5643aad59d30 2024-05-27T07:35:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde43400700 10 client.17028 handle_command_reply: tid=0 2024-05-27T07:35:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 client.17028 shutdown 2024-05-27T07:35:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aab55e40 msgr2=0x5643aacfa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aab55e40 0x5643aacfa190 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fde500aa5f0 tx=0x7fde500b1040 comp rx=0 tx=0).stop 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 shutdown_connections 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643aacdad30 0x5643aacf8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5643aab55e40 0x5643aacfa190 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5643aad59d30 0x5643aad5c150 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.631+0000 7fde72e05b80 1 --2- 172.21.15.12:0/3569451651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5643aacd4db0 0x5643aacf8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.632+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 >> 172.21.15.12:0/3569451651 conn(0x5643aabb5fc0 msgr2=0x5643aacd89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.632+0000 7fde43400700 0 client.17028 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.632+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 shutdown_connections 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.632+0000 7fde72e05b80 1 -- 172.21.15.12:0/3569451651 wait complete. 2024-05-27T07:35:47.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.632+0000 7fde72e05b80 20 client.17028 trim_cache size 0 max 16384 2024-05-27T07:35:47.629 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:35:47.629 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:35:47.629 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:35:47.629 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:35:47.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fde440b2a70 msgr2=0x7fde440b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fde440b2a70 0x7fde440b4f80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde5c103530 tx=0x7fde4800b410 comp rx=0 tx=0).stop 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 msgr2=0x7fde5c19cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c19cfd0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fde50000c00 tx=0x7fde500047b0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 shutdown_connections 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fde440b2a70 0x7fde440b4f80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde5c110bf0 0x7fde5c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde5c075af0 0x7fde5c19cfd0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 --2- 172.21.15.12:0/2256143753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde5c076440 0x7fde5c19d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 >> 172.21.15.12:0/2256143753 conn(0x7fde5c100030 msgr2=0x7fde5c100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 shutdown_connections 2024-05-27T07:35:47.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:47.633+0000 7fde64200700 1 -- 172.21.15.12:0/2256143753 wait complete. 2024-05-27T07:35:48.614 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:35:48.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:49.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:49 smithi012 ceph-mon[31473]: pgmap v461: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 947 KiB/s wr, 107 op/s 2024-05-27T07:35:49.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:49 smithi162 ceph-mon[35676]: pgmap v461: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 947 KiB/s wr, 107 op/s 2024-05-27T07:35:49.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:49 smithi062 ceph-mon[35304]: pgmap v461: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 947 KiB/s wr, 107 op/s 2024-05-27T07:35:51.329 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:51 smithi012 ceph-mon[31473]: pgmap v462: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-05-27T07:35:51.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:51 smithi162 ceph-mon[35676]: pgmap v462: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-05-27T07:35:51.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:51 smithi062 ceph-mon[35304]: pgmap v462: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-05-27T07:35:51.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 -- 172.21.15.12:0/3647304202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810b100 msgr2=0x7f5538117610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:51.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 --2- 172.21.15.12:0/3647304202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810b100 0x7f5538117610 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f553000b0a0 tx=0x7f553000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:51.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 -- 172.21.15.12:0/3647304202 shutdown_connections 2024-05-27T07:35:51.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 --2- 172.21.15.12:0/3647304202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810b100 0x7f5538117610 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:51.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 --2- 172.21.15.12:0/3647304202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f553810f6b0 0x7f553810fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:51.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 --2- 172.21.15.12:0/3647304202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f553810a7b0 0x7f553810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:51.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 -- 172.21.15.12:0/3647304202 >> 172.21.15.12:0/3647304202 conn(0x7f5538076af0 msgr2=0x7f5538076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:51.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 -- 172.21.15.12:0/3647304202 shutdown_connections 2024-05-27T07:35:51.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.725+0000 7f553de00700 1 -- 172.21.15.12:0/3647304202 wait complete. 2024-05-27T07:35:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 Processor -- start 2024-05-27T07:35:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 -- start start 2024-05-27T07:35:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810a7b0 0x7f55381a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f553810b100 0x7f55381a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f553810f6b0 0x7f55381a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f553811a830 con 0x7f553810a7b0 2024-05-27T07:35:51.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f553811a6b0 con 0x7f553810f6b0 2024-05-27T07:35:51.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.727+0000 7f553de00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f553811a9b0 con 0x7f553810b100 2024-05-27T07:35:51.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.728+0000 7f553ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810a7b0 0x7f55381a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:51.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.728+0000 7f553ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810a7b0 0x7f55381a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50004/0 (socket says 172.21.15.12:50004) 2024-05-27T07:35:51.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.728+0000 7f553ca00700 1 -- 172.21.15.12:0/776162268 learned_addr learned my addr 172.21.15.12:0/776162268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:51.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.728+0000 7f5537e00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f553810b100 0x7f55381a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.728+0000 7f553d400700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f553810f6b0 0x7f55381a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.729+0000 7f553ca00700 1 -- 172.21.15.12:0/776162268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f553810b100 msgr2=0x7f55381a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.729+0000 7f553ca00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f553810b100 0x7f55381a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.729+0000 7f553ca00700 1 -- 172.21.15.12:0/776162268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f553810f6b0 msgr2=0x7f55381a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.729+0000 7f553ca00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f553810f6b0 0x7f55381a59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:51.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.729+0000 7f553ca00700 1 -- 172.21.15.12:0/776162268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55381a6050 con 0x7f553810a7b0 2024-05-27T07:35:51.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.729+0000 7f553ca00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810a7b0 0x7f55381a1180 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f552c00b550 tx=0x7f552c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:51.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.729+0000 7f5535600700 1 -- 172.21.15.12:0/776162268 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c015020 con 0x7f553810a7b0 2024-05-27T07:35:51.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.730+0000 7f5535600700 1 -- 172.21.15.12:0/776162268 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f552c011cc0 con 0x7f553810a7b0 2024-05-27T07:35:51.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.730+0000 7f5535600700 1 -- 172.21.15.12:0/776162268 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c010c10 con 0x7f553810a7b0 2024-05-27T07:35:51.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.730+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55381a62e0 con 0x7f553810a7b0 2024-05-27T07:35:51.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.730+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55381a67d0 con 0x7f553810a7b0 2024-05-27T07:35:51.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.732+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55380512b0 con 0x7f553810a7b0 2024-05-27T07:35:51.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.735+0000 7f5535600700 1 -- 172.21.15.12:0/776162268 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f552c018020 con 0x7f553810a7b0 2024-05-27T07:35:51.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.737+0000 7f5535600700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55200b29f0 0x7f55200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:51.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.738+0000 7f5535600700 1 -- 172.21.15.12:0/776162268 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f552c0e2070 con 0x7f553810a7b0 2024-05-27T07:35:51.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.738+0000 7f5537e00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55200b29f0 0x7f55200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:51.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.739+0000 7f5537e00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55200b29f0 0x7f55200b4f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5538077f20 tx=0x7f552400b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:51.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:51.742+0000 7f5535600700 1 -- 172.21.15.12:0/776162268 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f552c01f020 con 0x7f553810a7b0 2024-05-27T07:35:52.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.102+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f553802e5e0 con 0x7f553810a7b0 2024-05-27T07:35:52.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.103+0000 7f5535600700 1 -- 172.21.15.12:0/776162268 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f552c0a96f0 con 0x7f553810a7b0 2024-05-27T07:35:52.100 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:35:52.100 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:35:52.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55200b29f0 msgr2=0x7f55200b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:52.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55200b29f0 0x7f55200b4f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5538077f20 tx=0x7f552400b410 comp rx=0 tx=0).stop 2024-05-27T07:35:52.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810a7b0 msgr2=0x7f55381a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:52.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810a7b0 0x7f55381a1180 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f552c00b550 tx=0x7f552c00b860 comp rx=0 tx=0).stop 2024-05-27T07:35:52.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 shutdown_connections 2024-05-27T07:35:52.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f55200b29f0 0x7f55200b4f00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f553810f6b0 0x7f55381a59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f553810a7b0 0x7f55381a1180 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 --2- 172.21.15.12:0/776162268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f553810b100 0x7f55381a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:52.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.106+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 >> 172.21.15.12:0/776162268 conn(0x7f5538076af0 msgr2=0x7f5538108850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:52.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.107+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 shutdown_connections 2024-05-27T07:35:52.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:52.107+0000 7f553de00700 1 -- 172.21.15.12:0/776162268 wait complete. 2024-05-27T07:35:52.106 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:35:52.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:52 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/776162268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:52.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:52 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/776162268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:52.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:52 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/776162268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:53.196 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:35:53.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:53 smithi062 ceph-mon[35304]: pgmap v463: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 946 KiB/s wr, 108 op/s 2024-05-27T07:35:53.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:53 smithi012 ceph-mon[31473]: pgmap v463: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 946 KiB/s wr, 108 op/s 2024-05-27T07:35:53.697 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:53.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:53 smithi162 ceph-mon[35676]: pgmap v463: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 946 KiB/s wr, 108 op/s 2024-05-27T07:35:55.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:55 smithi012 ceph-mon[31473]: pgmap v464: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 163 op/s 2024-05-27T07:35:55.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:55 smithi162 ceph-mon[35676]: pgmap v464: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 163 op/s 2024-05-27T07:35:55.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:55 smithi062 ceph-mon[35304]: pgmap v464: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 163 op/s 2024-05-27T07:35:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.089+0000 7f600d600700 1 -- 172.21.15.12:0/3056070018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60080764a0 msgr2=0x7f6008114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.089+0000 7f600d600700 1 --2- 172.21.15.12:0/3056070018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60080764a0 0x7f6008114e20 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4009990 tx=0x7f5ff4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.089+0000 7f600d600700 1 -- 172.21.15.12:0/3056070018 shutdown_connections 2024-05-27T07:35:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.089+0000 7f600d600700 1 --2- 172.21.15.12:0/3056070018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6008115360 0x7f6008117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.089+0000 7f600d600700 1 --2- 172.21.15.12:0/3056070018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6008075b50 0x7f6008075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.089+0000 7f600d600700 1 --2- 172.21.15.12:0/3056070018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60080764a0 0x7f6008114e20 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.089+0000 7f600d600700 1 -- 172.21.15.12:0/3056070018 >> 172.21.15.12:0/3056070018 conn(0x7f6008100030 msgr2=0x7f6008102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.090+0000 7f600d600700 1 -- 172.21.15.12:0/3056070018 shutdown_connections 2024-05-27T07:35:56.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.090+0000 7f600d600700 1 -- 172.21.15.12:0/3056070018 wait complete. 2024-05-27T07:35:56.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.091+0000 7f600d600700 1 Processor -- start 2024-05-27T07:35:56.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.092+0000 7f600d600700 1 -- start start 2024-05-27T07:35:56.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.092+0000 7f600d600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6008075b50 0x7f60081a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.092+0000 7f600d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60080764a0 0x7f60081a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.092+0000 7f600d600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6008115360 0x7f60081a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.092+0000 7f600d600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f600811a910 con 0x7f6008115360 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.092+0000 7f600d600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f600811a790 con 0x7f60080764a0 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.092+0000 7f600d600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f600811aa90 con 0x7f6008075b50 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.093+0000 7f6006000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60080764a0 0x7f60081a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.093+0000 7f6006a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6008075b50 0x7f60081a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.093+0000 7f6006000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60080764a0 0x7f60081a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:47142/0 (socket says 172.21.15.12:47142) 2024-05-27T07:35:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.093+0000 7f6006000700 1 -- 172.21.15.12:0/3943874555 learned_addr learned my addr 172.21.15.12:0/3943874555 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.094+0000 7f6007400700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6008115360 0x7f60081a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.094+0000 7f6006a00700 1 -- 172.21.15.12:0/3943874555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60080764a0 msgr2=0x7f60081a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.094+0000 7f6006a00700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60080764a0 0x7f60081a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.094+0000 7f6006a00700 1 -- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6008115360 msgr2=0x7f60081a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.094+0000 7f6006a00700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6008115360 0x7f60081a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.094+0000 7f6006a00700 1 -- 172.21.15.12:0/3943874555 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60081a6270 con 0x7f6008075b50 2024-05-27T07:35:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.095+0000 7f6006a00700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6008075b50 0x7f60081a13a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ff800d950 tx=0x7f5ff800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.095+0000 7f5fff400700 1 -- 172.21.15.12:0/3943874555 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8016070 con 0x7f6008075b50 2024-05-27T07:35:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.095+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60081a6560 con 0x7f6008075b50 2024-05-27T07:35:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.095+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60081ade80 con 0x7f6008075b50 2024-05-27T07:35:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.096+0000 7f5fff400700 1 -- 172.21.15.12:0/3943874555 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ff800fd20 con 0x7f6008075b50 2024-05-27T07:35:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.096+0000 7f5fff400700 1 -- 172.21.15.12:0/3943874555 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff800e3b0 con 0x7f6008075b50 2024-05-27T07:35:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.101+0000 7f5fff400700 1 -- 172.21.15.12:0/3943874555 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5ff8019070 con 0x7f6008075b50 2024-05-27T07:35:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.101+0000 7f601c1e1b80 1 Processor -- start 2024-05-27T07:35:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.102+0000 7f601c1e1b80 1 -- start start 2024-05-27T07:35:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.102+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e3970a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.102+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.102+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e39717110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.102+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558e39638e40 con 0x558e3970ae70 2024-05-27T07:35:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.102+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558e396289e0 con 0x558e396779f0 2024-05-27T07:35:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.102+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e3962ca20 con 0x558e3958be40 2024-05-27T07:35:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.103+0000 7f6007400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e3970a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.103+0000 7f6006000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e39717110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.103+0000 7f6006000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e39717110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48884/0 (socket says 172.21.15.12:48884) 2024-05-27T07:35:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.103+0000 7f6007400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e3970a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:47144/0 (socket says 172.21.15.12:47144) 2024-05-27T07:35:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.103+0000 7f6006000700 1 -- 172.21.15.12:0/3373997137 learned_addr learned my addr 172.21.15.12:0/3373997137 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.103+0000 7f6006a00700 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f6006000700 1 -- 172.21.15.12:0/3373997137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 msgr2=0x558e3970a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f6006000700 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f6006000700 1 -- 172.21.15.12:0/3373997137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 msgr2=0x558e3970a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f6006000700 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e3970a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f6006000700 1 -- 172.21.15.12:0/3373997137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e395f1420 con 0x558e3970ae70 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f5fff400700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fec0b2a70 0x7f5fec0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f5fff400700 1 -- 172.21.15.12:0/3943874555 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f5ff80e2660 con 0x7f6008075b50 2024-05-27T07:35:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.104+0000 7f6007400700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fec0b2a70 0x7f5fec0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.105+0000 7f6006000700 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e39717110 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4003750 tx=0x7f5ff4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2c72f923b1b6d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.105+0000 7f5ff3e00700 1 -- 172.21.15.12:0/3373997137 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4027070 con 0x558e3970ae70 2024-05-27T07:35:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.106+0000 7f5ff3e00700 1 -- 172.21.15.12:0/3373997137 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ff40037c0 con 0x558e3970ae70 2024-05-27T07:35:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.106+0000 7f5ff3e00700 1 -- 172.21.15.12:0/3373997137 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff400bb70 con 0x558e3970ae70 2024-05-27T07:35:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.106+0000 7f6007400700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fec0b2a70 0x7f5fec0b4f80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x558e3970cff0 tx=0x7f6000009100 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3373997137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 msgr2=0x558e39717110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e39717110 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4003750 tx=0x7f5ff4005050 comp rx=0 tx=0).stop 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3373997137 shutdown_connections 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e3970a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e39717110 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3373997137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3373997137 >> 172.21.15.12:0/3373997137 conn(0x558e39625390 msgr2=0x558e397064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3373997137 shutdown_connections 2024-05-27T07:35:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.107+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3373997137 wait complete. 2024-05-27T07:35:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.108+0000 7f601c1e1b80 1 -- start start 2024-05-27T07:35:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.108+0000 7f601c1e1b80 1 Processor -- start 2024-05-27T07:35:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3958be40 0x558e39714680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39714bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e39715100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558e3952efa0 con 0x558e3958be40 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558e395c9760 con 0x558e396779f0 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e39717b20 con 0x558e3970ae70 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f601c1e1b80 10 client.0 _ll_register_callbacks cb 0x558e395d2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.109+0000 7f6006a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3958be40 0x558e39714680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3958be40 0x558e39714680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48894/0 (socket says 172.21.15.12:48894) 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006a00700 1 -- 172.21.15.12:0/449962591 learned_addr learned my addr 172.21.15.12:0/449962591 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6007400700 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e39715100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006000700 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39714bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006a00700 1 -- 172.21.15.12:0/449962591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 msgr2=0x558e39715100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006a00700 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e39715100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006a00700 1 -- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 msgr2=0x558e39714bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006a00700 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39714bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.110+0000 7f6006a00700 1 -- 172.21.15.12:0/449962591 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e395788d0 con 0x558e3958be40 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.111+0000 7f6006a00700 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3958be40 0x558e39714680 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80df9a0 tx=0x7f5ff80a8e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dcc7dae216db4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.111+0000 7f5ff0c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.111+0000 7f5ff0c00700 1 -- 172.21.15.12:0/449962591 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8020070 con 0x558e3958be40 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.112+0000 7f5ff0c00700 1 -- 172.21.15.12:0/449962591 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f5ff80a9940 con 0x558e3958be40 2024-05-27T07:35:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.112+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ff40037c0 con 0x558e3958be40 2024-05-27T07:35:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.112+0000 7f5ff0c00700 1 -- 172.21.15.12:0/449962591 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5ff80a8fe0 con 0x558e3958be40 2024-05-27T07:35:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.112+0000 7f601c1e1b80 10 client.17046 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.112+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558e395f9060 con 0x558e3958be40 2024-05-27T07:35:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f5ff0c00700 1 -- 172.21.15.12:0/449962591 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ff40037c0 con 0x558e3958be40 2024-05-27T07:35:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f5ff0c00700 1 -- 172.21.15.12:0/449962591 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5ff80b1ea0 con 0x558e3958be40 2024-05-27T07:35:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f601c1e1b80 10 client.17046 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f601c1e1b80 10 client.17046 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:35:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f601c1e1b80 20 client.17046 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e39724ec0 0x558e39727370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f601c1e1b80 4 client.17046 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.113+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e3963ad90 con 0x558e39724ec0 2024-05-27T07:35:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.114+0000 7f6006000700 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e39724ec0 0x558e39727370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.115+0000 7f6006000700 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e39724ec0 0x558e39727370 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3055a6c64d1a7dc2 server_cookie=33780c454efbd4af in_seq=0 out_seq=0 2024-05-27T07:35:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.115+0000 7f5ff0c00700 10 client.17046 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.117+0000 7f5ff0c00700 1 -- 172.21.15.12:0/449962591 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558e3963ad90 con 0x558e39724ec0 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.117+0000 7f5ff0c00700 10 client.17046 handle_command_reply: tid=0 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.117+0000 7f601c1e1b80 1 client.17046 shutdown 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3958be40 msgr2=0x558e39714680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3958be40 0x558e39714680 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80df9a0 tx=0x7f5ff80a8e20 comp rx=0 tx=0).stop 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 shutdown_connections 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e39715100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3958be40 0x558e39714680 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e39724ec0 0x558e39727370 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/449962591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39714bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f5ff0c00700 0 client.17046 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:35:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 >> 172.21.15.12:0/449962591 conn(0x558e39625390 msgr2=0x558e3970a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 shutdown_connections 2024-05-27T07:35:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 1 -- 172.21.15.12:0/449962591 wait complete. 2024-05-27T07:35:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.118+0000 7f601c1e1b80 20 client.17046 trim_cache size 0 max 16384 2024-05-27T07:35:56.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.140+0000 7f601c1e1b80 1 Processor -- start 2024-05-27T07:35:56.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.140+0000 7f601c1e1b80 1 -- start start 2024-05-27T07:35:56.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.140+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.140+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39720c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.140+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e3972cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.140+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558e39704e50 con 0x558e3970ae70 2024-05-27T07:35:56.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.141+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe800e410 con 0x558e396779f0 2024-05-27T07:35:56.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.141+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe800d670 con 0x558e3958be40 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.141+0000 7f6007400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e3972cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.141+0000 7f6007400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e3972cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48896/0 (socket says 172.21.15.12:48896) 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.141+0000 7f6007400700 1 -- 172.21.15.12:0/3705263503 learned_addr learned my addr 172.21.15.12:0/3705263503 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.141+0000 7f6006000700 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39720c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.141+0000 7f6006a00700 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.142+0000 7f6007400700 1 -- 172.21.15.12:0/3705263503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 msgr2=0x558e3970a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.142+0000 7f6007400700 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.142+0000 7f6007400700 1 -- 172.21.15.12:0/3705263503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 msgr2=0x558e39720c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.142+0000 7f6007400700 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39720c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.142+0000 7f6007400700 1 -- 172.21.15.12:0/3705263503 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e39702240 con 0x558e3970ae70 2024-05-27T07:35:56.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.142+0000 7f6007400700 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e3972cb70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f6000004590 tx=0x7f6000016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c8dd67f75df60c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.143+0000 7f5ff3400700 1 -- 172.21.15.12:0/3705263503 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000007b10 con 0x558e3970ae70 2024-05-27T07:35:56.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.143+0000 7f5ff3400700 1 -- 172.21.15.12:0/3705263503 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f600001c030 con 0x558e3970ae70 2024-05-27T07:35:56.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.143+0000 7f5ff3400700 1 -- 172.21.15.12:0/3705263503 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000016e10 con 0x558e3970ae70 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3705263503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 msgr2=0x558e3972cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e3972cb70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f6000004590 tx=0x7f6000016000 comp rx=0 tx=0).stop 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3705263503 shutdown_connections 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e3970ae70 0x558e3972cb70 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3958be40 0x558e3970a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/3705263503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e396779f0 0x558e39720c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3705263503 >> 172.21.15.12:0/3705263503 conn(0x558e39625390 msgr2=0x558e3970a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3705263503 shutdown_connections 2024-05-27T07:35:56.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.144+0000 7f601c1e1b80 1 -- 172.21.15.12:0/3705263503 wait complete. 2024-05-27T07:35:56.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.145+0000 7f601c1e1b80 1 -- start start 2024-05-27T07:35:56.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.145+0000 7f601c1e1b80 1 Processor -- start 2024-05-27T07:35:56.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e3958be40 0x558e3971d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e396779f0 0x558e3971d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f601c1e1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e3970e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558e3963ad90 con 0x558e396779f0 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe800dd40 con 0x558e3958be40 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f601c1e1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e39715b10 con 0x558e3970ae70 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f601c1e1b80 10 client.0 _ll_register_callbacks cb 0x558e395d2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e396779f0 0x558e3971d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e396779f0 0x558e3971d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48900/0 (socket says 172.21.15.12:48900) 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006000700 1 -- 172.21.15.12:0/2055077193 learned_addr learned my addr 172.21.15.12:0/2055077193 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006a00700 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e3958be40 0x558e3971d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6007400700 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e3970e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006000700 1 -- 172.21.15.12:0/2055077193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 msgr2=0x558e3970e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006000700 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e3970e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006000700 1 -- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e3958be40 msgr2=0x558e3971d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.146+0000 7f6006000700 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e3958be40 0x558e3971d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.147+0000 7f6006000700 1 -- 172.21.15.12:0/2055077193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e3970eef0 con 0x558e396779f0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.147+0000 7f6006000700 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e396779f0 0x558e3971d5d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40050c0 tx=0x7f5ff4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553d27356ebf13ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.147+0000 7f5ff0c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.147+0000 7f5ff0c00700 1 -- 172.21.15.12:0/2055077193 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4027040 con 0x558e396779f0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.147+0000 7f5ff0c00700 1 -- 172.21.15.12:0/2055077193 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f5ff40051f0 con 0x558e396779f0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.147+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f600001c030 con 0x558e396779f0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.147+0000 7f5ff0c00700 1 -- 172.21.15.12:0/2055077193 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5ff40053f0 con 0x558e396779f0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 10 client.17052 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558e3971c870 con 0x558e396779f0 2024-05-27T07:35:56.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f5ff0c00700 1 -- 172.21.15.12:0/2055077193 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f600001c030 con 0x558e396779f0 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f5ff0c00700 1 -- 172.21.15.12:0/2055077193 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5ff4036070 con 0x558e396779f0 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 10 client.17052 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 10 client.17052 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 20 client.17052 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e3978fa90 0x558e39791eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 4 client.17052 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.148+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558e39634570 con 0x558e3978fa90 2024-05-27T07:35:56.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.149+0000 7f6006a00700 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e3978fa90 0x558e39791eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.150+0000 7f6006a00700 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e3978fa90 0x558e39791eb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b24907ea682c0d5a server_cookie=664b2a1688455f2d in_seq=0 out_seq=0 2024-05-27T07:35:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.150+0000 7f5ff0c00700 10 client.17052 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:35:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.150+0000 7f5ff0c00700 1 -- 172.21.15.12:0/2055077193 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558e39634570 con 0x558e3978fa90 2024-05-27T07:35:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.150+0000 7f5ff0c00700 10 client.17052 handle_command_reply: tid=0 2024-05-27T07:35:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.150+0000 7f601c1e1b80 1 client.17052 shutdown 2024-05-27T07:35:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.150+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e396779f0 msgr2=0x558e3971d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.150+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e396779f0 0x558e3971d5d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40050c0 tx=0x7f5ff4009310 comp rx=0 tx=0).stop 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 shutdown_connections 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558e3978fa90 0x558e39791eb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558e3958be40 0x558e3971d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e3970ae70 0x558e3970e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 --2- 172.21.15.12:0/2055077193 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558e396779f0 0x558e3971d5d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 >> 172.21.15.12:0/2055077193 conn(0x558e39625390 msgr2=0x558e3970bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f5ff0c00700 0 client.17052 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 shutdown_connections 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 1 -- 172.21.15.12:0/2055077193 wait complete. 2024-05-27T07:35:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.151+0000 7f601c1e1b80 20 client.17052 trim_cache size 0 max 16384 2024-05-27T07:35:56.149 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:35:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fec0b2a70 msgr2=0x7f5fec0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fec0b2a70 0x7f5fec0b4f80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x558e3970cff0 tx=0x7f6000009100 comp rx=0 tx=0).stop 2024-05-27T07:35:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6008075b50 msgr2=0x7f60081a13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6008075b50 0x7f60081a13a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ff800d950 tx=0x7f5ff800dc60 comp rx=0 tx=0).stop 2024-05-27T07:35:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 shutdown_connections 2024-05-27T07:35:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5fec0b2a70 0x7f5fec0b4f80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6008115360 0x7f60081a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6008075b50 0x7f60081a13a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 --2- 172.21.15.12:0/3943874555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60080764a0 0x7f60081a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 >> 172.21.15.12:0/3943874555 conn(0x7f6008100030 msgr2=0x7f6008100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 shutdown_connections 2024-05-27T07:35:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:56.152+0000 7f600d600700 1 -- 172.21.15.12:0/3943874555 wait complete. 2024-05-27T07:35:57.068 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:35:57.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:57 smithi012 ceph-mon[31473]: pgmap v465: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 945 KiB/s wr, 110 op/s 2024-05-27T07:35:57.563 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:57.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:57 smithi162 ceph-mon[35676]: pgmap v465: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 945 KiB/s wr, 110 op/s 2024-05-27T07:35:57.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:57 smithi062 ceph-mon[35304]: pgmap v465: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 945 KiB/s wr, 110 op/s 2024-05-27T07:35:59.521 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:35:59 smithi012 ceph-mon[31473]: pgmap v466: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 945 KiB/s wr, 110 op/s 2024-05-27T07:35:59.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:35:59 smithi162 ceph-mon[35676]: pgmap v466: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 945 KiB/s wr, 110 op/s 2024-05-27T07:35:59.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:35:59 smithi062 ceph-mon[35304]: pgmap v466: 129 pgs: 129 active+clean; 239 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 945 KiB/s wr, 110 op/s 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 -- 172.21.15.12:0/1166368244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc107e40 msgr2=0x7f99fc108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 --2- 172.21.15.12:0/1166368244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc107e40 0x7f99fc108250 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f99e80099c0 tx=0x7f99e8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 -- 172.21.15.12:0/1166368244 shutdown_connections 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 --2- 172.21.15.12:0/1166368244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99fc108790 0x7f99fc110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 --2- 172.21.15.12:0/1166368244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103b20 0x7f99fc103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 --2- 172.21.15.12:0/1166368244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc107e40 0x7f99fc108250 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 -- 172.21.15.12:0/1166368244 >> 172.21.15.12:0/1166368244 conn(0x7f99fc0fe510 msgr2=0x7f99fc100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 -- 172.21.15.12:0/1166368244 shutdown_connections 2024-05-27T07:35:59.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.966+0000 7f9a01800700 1 -- 172.21.15.12:0/1166368244 wait complete. 2024-05-27T07:35:59.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.968+0000 7f9a01800700 1 Processor -- start 2024-05-27T07:35:59.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.968+0000 7f9a01800700 1 -- start start 2024-05-27T07:35:59.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f9a01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc103b20 0x7f99fc0fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f9a01800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99fc107e40 0x7f99fc0fef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f9a01800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc108790 0x7f99fc0ff470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f9a01800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1132a0 con 0x7f99fc103b20 2024-05-27T07:35:59.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f9a01800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc113120 con 0x7f99fc107e40 2024-05-27T07:35:59.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f9a01800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc113420 con 0x7f99fc108790 2024-05-27T07:35:59.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f99faa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc103b20 0x7f99fc0fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f99faa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc103b20 0x7f99fc0fe9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48912/0 (socket says 172.21.15.12:48912) 2024-05-27T07:35:59.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f99faa00700 1 -- 172.21.15.12:0/11225398 learned_addr learned my addr 172.21.15.12:0/11225398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:59.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f99fb400700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc108790 0x7f99fc0ff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.969+0000 7f99fa000700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99fc107e40 0x7f99fc0fef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.970+0000 7f99faa00700 1 -- 172.21.15.12:0/11225398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc108790 msgr2=0x7f99fc0ff470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.970+0000 7f99faa00700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc108790 0x7f99fc0ff470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.970+0000 7f99faa00700 1 -- 172.21.15.12:0/11225398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99fc107e40 msgr2=0x7f99fc0fef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.970+0000 7f99faa00700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99fc107e40 0x7f99fc0fef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.970+0000 7f99faa00700 1 -- 172.21.15.12:0/11225398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc1a90f0 con 0x7f99fc103b20 2024-05-27T07:35:59.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.970+0000 7f99faa00700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc103b20 0x7f99fc0fe9f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00c970 tx=0x7f99ec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:59.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.971+0000 7f99f3400700 1 -- 172.21.15.12:0/11225398 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0078d0 con 0x7f99fc103b20 2024-05-27T07:35:59.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.971+0000 7f99f3400700 1 -- 172.21.15.12:0/11225398 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99ec018480 con 0x7f99fc103b20 2024-05-27T07:35:59.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.971+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc1a93e0 con 0x7f99fc103b20 2024-05-27T07:35:59.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.971+0000 7f99f3400700 1 -- 172.21.15.12:0/11225398 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec00f6b0 con 0x7f99fc103b20 2024-05-27T07:35:59.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.971+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99fc1a98a0 con 0x7f99fc103b20 2024-05-27T07:35:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.975+0000 7f99f3400700 1 -- 172.21.15.12:0/11225398 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f99ec018930 con 0x7f99fc103b20 2024-05-27T07:35:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.977+0000 7f9a1032cb80 1 Processor -- start 2024-05-27T07:35:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.977+0000 7f9a1032cb80 1 -- start start 2024-05-27T07:35:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f99f3400700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f99e00b29f0 0x7f99e00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28017f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27fb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af27fbdb0 0x561af27fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561af2729e40 con 0x561af2801d30 2024-05-27T07:35:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561af27199e0 con 0x561af267ce40 2024-05-27T07:35:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561af271da20 con 0x561af27fbdb0 2024-05-27T07:35:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.978+0000 7f99f3400700 1 -- 172.21.15.12:0/11225398 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f99ec0e2b50 con 0x7f99fc103b20 2024-05-27T07:35:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28017f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:47198/0 (socket says 172.21.15.12:47198) 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99faa00700 1 -- 172.21.15.12:0/950728041 learned_addr learned my addr 172.21.15.12:0/950728041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99fa000700 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27fb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99fb400700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f99e00b29f0 0x7f99e00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99fb400700 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af27fbdb0 0x561af27fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99faa00700 1 -- 172.21.15.12:0/950728041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af27fbdb0 msgr2=0x561af27fc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99faa00700 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af27fbdb0 0x561af27fc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99faa00700 1 -- 172.21.15.12:0/950728041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 msgr2=0x561af27fb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.979+0000 7f99faa00700 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27fb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.980+0000 7f99faa00700 1 -- 172.21.15.12:0/950728041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561af26e2420 con 0x561af267ce40 2024-05-27T07:35:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.980+0000 7f99fb400700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f99e00b29f0 0x7f99e00b4f00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f99f4007fd0 tx=0x7f99f4007dc0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.980+0000 7f99faa00700 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28017f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0a9750 tx=0x7f99ec0a94d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58787863386f189c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.981+0000 7f99e7e00700 1 -- 172.21.15.12:0/950728041 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0a5040 con 0x561af267ce40 2024-05-27T07:35:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.981+0000 7f99e7e00700 1 -- 172.21.15.12:0/950728041 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99ec0a9560 con 0x561af267ce40 2024-05-27T07:35:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.981+0000 7f99e7e00700 1 -- 172.21.15.12:0/950728041 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0b2820 con 0x561af267ce40 2024-05-27T07:35:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.982+0000 7f9a1032cb80 1 -- 172.21.15.12:0/950728041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 msgr2=0x561af28017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.982+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28017f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0a9750 tx=0x7f99ec0a94d0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.983+0000 7f9a1032cb80 1 -- 172.21.15.12:0/950728041 shutdown_connections 2024-05-27T07:35:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.983+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28017f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.983+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af27fbdb0 0x561af27fc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.983+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/950728041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.983+0000 7f9a1032cb80 1 -- 172.21.15.12:0/950728041 >> 172.21.15.12:0/950728041 conn(0x561af26dcfc0 msgr2=0x561af27f68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.984+0000 7f9a1032cb80 1 -- 172.21.15.12:0/950728041 shutdown_connections 2024-05-27T07:35:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.984+0000 7f9a1032cb80 1 -- 172.21.15.12:0/950728041 wait complete. 2024-05-27T07:35:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.985+0000 7f9a1032cb80 1 -- start start 2024-05-27T07:35:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.985+0000 7f9a1032cb80 1 Processor -- start 2024-05-27T07:35:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28055a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af27fbdb0 0x561af2805ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af2806020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561af261ffa0 con 0x561af27fbdb0 2024-05-27T07:35:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561af26ba760 con 0x561af267ce40 2024-05-27T07:35:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561af2808800 con 0x561af2801d30 2024-05-27T07:35:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f9a1032cb80 10 client.0 _ll_register_callbacks cb 0x561af26c3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f99fa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af27fbdb0 0x561af2805ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f99fa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af27fbdb0 0x561af2805ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48936/0 (socket says 172.21.15.12:48936) 2024-05-27T07:35:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f99fa000700 1 -- 172.21.15.12:0/2113236521 learned_addr learned my addr 172.21.15.12:0/2113236521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:35:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.986+0000 7f99fb400700 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af2806020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.987+0000 7f99fa000700 1 -- 172.21.15.12:0/2113236521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 msgr2=0x561af2806020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:59.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.987+0000 7f99fa000700 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af2806020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.987+0000 7f99fa000700 1 -- 172.21.15.12:0/2113236521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 msgr2=0x561af28055a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:59.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.987+0000 7f99fa000700 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28055a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.987+0000 7f99fa000700 1 -- 172.21.15.12:0/2113236521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561af26698d0 con 0x561af27fbdb0 2024-05-27T07:35:59.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.987+0000 7f99fa000700 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af27fbdb0 0x561af2805ae0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f99fc06c9b0 tx=0x7f99e800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307a0e18f87f9340 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:59.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.988+0000 7f99e4c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:35:59.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.988+0000 7f99e4c00700 1 -- 172.21.15.12:0/2113236521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e8003680 con 0x561af27fbdb0 2024-05-27T07:35:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.988+0000 7f99e4c00700 1 -- 172.21.15.12:0/2113236521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f99e8003820 con 0x561af27fbdb0 2024-05-27T07:35:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.989+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99ec0a9560 con 0x561af27fbdb0 2024-05-27T07:35:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.989+0000 7f99e4c00700 1 -- 172.21.15.12:0/2113236521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f99e8003a20 con 0x561af27fbdb0 2024-05-27T07:35:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.990+0000 7f9a1032cb80 10 client.17064 fetch_fsmap learned FSMap version 119 2024-05-27T07:35:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.990+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x561af26ea060 con 0x561af27fbdb0 2024-05-27T07:35:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f99e4c00700 1 -- 172.21.15.12:0/2113236521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99ec0a9560 con 0x561af27fbdb0 2024-05-27T07:35:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f99e4c00700 1 -- 172.21.15.12:0/2113236521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f99e8023b70 con 0x561af27fbdb0 2024-05-27T07:35:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f9a1032cb80 10 client.17064 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:35:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f9a1032cb80 10 client.17064 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:35:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f9a1032cb80 20 client.17064 populate_metadata read hostname 'smithi012' 2024-05-27T07:35:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af2815be0 0x561af28180b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f9a1032cb80 4 client.17064 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.992+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561af272bd90 con 0x561af2815be0 2024-05-27T07:35:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.993+0000 7f99faa00700 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af2815be0 0x561af28180b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.994+0000 7f99e4c00700 10 client.17064 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:35:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.994+0000 7f99faa00700 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af2815be0 0x561af28180b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=926a0e0597d73fd1 server_cookie=dd19d7ac22d910a6 in_seq=0 out_seq=0 2024-05-27T07:35:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f99e4c00700 1 -- 172.21.15.12:0/2113236521 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561af272bd90 con 0x561af2815be0 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f99e4c00700 10 client.17064 handle_command_reply: tid=0 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 client.17064 shutdown 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af27fbdb0 msgr2=0x561af2805ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af27fbdb0 0x561af2805ae0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f99fc06c9b0 tx=0x7f99e800b840 comp rx=0 tx=0).stop 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 shutdown_connections 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af2815be0 0x561af28180b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af28055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af2806020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2113236521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af27fbdb0 0x561af2805ae0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 >> 172.21.15.12:0/2113236521 conn(0x561af26dcfc0 msgr2=0x561af27f6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f99e4c00700 0 client.17064 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:35:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.997+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 shutdown_connections 2024-05-27T07:35:59.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.998+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2113236521 wait complete. 2024-05-27T07:35:59.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:35:59.998+0000 7f9a1032cb80 20 client.17064 trim_cache size 0 max 16384 2024-05-27T07:36:00.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 Processor -- start 2024-05-27T07:36:00.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 -- start start 2024-05-27T07:36:00.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af267ce40 0x561af27fb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:00.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27ff920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af2807bb0 0x561af2810850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1b1c90 con 0x561af2801d30 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1b0ef0 con 0x561af2807bb0 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.016+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1b15c0 con 0x561af267ce40 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27ff920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27ff920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48940/0 (socket says 172.21.15.12:48940) 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 -- 172.21.15.12:0/2271393748 learned_addr learned my addr 172.21.15.12:0/2271393748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99faa00700 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af267ce40 0x561af27fb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 -- 172.21.15.12:0/2271393748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af267ce40 msgr2=0x561af27fb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af267ce40 0x561af27fb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 -- 172.21.15.12:0/2271393748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af2807bb0 msgr2=0x561af2810850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fb400700 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af2807bb0 0x561af2810850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:00.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af2807bb0 0x561af2810850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fa000700 1 -- 172.21.15.12:0/2271393748 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561af2806390 con 0x561af2801d30 2024-05-27T07:36:00.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.017+0000 7f99fb400700 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af2807bb0 0x561af2810850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.018+0000 7f99fa000700 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27ff920 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f99e80058e0 tx=0x7f99e80057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e63ceee3bd3b76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.018+0000 7f99e7400700 1 -- 172.21.15.12:0/2271393748 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e803a070 con 0x561af2801d30 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.018+0000 7f99e7400700 1 -- 172.21.15.12:0/2271393748 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99e8028070 con 0x561af2801d30 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.018+0000 7f99e7400700 1 -- 172.21.15.12:0/2271393748 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e800f7d0 con 0x561af2801d30 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.019+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2271393748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 msgr2=0x561af27ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.019+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27ff920 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f99e80058e0 tx=0x7f99e80057e0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.019+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2271393748 shutdown_connections 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.019+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af2807bb0 0x561af2810850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.019+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af267ce40 0x561af27fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.019+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/2271393748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2801d30 0x561af27ff920 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.019+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2271393748 >> 172.21.15.12:0/2271393748 conn(0x561af26dcfc0 msgr2=0x561af27ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.020+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2271393748 shutdown_connections 2024-05-27T07:36:00.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.020+0000 7f9a1032cb80 1 -- 172.21.15.12:0/2271393748 wait complete. 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.020+0000 7f9a1032cb80 1 -- start start 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.020+0000 7f9a1032cb80 1 Processor -- start 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.021+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af280da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.021+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af280df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.021+0000 7f9a1032cb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2807bb0 0x561af280aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.021+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561af28083a0 con 0x561af2807bb0 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.021+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561af27fcb60 con 0x561af267ce40 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.021+0000 7f9a1032cb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561af2802140 con 0x561af2801d30 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.021+0000 7f9a1032cb80 10 client.0 _ll_register_callbacks cb 0x561af26c3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2807bb0 0x561af280aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af280da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:00.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2807bb0 0x561af280aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48942/0 (socket says 172.21.15.12:48942) 2024-05-27T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 -- 172.21.15.12:0/3221411989 learned_addr learned my addr 172.21.15.12:0/3221411989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fa000700 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af280df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 -- 172.21.15.12:0/3221411989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 msgr2=0x561af280df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af280df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 -- 172.21.15.12:0/3221411989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 msgr2=0x561af280da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af280da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.022+0000 7f99fb400700 1 -- 172.21.15.12:0/3221411989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561af27fb970 con 0x561af2807bb0 2024-05-27T07:36:00.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.023+0000 7f99fb400700 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2807bb0 0x561af280aa20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f99f4016e80 tx=0x7f99f4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335cc7ddd3191828 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:00.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.023+0000 7f99e4c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:00.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.023+0000 7f99e4c00700 1 -- 172.21.15.12:0/3221411989 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4010040 con 0x561af2807bb0 2024-05-27T07:36:00.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.023+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e8028070 con 0x561af2807bb0 2024-05-27T07:36:00.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.023+0000 7f99e4c00700 1 -- 172.21.15.12:0/3221411989 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7f99f4018d10 con 0x561af2807bb0 2024-05-27T07:36:00.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f99e4c00700 1 -- 172.21.15.12:0/3221411989 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f99f4033050 con 0x561af2807bb0 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 10 client.17070 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x561af2880a80 con 0x561af2807bb0 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f99e4c00700 1 -- 172.21.15.12:0/3221411989 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99e8028070 con 0x561af2807bb0 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f99e4c00700 1 -- 172.21.15.12:0/3221411989 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f99f4023430 con 0x561af2807bb0 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 10 client.17070 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 10 client.17070 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 20 client.17070 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af28811b0 0x561af28835d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 4 client.17070 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.024+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561af2725570 con 0x561af28811b0 2024-05-27T07:36:00.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.025+0000 7f99faa00700 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af28811b0 0x561af28835d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:00.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.026+0000 7f99faa00700 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af28811b0 0x561af28835d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27bc928e7b81f4c0 server_cookie=2ec63ebe776a51d8 in_seq=0 out_seq=0 2024-05-27T07:36:00.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.026+0000 7f99e4c00700 10 client.17070 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:36:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f99e4c00700 1 -- 172.21.15.12:0/3221411989 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561af2725570 con 0x561af28811b0 2024-05-27T07:36:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f99e4c00700 10 client.17070 handle_command_reply: tid=0 2024-05-27T07:36:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 client.17070 shutdown 2024-05-27T07:36:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2807bb0 msgr2=0x561af280aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:00.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2807bb0 0x561af280aa20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f99f4016e80 tx=0x7f99f4004b40 comp rx=0 tx=0).stop 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 shutdown_connections 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x561af28811b0 0x561af28835d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561af267ce40 0x561af280da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561af2807bb0 0x561af280aa20 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 --2- 172.21.15.12:0/3221411989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561af2801d30 0x561af280df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 >> 172.21.15.12:0/3221411989 conn(0x561af26dcfc0 msgr2=0x561af2806b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f99e4c00700 0 client.17070 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.027+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 shutdown_connections 2024-05-27T07:36:00.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.028+0000 7f9a1032cb80 1 -- 172.21.15.12:0/3221411989 wait complete. 2024-05-27T07:36:00.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.028+0000 7f9a1032cb80 20 client.17070 trim_cache size 0 max 16384 2024-05-27T07:36:00.025 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f99e00b29f0 msgr2=0x7f99e00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f99e00b29f0 0x7f99e00b4f00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f99f4007fd0 tx=0x7f99f4007dc0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc103b20 msgr2=0x7f99fc0fe9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc103b20 0x7f99fc0fe9f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00c970 tx=0x7f99ec00cc80 comp rx=0 tx=0).stop 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 shutdown_connections 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f99e00b29f0 0x7f99e00b4f00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc108790 0x7f99fc0ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99fc103b20 0x7f99fc0fe9f0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 --2- 172.21.15.12:0/11225398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99fc107e40 0x7f99fc0fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:00.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 >> 172.21.15.12:0/11225398 conn(0x7f99fc0fe510 msgr2=0x7f99fc102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:00.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 shutdown_connections 2024-05-27T07:36:00.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:00.029+0000 7f9a01800700 1 -- 172.21.15.12:0/11225398 wait complete. 2024-05-27T07:36:01.101 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:36:01.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:01 smithi012 ceph-mon[31473]: pgmap v467: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:36:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:01.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:01 smithi162 ceph-mon[35676]: pgmap v467: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:36:01.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:01 smithi062 ceph-mon[35304]: pgmap v467: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-27T07:36:03.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:03 smithi012 ceph-mon[31473]: pgmap v468: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1020 KiB/s wr, 113 op/s 2024-05-27T07:36:03.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:03 smithi162 ceph-mon[35676]: pgmap v468: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1020 KiB/s wr, 113 op/s 2024-05-27T07:36:03.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:03 smithi062 ceph-mon[35304]: pgmap v468: 129 pgs: 129 active+clean; 245 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1020 KiB/s wr, 113 op/s 2024-05-27T07:36:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.888+0000 7faa7e600700 1 -- 172.21.15.12:0/3999225235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa781054b0 msgr2=0x7faa781058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.888+0000 7faa7e600700 1 --2- 172.21.15.12:0/3999225235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa781054b0 0x7faa781058c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7faa68009990 tx=0x7faa68009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.889+0000 7faa7e600700 1 -- 172.21.15.12:0/3999225235 shutdown_connections 2024-05-27T07:36:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.889+0000 7faa7e600700 1 --2- 172.21.15.12:0/3999225235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa7810e8e0 0x7faa78110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.889+0000 7faa7e600700 1 --2- 172.21.15.12:0/3999225235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa781054b0 0x7faa781058c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.889+0000 7faa7e600700 1 --2- 172.21.15.12:0/3999225235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa78105e00 0x7faa7810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.889+0000 7faa7e600700 1 -- 172.21.15.12:0/3999225235 >> 172.21.15.12:0/3999225235 conn(0x7faa780fedb0 msgr2=0x7faa781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.890+0000 7faa7e600700 1 -- 172.21.15.12:0/3999225235 shutdown_connections 2024-05-27T07:36:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.890+0000 7faa7e600700 1 -- 172.21.15.12:0/3999225235 wait complete. 2024-05-27T07:36:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.891+0000 7faa7e600700 1 Processor -- start 2024-05-27T07:36:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.892+0000 7faa7e600700 1 -- start start 2024-05-27T07:36:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.892+0000 7faa7e600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa781054b0 0x7faa7819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.892+0000 7faa7e600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa78105e00 0x7faa7819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.892+0000 7faa7e600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa7810e8e0 0x7faa7819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.892+0000 7faa7e600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa781143b0 con 0x7faa78105e00 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.892+0000 7faa7e600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faa78114230 con 0x7faa781054b0 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.892+0000 7faa7e600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faa78114530 con 0x7faa7810e8e0 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa78105e00 0x7faa7819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa78105e00 0x7faa7819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32924/0 (socket says 172.21.15.12:32924) 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 -- 172.21.15.12:0/1908259646 learned_addr learned my addr 172.21.15.12:0/1908259646 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa77400700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa781054b0 0x7faa7819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa77e00700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa7810e8e0 0x7faa7819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 -- 172.21.15.12:0/1908259646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa7810e8e0 msgr2=0x7faa7819f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa7810e8e0 0x7faa7819f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 -- 172.21.15.12:0/1908259646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa781054b0 msgr2=0x7faa7819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa781054b0 0x7faa7819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.893+0000 7faa76a00700 1 -- 172.21.15.12:0/1908259646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7819fc50 con 0x7faa78105e00 2024-05-27T07:36:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.894+0000 7faa76a00700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa78105e00 0x7faa7819b320 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faa6400b530 tx=0x7faa6400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.894+0000 7faa6fe00700 1 -- 172.21.15.12:0/1908259646 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64015020 con 0x7faa78105e00 2024-05-27T07:36:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.894+0000 7faa6fe00700 1 -- 172.21.15.12:0/1908259646 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa64011cc0 con 0x7faa78105e00 2024-05-27T07:36:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.894+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7819ff40 con 0x7faa78105e00 2024-05-27T07:36:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.895+0000 7faa6fe00700 1 -- 172.21.15.12:0/1908259646 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64010dc0 con 0x7faa78105e00 2024-05-27T07:36:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.895+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa781a7860 con 0x7faa78105e00 2024-05-27T07:36:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.900+0000 7faa8d14db80 1 Processor -- start 2024-05-27T07:36:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.900+0000 7faa8d14db80 1 -- start start 2024-05-27T07:36:04.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa6fe00700 1 -- 172.21.15.12:0/1908259646 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7faa64018020 con 0x7faa78105e00 2024-05-27T07:36:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f13ee40 0x55611f2c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2bddb0 0x55611f2be1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa8d14db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55611f1ebe40 con 0x55611f13ee40 2024-05-27T07:36:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa8d14db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55611f1db9e0 con 0x55611f2c3d30 2024-05-27T07:36:04.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa8d14db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55611f1dfa20 con 0x55611f2bddb0 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f13ee40 0x55611f2c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f13ee40 0x55611f2c37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32936/0 (socket says 172.21.15.12:32936) 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa77e00700 1 -- 172.21.15.12:0/2644515290 learned_addr learned my addr 172.21.15.12:0/2644515290 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.901+0000 7faa77400700 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.902+0000 7faa76a00700 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2bddb0 0x55611f2be1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.902+0000 7faa77e00700 1 -- 172.21.15.12:0/2644515290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2bddb0 msgr2=0x55611f2be1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.902+0000 7faa77e00700 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2bddb0 0x55611f2be1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.902+0000 7faa77e00700 1 -- 172.21.15.12:0/2644515290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 msgr2=0x55611f2bd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.902+0000 7faa77e00700 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2bd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.902+0000 7faa77e00700 1 -- 172.21.15.12:0/2644515290 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55611f1a4420 con 0x55611f13ee40 2024-05-27T07:36:04.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.902+0000 7faa76a00700 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2bddb0 0x55611f2be1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:04.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.903+0000 7faa77e00700 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f13ee40 0x55611f2c37f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7faa70008550 tx=0x7faa70008860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186869677c1ec91b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:04.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.903+0000 7faa6fe00700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faa5c0b2a70 0x7faa5c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.903+0000 7faa6fe00700 1 -- 172.21.15.12:0/1908259646 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7faa640e2190 con 0x7faa78105e00 2024-05-27T07:36:04.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.903+0000 7faa6cc00700 1 -- 172.21.15.12:0/2644515290 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70003c00 con 0x55611f13ee40 2024-05-27T07:36:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.903+0000 7faa6cc00700 1 -- 172.21.15.12:0/2644515290 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa70003da0 con 0x55611f13ee40 2024-05-27T07:36:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.903+0000 7faa6cc00700 1 -- 172.21.15.12:0/2644515290 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa700127d0 con 0x55611f13ee40 2024-05-27T07:36:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.903+0000 7faa77400700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faa5c0b2a70 0x7faa5c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa77400700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faa5c0b2a70 0x7faa5c0b4f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faa68009750 tx=0x7faa68005040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 -- 172.21.15.12:0/2644515290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f13ee40 msgr2=0x55611f2c37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f13ee40 0x55611f2c37f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7faa70008550 tx=0x7faa70008860 comp rx=0 tx=0).stop 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 -- 172.21.15.12:0/2644515290 shutdown_connections 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2bddb0 0x55611f2be1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f13ee40 0x55611f2c37f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 --2- 172.21.15.12:0/2644515290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2bd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 -- 172.21.15.12:0/2644515290 >> 172.21.15.12:0/2644515290 conn(0x55611f19efc0 msgr2=0x55611f2b88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.905+0000 7faa8d14db80 1 -- 172.21.15.12:0/2644515290 shutdown_connections 2024-05-27T07:36:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.906+0000 7faa8d14db80 1 -- 172.21.15.12:0/2644515290 wait complete. 2024-05-27T07:36:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.907+0000 7faa8d14db80 1 -- start start 2024-05-27T07:36:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.907+0000 7faa8d14db80 1 Processor -- start 2024-05-27T07:36:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2bddb0 0x55611f2c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2c3d30 0x55611f2c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa8d14db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55611f0e1fa0 con 0x55611f2c3d30 2024-05-27T07:36:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa8d14db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55611f17c760 con 0x55611f2bddb0 2024-05-27T07:36:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa8d14db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55611f2ca800 con 0x55611f13ee40 2024-05-27T07:36:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa8d14db80 10 client.0 _ll_register_callbacks cb 0x55611f185c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2c3d30 0x55611f2c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2c3d30 0x55611f2c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32940/0 (socket says 172.21.15.12:32940) 2024-05-27T07:36:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.908+0000 7faa77e00700 1 -- 172.21.15.12:0/4056167576 learned_addr learned my addr 172.21.15.12:0/4056167576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.909+0000 7faa76a00700 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2bddb0 0x55611f2c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.909+0000 7faa77400700 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.909+0000 7faa77e00700 1 -- 172.21.15.12:0/4056167576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 msgr2=0x55611f2c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.909+0000 7faa77e00700 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.909+0000 7faa77e00700 1 -- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2bddb0 msgr2=0x55611f2c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.909+0000 7faa77e00700 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2bddb0 0x55611f2c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.909+0000 7faa77e00700 1 -- 172.21.15.12:0/4056167576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55611f12b8d0 con 0x55611f2c3d30 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.910+0000 7faa77e00700 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2c3d30 0x55611f2c7fc0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7faa70004260 tx=0x7faa70004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991d0ee6a108ec60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.910+0000 7faa61600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.910+0000 7faa61600700 1 -- 172.21.15.12:0/4056167576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70021020 con 0x55611f2c3d30 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.910+0000 7faa61600700 1 -- 172.21.15.12:0/4056167576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7faa70012710 con 0x55611f2c3d30 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.910+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa70003da0 con 0x55611f2c3d30 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa61600700 1 -- 172.21.15.12:0/4056167576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7faa7002c050 con 0x55611f2c3d30 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 10 client.17082 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55611f1ac060 con 0x55611f2c3d30 2024-05-27T07:36:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa61600700 1 -- 172.21.15.12:0/4056167576 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa70003da0 con 0x55611f2c3d30 2024-05-27T07:36:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa61600700 1 -- 172.21.15.12:0/4056167576 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7faa70013070 con 0x55611f2c3d30 2024-05-27T07:36:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 10 client.17082 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 10 client.17082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 20 client.17082 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f2e01d0 0x55611f2e2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 4 client.17082 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.911+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55611f1edd90 con 0x55611f2e01d0 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.912+0000 7faa77400700 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f2e01d0 0x55611f2e2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.913+0000 7faa77400700 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f2e01d0 0x55611f2e2660 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c54888cca4fb8b77 server_cookie=41faeaf1f8c30a3b in_seq=0 out_seq=0 2024-05-27T07:36:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.913+0000 7faa61600700 10 client.17082 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.915+0000 7faa61600700 1 -- 172.21.15.12:0/4056167576 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55611f1edd90 con 0x55611f2e01d0 2024-05-27T07:36:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.915+0000 7faa61600700 10 client.17082 handle_command_reply: tid=0 2024-05-27T07:36:04.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.915+0000 7faa8d14db80 1 client.17082 shutdown 2024-05-27T07:36:04.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2c3d30 msgr2=0x55611f2c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2c3d30 0x55611f2c7fc0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7faa70004260 tx=0x7faa70004290 comp rx=0 tx=0).stop 2024-05-27T07:36:04.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 shutdown_connections 2024-05-27T07:36:04.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2c3d30 0x55611f2c7fc0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f2e01d0 0x55611f2e2660 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 --2- 172.21.15.12:0/4056167576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2bddb0 0x55611f2c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 >> 172.21.15.12:0/4056167576 conn(0x55611f19efc0 msgr2=0x55611f2b8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:04.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa61600700 0 client.17082 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:04.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 shutdown_connections 2024-05-27T07:36:04.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 1 -- 172.21.15.12:0/4056167576 wait complete. 2024-05-27T07:36:04.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.916+0000 7faa8d14db80 20 client.17082 trim_cache size 0 max 16384 2024-05-27T07:36:04.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.937+0000 7faa8d14db80 1 Processor -- start 2024-05-27T07:36:04.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.937+0000 7faa8d14db80 1 -- start start 2024-05-27T07:36:04.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.938+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.938+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2bddb0 0x55611f2c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.938+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2dc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.938+0000 7faa8d14db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa781ad9d0 con 0x55611f2bddb0 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.938+0000 7faa8d14db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faa781acc30 con 0x55611f2c3d30 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.938+0000 7faa8d14db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faa781ad300 con 0x55611f13ee40 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.938+0000 7faa76a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2bddb0 0x55611f2c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa76a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2bddb0 0x55611f2c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32948/0 (socket says 172.21.15.12:32948) 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa76a00700 1 -- 172.21.15.12:0/3616785615 learned_addr learned my addr 172.21.15.12:0/3616785615 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:04.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa77400700 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa77e00700 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2dc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa76a00700 1 -- 172.21.15.12:0/3616785615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 msgr2=0x55611f2bd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa76a00700 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2bd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa76a00700 1 -- 172.21.15.12:0/3616785615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 msgr2=0x55611f2dc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa76a00700 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2dc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.939+0000 7faa76a00700 1 -- 172.21.15.12:0/3616785615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55611f2dffa0 con 0x55611f2bddb0 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.940+0000 7faa76a00700 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2bddb0 0x55611f2c99e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7faa640a8e10 tx=0x7faa64000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6904eff9e276367 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.941+0000 7faa63e00700 1 -- 172.21.15.12:0/3616785615 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa640a8880 con 0x55611f2bddb0 2024-05-27T07:36:04.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.941+0000 7faa63e00700 1 -- 172.21.15.12:0/3616785615 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa640a8c20 con 0x55611f2bddb0 2024-05-27T07:36:04.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.941+0000 7faa63e00700 1 -- 172.21.15.12:0/3616785615 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa640ad130 con 0x55611f2bddb0 2024-05-27T07:36:04.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.942+0000 7faa8d14db80 1 -- 172.21.15.12:0/3616785615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2bddb0 msgr2=0x55611f2c99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.942+0000 7faa8d14db80 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2bddb0 0x55611f2c99e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7faa640a8e10 tx=0x7faa64000bc0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.942+0000 7faa8d14db80 1 -- 172.21.15.12:0/3616785615 shutdown_connections 2024-05-27T07:36:04.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.942+0000 7faa8d14db80 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f2c3d30 0x55611f2dc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.942+0000 7faa8d14db80 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f13ee40 0x55611f2bd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.942+0000 7faa8d14db80 1 --2- 172.21.15.12:0/3616785615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2bddb0 0x55611f2c99e0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.942+0000 7faa8d14db80 1 -- 172.21.15.12:0/3616785615 >> 172.21.15.12:0/3616785615 conn(0x55611f19efc0 msgr2=0x55611f2c1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:04.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.943+0000 7faa8d14db80 1 -- 172.21.15.12:0/3616785615 shutdown_connections 2024-05-27T07:36:04.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.943+0000 7faa8d14db80 1 -- 172.21.15.12:0/3616785615 wait complete. 2024-05-27T07:36:04.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.943+0000 7faa8d14db80 1 -- start start 2024-05-27T07:36:04.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 1 Processor -- start 2024-05-27T07:36:04.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f13ee40 0x55611f2d8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2c3d30 0x55611f2e7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2e7c50 0x55611f2e8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55611f2c5ab0 con 0x55611f2e7c50 2024-05-27T07:36:04.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55611f2c6810 con 0x55611f13ee40 2024-05-27T07:36:04.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55611f349d80 con 0x55611f2c3d30 2024-05-27T07:36:04.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.944+0000 7faa8d14db80 10 client.0 _ll_register_callbacks cb 0x55611f185c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:04.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f13ee40 0x55611f2d8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2e7c50 0x55611f2e8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f13ee40 0x55611f2d8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:40300/0 (socket says 172.21.15.12:40300) 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77400700 1 -- 172.21.15.12:0/1333110310 learned_addr learned my addr 172.21.15.12:0/1333110310 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa76a00700 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2c3d30 0x55611f2e7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77e00700 1 -- 172.21.15.12:0/1333110310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2c3d30 msgr2=0x55611f2e7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77e00700 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2c3d30 0x55611f2e7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77e00700 1 -- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f13ee40 msgr2=0x55611f2d8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77e00700 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f13ee40 0x55611f2d8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77e00700 1 -- 172.21.15.12:0/1333110310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55611f349f80 con 0x55611f2e7c50 2024-05-27T07:36:04.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.945+0000 7faa77400700 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f13ee40 0x55611f2d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.946+0000 7faa77e00700 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2e7c50 0x55611f2e8060 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7faa70004260 tx=0x7faa7001ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7b3e748a9809e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.946+0000 7faa61600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.946+0000 7faa61600700 1 -- 172.21.15.12:0/1333110310 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70021040 con 0x55611f2e7c50 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.946+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa640a8c20 con 0x55611f2e7c50 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.946+0000 7faa61600700 1 -- 172.21.15.12:0/1333110310 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9554+0+0 (secure 0 0 0) 0x7faa70005c00 con 0x55611f2e7c50 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa61600700 1 -- 172.21.15.12:0/1333110310 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7faa70041050 con 0x55611f2e7c50 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 10 client.17088 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55611f34b1e0 con 0x55611f2e7c50 2024-05-27T07:36:04.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa61600700 1 -- 172.21.15.12:0/1333110310 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa640a8c20 con 0x55611f2e7c50 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa61600700 1 -- 172.21.15.12:0/1333110310 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7faa700137c0 con 0x55611f2e7c50 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 10 client.17088 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 10 client.17088 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 20 client.17088 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f34b860 0x55611f34dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 4 client.17088 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.947+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55611f1e7570 con 0x55611f34b860 2024-05-27T07:36:04.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.948+0000 7faa77400700 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f34b860 0x55611f34dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:04.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.949+0000 7faa77400700 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f34b860 0x55611f34dc80 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4549223a73dd30c server_cookie=e9d3f6d90e9a27b6 in_seq=0 out_seq=0 2024-05-27T07:36:04.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.949+0000 7faa61600700 10 client.17088 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:04.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.949+0000 7faa61600700 1 -- 172.21.15.12:0/1333110310 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55611f1e7570 con 0x55611f34b860 2024-05-27T07:36:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.949+0000 7faa61600700 10 client.17088 handle_command_reply: tid=0 2024-05-27T07:36:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.949+0000 7faa8d14db80 1 client.17088 shutdown 2024-05-27T07:36:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2e7c50 msgr2=0x55611f2e8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2e7c50 0x55611f2e8060 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7faa70004260 tx=0x7faa7001ee60 comp rx=0 tx=0).stop 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 shutdown_connections 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55611f34b860 0x55611f34dc80 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55611f13ee40 0x55611f2d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55611f2e7c50 0x55611f2e8060 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 --2- 172.21.15.12:0/1333110310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55611f2c3d30 0x55611f2e7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 >> 172.21.15.12:0/1333110310 conn(0x55611f19efc0 msgr2=0x55611f2c1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa61600700 0 client.17088 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 shutdown_connections 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 1 -- 172.21.15.12:0/1333110310 wait complete. 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.950+0000 7faa8d14db80 20 client.17088 trim_cache size 0 max 16384 2024-05-27T07:36:04.947 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:36:04.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faa5c0b2a70 msgr2=0x7faa5c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faa5c0b2a70 0x7faa5c0b4f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faa68009750 tx=0x7faa68005040 comp rx=0 tx=0).stop 2024-05-27T07:36:04.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa78105e00 msgr2=0x7faa7819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:04.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa78105e00 0x7faa7819b320 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faa6400b530 tx=0x7faa6400b840 comp rx=0 tx=0).stop 2024-05-27T07:36:04.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 shutdown_connections 2024-05-27T07:36:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7faa5c0b2a70 0x7faa5c0b4f80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa781054b0 0x7faa7819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa7810e8e0 0x7faa7819f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 --2- 172.21.15.12:0/1908259646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa78105e00 0x7faa7819b320 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.951+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 >> 172.21.15.12:0/1908259646 conn(0x7faa780fedb0 msgr2=0x7faa780ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.952+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 shutdown_connections 2024-05-27T07:36:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:04.952+0000 7faa7e600700 1 -- 172.21.15.12:0/1908259646 wait complete. 2024-05-27T07:36:05.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:05 smithi012 ceph-mon[31473]: pgmap v469: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 189 op/s 2024-05-27T07:36:05.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:05 smithi162 ceph-mon[35676]: pgmap v469: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 189 op/s 2024-05-27T07:36:05.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:05 smithi062 ceph-mon[35304]: pgmap v469: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 189 op/s 2024-05-27T07:36:06.941 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:36:06.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:36:07.444 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:07.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:07 smithi012 ceph-mon[31473]: pgmap v470: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-05-27T07:36:07.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:07 smithi012 ceph-mon[31473]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:36:07.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:07 smithi162 ceph-mon[35676]: pgmap v470: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-05-27T07:36:07.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:07 smithi162 ceph-mon[35676]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:36:07.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:07 smithi062 ceph-mon[35304]: pgmap v470: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-05-27T07:36:07.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:07 smithi062 ceph-mon[35304]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:36:09.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:09 smithi012 ceph-mon[31473]: pgmap v472: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-05-27T07:36:09.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:09 smithi162 ceph-mon[35676]: pgmap v472: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-05-27T07:36:09.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:09 smithi062 ceph-mon[35304]: pgmap v472: 129 pgs: 129 active+clean; 250 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-05-27T07:36:10.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 -- 172.21.15.12:0/2411739664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 msgr2=0x7f1fe0110740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 --2- 172.21.15.12:0/2411739664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe0110740 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009990 tx=0x7f1fcc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 -- 172.21.15.12:0/2411739664 shutdown_connections 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 --2- 172.21.15.12:0/2411739664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fe0110c80 0x7f1fe01130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 --2- 172.21.15.12:0/2411739664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fe01030d0 0x7f1fe01034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 --2- 172.21.15.12:0/2411739664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe0110740 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 -- 172.21.15.12:0/2411739664 >> 172.21.15.12:0/2411739664 conn(0x7f1fe00fedb0 msgr2=0x7f1fe01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.801+0000 7f1fe6800700 1 -- 172.21.15.12:0/2411739664 shutdown_connections 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.802+0000 7f1fe6800700 1 -- 172.21.15.12:0/2411739664 wait complete. 2024-05-27T07:36:10.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.803+0000 7f1fe6800700 1 Processor -- start 2024-05-27T07:36:10.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.803+0000 7f1fe6800700 1 -- start start 2024-05-27T07:36:10.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fe6800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fe01030d0 0x7f1fe019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fe6800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe019d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fe6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fe0110c80 0x7f1fe01a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fe6800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe0116540 con 0x7f1fe0103a20 2024-05-27T07:36:10.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fe6800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe01163c0 con 0x7f1fe0110c80 2024-05-27T07:36:10.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fe6800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe01166c0 con 0x7f1fe01030d0 2024-05-27T07:36:10.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe019d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe019d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32992/0 (socket says 172.21.15.12:32992) 2024-05-27T07:36:10.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.804+0000 7f1fdea00700 1 -- 172.21.15.12:0/3340631231 learned_addr learned my addr 172.21.15.12:0/3340631231 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:10.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdf400700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fe01030d0 0x7f1fe019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fe0110c80 0x7f1fe01a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdea00700 1 -- 172.21.15.12:0/3340631231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fe01030d0 msgr2=0x7f1fe019cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdea00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fe01030d0 0x7f1fe019cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdea00700 1 -- 172.21.15.12:0/3340631231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fe0110c80 msgr2=0x7f1fe01a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdea00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fe0110c80 0x7f1fe01a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdea00700 1 -- 172.21.15.12:0/3340631231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe01a1e40 con 0x7f1fe0103a20 2024-05-27T07:36:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.805+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fe0110c80 0x7f1fe01a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:10.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.806+0000 7f1fdea00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe019d510 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc000c00 tx=0x7f1fcc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:10.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.806+0000 7f1fd7e00700 1 -- 172.21.15.12:0/3340631231 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc027070 con 0x7f1fe0103a20 2024-05-27T07:36:10.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.806+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe01a20d0 con 0x7f1fe0103a20 2024-05-27T07:36:10.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.806+0000 7f1fd7e00700 1 -- 172.21.15.12:0/3340631231 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fcc00fb80 con 0x7f1fe0103a20 2024-05-27T07:36:10.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.806+0000 7f1fd7e00700 1 -- 172.21.15.12:0/3340631231 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc022c00 con 0x7f1fe0103a20 2024-05-27T07:36:10.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.806+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe01a9a50 con 0x7f1fe0103a20 2024-05-27T07:36:10.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.811+0000 7f1fd7e00700 1 -- 172.21.15.12:0/3340631231 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1fcc02a070 con 0x7f1fe0103a20 2024-05-27T07:36:10.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.812+0000 7f1ff533db80 1 Processor -- start 2024-05-27T07:36:10.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.812+0000 7f1ff533db80 1 -- start start 2024-05-27T07:36:10.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.813+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a30398dec0 0x55a303b12870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.813+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b0c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.813+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.813+0000 7f1ff533db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a303a3aec0 con 0x55a303b0ce30 2024-05-27T07:36:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.813+0000 7f1ff533db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a303a2aa60 con 0x55a303b12db0 2024-05-27T07:36:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.813+0000 7f1ff533db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a303a2eaa0 con 0x55a30398dec0 2024-05-27T07:36:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.814+0000 7f1fdf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b0c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.814+0000 7f1fdf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b0c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:40320/0 (socket says 172.21.15.12:40320) 2024-05-27T07:36:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.814+0000 7f1fdf400700 1 -- 172.21.15.12:0/3935208965 learned_addr learned my addr 172.21.15.12:0/3935208965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.814+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a30398dec0 0x55a303b12870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.814+0000 7f1fdea00700 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.814+0000 7f1fd7e00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fc40b2a70 0x7f1fc40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fd7e00700 1 -- 172.21.15.12:0/3340631231 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 9642+0+0 (secure 0 0 0) 0x7f1fcc0e3950 con 0x7f1fe0103a20 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fc40b2a70 0x7f1fc40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdf400700 1 -- 172.21.15.12:0/3935208965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a30398dec0 msgr2=0x55a303b12870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdf400700 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a30398dec0 0x55a303b12870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdf400700 1 -- 172.21.15.12:0/3935208965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 msgr2=0x55a303b0d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdf400700 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdf400700 1 -- 172.21.15.12:0/3935208965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3039f34a0 con 0x55a303b12db0 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdea00700 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:10.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.815+0000 7f1fdf400700 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b0c8f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000bd70 tx=0x7f1fd000f640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b353ad7ba7ff869 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.816+0000 7f1fd4c00700 1 -- 172.21.15.12:0/3935208965 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0015070 con 0x55a303b12db0 2024-05-27T07:36:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.816+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fc40b2a70 0x7f1fc40b4f80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009550 tx=0x7f1fd8004b40 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.816+0000 7f1fd4c00700 1 -- 172.21.15.12:0/3935208965 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fd000fda0 con 0x55a303b12db0 2024-05-27T07:36:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.816+0000 7f1fd4c00700 1 -- 172.21.15.12:0/3935208965 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00116a0 con 0x55a303b12db0 2024-05-27T07:36:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.817+0000 7f1ff533db80 1 -- 172.21.15.12:0/3935208965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 msgr2=0x55a303b0c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.817+0000 7f1ff533db80 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b0c8f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000bd70 tx=0x7f1fd000f640 comp rx=0 tx=0).stop 2024-05-27T07:36:10.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.817+0000 7f1ff533db80 1 -- 172.21.15.12:0/3935208965 shutdown_connections 2024-05-27T07:36:10.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.817+0000 7f1ff533db80 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.817+0000 7f1ff533db80 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a30398dec0 0x55a303b12870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.817+0000 7f1ff533db80 1 --2- 172.21.15.12:0/3935208965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b0c8f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.818+0000 7f1ff533db80 1 -- 172.21.15.12:0/3935208965 >> 172.21.15.12:0/3935208965 conn(0x55a3039ee040 msgr2=0x55a303b07950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:10.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.818+0000 7f1ff533db80 1 -- 172.21.15.12:0/3935208965 shutdown_connections 2024-05-27T07:36:10.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.818+0000 7f1ff533db80 1 -- 172.21.15.12:0/3935208965 wait complete. 2024-05-27T07:36:10.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.819+0000 7f1ff533db80 1 -- start start 2024-05-27T07:36:10.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.819+0000 7f1ff533db80 1 Processor -- start 2024-05-27T07:36:10.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.820+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30398dec0 0x55a303b165e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.820+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b0ce30 0x55a303b16b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.820+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b17060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.820+0000 7f1ff533db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a303931020 con 0x55a30398dec0 2024-05-27T07:36:10.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.820+0000 7f1ff533db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a3039cb7e0 con 0x55a303b12db0 2024-05-27T07:36:10.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.820+0000 7f1ff533db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a303b19880 con 0x55a303b0ce30 2024-05-27T07:36:10.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.820+0000 7f1ff533db80 10 client.0 _ll_register_callbacks cb 0x55a3039d4cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:10.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.821+0000 7f1fdf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30398dec0 0x55a303b165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.821+0000 7f1fdf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30398dec0 0x55a303b165e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33012/0 (socket says 172.21.15.12:33012) 2024-05-27T07:36:10.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.821+0000 7f1fdf400700 1 -- 172.21.15.12:0/2990151060 learned_addr learned my addr 172.21.15.12:0/2990151060 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:10.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.821+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b17060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.821+0000 7f1fdea00700 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b0ce30 0x55a303b16b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.822+0000 7f1fdf400700 1 -- 172.21.15.12:0/2990151060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b0ce30 msgr2=0x55a303b16b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.822+0000 7f1fdf400700 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b0ce30 0x55a303b16b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.822+0000 7f1fdf400700 1 -- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 msgr2=0x55a303b17060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.822+0000 7f1fdf400700 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b17060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.822+0000 7f1fdf400700 1 -- 172.21.15.12:0/2990151060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a30397a950 con 0x55a30398dec0 2024-05-27T07:36:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1fdf400700 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30398dec0 0x55a303b165e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000ff90 tx=0x7f1fd001a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1282ef7a5a653d8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1fc9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1fc9600700 1 -- 172.21.15.12:0/2990151060 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0015070 con 0x55a30398dec0 2024-05-27T07:36:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1fc9600700 1 -- 172.21.15.12:0/2990151060 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9642+0+0 (secure 0 0 0) 0x7f1fd000fba0 con 0x55a30398dec0 2024-05-27T07:36:10.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fd000fda0 con 0x55a30398dec0 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1fc9600700 1 -- 172.21.15.12:0/2990151060 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f1fd002e050 con 0x55a30398dec0 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1ff533db80 10 client.17100 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.823+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a3039fb0e0 con 0x55a30398dec0 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1fc9600700 1 -- 172.21.15.12:0/2990151060 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fd000fda0 con 0x55a30398dec0 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1fc9600700 1 -- 172.21.15.12:0/2990151060 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f1fd0004ac0 con 0x55a30398dec0 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1ff533db80 10 client.17100 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1ff533db80 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1ff533db80 20 client.17100 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:10.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1ff533db80 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b26b30 0x55a303b28fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1ff533db80 4 client.17100 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:10.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.824+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a303a3ce10 con 0x55a303b26b30 2024-05-27T07:36:10.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.825+0000 7f1fdea00700 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b26b30 0x55a303b28fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.826+0000 7f1fdea00700 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b26b30 0x55a303b28fe0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e92b7f0ea89ae700 server_cookie=3845c28d76b9626c in_seq=0 out_seq=0 2024-05-27T07:36:10.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.826+0000 7f1fc9600700 10 client.17100 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.829+0000 7f1fc9600700 1 -- 172.21.15.12:0/2990151060 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a303a3ce10 con 0x55a303b26b30 2024-05-27T07:36:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.829+0000 7f1fc9600700 10 client.17100 handle_command_reply: tid=0 2024-05-27T07:36:10.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.829+0000 7f1ff533db80 1 client.17100 shutdown 2024-05-27T07:36:10.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30398dec0 msgr2=0x55a303b165e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30398dec0 0x55a303b165e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000ff90 tx=0x7f1fd001a690 comp rx=0 tx=0).stop 2024-05-27T07:36:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 shutdown_connections 2024-05-27T07:36:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b26b30 0x55a303b28fe0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b12db0 0x55a303b17060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a30398dec0 0x55a303b165e0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 --2- 172.21.15.12:0/2990151060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b0ce30 0x55a303b16b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 >> 172.21.15.12:0/2990151060 conn(0x55a3039ee040 msgr2=0x55a303b07600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.830+0000 7f1fc9600700 0 client.17100 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.831+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 shutdown_connections 2024-05-27T07:36:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.831+0000 7f1ff533db80 1 -- 172.21.15.12:0/2990151060 wait complete. 2024-05-27T07:36:10.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.831+0000 7f1ff533db80 20 client.17100 trim_cache size 0 max 16384 2024-05-27T07:36:10.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.860+0000 7f1ff533db80 1 Processor -- start 2024-05-27T07:36:10.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.860+0000 7f1ff533db80 1 -- start start 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.860+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.860+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.861+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b22980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.861+0000 7f1ff533db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a303b158e0 con 0x55a303b0ce30 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.861+0000 7f1ff533db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc000e530 con 0x55a303b07be0 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.861+0000 7f1ff533db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc000d790 con 0x55a303b12db0 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.861+0000 7f1fdf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.861+0000 7f1fdea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b141d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:50202/0 (socket says 172.21.15.12:50202) 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdea00700 1 -- 172.21.15.12:0/1222966595 learned_addr learned my addr 172.21.15.12:0/1222966595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b22980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdea00700 1 -- 172.21.15.12:0/1222966595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 msgr2=0x55a303b22980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdea00700 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b22980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdea00700 1 -- 172.21.15.12:0/1222966595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 msgr2=0x55a303b0c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdea00700 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.862+0000 7f1fdea00700 1 -- 172.21.15.12:0/1222966595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a303b14b50 con 0x55a303b12db0 2024-05-27T07:36:10.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.863+0000 7f1fdea00700 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b141d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0e1240 tx=0x7f1fcc0e5e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ce3d3ae2f5579f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.864+0000 7f1fcbe00700 1 -- 172.21.15.12:0/1222966595 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc0a9fd0 con 0x55a303b12db0 2024-05-27T07:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.864+0000 7f1fcbe00700 1 -- 172.21.15.12:0/1222966595 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fcc0aa370 con 0x55a303b12db0 2024-05-27T07:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.864+0000 7f1fcbe00700 1 -- 172.21.15.12:0/1222966595 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc0b3630 con 0x55a303b12db0 2024-05-27T07:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 -- 172.21.15.12:0/1222966595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 msgr2=0x55a303b141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b141d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0e1240 tx=0x7f1fcc0e5e60 comp rx=0 tx=0).stop 2024-05-27T07:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 -- 172.21.15.12:0/1222966595 shutdown_connections 2024-05-27T07:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b22980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b0c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1222966595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b141d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 -- 172.21.15.12:0/1222966595 >> 172.21.15.12:0/1222966595 conn(0x55a3039ee040 msgr2=0x55a303b16df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.865+0000 7f1ff533db80 1 -- 172.21.15.12:0/1222966595 shutdown_connections 2024-05-27T07:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.868+0000 7f1ff533db80 1 -- 172.21.15.12:0/1222966595 wait complete. 2024-05-27T07:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.868+0000 7f1ff533db80 1 -- start start 2024-05-27T07:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.869+0000 7f1ff533db80 1 Processor -- start 2024-05-27T07:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.869+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b1ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.869+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b1f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.869+0000 7f1ff533db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b1bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.869+0000 7f1ff533db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc000de60 con 0x55a303b0ce30 2024-05-27T07:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.869+0000 7f1ff533db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a303b08e80 con 0x55a303b07be0 2024-05-27T07:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.869+0000 7f1ff533db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a303b1c360 con 0x55a303b12db0 2024-05-27T07:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1ff533db80 10 client.0 _ll_register_callbacks cb 0x55a3039d4cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b1f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b1f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33016/0 (socket says 172.21.15.12:33016) 2024-05-27T07:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 -- 172.21.15.12:0/1432834371 learned_addr learned my addr 172.21.15.12:0/1432834371 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdf400700 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b1ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdfe00700 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b1bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 -- 172.21.15.12:0/1432834371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 msgr2=0x55a303b1bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b1bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 -- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 msgr2=0x55a303b1ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b1ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.870+0000 7f1fdea00700 1 -- 172.21.15.12:0/1432834371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a303b1c560 con 0x55a303b0ce30 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.871+0000 7f1fdea00700 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b1f3b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0a7ad0 tx=0x7f1fcc0aa270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29bf4df8f636b6f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.871+0000 7f1fc9600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.871+0000 7f1fc9600700 1 -- 172.21.15.12:0/1432834371 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc0aaf60 con 0x55a303b0ce30 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.871+0000 7f1fc9600700 1 -- 172.21.15.12:0/1432834371 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 9642+0+0 (secure 0 0 0) 0x7f1fcc0a6af0 con 0x55a303b0ce30 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.871+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fcc0aa370 con 0x55a303b0ce30 2024-05-27T07:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.872+0000 7f1fc9600700 1 -- 172.21.15.12:0/1432834371 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f1fcc0cd0f0 con 0x55a303b0ce30 2024-05-27T07:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.872+0000 7f1ff533db80 10 client.17106 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.872+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a303b9ab70 con 0x55a303b0ce30 2024-05-27T07:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.872+0000 7f1fc9600700 1 -- 172.21.15.12:0/1432834371 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fcc0aa370 con 0x55a303b0ce30 2024-05-27T07:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.872+0000 7f1fc9600700 1 -- 172.21.15.12:0/1432834371 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f1fcc0b31a0 con 0x55a303b0ce30 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.873+0000 7f1ff533db80 10 client.17106 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.873+0000 7f1ff533db80 10 client.17106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.873+0000 7f1ff533db80 20 client.17106 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.873+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b9aed0 0x55a303b9d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.873+0000 7f1ff533db80 4 client.17106 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.873+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a303a365f0 con 0x55a303b9aed0 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.873+0000 7f1fdf400700 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b9aed0 0x55a303b9d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.874+0000 7f1fdf400700 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b9aed0 0x55a303b9d2f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fed07a0656cae7a server_cookie=803c3a6ae34fb15d in_seq=0 out_seq=0 2024-05-27T07:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.874+0000 7f1fc9600700 10 client.17106 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1fc9600700 1 -- 172.21.15.12:0/1432834371 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a303a365f0 con 0x55a303b9aed0 2024-05-27T07:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1fc9600700 10 client.17106 handle_command_reply: tid=0 2024-05-27T07:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 client.17106 shutdown 2024-05-27T07:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 msgr2=0x55a303b1f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b1f3b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0a7ad0 tx=0x7f1fcc0aa270 comp rx=0 tx=0).stop 2024-05-27T07:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 shutdown_connections 2024-05-27T07:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a303b9aed0 0x55a303b9d2f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a303b07be0 0x55a303b1ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a303b12db0 0x55a303b1bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 --2- 172.21.15.12:0/1432834371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a303b0ce30 0x55a303b1f3b0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 >> 172.21.15.12:0/1432834371 conn(0x55a3039ee040 msgr2=0x55a303b163c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.878+0000 7f1fc9600700 0 client.17106 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:10.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.880+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 shutdown_connections 2024-05-27T07:36:10.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.880+0000 7f1ff533db80 1 -- 172.21.15.12:0/1432834371 wait complete. 2024-05-27T07:36:10.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.880+0000 7f1ff533db80 20 client.17106 trim_cache size 0 max 16384 2024-05-27T07:36:10.877 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:36:10.877 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:36:10.877 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "0acb88d9-6d78-4937-addb-17cca87dbb4b", 2024-05-27T07:36:10.877 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:36:10.877 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.881+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fc40b2a70 msgr2=0x7f1fc40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.881+0000 7f1fe6800700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fc40b2a70 0x7f1fc40b4f80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009550 tx=0x7f1fd8004b40 comp rx=0 tx=0).stop 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.881+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 msgr2=0x7f1fe019d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.881+0000 7f1fe6800700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe019d510 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc000c00 tx=0x7f1fcc005ef0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 shutdown_connections 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fc40b2a70 0x7f1fc40b4f80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fe0110c80 0x7f1fe01a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fe01030d0 0x7f1fe019cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 --2- 172.21.15.12:0/3340631231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fe0103a20 0x7f1fe019d510 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 >> 172.21.15.12:0/3340631231 conn(0x7f1fe00fedb0 msgr2=0x7f1fe0112880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 shutdown_connections 2024-05-27T07:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:10.882+0000 7f1fe6800700 1 -- 172.21.15.12:0/3340631231 wait complete. 2024-05-27T07:36:11.595 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:11 smithi012 ceph-mon[31473]: pgmap v473: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:36:11.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:11 smithi162 ceph-mon[35676]: pgmap v473: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:36:11.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:11 smithi062 ceph-mon[35304]: pgmap v473: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:36:11.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:36:12.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:12 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:36:12.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:12 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:12 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:36:12.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:12 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:36:12.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:12 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:36:12.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:12 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:12 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.411 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:12.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:12 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:36:12.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:12 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:12 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:36:12.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:12 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:36:12.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:12 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:36:12.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:12 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:12 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:12 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:36:12.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:12 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:12 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:36:12.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:12 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:36:12.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:12 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:36:12.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:12 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:12.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:12 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:13.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:13 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:13.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:13 smithi012 ceph-mon[31473]: pgmap v474: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:36:13.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:13 smithi012 ceph-mon[31473]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:36:13.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:13 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:13.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:13 smithi162 ceph-mon[35676]: pgmap v474: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:36:13.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:13 smithi162 ceph-mon[35676]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:36:13.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:13 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:13.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:13 smithi062 ceph-mon[35304]: pgmap v474: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-05-27T07:36:13.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:13 smithi062 ceph-mon[35304]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:36:15.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.087+0000 7fc8e2400700 1 -- 172.21.15.12:0/4059099747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 msgr2=0x7fc8dc110dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.087+0000 7fc8e2400700 1 --2- 172.21.15.12:0/4059099747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc110dd0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400b0a0 tx=0x7fc8d400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.087+0000 7fc8e2400700 1 -- 172.21.15.12:0/4059099747 shutdown_connections 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.087+0000 7fc8e2400700 1 --2- 172.21.15.12:0/4059099747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc105450 0x7fc8dc105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.087+0000 7fc8e2400700 1 --2- 172.21.15.12:0/4059099747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc110dd0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.087+0000 7fc8e2400700 1 --2- 172.21.15.12:0/4059099747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8dc105da0 0x7fc8dc10e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.087+0000 7fc8e2400700 1 -- 172.21.15.12:0/4059099747 >> 172.21.15.12:0/4059099747 conn(0x7fc8dc0fede0 msgr2=0x7fc8dc101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.088+0000 7fc8e2400700 1 -- 172.21.15.12:0/4059099747 shutdown_connections 2024-05-27T07:36:15.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.088+0000 7fc8e2400700 1 -- 172.21.15.12:0/4059099747 wait complete. 2024-05-27T07:36:15.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.089+0000 7fc8e2400700 1 Processor -- start 2024-05-27T07:36:15.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.089+0000 7fc8e2400700 1 -- start start 2024-05-27T07:36:15.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8e2400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc105450 0x7fc8dc103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8e2400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8dc105da0 0x7fc8dc103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8e2400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8e2400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc114320 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8e2400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc1141a0 con 0x7fc8dc105450 2024-05-27T07:36:15.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8e2400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc1144a0 con 0x7fc8dc105da0 2024-05-27T07:36:15.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8dbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc103bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45896/0 (socket says 172.21.15.12:45896) 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8dbe00700 1 -- 172.21.15.12:0/3907632177 learned_addr learned my addr 172.21.15.12:0/3907632177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.090+0000 7fc8db400700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc105450 0x7fc8dc103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.091+0000 7fc8daa00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8dc105da0 0x7fc8dc103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.091+0000 7fc8dbe00700 1 -- 172.21.15.12:0/3907632177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8dc105da0 msgr2=0x7fc8dc103680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.091+0000 7fc8dbe00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8dc105da0 0x7fc8dc103680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.091+0000 7fc8dbe00700 1 -- 172.21.15.12:0/3907632177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc105450 msgr2=0x7fc8dc103140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.091+0000 7fc8dbe00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc105450 0x7fc8dc103140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.091+0000 7fc8dbe00700 1 -- 172.21.15.12:0/3907632177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc1a9070 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.091+0000 7fc8daa00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8dc105da0 0x7fc8dc103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.092+0000 7fc8dbe00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc103bc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400b830 tx=0x7fc8d4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.092+0000 7fc8d3e00700 1 -- 172.21.15.12:0/3907632177 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4021070 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.092+0000 7fc8d3e00700 1 -- 172.21.15.12:0/3907632177 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8d4007b80 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.092+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8dc1a9300 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.092+0000 7fc8d3e00700 1 -- 172.21.15.12:0/3907632177 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d40052a0 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.092+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8dc1a97b0 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.098+0000 7fc8d3e00700 1 -- 172.21.15.12:0/3907632177 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc8d4024070 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.098+0000 7fc8f0ff1b80 1 Processor -- start 2024-05-27T07:36:15.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.100+0000 7fc8f0ff1b80 1 -- start start 2024-05-27T07:36:15.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.100+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6af250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.100+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.100+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6c4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.100+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56072f5dde40 con 0x56072f6afdb0 2024-05-27T07:36:15.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.100+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56072f5cd9e0 con 0x56072f61c9f0 2024-05-27T07:36:15.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.100+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56072f5d1a20 con 0x56072f530e40 2024-05-27T07:36:15.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.101+0000 7fc8d3e00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc8c00b2af0 0x7fc8c00b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.101+0000 7fc8d3e00700 1 -- 172.21.15.12:0/3907632177 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7fc8d400f020 con 0x7fc8dc10e8c0 2024-05-27T07:36:15.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.101+0000 7fc8dbe00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc8c00b2af0 0x7fc8c00b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.101+0000 7fc8db400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.102+0000 7fc8daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.102+0000 7fc8daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:36374/0 (socket says 172.21.15.12:36374) 2024-05-27T07:36:15.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.102+0000 7fc8daa00700 1 -- 172.21.15.12:0/1604076654 learned_addr learned my addr 172.21.15.12:0/1604076654 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:15.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.102+0000 7fc8daa00700 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8db400700 1 -- 172.21.15.12:0/1604076654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 msgr2=0x56072f6af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8db400700 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6af250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8db400700 1 -- 172.21.15.12:0/1604076654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 msgr2=0x56072f6c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8db400700 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8db400700 1 -- 172.21.15.12:0/1604076654 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56072f596420 con 0x56072f530e40 2024-05-27T07:36:15.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8daa00700 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:15.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8dbe00700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc8c00b2af0 0x7fc8c00b5000 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40aaf30 tx=0x7fc8d401df90 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:15.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.103+0000 7fc8db400700 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6af7b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc00bc60 tx=0x7fc8cc00bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdddcf7f834f9d03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:15.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.104+0000 7fc8d0c00700 1 -- 172.21.15.12:0/1604076654 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc010ca0 con 0x56072f530e40 2024-05-27T07:36:15.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.104+0000 7fc8d0c00700 1 -- 172.21.15.12:0/1604076654 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8cc010e40 con 0x56072f530e40 2024-05-27T07:36:15.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.104+0000 7fc8d0c00700 1 -- 172.21.15.12:0/1604076654 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc011440 con 0x56072f530e40 2024-05-27T07:36:15.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.107+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/1604076654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 msgr2=0x56072f6af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.107+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6af7b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc00bc60 tx=0x7fc8cc00bf70 comp rx=0 tx=0).stop 2024-05-27T07:36:15.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.107+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/1604076654 shutdown_connections 2024-05-27T07:36:15.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.107+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.107+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.107+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/1604076654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6af7b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.108+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/1604076654 >> 172.21.15.12:0/1604076654 conn(0x56072f5ca390 msgr2=0x56072f6ab4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:15.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.108+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/1604076654 shutdown_connections 2024-05-27T07:36:15.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.108+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/1604076654 wait complete. 2024-05-27T07:36:15.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.109+0000 7fc8f0ff1b80 1 -- start start 2024-05-27T07:36:15.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.109+0000 7fc8f0ff1b80 1 Processor -- start 2024-05-27T07:36:15.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.110+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6bf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.110+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.110+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f6afdb0 0x56072f6c0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.110+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56072f4d3fa0 con 0x56072f61c9f0 2024-05-27T07:36:15.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.110+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56072f56e760 con 0x56072f6afdb0 2024-05-27T07:36:15.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.110+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56072f6c53e0 con 0x56072f530e40 2024-05-27T07:36:15.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.110+0000 7fc8f0ff1b80 10 client.0 _ll_register_callbacks cb 0x56072f577c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6bfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45914/0 (socket says 172.21.15.12:45914) 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 -- 172.21.15.12:0/2990857008 learned_addr learned my addr 172.21.15.12:0/2990857008 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8daa00700 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6bf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8daa00700 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f6afdb0 0x56072f6c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 -- 172.21.15.12:0/2990857008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 msgr2=0x56072f6bf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6bf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 -- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f6afdb0 msgr2=0x56072f6c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f6afdb0 0x56072f6c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.111+0000 7fc8db400700 1 -- 172.21.15.12:0/2990857008 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56072f51d8d0 con 0x56072f61c9f0 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.112+0000 7fc8db400700 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6bfe00 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005910 tx=0x7fc8cc0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02f80285c52d92f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:15.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.112+0000 7fc8c5600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:15.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.112+0000 7fc8c5600700 1 -- 172.21.15.12:0/2990857008 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc005310 con 0x56072f61c9f0 2024-05-27T07:36:15.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.112+0000 7fc8c5600700 1 -- 172.21.15.12:0/2990857008 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7fc8cc0054b0 con 0x56072f61c9f0 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.112+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8cc010e40 con 0x56072f61c9f0 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8c5600700 1 -- 172.21.15.12:0/2990857008 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc8cc0056b0 con 0x56072f61c9f0 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8f0ff1b80 10 client.17118 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56072f59e060 con 0x56072f61c9f0 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8c5600700 1 -- 172.21.15.12:0/2990857008 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8cc010e40 con 0x56072f61c9f0 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8c5600700 1 -- 172.21.15.12:0/2990857008 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fc8cc011870 con 0x56072f61c9f0 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8f0ff1b80 10 client.17118 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8f0ff1b80 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8f0ff1b80 20 client.17118 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.113+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f6d2610 0x56072f6d4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.114+0000 7fc8f0ff1b80 4 client.17118 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:15.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.114+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56072f5dfd90 con 0x56072f6d2610 2024-05-27T07:36:15.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.115+0000 7fc8daa00700 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f6d2610 0x56072f6d4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.116+0000 7fc8daa00700 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f6d2610 0x56072f6d4a80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=259c60db64c4ba2e server_cookie=cfd69e5067105a73 in_seq=0 out_seq=0 2024-05-27T07:36:15.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.116+0000 7fc8c5600700 10 client.17118 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:15.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.118+0000 7fc8c5600700 1 -- 172.21.15.12:0/2990857008 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56072f5dfd90 con 0x56072f6d2610 2024-05-27T07:36:15.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.118+0000 7fc8c5600700 10 client.17118 handle_command_reply: tid=0 2024-05-27T07:36:15.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.118+0000 7fc8f0ff1b80 1 client.17118 shutdown 2024-05-27T07:36:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 msgr2=0x56072f6bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6bfe00 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005910 tx=0x7fc8cc0036f0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 shutdown_connections 2024-05-27T07:36:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f6d2610 0x56072f6d4a80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f6afdb0 0x56072f6c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6bf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/2990857008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6bfe00 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.120+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 >> 172.21.15.12:0/2990857008 conn(0x56072f5ca390 msgr2=0x56072f6ab010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:15.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.121+0000 7fc8c5600700 0 client.17118 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.121+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 shutdown_connections 2024-05-27T07:36:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.121+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/2990857008 wait complete. 2024-05-27T07:36:15.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.121+0000 7fc8f0ff1b80 20 client.17118 trim_cache size 0 max 16384 2024-05-27T07:36:15.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 Processor -- start 2024-05-27T07:36:15.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 -- start start 2024-05-27T07:36:15.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6d4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56072f6d4c60 con 0x56072f6afdb0 2024-05-27T07:36:15.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc1ad880 con 0x56072f61c9f0 2024-05-27T07:36:15.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.152+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc1acae0 con 0x56072f530e40 2024-05-27T07:36:15.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45922/0 (socket says 172.21.15.12:45922) 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8db400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6d4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 -- 172.21.15.12:0/104548935 learned_addr learned my addr 172.21.15.12:0/104548935 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 -- 172.21.15.12:0/104548935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 msgr2=0x56072f6ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 -- 172.21.15.12:0/104548935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 msgr2=0x56072f6d4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6d4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.153+0000 7fc8daa00700 1 -- 172.21.15.12:0/104548935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56072f6b01c0 con 0x56072f6afdb0 2024-05-27T07:36:15.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.154+0000 7fc8daa00700 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6ce4b0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8005950 tx=0x7fc8c8016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b781499234080545 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:15.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.154+0000 7fc8c7e00700 1 -- 172.21.15.12:0/104548935 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8028040 con 0x56072f6afdb0 2024-05-27T07:36:15.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.154+0000 7fc8c7e00700 1 -- 172.21.15.12:0/104548935 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8c8004830 con 0x56072f6afdb0 2024-05-27T07:36:15.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.154+0000 7fc8c7e00700 1 -- 172.21.15.12:0/104548935 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c80052f0 con 0x56072f6afdb0 2024-05-27T07:36:15.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.155+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/104548935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 msgr2=0x56072f6ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.155+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6ce4b0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8005950 tx=0x7fc8c8016040 comp rx=0 tx=0).stop 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.155+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/104548935 shutdown_connections 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.155+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f6afdb0 0x56072f6ce4b0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.155+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f530e40 0x56072f6ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.155+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/104548935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f61c9f0 0x56072f6d4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.155+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/104548935 >> 172.21.15.12:0/104548935 conn(0x56072f5ca390 msgr2=0x56072f6b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.156+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/104548935 shutdown_connections 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.156+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/104548935 wait complete. 2024-05-27T07:36:15.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.156+0000 7fc8f0ff1b80 1 -- start start 2024-05-27T07:36:15.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.156+0000 7fc8f0ff1b80 1 Processor -- start 2024-05-27T07:36:15.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f530e40 0x56072f6cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8f0ff1b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f6afdb0 0x56072f6c3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56072f6afaf0 con 0x56072f61c9f0 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc1ad1b0 con 0x56072f530e40 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8f0ff1b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56072f6bde30 con 0x56072f6afdb0 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8f0ff1b80 10 client.0 _ll_register_callbacks cb 0x56072f577c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.157+0000 7fc8db400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6c2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45936/0 (socket says 172.21.15.12:45936) 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8db400700 1 -- 172.21.15.12:0/4065086481 learned_addr learned my addr 172.21.15.12:0/4065086481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8daa00700 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f6afdb0 0x56072f6c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8db400700 1 -- 172.21.15.12:0/4065086481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f6afdb0 msgr2=0x56072f6c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8db400700 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f6afdb0 0x56072f6c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8db400700 1 -- 172.21.15.12:0/4065086481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f530e40 msgr2=0x56072f6cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8db400700 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f530e40 0x56072f6cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8db400700 1 -- 172.21.15.12:0/4065086481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56072f6a7240 con 0x56072f61c9f0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8daa00700 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f6afdb0 0x56072f6c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8db400700 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6c2ee0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc011c90 tx=0x7fc8cc005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae0ef0e9f5a8708 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.158+0000 7fc8c5600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.159+0000 7fc8c5600700 1 -- 172.21.15.12:0/4065086481 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc011650 con 0x56072f61c9f0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.159+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8c8004830 con 0x56072f61c9f0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.159+0000 7fc8c5600700 1 -- 172.21.15.12:0/4065086481 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7fc8cc021040 con 0x56072f61c9f0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.159+0000 7fc8c5600700 1 -- 172.21.15.12:0/4065086481 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc8cc042050 con 0x56072f61c9f0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.159+0000 7fc8f0ff1b80 10 client.17124 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.159+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56072f6c9220 con 0x56072f61c9f0 2024-05-27T07:36:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8c5600700 1 -- 172.21.15.12:0/4065086481 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8c8004830 con 0x56072f61c9f0 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8c5600700 1 -- 172.21.15.12:0/4065086481 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fc8cc004690 con 0x56072f61c9f0 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8f0ff1b80 10 client.17124 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8f0ff1b80 10 client.17124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8f0ff1b80 20 client.17124 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f73d160 0x56072f73f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8f0ff1b80 4 client.17124 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56072f5d9570 con 0x56072f73d160 2024-05-27T07:36:15.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.160+0000 7fc8daa00700 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f73d160 0x56072f73f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.161+0000 7fc8c5600700 10 client.17124 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.161+0000 7fc8daa00700 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f73d160 0x56072f73f580 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f464864c15e66fe server_cookie=7b775cb644f3f725 in_seq=0 out_seq=0 2024-05-27T07:36:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8c5600700 1 -- 172.21.15.12:0/4065086481 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56072f5d9570 con 0x56072f73d160 2024-05-27T07:36:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8c5600700 10 client.17124 handle_command_reply: tid=0 2024-05-27T07:36:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 client.17124 shutdown 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 msgr2=0x56072f6c2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6c2ee0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc011c90 tx=0x7fc8cc005640 comp rx=0 tx=0).stop 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 shutdown_connections 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56072f73d160 0x56072f73f580 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56072f530e40 0x56072f6cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56072f6afdb0 0x56072f6c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 --2- 172.21.15.12:0/4065086481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56072f61c9f0 0x56072f6c2ee0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8c5600700 0 client.17124 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.162+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 >> 172.21.15.12:0/4065086481 conn(0x56072f5ca390 msgr2=0x56072f6b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.163+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 shutdown_connections 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.163+0000 7fc8f0ff1b80 1 -- 172.21.15.12:0/4065086481 wait complete. 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.163+0000 7fc8f0ff1b80 20 client.17124 trim_cache size 0 max 16384 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stdout: "0acb88d9-6d78-4937-addb-17cca87dbb4b": { 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "0acb88d9-6d78-4937-addb-17cca87dbb4b", 2024-05-27T07:36:15.159 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc8c00b2af0 msgr2=0x7fc8c00b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc8c00b2af0 0x7fc8c00b5000 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40aaf30 tx=0x7fc8d401df90 comp rx=0 tx=0).stop 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 msgr2=0x7fc8dc103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc103bc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400b830 tx=0x7fc8d4004d00 comp rx=0 tx=0).stop 2024-05-27T07:36:15.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 shutdown_connections 2024-05-27T07:36:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc8c00b2af0 0x7fc8c00b5000 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc105450 0x7fc8dc103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8dc10e8c0 0x7fc8dc103bc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 --2- 172.21.15.12:0/3907632177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8dc105da0 0x7fc8dc103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 >> 172.21.15.12:0/3907632177 conn(0x7fc8dc0fede0 msgr2=0x7fc8dc100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 shutdown_connections 2024-05-27T07:36:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:15.164+0000 7fc8e2400700 1 -- 172.21.15.12:0/3907632177 wait complete. 2024-05-27T07:36:15.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:15 smithi012 ceph-mon[31473]: pgmap v476: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-05-27T07:36:15.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:15 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:15.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:15 smithi162 ceph-mon[35676]: pgmap v476: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-05-27T07:36:15.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:15 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:15.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:15 smithi062 ceph-mon[35304]: pgmap v476: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-05-27T07:36:15.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:15 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:16.089 INFO:tasks.cephfs.filesystem:scrub status for tag:0acb88d9-6d78-4937-addb-17cca87dbb4b - {'path': '/', 'tag': '0acb88d9-6d78-4937-addb-17cca87dbb4b', 'options': 'recursive,force'} 2024-05-27T07:36:16.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:16 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:16.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:16 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:16.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:16 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:36:16.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:16 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:16.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:16 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:16.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:16 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:36:16.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:16 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:16.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:16 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:16.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:16 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:36:17.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:17 smithi012 ceph-mon[31473]: pgmap v477: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-05-27T07:36:17.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:17 smithi012 ceph-mon[31473]: pgmap v478: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-05-27T07:36:17.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:17 smithi012 ceph-mon[31473]: pgmap v479: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:36:17.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:17 smithi162 ceph-mon[35676]: pgmap v477: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-05-27T07:36:17.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:17 smithi162 ceph-mon[35676]: pgmap v478: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-05-27T07:36:17.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:17 smithi162 ceph-mon[35676]: pgmap v479: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:36:17.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:17 smithi062 ceph-mon[35304]: pgmap v477: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-05-27T07:36:17.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:17 smithi062 ceph-mon[35304]: pgmap v478: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-05-27T07:36:17.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:17 smithi062 ceph-mon[35304]: pgmap v479: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:36:19.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:19 smithi012 ceph-mon[31473]: pgmap v480: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:36:19.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:19 smithi162 ceph-mon[35676]: pgmap v480: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:36:19.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:19 smithi062 ceph-mon[35304]: pgmap v480: 129 pgs: 129 active+clean; 246 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-05-27T07:36:21.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:21 smithi162 ceph-mon[35676]: pgmap v481: 129 pgs: 129 active+clean; 248 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 193 op/s 2024-05-27T07:36:21.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:21 smithi062 ceph-mon[35304]: pgmap v481: 129 pgs: 129 active+clean; 248 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 193 op/s 2024-05-27T07:36:21.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:21 smithi012 ceph-mon[31473]: pgmap v481: 129 pgs: 129 active+clean; 248 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 193 op/s 2024-05-27T07:36:23.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:23 smithi162 ceph-mon[35676]: pgmap v482: 129 pgs: 129 active+clean; 248 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 810 KiB/s wr, 92 op/s 2024-05-27T07:36:23.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:23 smithi062 ceph-mon[35304]: pgmap v482: 129 pgs: 129 active+clean; 248 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 810 KiB/s wr, 92 op/s 2024-05-27T07:36:23.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:23 smithi012 ceph-mon[31473]: pgmap v482: 129 pgs: 129 active+clean; 248 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 810 KiB/s wr, 92 op/s 2024-05-27T07:36:25.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:25 smithi162 ceph-mon[35676]: pgmap v483: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-05-27T07:36:25.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:25 smithi062 ceph-mon[35304]: pgmap v483: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-05-27T07:36:25.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:25 smithi012 ceph-mon[31473]: pgmap v483: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-05-27T07:36:26.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:36:26.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:36:26.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:36:26.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:36:26.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:36:26.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:36:27.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:27 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:36:27.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:27 smithi162 ceph-mon[35676]: pgmap v484: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:36:27.759 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:27 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:36:27.759 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:27 smithi062 ceph-mon[35304]: pgmap v484: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:36:27.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:27 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:36:27.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:27 smithi012 ceph-mon[31473]: pgmap v484: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:36:29.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:29 smithi162 ceph-mon[35676]: pgmap v485: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-05-27T07:36:29.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:29 smithi062 ceph-mon[35304]: pgmap v485: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-05-27T07:36:29.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:29 smithi012 ceph-mon[31473]: pgmap v485: 129 pgs: 129 active+clean; 254 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-05-27T07:36:31.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:31 smithi162 ceph-mon[35676]: pgmap v486: 129 pgs: 129 active+clean; 255 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 165 op/s 2024-05-27T07:36:31.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:31 smithi062 ceph-mon[35304]: pgmap v486: 129 pgs: 129 active+clean; 255 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 165 op/s 2024-05-27T07:36:31.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:31 smithi012 ceph-mon[31473]: pgmap v486: 129 pgs: 129 active+clean; 255 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 165 op/s 2024-05-27T07:36:33.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:33 smithi162 ceph-mon[35676]: pgmap v487: 129 pgs: 129 active+clean; 255 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 102 op/s 2024-05-27T07:36:33.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:33 smithi062 ceph-mon[35304]: pgmap v487: 129 pgs: 129 active+clean; 255 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 102 op/s 2024-05-27T07:36:33.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:33 smithi012 ceph-mon[31473]: pgmap v487: 129 pgs: 129 active+clean; 255 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 102 op/s 2024-05-27T07:36:35.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:35 smithi162 ceph-mon[35676]: pgmap v488: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-05-27T07:36:35.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:35 smithi062 ceph-mon[35304]: pgmap v488: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-05-27T07:36:35.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:35 smithi012 ceph-mon[31473]: pgmap v488: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-05-27T07:36:37.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:37 smithi162 ceph-mon[35676]: pgmap v489: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-27T07:36:37.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:37 smithi062 ceph-mon[35304]: pgmap v489: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-27T07:36:37.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:37 smithi012 ceph-mon[31473]: pgmap v489: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-27T07:36:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:39 smithi162 ceph-mon[35676]: pgmap v490: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-27T07:36:39.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:39 smithi062 ceph-mon[35304]: pgmap v490: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-27T07:36:39.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:39 smithi012 ceph-mon[31473]: pgmap v490: 129 pgs: 129 active+clean; 258 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-27T07:36:41.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:41 smithi162 ceph-mon[35676]: pgmap v491: 129 pgs: 129 active+clean; 256 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-05-27T07:36:41.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:41 smithi062 ceph-mon[35304]: pgmap v491: 129 pgs: 129 active+clean; 256 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-05-27T07:36:41.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:41 smithi012 ceph-mon[31473]: pgmap v491: 129 pgs: 129 active+clean; 256 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-05-27T07:36:43.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:43 smithi162 ceph-mon[35676]: pgmap v492: 129 pgs: 129 active+clean; 256 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 970 KiB/s wr, 101 op/s 2024-05-27T07:36:43.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:43 smithi062 ceph-mon[35304]: pgmap v492: 129 pgs: 129 active+clean; 256 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 970 KiB/s wr, 101 op/s 2024-05-27T07:36:43.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:43 smithi012 ceph-mon[31473]: pgmap v492: 129 pgs: 129 active+clean; 256 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 970 KiB/s wr, 101 op/s 2024-05-27T07:36:45.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:45 smithi162 ceph-mon[35676]: pgmap v493: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-05-27T07:36:45.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:45 smithi062 ceph-mon[35304]: pgmap v493: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-05-27T07:36:45.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:45 smithi012 ceph-mon[31473]: pgmap v493: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.4 MiB/s wr, 149 op/s 2024-05-27T07:36:46.090 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:36:46.584 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:47.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:47 smithi162 ceph-mon[35676]: pgmap v494: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 KiB/s wr, 97 op/s 2024-05-27T07:36:47.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:47 smithi062 ceph-mon[35304]: pgmap v494: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 KiB/s wr, 97 op/s 2024-05-27T07:36:47.788 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:47 smithi012 ceph-mon[31473]: pgmap v494: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 KiB/s wr, 97 op/s 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.030+0000 7f3839800700 1 -- 172.21.15.12:0/1843273511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834105e30 msgr2=0x7f383410e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.030+0000 7f3839800700 1 --2- 172.21.15.12:0/1843273511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834105e30 0x7f383410e380 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f381c009990 tx=0x7f381c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.031+0000 7f3839800700 1 -- 172.21.15.12:0/1843273511 shutdown_connections 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.031+0000 7f3839800700 1 --2- 172.21.15.12:0/1843273511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f383410e910 0x7f3834110df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.031+0000 7f3839800700 1 --2- 172.21.15.12:0/1843273511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38341054e0 0x7f38341058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.031+0000 7f3839800700 1 --2- 172.21.15.12:0/1843273511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834105e30 0x7f383410e380 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.031+0000 7f3839800700 1 -- 172.21.15.12:0/1843273511 >> 172.21.15.12:0/1843273511 conn(0x7f3834078df0 msgr2=0x7f3834079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.031+0000 7f3839800700 1 -- 172.21.15.12:0/1843273511 shutdown_connections 2024-05-27T07:36:49.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.031+0000 7f3839800700 1 -- 172.21.15.12:0/1843273511 wait complete. 2024-05-27T07:36:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.032+0000 7f3839800700 1 Processor -- start 2024-05-27T07:36:49.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.033+0000 7f3839800700 1 -- start start 2024-05-27T07:36:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.033+0000 7f3839800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38341054e0 0x7f383419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.033+0000 7f3839800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3834105e30 0x7f383419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.033+0000 7f3839800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f383410e910 0x7f38341a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.033+0000 7f3839800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3834114370 con 0x7f383410e910 2024-05-27T07:36:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.033+0000 7f3839800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f38341141f0 con 0x7f38341054e0 2024-05-27T07:36:49.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.033+0000 7f3839800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f38341144f0 con 0x7f3834105e30 2024-05-27T07:36:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f383410e910 0x7f38341a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f383410e910 0x7f38341a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33068/0 (socket says 172.21.15.12:33068) 2024-05-27T07:36:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 -- 172.21.15.12:0/3885300740 learned_addr learned my addr 172.21.15.12:0/3885300740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3832a00700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38341054e0 0x7f383419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f382be00700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3834105e30 0x7f383419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 -- 172.21.15.12:0/3885300740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3834105e30 msgr2=0x7f383419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3834105e30 0x7f383419d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 -- 172.21.15.12:0/3885300740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38341054e0 msgr2=0x7f383419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38341054e0 0x7f383419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3833400700 1 -- 172.21.15.12:0/3885300740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38341a1ed0 con 0x7f383410e910 2024-05-27T07:36:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.034+0000 7f3832a00700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38341054e0 0x7f383419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.035+0000 7f3833400700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f383410e910 0x7f38341a1830 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f382c007ee0 tx=0x7f382c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.035+0000 7f382b400700 1 -- 172.21.15.12:0/3885300740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c00dbd0 con 0x7f383410e910 2024-05-27T07:36:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.035+0000 7f382b400700 1 -- 172.21.15.12:0/3885300740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f382c00dd70 con 0x7f383410e910 2024-05-27T07:36:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.035+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38341a21c0 con 0x7f383410e910 2024-05-27T07:36:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.035+0000 7f382b400700 1 -- 172.21.15.12:0/3885300740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c0058d0 con 0x7f383410e910 2024-05-27T07:36:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.035+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38341a9ae0 con 0x7f383410e910 2024-05-27T07:36:49.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.040+0000 7f382b400700 1 -- 172.21.15.12:0/3885300740 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f382c01a020 con 0x7f383410e910 2024-05-27T07:36:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.042+0000 7f382b400700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f38180b29f0 0x7f38180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.042+0000 7f3848261b80 1 Processor -- start 2024-05-27T07:36:49.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.042+0000 7f382b400700 1 -- 172.21.15.12:0/3885300740 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7f382c0e22a0 con 0x7f383410e910 2024-05-27T07:36:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.042+0000 7f3848261b80 1 -- start start 2024-05-27T07:36:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.042+0000 7f3832a00700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f38180b29f0 0x7f38180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.043+0000 7f3848261b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11b3f9f0 0x55ce11bd2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.043+0000 7f3848261b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11a53e40 0x55ce11bd2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.043+0000 7f3848261b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bdf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.043+0000 7f3848261b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ce11b00e40 con 0x55ce11bd2e70 2024-05-27T07:36:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.043+0000 7f3848261b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ce11af09e0 con 0x55ce11b3f9f0 2024-05-27T07:36:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.043+0000 7f3848261b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ce11af4a20 con 0x55ce11a53e40 2024-05-27T07:36:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3832a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11b3f9f0 0x55ce11bd2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3832a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11b3f9f0 0x55ce11bd2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:57428/0 (socket says 172.21.15.12:57428) 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f382be00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11a53e40 0x55ce11bd2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3832a00700 1 -- 172.21.15.12:0/593922791 learned_addr learned my addr 172.21.15.12:0/593922791 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3833400700 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bdf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3832a00700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f38180b29f0 0x7f38180b4f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3824005950 tx=0x7f38240058e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3833400700 1 -- 172.21.15.12:0/593922791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11a53e40 msgr2=0x55ce11bd2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3833400700 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11a53e40 0x55ce11bd2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3833400700 1 -- 172.21.15.12:0/593922791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11b3f9f0 msgr2=0x55ce11bd2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3833400700 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11b3f9f0 0x55ce11bd2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.044+0000 7f3833400700 1 -- 172.21.15.12:0/593922791 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce11ab9420 con 0x55ce11bd2e70 2024-05-27T07:36:49.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.046+0000 7f3833400700 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bdf110 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f382c0df580 tx=0x7f382c0a8b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed3070d859eb2eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.046+0000 7f3823e00700 1 -- 172.21.15.12:0/593922791 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c0a9110 con 0x55ce11bd2e70 2024-05-27T07:36:49.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.046+0000 7f3823e00700 1 -- 172.21.15.12:0/593922791 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f382c0ad4c0 con 0x55ce11bd2e70 2024-05-27T07:36:49.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.046+0000 7f3823e00700 1 -- 172.21.15.12:0/593922791 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c095670 con 0x55ce11bd2e70 2024-05-27T07:36:49.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 -- 172.21.15.12:0/593922791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 msgr2=0x55ce11bdf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bdf110 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f382c0df580 tx=0x7f382c0a8b30 comp rx=0 tx=0).stop 2024-05-27T07:36:49.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 -- 172.21.15.12:0/593922791 shutdown_connections 2024-05-27T07:36:49.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11b3f9f0 0x55ce11bd2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bdf110 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 --2- 172.21.15.12:0/593922791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11a53e40 0x55ce11bd2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 -- 172.21.15.12:0/593922791 >> 172.21.15.12:0/593922791 conn(0x55ce11aed390 msgr2=0x55ce11bce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.047+0000 7f3848261b80 1 -- 172.21.15.12:0/593922791 shutdown_connections 2024-05-27T07:36:49.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.048+0000 7f3848261b80 1 -- 172.21.15.12:0/593922791 wait complete. 2024-05-27T07:36:49.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.048+0000 7f3848261b80 1 -- start start 2024-05-27T07:36:49.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.049+0000 7f3848261b80 1 Processor -- start 2024-05-27T07:36:49.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3848261b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11beb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3848261b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11beb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3848261b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11be56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3848261b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ce119f6fa0 con 0x55ce11bd2e70 2024-05-27T07:36:49.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3848261b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ce11a91760 con 0x55ce11a53e40 2024-05-27T07:36:49.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3848261b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ce11bdfb80 con 0x55ce11b3f9f0 2024-05-27T07:36:49.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3848261b80 10 client.0 _ll_register_callbacks cb 0x55ce11a9ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:49.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3832a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11beb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3832a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11beb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:57434/0 (socket says 172.21.15.12:57434) 2024-05-27T07:36:49.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f3832a00700 1 -- 172.21.15.12:0/1134748350 learned_addr learned my addr 172.21.15.12:0/1134748350 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:49.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.050+0000 7f382be00700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11beb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.051+0000 7f3833400700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11be56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.051+0000 7f3832a00700 1 -- 172.21.15.12:0/1134748350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 msgr2=0x55ce11beb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.051+0000 7f3832a00700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11beb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.051+0000 7f3832a00700 1 -- 172.21.15.12:0/1134748350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 msgr2=0x55ce11be56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.051+0000 7f3832a00700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11be56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.051+0000 7f3832a00700 1 -- 172.21.15.12:0/1134748350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce11a408d0 con 0x55ce11a53e40 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.051+0000 7f3833400700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11be56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.052+0000 7f3832a00700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11beb2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3824011470 tx=0x7f3824004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415c3ab516b5645 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.052+0000 7f3820c00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.053+0000 7f3820c00700 1 -- 172.21.15.12:0/1134748350 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824028670 con 0x55ce11a53e40 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.053+0000 7f3820c00700 1 -- 172.21.15.12:0/1134748350 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7f3824028810 con 0x55ce11a53e40 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.053+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f382c0ad4c0 con 0x55ce11a53e40 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.054+0000 7f3820c00700 1 -- 172.21.15.12:0/1134748350 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f3824028a10 con 0x55ce11a53e40 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.054+0000 7f3848261b80 10 client.36572 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:49.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.054+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55ce11ac1060 con 0x55ce11a53e40 2024-05-27T07:36:49.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3820c00700 1 -- 172.21.15.12:0/1134748350 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f382c0ad4c0 con 0x55ce11a53e40 2024-05-27T07:36:49.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3820c00700 1 -- 172.21.15.12:0/1134748350 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f38240103d0 con 0x55ce11a53e40 2024-05-27T07:36:49.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3848261b80 10 client.36572 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:49.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3848261b80 10 client.36572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:49.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3848261b80 20 client.36572 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:49.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3848261b80 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11bf5760 0x55ce11bf7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3848261b80 4 client.36572 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:49.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.055+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce11b02d90 con 0x55ce11bf5760 2024-05-27T07:36:49.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.056+0000 7f382be00700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11bf5760 0x55ce11bf7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.057+0000 7f382be00700 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11bf5760 0x55ce11bf7c50 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebdf29d7e0ef6485 server_cookie=e00395b596a8dee2 in_seq=0 out_seq=0 2024-05-27T07:36:49.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.057+0000 7f3820c00700 10 client.36572 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3820c00700 1 -- 172.21.15.12:0/1134748350 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ce11b02d90 con 0x55ce11bf5760 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3820c00700 10 client.36572 handle_command_reply: tid=0 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 client.36572 shutdown 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 msgr2=0x55ce11beb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11beb2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3824011470 tx=0x7f3824004750 comp rx=0 tx=0).stop 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 shutdown_connections 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11bf5760 0x55ce11bf7c50 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11beb2c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11be56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 --2- 172.21.15.12:0/1134748350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11beb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.059+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 >> 172.21.15.12:0/1134748350 conn(0x55ce11aed390 msgr2=0x55ce11bd2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.060+0000 7f3820c00700 0 client.36572 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:49.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.060+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 shutdown_connections 2024-05-27T07:36:49.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.060+0000 7f3848261b80 1 -- 172.21.15.12:0/1134748350 wait complete. 2024-05-27T07:36:49.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.060+0000 7f3848261b80 20 client.36572 trim_cache size 0 max 16384 2024-05-27T07:36:49.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.088+0000 7f3848261b80 1 Processor -- start 2024-05-27T07:36:49.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.088+0000 7f3848261b80 1 -- start start 2024-05-27T07:36:49.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.089+0000 7f3848261b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11bcdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.089+0000 7f3848261b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.089+0000 7f3848261b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bd20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.089+0000 7f3848261b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f38341adaa0 con 0x55ce11bd2e70 2024-05-27T07:36:49.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.089+0000 7f3848261b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f38341acd40 con 0x55ce11a53e40 2024-05-27T07:36:49.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.089+0000 7f3848261b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f38341ad3d0 con 0x55ce11b3f9f0 2024-05-27T07:36:49.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bd20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bd20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33084/0 (socket says 172.21.15.12:33084) 2024-05-27T07:36:49.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 -- 172.21.15.12:0/1742864301 learned_addr learned my addr 172.21.15.12:0/1742864301 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:49.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3832a00700 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11bcdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f382be00700 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 -- 172.21.15.12:0/1742864301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 msgr2=0x55ce11bce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 -- 172.21.15.12:0/1742864301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 msgr2=0x55ce11bcdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11bcdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.090+0000 7f3833400700 1 -- 172.21.15.12:0/1742864301 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce11bce9d0 con 0x55ce11bd2e70 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.091+0000 7f3833400700 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bd20b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f382c0a90a0 tx=0x7f382c005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af6bcc96df008d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.091+0000 7f3823400700 1 -- 172.21.15.12:0/1742864301 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c0b1f70 con 0x55ce11bd2e70 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.091+0000 7f3823400700 1 -- 172.21.15.12:0/1742864301 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f382c0a9110 con 0x55ce11bd2e70 2024-05-27T07:36:49.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.091+0000 7f3823400700 1 -- 172.21.15.12:0/1742864301 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c095670 con 0x55ce11bd2e70 2024-05-27T07:36:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.092+0000 7f3848261b80 1 -- 172.21.15.12:0/1742864301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 msgr2=0x55ce11bd20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.092+0000 7f3848261b80 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bd20b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f382c0a90a0 tx=0x7f382c005b90 comp rx=0 tx=0).stop 2024-05-27T07:36:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.092+0000 7f3848261b80 1 -- 172.21.15.12:0/1742864301 shutdown_connections 2024-05-27T07:36:49.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.092+0000 7f3848261b80 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11a53e40 0x55ce11bcdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.092+0000 7f3848261b80 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11bd2e70 0x55ce11bd20b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.092+0000 7f3848261b80 1 --2- 172.21.15.12:0/1742864301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.092+0000 7f3848261b80 1 -- 172.21.15.12:0/1742864301 >> 172.21.15.12:0/1742864301 conn(0x55ce11aed390 msgr2=0x55ce11bf7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.093+0000 7f3848261b80 1 -- 172.21.15.12:0/1742864301 shutdown_connections 2024-05-27T07:36:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.093+0000 7f3848261b80 1 -- 172.21.15.12:0/1742864301 wait complete. 2024-05-27T07:36:49.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.093+0000 7f3848261b80 1 -- start start 2024-05-27T07:36:49.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.094+0000 7f3848261b80 1 Processor -- start 2024-05-27T07:36:49.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.094+0000 7f3848261b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11a53e40 0x55ce11bf1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.094+0000 7f3848261b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bed000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.094+0000 7f3848261b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11bd2e70 0x55ce11bed540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.094+0000 7f3848261b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ce11be3c30 con 0x55ce11a53e40 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.094+0000 7f3848261b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ce11be67d0 con 0x55ce11bd2e70 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.094+0000 7f3848261b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ce11beae90 con 0x55ce11b3f9f0 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.095+0000 7f3848261b80 10 client.0 _ll_register_callbacks cb 0x55ce11a9ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.095+0000 7f3832a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11a53e40 0x55ce11bf1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.095+0000 7f3832a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11a53e40 0x55ce11bf1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33092/0 (socket says 172.21.15.12:33092) 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.095+0000 7f3832a00700 1 -- 172.21.15.12:0/493405090 learned_addr learned my addr 172.21.15.12:0/493405090 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.095+0000 7f382be00700 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.095+0000 7f3833400700 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11bd2e70 0x55ce11bed540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.095+0000 7f3832a00700 1 -- 172.21.15.12:0/493405090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 msgr2=0x55ce11bed000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3832a00700 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bed000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3832a00700 1 -- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11bd2e70 msgr2=0x55ce11bed540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3832a00700 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11bd2e70 0x55ce11bed540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3832a00700 1 -- 172.21.15.12:0/493405090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce11bedcc0 con 0x55ce11a53e40 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3832a00700 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11a53e40 0x55ce11bf1ca0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3824026690 tx=0x7f382400b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee80f506eee7660 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3820c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3820c00700 1 -- 172.21.15.12:0/493405090 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824028470 con 0x55ce11a53e40 2024-05-27T07:36:49.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3820c00700 1 -- 172.21.15.12:0/493405090 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7f3824028610 con 0x55ce11a53e40 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.096+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f382c0a9110 con 0x55ce11a53e40 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3820c00700 1 -- 172.21.15.12:0/493405090 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f3824028810 con 0x55ce11a53e40 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 10 client.17139 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55ce11c601e0 con 0x55ce11a53e40 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3820c00700 1 -- 172.21.15.12:0/493405090 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f382c0a9110 con 0x55ce11a53e40 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3820c00700 1 -- 172.21.15.12:0/493405090 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f3824026c70 con 0x55ce11a53e40 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 10 client.17139 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 10 client.17139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 20 client.17139 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11c60820 0x55ce11c62c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 4 client.17139 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:36:49.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.097+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ce11afc570 con 0x55ce11c60820 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.098+0000 7f382be00700 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11c60820 0x55ce11c62c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.099+0000 7f382be00700 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11c60820 0x55ce11c62c40 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fdba5b7fb4c67b3 server_cookie=dd94b36a24ced010 in_seq=0 out_seq=0 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.099+0000 7f3820c00700 10 client.17139 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.099+0000 7f3820c00700 1 -- 172.21.15.12:0/493405090 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ce11afc570 con 0x55ce11c60820 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.099+0000 7f3820c00700 10 client.17139 handle_command_reply: tid=0 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 client.17139 shutdown 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11a53e40 msgr2=0x55ce11bf1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11a53e40 0x55ce11bf1ca0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3824026690 tx=0x7f382400b450 comp rx=0 tx=0).stop 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 shutdown_connections 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ce11c60820 0x55ce11c62c40 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce11bd2e70 0x55ce11bed540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ce11a53e40 0x55ce11bf1ca0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 --2- 172.21.15.12:0/493405090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce11b3f9f0 0x55ce11bed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3820c00700 0 client.17139 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 >> 172.21.15.12:0/493405090 conn(0x55ce11aed390 msgr2=0x55ce11beb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 shutdown_connections 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 1 -- 172.21.15.12:0/493405090 wait complete. 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.100+0000 7f3848261b80 20 client.17139 trim_cache size 0 max 16384 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:36:49.098 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:36:49.099 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:36:49.099 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:36:49.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f38180b29f0 msgr2=0x7f38180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f38180b29f0 0x7f38180b4f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3824005950 tx=0x7f38240058e0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f383410e910 msgr2=0x7f38341a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f383410e910 0x7f38341a1830 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f382c007ee0 tx=0x7f382c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 shutdown_connections 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f38180b29f0 0x7f38180b4f00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38341054e0 0x7f383419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f383410e910 0x7f38341a1830 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 --2- 172.21.15.12:0/3885300740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3834105e30 0x7f383419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.101+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 >> 172.21.15.12:0/3885300740 conn(0x7f3834078df0 msgr2=0x7f3834103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.102+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 shutdown_connections 2024-05-27T07:36:49.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:49.102+0000 7f3839800700 1 -- 172.21.15.12:0/3885300740 wait complete. 2024-05-27T07:36:49.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:49 smithi062 ceph-mon[35304]: pgmap v495: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 KiB/s wr, 97 op/s 2024-05-27T07:36:49.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:49 smithi012 ceph-mon[31473]: pgmap v495: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 KiB/s wr, 97 op/s 2024-05-27T07:36:49.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:49 smithi162 ceph-mon[35676]: pgmap v495: 129 pgs: 129 active+clean; 258 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 KiB/s wr, 97 op/s 2024-05-27T07:36:50.107 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:36:50.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:36:50.643 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:51.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:51 smithi012 ceph-mon[31473]: pgmap v496: 129 pgs: 129 active+clean; 262 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-05-27T07:36:51.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:51 smithi062 ceph-mon[35304]: pgmap v496: 129 pgs: 129 active+clean; 262 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-05-27T07:36:51.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:51 smithi162 ceph-mon[35676]: pgmap v496: 129 pgs: 129 active+clean; 262 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 148 op/s 2024-05-27T07:36:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.374+0000 7f36cc800700 1 -- 172.21.15.12:0/1709893394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 msgr2=0x7f36c4106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.374+0000 7f36cc800700 1 --2- 172.21.15.12:0/1709893394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c4106970 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009990 tx=0x7f36b4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.375+0000 7f36cc800700 1 -- 172.21.15.12:0/1709893394 shutdown_connections 2024-05-27T07:36:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.375+0000 7f36cc800700 1 --2- 172.21.15.12:0/1709893394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c4106eb0 0x7f36c4113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.375+0000 7f36cc800700 1 --2- 172.21.15.12:0/1709893394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c410c5a0 0x7f36c410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.375+0000 7f36cc800700 1 --2- 172.21.15.12:0/1709893394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c4106970 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.375+0000 7f36cc800700 1 -- 172.21.15.12:0/1709893394 >> 172.21.15.12:0/1709893394 conn(0x7f36c4102040 msgr2=0x7f36c4104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.375+0000 7f36cc800700 1 -- 172.21.15.12:0/1709893394 shutdown_connections 2024-05-27T07:36:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.375+0000 7f36cc800700 1 -- 172.21.15.12:0/1709893394 wait complete. 2024-05-27T07:36:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.376+0000 7f36cc800700 1 Processor -- start 2024-05-27T07:36:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.376+0000 7f36cc800700 1 -- start start 2024-05-27T07:36:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36cc800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36cc800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c4106eb0 0x7f36c419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36cc800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c410c5a0 0x7f36c41a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36cc800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f36c4116580 con 0x7f36c4106560 2024-05-27T07:36:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36cc800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f36c4116400 con 0x7f36c4106eb0 2024-05-27T07:36:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36cc800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36c4116700 con 0x7f36c410c5a0 2024-05-27T07:36:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36c9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36c9e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c419cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41146/0 (socket says 172.21.15.12:41146) 2024-05-27T07:36:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36c9e00700 1 -- 172.21.15.12:0/3443042587 learned_addr learned my addr 172.21.15.12:0/3443042587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.377+0000 7f36c9400700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c4106eb0 0x7f36c419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36ca800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c410c5a0 0x7f36c41a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36c9e00700 1 -- 172.21.15.12:0/3443042587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c410c5a0 msgr2=0x7f36c41a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36c9e00700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c410c5a0 0x7f36c41a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36c9e00700 1 -- 172.21.15.12:0/3443042587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c4106eb0 msgr2=0x7f36c419d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36c9e00700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c4106eb0 0x7f36c419d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36c9e00700 1 -- 172.21.15.12:0/3443042587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c41a1e00 con 0x7f36c4106560 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36ca800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c410c5a0 0x7f36c41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.378+0000 7f36c9e00700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c419cf30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f36b800d980 tx=0x7f36b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.379+0000 7f36bea00700 1 -- 172.21.15.12:0/3443042587 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b8017070 con 0x7f36c4106560 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.379+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c41a20f0 con 0x7f36c4106560 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.379+0000 7f36bea00700 1 -- 172.21.15.12:0/3443042587 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36b8009dc0 con 0x7f36c4106560 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.379+0000 7f36bea00700 1 -- 172.21.15.12:0/3443042587 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b800f3c0 con 0x7f36c4106560 2024-05-27T07:36:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.379+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c41a9a10 con 0x7f36c4106560 2024-05-27T07:36:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.380+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36c410e960 con 0x7f36c4106560 2024-05-27T07:36:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.383+0000 7f36bea00700 1 -- 172.21.15.12:0/3443042587 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f36b8009930 con 0x7f36c4106560 2024-05-27T07:36:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.384+0000 7f36bea00700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f36ac0b2a70 0x7f36ac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.385+0000 7f36bea00700 1 -- 172.21.15.12:0/3443042587 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7f36b80e24d0 con 0x7f36c4106560 2024-05-27T07:36:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.385+0000 7f36c9400700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f36ac0b2a70 0x7f36ac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.385+0000 7f36c9400700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f36ac0b2a70 0x7f36ac0b4f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36b400b5c0 tx=0x7f36b4005fd0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.391+0000 7f36bea00700 1 -- 172.21.15.12:0/3443042587 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f36b80a9a40 con 0x7f36c4106560 2024-05-27T07:36:53.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.744+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36c40512b0 con 0x7f36c4106560 2024-05-27T07:36:53.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:53 smithi012 ceph-mon[31473]: pgmap v497: 129 pgs: 129 active+clean; 262 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 933 KiB/s wr, 99 op/s 2024-05-27T07:36:53.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.746+0000 7f36bea00700 1 -- 172.21.15.12:0/3443042587 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f36b80a9340 con 0x7f36c4106560 2024-05-27T07:36:53.743 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:36:53.743 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:36:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.750+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f36ac0b2a70 msgr2=0x7f36ac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.750+0000 7f36cc800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f36ac0b2a70 0x7f36ac0b4f80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36b400b5c0 tx=0x7f36b4005fd0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.750+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 msgr2=0x7f36c419cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.750+0000 7f36cc800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c419cf30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f36b800d980 tx=0x7f36b800dc90 comp rx=0 tx=0).stop 2024-05-27T07:36:53.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.752+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 shutdown_connections 2024-05-27T07:36:53.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.752+0000 7f36cc800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f36ac0b2a70 0x7f36ac0b4f80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.752+0000 7f36cc800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c410c5a0 0x7f36c41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.752+0000 7f36cc800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36c4106560 0x7f36c419cf30 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.752+0000 7f36cc800700 1 --2- 172.21.15.12:0/3443042587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c4106eb0 0x7f36c419d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.752+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 >> 172.21.15.12:0/3443042587 conn(0x7f36c4102040 msgr2=0x7f36c41038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:53.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.753+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 shutdown_connections 2024-05-27T07:36:53.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:53.753+0000 7f36cc800700 1 -- 172.21.15.12:0/3443042587 wait complete. 2024-05-27T07:36:53.751 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:36:53.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:53 smithi162 ceph-mon[35676]: pgmap v497: 129 pgs: 129 active+clean; 262 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 933 KiB/s wr, 99 op/s 2024-05-27T07:36:53.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:53 smithi062 ceph-mon[35304]: pgmap v497: 129 pgs: 129 active+clean; 262 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 933 KiB/s wr, 99 op/s 2024-05-27T07:36:54.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:36:54.756 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:54 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3443042587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:54.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:54 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3443042587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:54.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:54 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3443042587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:55.176 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:55.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:55 smithi012 ceph-mon[31473]: pgmap v498: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-05-27T07:36:55.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:55 smithi162 ceph-mon[35676]: pgmap v498: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-05-27T07:36:55.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:55 smithi062 ceph-mon[35304]: pgmap v498: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.749+0000 7ff183e00700 1 -- 172.21.15.12:0/40465474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105280 msgr2=0x7ff184105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.749+0000 7ff183e00700 1 --2- 172.21.15.12:0/40465474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105280 0x7ff184105690 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff174009990 tx=0x7ff174009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.750+0000 7ff183e00700 1 -- 172.21.15.12:0/40465474 shutdown_connections 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.750+0000 7ff183e00700 1 --2- 172.21.15.12:0/40465474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18410e6f0 0x7ff184110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.750+0000 7ff183e00700 1 --2- 172.21.15.12:0/40465474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105280 0x7ff184105690 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.750+0000 7ff183e00700 1 --2- 172.21.15.12:0/40465474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff184105bd0 0x7ff18410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.750+0000 7ff183e00700 1 -- 172.21.15.12:0/40465474 >> 172.21.15.12:0/40465474 conn(0x7ff184078af0 msgr2=0x7ff184078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.750+0000 7ff183e00700 1 -- 172.21.15.12:0/40465474 shutdown_connections 2024-05-27T07:36:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.750+0000 7ff183e00700 1 -- 172.21.15.12:0/40465474 wait complete. 2024-05-27T07:36:57.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.751+0000 7ff183e00700 1 Processor -- start 2024-05-27T07:36:57.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.751+0000 7ff183e00700 1 -- start start 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff183e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff184105280 0x7ff18419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff183e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105bd0 0x7ff18419d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff183e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18410e6f0 0x7ff1841a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff183e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff184114190 con 0x7ff184105bd0 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff183e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff184114010 con 0x7ff18410e6f0 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff183e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff184114310 con 0x7ff184105280 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff17a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105bd0 0x7ff18419d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff17a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105bd0 0x7ff18419d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41164/0 (socket says 172.21.15.12:41164) 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff17a000700 1 -- 172.21.15.12:0/4216565941 learned_addr learned my addr 172.21.15.12:0/4216565941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.752+0000 7ff183400700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18410e6f0 0x7ff1841a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff182a00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff184105280 0x7ff18419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff17a000700 1 -- 172.21.15.12:0/4216565941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff184105280 msgr2=0x7ff18419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff17a000700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff184105280 0x7ff18419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff17a000700 1 -- 172.21.15.12:0/4216565941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18410e6f0 msgr2=0x7ff1841a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff17a000700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18410e6f0 0x7ff1841a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff17a000700 1 -- 172.21.15.12:0/4216565941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1841a1bf0 con 0x7ff184105bd0 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff182a00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff184105280 0x7ff18419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.753+0000 7ff17a000700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105bd0 0x7ff18419d2c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00d950 tx=0x7ff16c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.754+0000 7ff17be00700 1 -- 172.21.15.12:0/4216565941 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c016070 con 0x7ff184105bd0 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.754+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1841a1ee0 con 0x7ff184105bd0 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.754+0000 7ff17be00700 1 -- 172.21.15.12:0/4216565941 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff16c009dd0 con 0x7ff184105bd0 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.754+0000 7ff17be00700 1 -- 172.21.15.12:0/4216565941 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c020060 con 0x7ff184105bd0 2024-05-27T07:36:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.754+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1841a9800 con 0x7ff184105bd0 2024-05-27T07:36:57.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.759+0000 7ff1986d3b80 1 Processor -- start 2024-05-27T07:36:57.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.759+0000 7ff1986d3b80 1 -- start start 2024-05-27T07:36:57.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.760+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.760+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6410 0x555690900ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.760+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6820 0x555690903020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.760+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555690824e40 con 0x555690777e40 2024-05-27T07:36:57.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.760+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5556908149e0 con 0x5556908f6410 2024-05-27T07:36:57.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.760+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555690818a20 con 0x5556908f6820 2024-05-27T07:36:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.761+0000 7ff183400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.761+0000 7ff183400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908f5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41178/0 (socket says 172.21.15.12:41178) 2024-05-27T07:36:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.761+0000 7ff183400700 1 -- 172.21.15.12:0/2555499435 learned_addr learned my addr 172.21.15.12:0/2555499435 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.761+0000 7ff17be00700 1 -- 172.21.15.12:0/4216565941 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff16c019070 con 0x7ff184105bd0 2024-05-27T07:36:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.761+0000 7ff182a00700 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6410 0x555690900ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.762+0000 7ff183400700 1 -- 172.21.15.12:0/2555499435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6820 msgr2=0x555690903020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.762+0000 7ff183400700 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6820 0x555690903020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.762+0000 7ff183400700 1 -- 172.21.15.12:0/2555499435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6410 msgr2=0x555690900ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.762+0000 7ff183400700 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6410 0x555690900ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.762+0000 7ff183400700 1 -- 172.21.15.12:0/2555499435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556907dd420 con 0x555690777e40 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.762+0000 7ff183400700 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908f5ed0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff17c003fa0 tx=0x7ff17c009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bb6adfafe1f72 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.762+0000 7ff173e00700 1 -- 172.21.15.12:0/2555499435 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c004bb0 con 0x555690777e40 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.763+0000 7ff173e00700 1 -- 172.21.15.12:0/2555499435 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff17c004d50 con 0x555690777e40 2024-05-27T07:36:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.763+0000 7ff173e00700 1 -- 172.21.15.12:0/2555499435 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c007d90 con 0x555690777e40 2024-05-27T07:36:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff17be00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff1680b2a20 0x7ff1680b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2555499435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 msgr2=0x5556908f5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908f5ed0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff17c003fa0 tx=0x7ff17c009ba0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff17be00700 1 -- 172.21.15.12:0/4216565941 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7ff16c0e29f0 con 0x7ff184105bd0 2024-05-27T07:36:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2555499435 shutdown_connections 2024-05-27T07:36:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6820 0x555690903020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908f5ed0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2555499435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6410 0x555690900ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.764+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2555499435 >> 172.21.15.12:0/2555499435 conn(0x5556907d7fc0 msgr2=0x5556908f1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.765+0000 7ff182a00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff1680b2a20 0x7ff1680b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.765+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2555499435 shutdown_connections 2024-05-27T07:36:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.765+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2555499435 wait complete. 2024-05-27T07:36:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.766+0000 7ff1986d3b80 1 -- start start 2024-05-27T07:36:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.767+0000 7ff1986d3b80 1 Processor -- start 2024-05-27T07:36:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.767+0000 7ff182a00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff1680b2a20 0x7ff1680b4f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5556908f7430 tx=0x7ff174009380 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555690777e40 0x555690900570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690900ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556908f6820 0x555690900ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55569071afa0 con 0x5556908f6820 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5556907b5760 con 0x555690777e40 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555690903990 con 0x5556908f6410 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff1986d3b80 10 client.0 _ll_register_callbacks cb 0x5556907bec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff17a000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690900ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff17a000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690900ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:47310/0 (socket says 172.21.15.12:47310) 2024-05-27T07:36:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.768+0000 7ff17a000700 1 -- 172.21.15.12:0/1375982600 learned_addr learned my addr 172.21.15.12:0/1375982600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff183400700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556908f6820 0x555690900ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff182a00700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555690777e40 0x555690900570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff17a000700 1 -- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555690777e40 msgr2=0x555690900570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff17a000700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555690777e40 0x555690900570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff17a000700 1 -- 172.21.15.12:0/1375982600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556908f6820 msgr2=0x555690900ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff17a000700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556908f6820 0x555690900ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff17a000700 1 -- 172.21.15.12:0/1375982600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556907648d0 con 0x5556908f6410 2024-05-27T07:36:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.769+0000 7ff183400700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556908f6820 0x555690900ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.770+0000 7ff17a000700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690900ab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff16c000c00 tx=0x7ff16c0b2110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f8e66899dbd1e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.770+0000 7ff170c00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:36:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.770+0000 7ff170c00700 1 -- 172.21.15.12:0/1375982600 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0a98c0 con 0x5556908f6410 2024-05-27T07:36:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.770+0000 7ff170c00700 1 -- 172.21.15.12:0/1375982600 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7ff16c0a5b50 con 0x5556908f6410 2024-05-27T07:36:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.771+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff17c004d50 con 0x5556908f6410 2024-05-27T07:36:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.771+0000 7ff170c00700 1 -- 172.21.15.12:0/1375982600 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7ff16c0adc70 con 0x5556908f6410 2024-05-27T07:36:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.771+0000 7ff1986d3b80 10 client.26614 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.771+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5556907e5060 con 0x5556908f6410 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff170c00700 1 -- 172.21.15.12:0/1375982600 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff17c004d50 con 0x5556908f6410 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff170c00700 1 -- 172.21.15.12:0/1375982600 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7ff16c0b2290 con 0x5556908f6410 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff1986d3b80 10 client.26614 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff1986d3b80 10 client.26614 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff1986d3b80 20 client.26614 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x55569091b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff1986d3b80 4 client.26614 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.772+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555690826d90 con 0x555690919530 2024-05-27T07:36:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.773+0000 7ff182a00700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x55569091b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.774+0000 7ff182a00700 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x55569091b980 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a29dfe3fdadc6fd server_cookie=bb29a457692514f7 in_seq=0 out_seq=0 2024-05-27T07:36:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.774+0000 7ff170c00700 10 client.26614 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:36:57.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.776+0000 7ff170c00700 1 -- 172.21.15.12:0/1375982600 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555690826d90 con 0x555690919530 2024-05-27T07:36:57.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.776+0000 7ff170c00700 10 client.26614 handle_command_reply: tid=0 2024-05-27T07:36:57.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.776+0000 7ff1986d3b80 1 client.26614 shutdown 2024-05-27T07:36:57.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.776+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 msgr2=0x555690900ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.776+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690900ab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff16c000c00 tx=0x7ff16c0b2110 comp rx=0 tx=0).stop 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 shutdown_connections 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x55569091b980 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555690777e40 0x555690900570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff170c00700 0 client.26614 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556908f6820 0x555690900ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/1375982600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690900ab0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 >> 172.21.15.12:0/1375982600 conn(0x5556907d7fc0 msgr2=0x5556908f1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 shutdown_connections 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 1 -- 172.21.15.12:0/1375982600 wait complete. 2024-05-27T07:36:57.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.777+0000 7ff1986d3b80 20 client.26614 trim_cache size 0 max 16384 2024-05-27T07:36:57.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.795+0000 7ff1986d3b80 1 Processor -- start 2024-05-27T07:36:57.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.795+0000 7ff1986d3b80 1 -- start start 2024-05-27T07:36:57.792 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:57 smithi012 ceph-mon[31473]: pgmap v499: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1007 KiB/s wr, 108 op/s 2024-05-27T07:36:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.796+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908fbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.796+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690902a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.796+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 0x555690900a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.796+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff1841ad800 con 0x555690777e40 2024-05-27T07:36:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.796+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff1841aca60 con 0x5556908f6820 2024-05-27T07:36:57.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.796+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff1841ad130 con 0x5556908f6410 2024-05-27T07:36:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.796+0000 7ff182a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908fbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff182a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908fbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41192/0 (socket says 172.21.15.12:41192) 2024-05-27T07:36:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff182a00700 1 -- 172.21.15.12:0/4146602311 learned_addr learned my addr 172.21.15.12:0/4146602311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff17a000700 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690902a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff183400700 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 0x555690900a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff182a00700 1 -- 172.21.15.12:0/4146602311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 msgr2=0x555690902a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff182a00700 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690902a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff182a00700 1 -- 172.21.15.12:0/4146602311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 msgr2=0x555690900a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff182a00700 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 0x555690900a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.797+0000 7ff182a00700 1 -- 172.21.15.12:0/4146602311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556908feb00 con 0x555690777e40 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.798+0000 7ff182a00700 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908fbe50 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff174009750 tx=0x7ff17401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785eb493c1c9b27f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.798+0000 7ff173400700 1 -- 172.21.15.12:0/4146602311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174003da0 con 0x555690777e40 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.798+0000 7ff173400700 1 -- 172.21.15.12:0/4146602311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff174005250 con 0x555690777e40 2024-05-27T07:36:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.798+0000 7ff173400700 1 -- 172.21.15.12:0/4146602311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1740319f0 con 0x555690777e40 2024-05-27T07:36:57.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 -- 172.21.15.12:0/4146602311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 msgr2=0x5556908fbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908fbe50 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff174009750 tx=0x7ff17401f000 comp rx=0 tx=0).stop 2024-05-27T07:36:57.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 -- 172.21.15.12:0/4146602311 shutdown_connections 2024-05-27T07:36:57.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 0x555690900a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x5556908fbe50 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/4146602311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690902a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 -- 172.21.15.12:0/4146602311 >> 172.21.15.12:0/4146602311 conn(0x5556907d7fc0 msgr2=0x555690900350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 -- 172.21.15.12:0/4146602311 shutdown_connections 2024-05-27T07:36:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 -- 172.21.15.12:0/4146602311 wait complete. 2024-05-27T07:36:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.800+0000 7ff1986d3b80 1 -- start start 2024-05-27T07:36:57.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 1 Processor -- start 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x555690911540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690911a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 0x55569090e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555690901630 con 0x555690777e40 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555690826d90 con 0x5556908f6820 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555690915350 con 0x5556908f6410 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.801+0000 7ff1986d3b80 10 client.0 _ll_register_callbacks cb 0x5556907bec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:57.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x555690911540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x555690911540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41202/0 (socket says 172.21.15.12:41202) 2024-05-27T07:36:57.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 -- 172.21.15.12:0/2422660528 learned_addr learned my addr 172.21.15.12:0/2422660528 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 -- 172.21.15.12:0/2422660528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 msgr2=0x555690911a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690911a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 -- 172.21.15.12:0/2422660528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 msgr2=0x55569090e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 0x55569090e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.802+0000 7ff182a00700 1 -- 172.21.15.12:0/2422660528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55569090ea50 con 0x555690777e40 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff182a00700 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x555690911540 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff174009350 tx=0x7ff174003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda7fbf628817d05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff170c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff170c00700 1 -- 172.21.15.12:0/2422660528 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17400bc20 con 0x555690777e40 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff170c00700 1 -- 172.21.15.12:0/2422660528 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7ff17400bdc0 con 0x555690777e40 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555690820570 con 0x555690777e40 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff170c00700 1 -- 172.21.15.12:0/2422660528 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7ff174048050 con 0x555690777e40 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff1986d3b80 10 client.17163 fetch_fsmap learned FSMap version 119 2024-05-27T07:36:57.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.803+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x555690984720 con 0x555690777e40 2024-05-27T07:36:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff170c00700 1 -- 172.21.15.12:0/2422660528 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x555690820570 con 0x555690777e40 2024-05-27T07:36:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff170c00700 1 -- 172.21.15.12:0/2422660528 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7ff17403a4a0 con 0x555690777e40 2024-05-27T07:36:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff1986d3b80 10 client.17163 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:36:57.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff1986d3b80 10 client.17163 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:36:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff1986d3b80 20 client.17163 populate_metadata read hostname 'smithi012' 2024-05-27T07:36:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x555690986bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff1986d3b80 4 client.17163 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.804+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555690697250 con 0x555690919530 2024-05-27T07:36:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.805+0000 7ff17a000700 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x555690986bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.806+0000 7ff170c00700 10 client.17163 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:36:57.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.806+0000 7ff17a000700 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x555690986bc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70d2563cbeefe09b server_cookie=7b8e1917edec4feb in_seq=0 out_seq=0 2024-05-27T07:36:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff170c00700 1 -- 172.21.15.12:0/2422660528 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555690697250 con 0x555690919530 2024-05-27T07:36:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff170c00700 10 client.17163 handle_command_reply: tid=0 2024-05-27T07:36:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 client.17163 shutdown 2024-05-27T07:36:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 msgr2=0x555690911540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x555690911540 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff174009350 tx=0x7ff174003f80 comp rx=0 tx=0).stop 2024-05-27T07:36:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 shutdown_connections 2024-05-27T07:36:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x555690919530 0x555690986bc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556908f6820 0x55569090e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555690777e40 0x555690911540 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 --2- 172.21.15.12:0/2422660528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556908f6410 0x555690911a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff170c00700 0 client.17163 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:36:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 >> 172.21.15.12:0/2422660528 conn(0x5556907d7fc0 msgr2=0x555690900350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 shutdown_connections 2024-05-27T07:36:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.807+0000 7ff1986d3b80 1 -- 172.21.15.12:0/2422660528 wait complete. 2024-05-27T07:36:57.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.808+0000 7ff1986d3b80 20 client.17163 trim_cache size 0 max 16384 2024-05-27T07:36:57.805 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:36:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.808+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff1680b2a20 msgr2=0x7ff1680b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.808+0000 7ff183e00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff1680b2a20 0x7ff1680b4f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5556908f7430 tx=0x7ff174009380 comp rx=0 tx=0).stop 2024-05-27T07:36:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.808+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105bd0 msgr2=0x7ff18419d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.808+0000 7ff183e00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105bd0 0x7ff18419d2c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00d950 tx=0x7ff16c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:36:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 shutdown_connections 2024-05-27T07:36:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff1680b2a20 0x7ff1680b4f30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18410e6f0 0x7ff1841a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff184105280 0x7ff18419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 --2- 172.21.15.12:0/4216565941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff184105bd0 0x7ff18419d2c0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 >> 172.21.15.12:0/4216565941 conn(0x7ff184078af0 msgr2=0x7ff184102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 shutdown_connections 2024-05-27T07:36:57.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:36:57.809+0000 7ff183e00700 1 -- 172.21.15.12:0/4216565941 wait complete. 2024-05-27T07:36:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:57 smithi162 ceph-mon[35676]: pgmap v499: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1007 KiB/s wr, 108 op/s 2024-05-27T07:36:57.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:57 smithi062 ceph-mon[35304]: pgmap v499: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1007 KiB/s wr, 108 op/s 2024-05-27T07:36:58.041 INFO:tasks.workunit.client.0.smithi012.stdout:All 50000 operations completed A-OK! 2024-05-27T07:36:58.052 INFO:tasks.workunit.client.0.smithi012.stderr:+ ./fsx -z 100MB -N 50000 -p 10000 -l 104857600 2024-05-27T07:36:58.059 INFO:tasks.workunit.client.0.smithi012.stdout:Seed set to 1 2024-05-27T07:36:58.749 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:36:59.213 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:59.588 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-05-27T07:36:59.588 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-05-27T07:36:59.588 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support clone range, disabling! 2024-05-27T07:36:59.589 INFO:tasks.workunit.client.0.smithi012.stderr:main: filesystem does not support dedupe range, disabling! 2024-05-27T07:36:59.604 INFO:tasks.workunit.client.0.smithi012.stdout:skipping zero size read 2024-05-27T07:36:59.634 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0x1c37b90 2024-05-27T07:36:59.668 INFO:tasks.workunit.client.0.smithi012.stdout:truncating to largest ever: 0x5bd330d 2024-05-27T07:36:59.721 INFO:tasks.workunit.client.0.smithi012.stdout:copying to largest ever: 0x637bf99 2024-05-27T07:36:59.723 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:36:59 smithi012 ceph-mon[31473]: pgmap v500: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1007 KiB/s wr, 108 op/s 2024-05-27T07:36:59.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:36:59 smithi162 ceph-mon[35676]: pgmap v500: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1007 KiB/s wr, 108 op/s 2024-05-27T07:36:59.991 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:36:59 smithi062 ceph-mon[35304]: pgmap v500: 129 pgs: 129 active+clean; 263 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1007 KiB/s wr, 108 op/s 2024-05-27T07:37:00.794 INFO:tasks.workunit.client.0.smithi012.stdout:copying to largest ever: 0x63d9b9c 2024-05-27T07:37:00.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:00 smithi012 ceph-mon[31473]: pgmap v501: 129 pgs: 129 active+clean; 264 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-05-27T07:37:00.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:00 smithi162 ceph-mon[35676]: pgmap v501: 129 pgs: 129 active+clean; 264 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-05-27T07:37:00.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:00 smithi062 ceph-mon[35304]: pgmap v501: 129 pgs: 129 active+clean; 264 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-05-27T07:37:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.925+0000 7f905ca00700 1 -- 172.21.15.12:0/2018696373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058075ad0 msgr2=0x7f9058075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.925+0000 7f905ca00700 1 --2- 172.21.15.12:0/2018696373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058075ad0 0x7f9058075ee0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9050009990 tx=0x7f9050009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.927+0000 7f905ca00700 1 -- 172.21.15.12:0/2018696373 shutdown_connections 2024-05-27T07:37:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.927+0000 7f905ca00700 1 --2- 172.21.15.12:0/2018696373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9058110c70 0x7f9058113090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.927+0000 7f905ca00700 1 --2- 172.21.15.12:0/2018696373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058075ad0 0x7f9058075ee0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.927+0000 7f905ca00700 1 --2- 172.21.15.12:0/2018696373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9058076420 0x7f9058110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.927+0000 7f905ca00700 1 -- 172.21.15.12:0/2018696373 >> 172.21.15.12:0/2018696373 conn(0x7f9058079ef0 msgr2=0x7f905807a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.928+0000 7f905ca00700 1 -- 172.21.15.12:0/2018696373 shutdown_connections 2024-05-27T07:37:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.928+0000 7f905ca00700 1 -- 172.21.15.12:0/2018696373 wait complete. 2024-05-27T07:37:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.930+0000 7f905ca00700 1 Processor -- start 2024-05-27T07:37:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.931+0000 7f905ca00700 1 -- start start 2024-05-27T07:37:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.931+0000 7f905ca00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9058075ad0 0x7f905819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.931+0000 7f905ca00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058076420 0x7f905819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.931+0000 7f905ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9058110c70 0x7f90581a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.931+0000 7f905ca00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9058116510 con 0x7f9058076420 2024-05-27T07:37:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.931+0000 7f905ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9058116390 con 0x7f9058110c70 2024-05-27T07:37:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.931+0000 7f905ca00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9058116690 con 0x7f9058075ad0 2024-05-27T07:37:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.932+0000 7f9055400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058076420 0x7f905819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.933+0000 7f9055400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058076420 0x7f905819d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39026/0 (socket says 172.21.15.12:39026) 2024-05-27T07:37:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.933+0000 7f9055400700 1 -- 172.21.15.12:0/49287826 learned_addr learned my addr 172.21.15.12:0/49287826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.933+0000 7f9055e00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9058075ad0 0x7f905819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.933+0000 7f9056800700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9058110c70 0x7f90581a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.934+0000 7f9055400700 1 -- 172.21.15.12:0/49287826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9058075ad0 msgr2=0x7f905819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.934+0000 7f9055400700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9058075ad0 0x7f905819cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.934+0000 7f9055400700 1 -- 172.21.15.12:0/49287826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9058110c70 msgr2=0x7f90581a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.934+0000 7f9055400700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9058110c70 0x7f90581a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.934+0000 7f9055400700 1 -- 172.21.15.12:0/49287826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90581a1df0 con 0x7f9058076420 2024-05-27T07:37:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.935+0000 7f9055400700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058076420 0x7f905819d4c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f904000d950 tx=0x7f904000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.936+0000 7f904ea00700 1 -- 172.21.15.12:0/49287826 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040016070 con 0x7f9058076420 2024-05-27T07:37:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.936+0000 7f904ea00700 1 -- 172.21.15.12:0/49287826 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9040009dd0 con 0x7f9058076420 2024-05-27T07:37:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.936+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90581a20e0 con 0x7f9058076420 2024-05-27T07:37:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.936+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905806d1d0 con 0x7f9058076420 2024-05-27T07:37:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.936+0000 7f904ea00700 1 -- 172.21.15.12:0/49287826 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904000f3c0 con 0x7f9058076420 2024-05-27T07:37:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.943+0000 7f906b571b80 1 Processor -- start 2024-05-27T07:37:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.943+0000 7f906b571b80 1 -- start start 2024-05-27T07:37:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.944+0000 7f906b571b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e0c4e40 0x555e3e2497f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.944+0000 7f906b571b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e243870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.944+0000 7f906b571b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e243db0 0x555e3e2441c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.944+0000 7f906b571b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555e3e171e40 con 0x555e3e0c4e40 2024-05-27T07:37:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.944+0000 7f906b571b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555e3e1619e0 con 0x555e3e249d30 2024-05-27T07:37:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.944+0000 7f906b571b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555e3e165a20 con 0x555e3e243db0 2024-05-27T07:37:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.945+0000 7f9055e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e243870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.945+0000 7f9055e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e243870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41348/0 (socket says 172.21.15.12:41348) 2024-05-27T07:37:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.945+0000 7f9055e00700 1 -- 172.21.15.12:0/1215567177 learned_addr learned my addr 172.21.15.12:0/1215567177 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.945+0000 7f9056800700 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e0c4e40 0x555e3e2497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.945+0000 7f904ea00700 1 -- 172.21.15.12:0/49287826 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9040019070 con 0x7f9058076420 2024-05-27T07:37:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.945+0000 7f9055400700 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e243db0 0x555e3e2441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.946+0000 7f9055e00700 1 -- 172.21.15.12:0/1215567177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e243db0 msgr2=0x555e3e2441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.946+0000 7f9055e00700 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e243db0 0x555e3e2441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.946+0000 7f9055e00700 1 -- 172.21.15.12:0/1215567177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e0c4e40 msgr2=0x555e3e2497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.946+0000 7f9055e00700 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e0c4e40 0x555e3e2497f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.946+0000 7f9055e00700 1 -- 172.21.15.12:0/1215567177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e3e12a420 con 0x555e3e249d30 2024-05-27T07:37:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.947+0000 7f9055e00700 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e243870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9050009990 tx=0x7f9050009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb31b9cb2e75f698 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.947+0000 7f9047400700 1 -- 172.21.15.12:0/1215567177 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050027070 con 0x555e3e249d30 2024-05-27T07:37:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.948+0000 7f904ea00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f903c0b2af0 0x7f903c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.948+0000 7f904ea00700 1 -- 172.21.15.12:0/49287826 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7f90400e28f0 con 0x7f9058076420 2024-05-27T07:37:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.948+0000 7f9047400700 1 -- 172.21.15.12:0/1215567177 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9050003c50 con 0x555e3e249d30 2024-05-27T07:37:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.948+0000 7f9047400700 1 -- 172.21.15.12:0/1215567177 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905000bb50 con 0x555e3e249d30 2024-05-27T07:37:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.949+0000 7f9056800700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f903c0b2af0 0x7f903c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 -- 172.21.15.12:0/1215567177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 msgr2=0x555e3e243870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e243870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9050009990 tx=0x7f9050009ee0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f9056800700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f903c0b2af0 0x7f903c0b5000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x555e3e244b30 tx=0x7f904800c040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 -- 172.21.15.12:0/1215567177 shutdown_connections 2024-05-27T07:37:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e243db0 0x555e3e2441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e0c4e40 0x555e3e2497f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 --2- 172.21.15.12:0/1215567177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e243870 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 -- 172.21.15.12:0/1215567177 >> 172.21.15.12:0/1215567177 conn(0x555e3e124fc0 msgr2=0x555e3e23e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 -- 172.21.15.12:0/1215567177 shutdown_connections 2024-05-27T07:37:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.950+0000 7f906b571b80 1 -- 172.21.15.12:0/1215567177 wait complete. 2024-05-27T07:37:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.951+0000 7f906b571b80 1 -- start start 2024-05-27T07:37:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.951+0000 7f906b571b80 1 Processor -- start 2024-05-27T07:37:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f906b571b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e24d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f906b571b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e243db0 0x555e3e24d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f906b571b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e24ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f906b571b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555e3e067fa0 con 0x555e3e243db0 2024-05-27T07:37:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f906b571b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555e3e102760 con 0x555e3e249d30 2024-05-27T07:37:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f906b571b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555e3e2507a0 con 0x555e3e0c4e40 2024-05-27T07:37:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f906b571b80 10 client.0 _ll_register_callbacks cb 0x555e3e10bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f9055e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e24d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f9055e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e24d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:41006/0 (socket says 172.21.15.12:41006) 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f9055e00700 1 -- 172.21.15.12:0/688178327 learned_addr learned my addr 172.21.15.12:0/688178327 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f9055400700 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e243db0 0x555e3e24d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.952+0000 7f9056800700 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e24ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.953+0000 7f9055e00700 1 -- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 msgr2=0x555e3e24ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.953+0000 7f9055e00700 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e24ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.953+0000 7f9055e00700 1 -- 172.21.15.12:0/688178327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e243db0 msgr2=0x555e3e24d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.953+0000 7f9055e00700 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e243db0 0x555e3e24d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.953+0000 7f9055e00700 1 -- 172.21.15.12:0/688178327 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e3e0b18d0 con 0x555e3e0c4e40 2024-05-27T07:37:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.953+0000 7f9055e00700 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e24d450 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9050006010 tx=0x7f90500054c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bdfac0843b1db44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.953+0000 7f903be00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:37:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.954+0000 7f903be00700 1 -- 172.21.15.12:0/688178327 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050030020 con 0x555e3e0c4e40 2024-05-27T07:37:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.954+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9050003c50 con 0x555e3e0c4e40 2024-05-27T07:37:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.954+0000 7f903be00700 1 -- 172.21.15.12:0/688178327 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7f90500037c0 con 0x555e3e0c4e40 2024-05-27T07:37:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.954+0000 7f903be00700 1 -- 172.21.15.12:0/688178327 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f9050027040 con 0x555e3e0c4e40 2024-05-27T07:37:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.954+0000 7f906b571b80 10 client.26626 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.954+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x555e3e132060 con 0x555e3e0c4e40 2024-05-27T07:37:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f903be00700 1 -- 172.21.15.12:0/688178327 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9050003c50 con 0x555e3e0c4e40 2024-05-27T07:37:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f903be00700 1 -- 172.21.15.12:0/688178327 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f905000f580 con 0x555e3e0c4e40 2024-05-27T07:37:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f906b571b80 10 client.26626 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f906b571b80 10 client.26626 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:37:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f906b571b80 20 client.26626 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f906b571b80 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e25d610 0x555e3e25fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f906b571b80 4 client.26626 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.955+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e3e173d90 con 0x555e3e25d610 2024-05-27T07:37:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.956+0000 7f9055400700 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e25d610 0x555e3e25fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.957+0000 7f9055400700 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e25d610 0x555e3e25fae0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c92fa5f089b367e0 server_cookie=a67027f580e0e561 in_seq=0 out_seq=0 2024-05-27T07:37:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.957+0000 7f903be00700 10 client.26626 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:37:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.958+0000 7f903be00700 1 -- 172.21.15.12:0/688178327 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555e3e173d90 con 0x555e3e25d610 2024-05-27T07:37:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.958+0000 7f903be00700 10 client.26626 handle_command_reply: tid=0 2024-05-27T07:37:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.958+0000 7f906b571b80 1 client.26626 shutdown 2024-05-27T07:37:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 msgr2=0x555e3e24d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e24d450 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9050006010 tx=0x7f90500054c0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 shutdown_connections 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e25d610 0x555e3e25fae0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e249d30 0x555e3e24ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e24d450 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 --2- 172.21.15.12:0/688178327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e243db0 0x555e3e24d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 >> 172.21.15.12:0/688178327 conn(0x555e3e124fc0 msgr2=0x555e3e23e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f903be00700 0 client.26626 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 shutdown_connections 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 1 -- 172.21.15.12:0/688178327 wait complete. 2024-05-27T07:37:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.959+0000 7f906b571b80 20 client.26626 trim_cache size 0 max 16384 2024-05-27T07:37:01.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.984+0000 7f906b571b80 1 Processor -- start 2024-05-27T07:37:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.984+0000 7f906b571b80 1 -- start start 2024-05-27T07:37:01.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.985+0000 7f906b571b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e243870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.985+0000 7f906b571b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e243db0 0x555e3e24c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.985+0000 7f906b571b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e249d30 0x555e3e24f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.985+0000 7f906b571b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f903400e410 con 0x555e3e249d30 2024-05-27T07:37:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.985+0000 7f906b571b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f903400d670 con 0x555e3e243db0 2024-05-27T07:37:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.985+0000 7f906b571b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f903400dd40 con 0x555e3e0c4e40 2024-05-27T07:37:01.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.986+0000 7f9055400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e243db0 0x555e3e24c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.986+0000 7f9055400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e243db0 0x555e3e24c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41362/0 (socket says 172.21.15.12:41362) 2024-05-27T07:37:01.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.986+0000 7f9055400700 1 -- 172.21.15.12:0/1707348392 learned_addr learned my addr 172.21.15.12:0/1707348392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:01.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.987+0000 7f9055400700 1 -- 172.21.15.12:0/1707348392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 msgr2=0x555e3e243870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:37:01.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.987+0000 7f9055400700 1 --2- 172.21.15.12:0/1707348392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e243870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.987+0000 7f9055400700 1 -- 172.21.15.12:0/1707348392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e249d30 msgr2=0x555e3e24f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.987+0000 7f9055400700 1 --2- 172.21.15.12:0/1707348392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e249d30 0x555e3e24f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.987+0000 7f9055400700 1 -- 172.21.15.12:0/1707348392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e3e24fcf0 con 0x555e3e243db0 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.987+0000 7f9055400700 1 --2- 172.21.15.12:0/1707348392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e243db0 0x555e3e24c770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90400a69d0 tx=0x7f90400cc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa5ae930439db3ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.988+0000 7f9046a00700 1 -- 172.21.15.12:0/1707348392 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040020050 con 0x555e3e243db0 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.988+0000 7f9046a00700 1 -- 172.21.15.12:0/1707348392 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f90400a9270 con 0x555e3e243db0 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.988+0000 7f9046a00700 1 -- 172.21.15.12:0/1707348392 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90400910b0 con 0x555e3e243db0 2024-05-27T07:37:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.989+0000 7f906b571b80 1 -- 172.21.15.12:0/1707348392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e243db0 msgr2=0x555e3e24c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 --2- 172.21.15.12:0/1707348392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e243db0 0x555e3e24c770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90400a69d0 tx=0x7f90400cc040 comp rx=0 tx=0).stop 2024-05-27T07:37:01.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 -- 172.21.15.12:0/1707348392 shutdown_connections 2024-05-27T07:37:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 --2- 172.21.15.12:0/1707348392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e249d30 0x555e3e24f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 --2- 172.21.15.12:0/1707348392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e0c4e40 0x555e3e243870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 --2- 172.21.15.12:0/1707348392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e243db0 0x555e3e24c770 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90400a69d0 tx=0x7f90400cc040 comp rx=0 tx=0).stop 2024-05-27T07:37:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 -- 172.21.15.12:0/1707348392 >> 172.21.15.12:0/1707348392 conn(0x555e3e124fc0 msgr2=0x555e3e247140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:01.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 -- 172.21.15.12:0/1707348392 shutdown_connections 2024-05-27T07:37:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.990+0000 7f906b571b80 1 -- 172.21.15.12:0/1707348392 wait complete. 2024-05-27T07:37:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.991+0000 7f906b571b80 1 -- start start 2024-05-27T07:37:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.991+0000 7f906b571b80 1 Processor -- start 2024-05-27T07:37:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f906b571b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e0c4e40 0x555e3e256060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f906b571b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e249d30 0x555e3e2649a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f906b571b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e264ee0 0x555e3e2652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f906b571b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555e3e24e510 con 0x555e3e264ee0 2024-05-27T07:37:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f906b571b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555e3e24b440 con 0x555e3e0c4e40 2024-05-27T07:37:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f906b571b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555e3e249240 con 0x555e3e249d30 2024-05-27T07:37:01.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f906b571b80 10 client.0 _ll_register_callbacks cb 0x555e3e10bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f9055e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e0c4e40 0x555e3e256060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f9055e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e0c4e40 0x555e3e256060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41370/0 (socket says 172.21.15.12:41370) 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.992+0000 7f9055e00700 1 -- 172.21.15.12:0/698192574 learned_addr learned my addr 172.21.15.12:0/698192574 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.993+0000 7f9055400700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e249d30 0x555e3e2649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.993+0000 7f9056800700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e264ee0 0x555e3e2652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.993+0000 7f9055e00700 1 -- 172.21.15.12:0/698192574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e249d30 msgr2=0x555e3e2649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.993+0000 7f9055e00700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e249d30 0x555e3e2649a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.993+0000 7f9055e00700 1 -- 172.21.15.12:0/698192574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e264ee0 msgr2=0x555e3e2652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:01.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.993+0000 7f9055e00700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e264ee0 0x555e3e2652f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.993+0000 7f9055e00700 1 -- 172.21.15.12:0/698192574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e3e2cf980 con 0x555e3e0c4e40 2024-05-27T07:37:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.994+0000 7f9056800700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e264ee0 0x555e3e2652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:01.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.994+0000 7f9055e00700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e0c4e40 0x555e3e256060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f905000f9f0 tx=0x7f905000fe90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a1566a99dee6ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.994+0000 7f903be00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:37:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.994+0000 7f903be00700 1 -- 172.21.15.12:0/698192574 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050030040 con 0x555e3e0c4e40 2024-05-27T07:37:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.995+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90400a9270 con 0x555e3e0c4e40 2024-05-27T07:37:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.995+0000 7f903be00700 1 -- 172.21.15.12:0/698192574 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9554+0+0 (secure 0 0 0) 0x7f90500343a0 con 0x555e3e0c4e40 2024-05-27T07:37:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.995+0000 7f903be00700 1 -- 172.21.15.12:0/698192574 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f90500058e0 con 0x555e3e0c4e40 2024-05-27T07:37:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.995+0000 7f906b571b80 10 client.36617 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:01.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.995+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x555e3e2d0b40 con 0x555e3e0c4e40 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f903be00700 1 -- 172.21.15.12:0/698192574 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f90400a9270 con 0x555e3e0c4e40 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f903be00700 1 -- 172.21.15.12:0/698192574 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f9050003b20 con 0x555e3e0c4e40 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f906b571b80 10 client.36617 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f906b571b80 10 client.36617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f906b571b80 20 client.36617 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f906b571b80 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e2d0fb0 0x555e3e2d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f906b571b80 4 client.36617 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.996+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555e3e16d570 con 0x555e3e2d0fb0 2024-05-27T07:37:01.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.997+0000 7f9055400700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e2d0fb0 0x555e3e2d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.998+0000 7f9055400700 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e2d0fb0 0x555e3e2d33d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e217b1337a89ca3 server_cookie=d10af78d09dc1cac in_seq=0 out_seq=0 2024-05-27T07:37:01.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.998+0000 7f903be00700 10 client.36617 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:37:01.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.998+0000 7f903be00700 1 -- 172.21.15.12:0/698192574 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555e3e16d570 con 0x555e3e2d0fb0 2024-05-27T07:37:01.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.998+0000 7f903be00700 10 client.36617 handle_command_reply: tid=0 2024-05-27T07:37:01.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.998+0000 7f906b571b80 1 client.36617 shutdown 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e0c4e40 msgr2=0x555e3e256060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e0c4e40 0x555e3e256060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f905000f9f0 tx=0x7f905000fe90 comp rx=0 tx=0).stop 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 shutdown_connections 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x555e3e2d0fb0 0x555e3e2d33d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e3e0c4e40 0x555e3e256060 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x555e3e264ee0 0x555e3e2652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 --2- 172.21.15.12:0/698192574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e3e249d30 0x555e3e2649a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 >> 172.21.15.12:0/698192574 conn(0x555e3e124fc0 msgr2=0x555e3e24c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f903be00700 0 client.36617 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:37:01.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:01.999+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 shutdown_connections 2024-05-27T07:37:01.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.000+0000 7f906b571b80 1 -- 172.21.15.12:0/698192574 wait complete. 2024-05-27T07:37:01.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.000+0000 7f906b571b80 20 client.36617 trim_cache size 0 max 16384 2024-05-27T07:37:01.997 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f903c0b2af0 msgr2=0x7f903c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f903c0b2af0 0x7f903c0b5000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x555e3e244b30 tx=0x7f904800c040 comp rx=0 tx=0).stop 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058076420 msgr2=0x7f905819d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058076420 0x7f905819d4c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f904000d950 tx=0x7f904000dc60 comp rx=0 tx=0).stop 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 shutdown_connections 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f903c0b2af0 0x7f903c0b5000 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9058110c70 0x7f90581a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9058075ad0 0x7f905819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 --2- 172.21.15.12:0/49287826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9058076420 0x7f905819d4c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:01.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.001+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 >> 172.21.15.12:0/49287826 conn(0x7f9058079ef0 msgr2=0x7f9058101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:01.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.002+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 shutdown_connections 2024-05-27T07:37:01.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:02.002+0000 7f905ca00700 1 -- 172.21.15.12:0/49287826 wait complete. 2024-05-27T07:37:03.077 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:37:03.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:03 smithi012 ceph-mon[31473]: pgmap v502: 129 pgs: 129 active+clean; 264 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1015 KiB/s wr, 104 op/s 2024-05-27T07:37:03.562 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:03.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:03 smithi162 ceph-mon[35676]: pgmap v502: 129 pgs: 129 active+clean; 264 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1015 KiB/s wr, 104 op/s 2024-05-27T07:37:03.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:03 smithi062 ceph-mon[35304]: pgmap v502: 129 pgs: 129 active+clean; 264 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1015 KiB/s wr, 104 op/s 2024-05-27T07:37:05.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:05 smithi012 ceph-mon[31473]: pgmap v503: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-05-27T07:37:05.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:05 smithi162 ceph-mon[35676]: pgmap v503: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-05-27T07:37:05.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:05 smithi062 ceph-mon[35304]: pgmap v503: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-05-27T07:37:06.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 -- 172.21.15.12:0/1263862507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad501045e0 msgr2=0x7fad501049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 --2- 172.21.15.12:0/1263862507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad501045e0 0x7fad501049f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fad400099c0 tx=0x7fad40009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 -- 172.21.15.12:0/1263862507 shutdown_connections 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 --2- 172.21.15.12:0/1263862507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad50109860 0x7fad501111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 --2- 172.21.15.12:0/1263862507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad501045e0 0x7fad501049f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 --2- 172.21.15.12:0/1263862507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad50108f10 0x7fad50109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 -- 172.21.15.12:0/1263862507 >> 172.21.15.12:0/1263862507 conn(0x7fad50100030 msgr2=0x7fad50102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.784+0000 7fad55a00700 1 -- 172.21.15.12:0/1263862507 shutdown_connections 2024-05-27T07:37:06.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.785+0000 7fad55a00700 1 -- 172.21.15.12:0/1263862507 wait complete. 2024-05-27T07:37:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 Processor -- start 2024-05-27T07:37:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 -- start start 2024-05-27T07:37:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad501045e0 0x7fad5019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad50108f10 0x7fad5019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad50109860 0x7fad501a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fad50114410 con 0x7fad50108f10 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad50114290 con 0x7fad50109860 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.786+0000 7fad55a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad50114590 con 0x7fad501045e0 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad50108f10 0x7fad5019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad50108f10 0x7fad5019d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39088/0 (socket says 172.21.15.12:39088) 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 -- 172.21.15.12:0/1122533053 learned_addr learned my addr 172.21.15.12:0/1122533053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:06.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4f400700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad50109860 0x7fad501a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 -- 172.21.15.12:0/1122533053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad501045e0 msgr2=0x7fad5019d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad501045e0 0x7fad5019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 -- 172.21.15.12:0/1122533053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad50109860 msgr2=0x7fad501a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad50109860 0x7fad501a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.787+0000 7fad4e000700 1 -- 172.21.15.12:0/1122533053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad501a1ef0 con 0x7fad50108f10 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.788+0000 7fad4e000700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad50108f10 0x7fad5019d5c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00e970 tx=0x7fad3c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.788+0000 7fad47400700 1 -- 172.21.15.12:0/1122533053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c00cbd0 con 0x7fad50108f10 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.788+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad501a21e0 con 0x7fad50108f10 2024-05-27T07:37:06.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.788+0000 7fad47400700 1 -- 172.21.15.12:0/1122533053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad3c00cd70 con 0x7fad50108f10 2024-05-27T07:37:06.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.788+0000 7fad47400700 1 -- 172.21.15.12:0/1122533053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c005570 con 0x7fad50108f10 2024-05-27T07:37:06.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.788+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad501a9b00 con 0x7fad50108f10 2024-05-27T07:37:06.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.794+0000 7fad47400700 1 -- 172.21.15.12:0/1122533053 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fad3c004750 con 0x7fad50108f10 2024-05-27T07:37:06.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.794+0000 7fad64558b80 1 Processor -- start 2024-05-27T07:37:06.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.794+0000 7fad64558b80 1 -- start start 2024-05-27T07:37:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.794+0000 7fad64558b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad64558b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185a72e40 0x55e185bf1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad64558b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bfe110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad64558b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e185b1fe40 con 0x55e185a72e40 2024-05-27T07:37:06.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad64558b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e185b0f9e0 con 0x55e185b5e9f0 2024-05-27T07:37:06.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad64558b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e185b13a20 con 0x55e185bf1e70 2024-05-27T07:37:06.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad4ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185a72e40 0x55e185bf1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad4e000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bfe110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad4e000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bfe110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:41072/0 (socket says 172.21.15.12:41072) 2024-05-27T07:37:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad4e000700 1 -- 172.21.15.12:0/4028987333 learned_addr learned my addr 172.21.15.12:0/4028987333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad4ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185a72e40 0x55e185bf1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39090/0 (socket says 172.21.15.12:39090) 2024-05-27T07:37:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.795+0000 7fad4f400700 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad47400700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad340b29f0 0x7fad340b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad47400700 1 -- 172.21.15.12:0/1122533053 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 9642+0+0 (secure 0 0 0) 0x7fad3c0e2d40 con 0x7fad50108f10 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad4e000700 1 -- 172.21.15.12:0/4028987333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 msgr2=0x55e185bf1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad4e000700 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad4e000700 1 -- 172.21.15.12:0/4028987333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185a72e40 msgr2=0x55e185bf1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad4e000700 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185a72e40 0x55e185bf1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad4e000700 1 -- 172.21.15.12:0/4028987333 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e185ad8420 con 0x55e185bf1e70 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad4f400700 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.796+0000 7fad4f400700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad340b29f0 0x7fad340b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.797+0000 7fad4e000700 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bfe110 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fad3c0dff10 tx=0x7fad3c00ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44ea688ddf43eeb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.797+0000 7fad4f400700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad340b29f0 0x7fad340b4f00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55e185bf3ff0 tx=0x7fad48004be0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.797+0000 7fad3be00700 1 -- 172.21.15.12:0/4028987333 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c0a9930 con 0x55e185bf1e70 2024-05-27T07:37:06.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.797+0000 7fad3be00700 1 -- 172.21.15.12:0/4028987333 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad3c0a9670 con 0x55e185bf1e70 2024-05-27T07:37:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.797+0000 7fad3be00700 1 -- 172.21.15.12:0/4028987333 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c0a52f0 con 0x55e185bf1e70 2024-05-27T07:37:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.798+0000 7fad64558b80 1 -- 172.21.15.12:0/4028987333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 msgr2=0x55e185bfe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.798+0000 7fad64558b80 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bfe110 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fad3c0dff10 tx=0x7fad3c00ef50 comp rx=0 tx=0).stop 2024-05-27T07:37:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 -- 172.21.15.12:0/4028987333 shutdown_connections 2024-05-27T07:37:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bfe110 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 --2- 172.21.15.12:0/4028987333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185a72e40 0x55e185bf1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 -- 172.21.15.12:0/4028987333 >> 172.21.15.12:0/4028987333 conn(0x55e185b0c390 msgr2=0x55e185bed4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 -- 172.21.15.12:0/4028987333 shutdown_connections 2024-05-27T07:37:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 -- 172.21.15.12:0/4028987333 wait complete. 2024-05-27T07:37:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.799+0000 7fad64558b80 1 -- start start 2024-05-27T07:37:06.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 1 Processor -- start 2024-05-27T07:37:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bfb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bfbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bfc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e185a15fa0 con 0x55e185bf1e70 2024-05-27T07:37:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e185ab0760 con 0x55e185b5e9f0 2024-05-27T07:37:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e185bfeb20 con 0x55e185a72e40 2024-05-27T07:37:06.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.800+0000 7fad64558b80 10 client.0 _ll_register_callbacks cb 0x55e185ab9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.801+0000 7fad4f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bfc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.801+0000 7fad4e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bfbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.801+0000 7fad4ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bfb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.801+0000 7fad4e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bfbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41422/0 (socket says 172.21.15.12:41422) 2024-05-27T07:37:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.801+0000 7fad4e000700 1 -- 172.21.15.12:0/1774530272 learned_addr learned my addr 172.21.15.12:0/1774530272 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:06.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.802+0000 7fad4ea00700 1 -- 172.21.15.12:0/1774530272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 msgr2=0x55e185bfbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.802+0000 7fad4ea00700 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bfbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.802+0000 7fad4ea00700 1 -- 172.21.15.12:0/1774530272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 msgr2=0x55e185bfc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.802+0000 7fad4ea00700 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bfc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.802+0000 7fad4ea00700 1 -- 172.21.15.12:0/1774530272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e185a5f8d0 con 0x55e185a72e40 2024-05-27T07:37:06.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.806+0000 7fad4ea00700 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bfb690 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad40006010 tx=0x7fad40003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4685c3255587a6a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.807+0000 7fad38c00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:37:06.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.808+0000 7fad38c00700 1 -- 172.21.15.12:0/1774530272 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40005240 con 0x55e185a72e40 2024-05-27T07:37:06.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.808+0000 7fad38c00700 1 -- 172.21.15.12:0/1774530272 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 9642+0+0 (secure 0 0 0) 0x7fad400053e0 con 0x55e185a72e40 2024-05-27T07:37:06.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.808+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad3c0a9670 con 0x55e185a72e40 2024-05-27T07:37:06.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.809+0000 7fad38c00700 1 -- 172.21.15.12:0/1774530272 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fad4002f050 con 0x55e185a72e40 2024-05-27T07:37:06.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.809+0000 7fad64558b80 10 client.26635 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:06.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.809+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55e185ae0060 con 0x55e185a72e40 2024-05-27T07:37:06.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.810+0000 7fad38c00700 1 -- 172.21.15.12:0/1774530272 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad3c0a9670 con 0x55e185a72e40 2024-05-27T07:37:06.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.810+0000 7fad38c00700 1 -- 172.21.15.12:0/1774530272 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fad4002a030 con 0x55e185a72e40 2024-05-27T07:37:06.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.810+0000 7fad64558b80 10 client.26635 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:06.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.810+0000 7fad64558b80 10 client.26635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:06.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.810+0000 7fad64558b80 20 client.26635 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:06.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.810+0000 7fad64558b80 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c0be50 0x55e185c0e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.811+0000 7fad64558b80 4 client.26635 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:06.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.811+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e185b21d90 con 0x55e185c0be50 2024-05-27T07:37:06.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.812+0000 7fad4e000700 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c0be50 0x55e185c0e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.814+0000 7fad38c00700 10 client.26635 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:06.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.814+0000 7fad4e000700 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c0be50 0x55e185c0e300 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=689fdac00b272f10 server_cookie=b7bcffe2d395ea47 in_seq=0 out_seq=0 2024-05-27T07:37:06.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.815+0000 7fad38c00700 1 -- 172.21.15.12:0/1774530272 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e185b21d90 con 0x55e185c0be50 2024-05-27T07:37:06.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.815+0000 7fad38c00700 10 client.26635 handle_command_reply: tid=0 2024-05-27T07:37:06.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 client.26635 shutdown 2024-05-27T07:37:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 msgr2=0x55e185bfb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bfb690 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad40006010 tx=0x7fad40003f10 comp rx=0 tx=0).stop 2024-05-27T07:37:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 shutdown_connections 2024-05-27T07:37:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bfc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bfb690 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c0be50 0x55e185c0e300 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 --2- 172.21.15.12:0/1774530272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bfbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.816+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 >> 172.21.15.12:0/1774530272 conn(0x55e185b0c390 msgr2=0x55e185bf13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.817+0000 7fad38c00700 0 client.26635 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.817+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 shutdown_connections 2024-05-27T07:37:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.817+0000 7fad64558b80 1 -- 172.21.15.12:0/1774530272 wait complete. 2024-05-27T07:37:06.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.817+0000 7fad64558b80 20 client.26635 trim_cache size 0 max 16384 2024-05-27T07:37:06.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.840+0000 7fad64558b80 1 Processor -- start 2024-05-27T07:37:06.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.840+0000 7fad64558b80 1 -- start start 2024-05-27T07:37:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad64558b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185a72e40 0x55e185bf6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad64558b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185b5e9f0 0x55e185c0e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad64558b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad64558b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e185bfa810 con 0x55e185b5e9f0 2024-05-27T07:37:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad64558b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad501b2100 con 0x55e185a72e40 2024-05-27T07:37:06.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad64558b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad501b1360 con 0x55e185bf1e70 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad4f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad4f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bed5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:41096/0 (socket says 172.21.15.12:41096) 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.841+0000 7fad4f400700 1 -- 172.21.15.12:0/3548009672 learned_addr learned my addr 172.21.15.12:0/3548009672 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.842+0000 7fad4e000700 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185b5e9f0 0x55e185c0e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.842+0000 7fad4ea00700 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185a72e40 0x55e185bf6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.842+0000 7fad4f400700 1 -- 172.21.15.12:0/3548009672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185a72e40 msgr2=0x55e185bf6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.842+0000 7fad4f400700 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185a72e40 0x55e185bf6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.842+0000 7fad4f400700 1 -- 172.21.15.12:0/3548009672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185b5e9f0 msgr2=0x55e185c0e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.842+0000 7fad4f400700 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185b5e9f0 0x55e185c0e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.842+0000 7fad4f400700 1 -- 172.21.15.12:0/3548009672 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e185bee850 con 0x55e185bf1e70 2024-05-27T07:37:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.843+0000 7fad4f400700 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bed5c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad4800aef0 tx=0x7fad48015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2591eaf187682825 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.843+0000 7fad3b400700 1 -- 172.21.15.12:0/3548009672 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48017d80 con 0x55e185bf1e70 2024-05-27T07:37:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.843+0000 7fad3b400700 1 -- 172.21.15.12:0/3548009672 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad4801b040 con 0x55e185bf1e70 2024-05-27T07:37:06.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.843+0000 7fad3b400700 1 -- 172.21.15.12:0/3548009672 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480225e0 con 0x55e185bf1e70 2024-05-27T07:37:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.844+0000 7fad64558b80 1 -- 172.21.15.12:0/3548009672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 msgr2=0x55e185bed5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.844+0000 7fad64558b80 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bed5c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad4800aef0 tx=0x7fad48015000 comp rx=0 tx=0).stop 2024-05-27T07:37:06.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.845+0000 7fad64558b80 1 -- 172.21.15.12:0/3548009672 shutdown_connections 2024-05-27T07:37:06.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.845+0000 7fad64558b80 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185a72e40 0x55e185bf6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.845+0000 7fad64558b80 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185bf1e70 0x55e185bed5c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.845+0000 7fad64558b80 1 --2- 172.21.15.12:0/3548009672 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185b5e9f0 0x55e185c0e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.845+0000 7fad64558b80 1 -- 172.21.15.12:0/3548009672 >> 172.21.15.12:0/3548009672 conn(0x55e185b0c390 msgr2=0x55e185bf2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:06.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.845+0000 7fad64558b80 1 -- 172.21.15.12:0/3548009672 shutdown_connections 2024-05-27T07:37:06.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.846+0000 7fad64558b80 1 -- 172.21.15.12:0/3548009672 wait complete. 2024-05-27T07:37:06.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.846+0000 7fad64558b80 1 -- start start 2024-05-27T07:37:06.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.847+0000 7fad64558b80 1 Processor -- start 2024-05-27T07:37:06.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.848+0000 7fad64558b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bf6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.848+0000 7fad64558b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.848+0000 7fad64558b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bf5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.848+0000 7fad64558b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fad501b1a30 con 0x55e185bf1e70 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.848+0000 7fad64558b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e185bf5b20 con 0x55e185b5e9f0 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.848+0000 7fad64558b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e185bf5d20 con 0x55e185a72e40 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.848+0000 7fad64558b80 10 client.0 _ll_register_callbacks cb 0x55e185ab9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.849+0000 7fad4e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.849+0000 7fad4e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41430/0 (socket says 172.21.15.12:41430) 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.849+0000 7fad4e000700 1 -- 172.21.15.12:0/2829978827 learned_addr learned my addr 172.21.15.12:0/2829978827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.849+0000 7fad4f400700 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bf5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.850+0000 7fad4ea00700 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bf6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.851+0000 7fad4e000700 1 -- 172.21.15.12:0/2829978827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 msgr2=0x55e185bf6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.851+0000 7fad4e000700 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bf6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.851+0000 7fad4e000700 1 -- 172.21.15.12:0/2829978827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 msgr2=0x55e185bf5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:06.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.851+0000 7fad4e000700 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bf5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.851+0000 7fad4e000700 1 -- 172.21.15.12:0/2829978827 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e185be9240 con 0x55e185b5e9f0 2024-05-27T07:37:06.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.852+0000 7fad4e000700 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf5040 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad3c0e3000 tx=0x7fad3c0092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8240966ab3e3d71 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:06.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.852+0000 7fad38c00700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:37:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.853+0000 7fad38c00700 1 -- 172.21.15.12:0/2829978827 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c0a92d0 con 0x55e185b5e9f0 2024-05-27T07:37:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.853+0000 7fad38c00700 1 -- 172.21.15.12:0/2829978827 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 9642+0+0 (secure 0 0 0) 0x7fad3c0a9fd0 con 0x55e185b5e9f0 2024-05-27T07:37:06.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.854+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad4801b040 con 0x55e185b5e9f0 2024-05-27T07:37:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.854+0000 7fad38c00700 1 -- 172.21.15.12:0/2829978827 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fad3c0a94b0 con 0x55e185b5e9f0 2024-05-27T07:37:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.854+0000 7fad64558b80 10 client.36635 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:06.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.854+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55e185c03d90 con 0x55e185b5e9f0 2024-05-27T07:37:06.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad38c00700 1 -- 172.21.15.12:0/2829978827 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad4801b040 con 0x55e185b5e9f0 2024-05-27T07:37:06.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad38c00700 1 -- 172.21.15.12:0/2829978827 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fad3c0b2530 con 0x55e185b5e9f0 2024-05-27T07:37:06.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad64558b80 10 client.36635 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:06.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad64558b80 10 client.36635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:06.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad64558b80 20 client.36635 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:06.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad64558b80 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c04120 0x55e185c04530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:06.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad64558b80 4 client.36635 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:37:06.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.855+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e185b1b570 con 0x55e185c04120 2024-05-27T07:37:06.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.856+0000 7fad4ea00700 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c04120 0x55e185c04530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:06.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.857+0000 7fad4ea00700 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c04120 0x55e185c04530 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3174dbb51b29e server_cookie=6202da034a643aa5 in_seq=0 out_seq=0 2024-05-27T07:37:06.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.857+0000 7fad38c00700 10 client.36635 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:06.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.857+0000 7fad38c00700 1 -- 172.21.15.12:0/2829978827 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e185b1b570 con 0x55e185c04120 2024-05-27T07:37:06.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad38c00700 10 client.36635 handle_command_reply: tid=0 2024-05-27T07:37:06.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad64558b80 1 client.36635 shutdown 2024-05-27T07:37:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 msgr2=0x55e185bf5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad64558b80 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf5040 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad3c0e3000 tx=0x7fad3c0092d0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 shutdown_connections 2024-05-27T07:37:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad64558b80 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e185bf1e70 0x55e185bf5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad64558b80 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e185a72e40 0x55e185bf6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.858+0000 7fad64558b80 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55e185c04120 0x55e185c04530 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.859+0000 7fad64558b80 1 --2- 172.21.15.12:0/2829978827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e185b5e9f0 0x55e185bf5040 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.859+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 >> 172.21.15.12:0/2829978827 conn(0x55e185b0c390 msgr2=0x55e185bfcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.859+0000 7fad38c00700 0 client.36635 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.859+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 shutdown_connections 2024-05-27T07:37:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.859+0000 7fad64558b80 1 -- 172.21.15.12:0/2829978827 wait complete. 2024-05-27T07:37:06.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.859+0000 7fad64558b80 20 client.36635 trim_cache size 0 max 16384 2024-05-27T07:37:06.857 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:37:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad340b29f0 msgr2=0x7fad340b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad340b29f0 0x7fad340b4f00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55e185bf3ff0 tx=0x7fad48004be0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad50108f10 msgr2=0x7fad5019d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad50108f10 0x7fad5019d5c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00e970 tx=0x7fad3c00ec80 comp rx=0 tx=0).stop 2024-05-27T07:37:06.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 shutdown_connections 2024-05-27T07:37:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad340b29f0 0x7fad340b4f00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad50109860 0x7fad501a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.860+0000 7fad55a00700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad501045e0 0x7fad5019d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.861+0000 7fad55a00700 1 --2- 172.21.15.12:0/1122533053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad50108f10 0x7fad5019d5c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.861+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 >> 172.21.15.12:0/1122533053 conn(0x7fad50100030 msgr2=0x7fad5010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.861+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 shutdown_connections 2024-05-27T07:37:06.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:06.861+0000 7fad55a00700 1 -- 172.21.15.12:0/1122533053 wait complete. 2024-05-27T07:37:07.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:07 smithi012 ceph-mon[31473]: pgmap v504: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 991 KiB/s wr, 134 op/s 2024-05-27T07:37:07.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:07 smithi012 ceph-mon[31473]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:37:07.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:07 smithi162 ceph-mon[35676]: pgmap v504: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 991 KiB/s wr, 134 op/s 2024-05-27T07:37:07.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:07 smithi162 ceph-mon[35676]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:37:07.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:07 smithi062 ceph-mon[35304]: pgmap v504: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 991 KiB/s wr, 134 op/s 2024-05-27T07:37:07.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:07 smithi062 ceph-mon[35304]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:37:08.978 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:37:08.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:37:09.349 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:09 smithi012 ceph-mon[31473]: pgmap v506: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-05-27T07:37:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:09.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:09 smithi162 ceph-mon[35676]: pgmap v506: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-05-27T07:37:09.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:09 smithi062 ceph-mon[35304]: pgmap v506: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-05-27T07:37:11.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:11 smithi012 ceph-mon[31473]: pgmap v507: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2024-05-27T07:37:11.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:11 smithi162 ceph-mon[35676]: pgmap v507: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2024-05-27T07:37:11.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:11 smithi062 ceph-mon[35304]: pgmap v507: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2024-05-27T07:37:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 -- 172.21.15.12:0/1861484089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e0109860 msgr2=0x7fa0e01111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/1861484089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e0109860 0x7fa0e01111c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800b0d0 tx=0x7fa0d800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:37:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 -- 172.21.15.12:0/1861484089 shutdown_connections 2024-05-27T07:37:11.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/1861484089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e0109860 0x7fa0e01111c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/1861484089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0e01045e0 0x7fa0e01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/1861484089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e0108f10 0x7fa0e0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 -- 172.21.15.12:0/1861484089 >> 172.21.15.12:0/1861484089 conn(0x7fa0e0100030 msgr2=0x7fa0e0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 -- 172.21.15.12:0/1861484089 shutdown_connections 2024-05-27T07:37:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.992+0000 7fa0e7e00700 1 -- 172.21.15.12:0/1861484089 wait complete. 2024-05-27T07:37:11.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.994+0000 7fa0e7e00700 1 Processor -- start 2024-05-27T07:37:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.994+0000 7fa0e7e00700 1 -- start start 2024-05-27T07:37:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.995+0000 7fa0e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e01045e0 0x7fa0e0075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:11.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.995+0000 7fa0e7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0e0108f10 0x7fa0e0075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.995+0000 7fa0e7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e0109860 0x7fa0e0076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.995+0000 7fa0e7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e0114340 con 0x7fa0e01045e0 2024-05-27T07:37:11.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.995+0000 7fa0e7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e01141c0 con 0x7fa0e0109860 2024-05-27T07:37:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.995+0000 7fa0e7e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e01144c0 con 0x7fa0e0108f10 2024-05-27T07:37:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.995+0000 7fa0e5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e01045e0 0x7fa0e0075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e01045e0 0x7fa0e0075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44778/0 (socket says 172.21.15.12:44778) 2024-05-27T07:37:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5400700 1 -- 172.21.15.12:0/539659198 learned_addr learned my addr 172.21.15.12:0/539659198 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0e0108f10 0x7fa0e0075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e0109860 0x7fa0e0076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5400700 1 -- 172.21.15.12:0/539659198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0e0108f10 msgr2=0x7fa0e0075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5400700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0e0108f10 0x7fa0e0075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5400700 1 -- 172.21.15.12:0/539659198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e0109860 msgr2=0x7fa0e0076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5400700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e0109860 0x7fa0e0076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.996+0000 7fa0e5400700 1 -- 172.21.15.12:0/539659198 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e01a90f0 con 0x7fa0e01045e0 2024-05-27T07:37:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.997+0000 7fa0e5400700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e01045e0 0x7fa0e0075ab0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400d950 tx=0x7fa0d400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.997+0000 7fa0de000700 1 -- 172.21.15.12:0/539659198 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4016070 con 0x7fa0e01045e0 2024-05-27T07:37:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.997+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e01a93e0 con 0x7fa0e01045e0 2024-05-27T07:37:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.997+0000 7fa0de000700 1 -- 172.21.15.12:0/539659198 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0d400fd20 con 0x7fa0e01045e0 2024-05-27T07:37:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.997+0000 7fa0de000700 1 -- 172.21.15.12:0/539659198 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400e3b0 con 0x7fa0e01045e0 2024-05-27T07:37:11.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:11.998+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e01a9970 con 0x7fa0e01045e0 2024-05-27T07:37:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.002+0000 7fa0de000700 1 -- 172.21.15.12:0/539659198 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa0d4019070 con 0x7fa0e01045e0 2024-05-27T07:37:11.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.003+0000 7fa0f68b0b80 1 Processor -- start 2024-05-27T07:37:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.003+0000 7fa0f68b0b80 1 -- start start 2024-05-27T07:37:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed567eec0 0x55bed5803870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed57fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed57fe240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55bed572bec0 con 0x55bed567eec0 2024-05-27T07:37:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bed571ba60 con 0x55bed57fde30 2024-05-27T07:37:12.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bed571faa0 con 0x55bed5803db0 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0de000700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa0c80b2a70 0x7fa0c80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0e5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed567eec0 0x55bed5803870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0e5e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed567eec0 0x55bed5803870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44780/0 (socket says 172.21.15.12:44780) 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0e5e00700 1 -- 172.21.15.12:0/713094453 learned_addr learned my addr 172.21.15.12:0/713094453 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed57fe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0de000700 1 -- 172.21.15.12:0/539659198 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 9642+0+0 (secure 0 0 0) 0x7fa0d40e2590 con 0x7fa0e01045e0 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.004+0000 7fa0e5400700 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed57fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.005+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa0c80b2a70 0x7fa0c80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.005+0000 7fa0e5e00700 1 -- 172.21.15.12:0/713094453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 msgr2=0x55bed57fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.005+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed57fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.005+0000 7fa0e5e00700 1 -- 172.21.15.12:0/713094453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 msgr2=0x55bed57fe240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.005+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed57fe240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.005+0000 7fa0e5e00700 1 -- 172.21.15.12:0/713094453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bed56e44a0 con 0x55bed567eec0 2024-05-27T07:37:12.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.005+0000 7fa0e5400700 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed57fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.006+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed567eec0 0x55bed5803870 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009490 tx=0x7fa0d80094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f708aa1cd791d2ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.006+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa0c80b2a70 0x7fa0c80b4f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005000 tx=0x7fa0d8028000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.006+0000 7fa0d2a00700 1 -- 172.21.15.12:0/713094453 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8007800 con 0x55bed567eec0 2024-05-27T07:37:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.006+0000 7fa0d2a00700 1 -- 172.21.15.12:0/713094453 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0d80050a0 con 0x55bed567eec0 2024-05-27T07:37:12.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.006+0000 7fa0d2a00700 1 -- 172.21.15.12:0/713094453 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d801c6a0 con 0x55bed567eec0 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/713094453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed567eec0 msgr2=0x55bed5803870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed567eec0 0x55bed5803870 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009490 tx=0x7fa0d80094c0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/713094453 shutdown_connections 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed57fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed567eec0 0x55bed5803870 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/713094453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed57fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/713094453 >> 172.21.15.12:0/713094453 conn(0x55bed56df040 msgr2=0x55bed57f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.007+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/713094453 shutdown_connections 2024-05-27T07:37:12.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.008+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/713094453 wait complete. 2024-05-27T07:37:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.008+0000 7fa0f68b0b80 1 -- start start 2024-05-27T07:37:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.008+0000 7fa0f68b0b80 1 Processor -- start 2024-05-27T07:37:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed5807640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5807b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed58080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55bed5622020 con 0x55bed57fde30 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bed56bc7e0 con 0x55bed567eec0 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bed580a790 con 0x55bed5803db0 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0f68b0b80 10 client.0 _ll_register_callbacks cb 0x55bed56c5cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0e4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5807b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0e4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5807b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44790/0 (socket says 172.21.15.12:44790) 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.009+0000 7fa0e4a00700 1 -- 172.21.15.12:0/818028172 learned_addr learned my addr 172.21.15.12:0/818028172 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e5400700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed5807640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed58080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e4a00700 1 -- 172.21.15.12:0/818028172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 msgr2=0x55bed58080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed58080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e4a00700 1 -- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 msgr2=0x55bed5807640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed5807640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e4a00700 1 -- 172.21.15.12:0/818028172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bed566b950 con 0x55bed57fde30 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.010+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed58080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5807b80 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00bf90 tx=0x7fa0cc011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92627aa2b53afc2b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0c7400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0c7400700 1 -- 172.21.15.12:0/818028172 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc004830 con 0x55bed57fde30 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0d80050a0 con 0x55bed57fde30 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0c7400700 1 -- 172.21.15.12:0/818028172 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 9642+0+0 (secure 0 0 0) 0x7fa0cc0049d0 con 0x55bed57fde30 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0c7400700 1 -- 172.21.15.12:0/818028172 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa0cc025050 con 0x55bed57fde30 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0f68b0b80 10 client.17208 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.011+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55bed56ec0e0 con 0x55bed57fde30 2024-05-27T07:37:12.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0c7400700 1 -- 172.21.15.12:0/818028172 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0d80050a0 con 0x55bed57fde30 2024-05-27T07:37:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0c7400700 1 -- 172.21.15.12:0/818028172 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa0cc01a780 con 0x55bed57fde30 2024-05-27T07:37:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0f68b0b80 10 client.17208 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0f68b0b80 10 client.17208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0f68b0b80 20 client.17208 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5817b10 0x55bed5819f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0f68b0b80 4 client.17208 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:12.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.012+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bed572de10 con 0x55bed5817b10 2024-05-27T07:37:12.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.013+0000 7fa0e5400700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5817b10 0x55bed5819f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.014+0000 7fa0e5400700 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5817b10 0x55bed5819f60 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a57ff574519a067b server_cookie=72e3c898457fc3ad in_seq=0 out_seq=0 2024-05-27T07:37:12.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.014+0000 7fa0c7400700 10 client.17208 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0c7400700 1 -- 172.21.15.12:0/818028172 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bed572de10 con 0x55bed5817b10 2024-05-27T07:37:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0c7400700 10 client.17208 handle_command_reply: tid=0 2024-05-27T07:37:12.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 client.17208 shutdown 2024-05-27T07:37:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 msgr2=0x55bed5807b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5807b80 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00bf90 tx=0x7fa0cc011720 comp rx=0 tx=0).stop 2024-05-27T07:37:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 shutdown_connections 2024-05-27T07:37:12.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5817b10 0x55bed5819f60 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed5807640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed58080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/818028172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5807b80 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0c7400700 0 client.17208 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.017+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 >> 172.21.15.12:0/818028172 conn(0x55bed56df040 msgr2=0x55bed57f8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.018+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 shutdown_connections 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.018+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/818028172 wait complete. 2024-05-27T07:37:12.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.018+0000 7fa0f68b0b80 20 client.17208 trim_cache size 0 max 16384 2024-05-27T07:37:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.050+0000 7fa0f68b0b80 1 Processor -- start 2024-05-27T07:37:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.050+0000 7fa0f68b0b80 1 -- start start 2024-05-27T07:37:12.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.051+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed57fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.051+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5809a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.051+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed5807e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.051+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55bed58029c0 con 0x55bed57fde30 2024-05-27T07:37:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.051+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e01b1f50 con 0x55bed567eec0 2024-05-27T07:37:12.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.051+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e01b11b0 con 0x55bed5803db0 2024-05-27T07:37:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5809a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5809a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44806/0 (socket says 172.21.15.12:44806) 2024-05-27T07:37:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 -- 172.21.15.12:0/1911688982 learned_addr learned my addr 172.21.15.12:0/1911688982 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed5807e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e5400700 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed57fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 -- 172.21.15.12:0/1911688982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 msgr2=0x55bed5807e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed5807e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 -- 172.21.15.12:0/1911688982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 msgr2=0x55bed57fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed57fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e4a00700 1 -- 172.21.15.12:0/1911688982 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bed5805bb0 con 0x55bed57fde30 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.052+0000 7fa0e5400700 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed57fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.053+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5809a90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc01ac70 tx=0x7fa0cc011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1528f51e136d4b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.053+0000 7fa0d2000700 1 -- 172.21.15.12:0/1911688982 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc02d070 con 0x55bed57fde30 2024-05-27T07:37:12.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.053+0000 7fa0d2000700 1 -- 172.21.15.12:0/1911688982 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0cc005570 con 0x55bed57fde30 2024-05-27T07:37:12.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.053+0000 7fa0d2000700 1 -- 172.21.15.12:0/1911688982 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc0038e0 con 0x55bed57fde30 2024-05-27T07:37:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.055+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1911688982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 msgr2=0x55bed5809a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.055+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5809a90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc01ac70 tx=0x7fa0cc011be0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.055+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1911688982 shutdown_connections 2024-05-27T07:37:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.055+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed567eec0 0x55bed57fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.055+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed5803db0 0x55bed5807e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.055+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1911688982 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed57fde30 0x55bed5809a90 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.055+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1911688982 >> 172.21.15.12:0/1911688982 conn(0x55bed56df040 msgr2=0x55bed5801b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.056+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1911688982 shutdown_connections 2024-05-27T07:37:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.056+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1911688982 wait complete. 2024-05-27T07:37:12.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.057+0000 7fa0f68b0b80 1 -- start start 2024-05-27T07:37:12.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.057+0000 7fa0f68b0b80 1 Processor -- start 2024-05-27T07:37:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.058+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed567eec0 0x55bed580fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.058+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed5810120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.058+0000 7fa0f68b0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed5803db0 0x55bed580c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.058+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55bed572de10 con 0x55bed5803db0 2024-05-27T07:37:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.058+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e01b1880 con 0x55bed57fde30 2024-05-27T07:37:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.058+0000 7fa0f68b0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bed58145c0 con 0x55bed567eec0 2024-05-27T07:37:12.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.058+0000 7fa0f68b0b80 10 client.0 _ll_register_callbacks cb 0x55bed56c5cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed567eec0 0x55bed580fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed5810120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed567eec0 0x55bed580fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:49158/0 (socket says 172.21.15.12:49158) 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 -- 172.21.15.12:0/1870420644 learned_addr learned my addr 172.21.15.12:0/1870420644 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5e00700 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed5803db0 0x55bed580c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 -- 172.21.15.12:0/1870420644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 msgr2=0x55bed5810120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed5810120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 -- 172.21.15.12:0/1870420644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed5803db0 msgr2=0x55bed580c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:12.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed5803db0 0x55bed580c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.059+0000 7fa0e5400700 1 -- 172.21.15.12:0/1870420644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bed580d090 con 0x55bed567eec0 2024-05-27T07:37:12.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.060+0000 7fa0e5400700 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed567eec0 0x55bed580fbe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40a6620 tx=0x7fa0d4005910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=948e1d57aa1d364 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:12.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.060+0000 7fa0c7400700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.060+0000 7fa0c7400700 1 -- 172.21.15.12:0/1870420644 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4020040 con 0x55bed567eec0 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.060+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0cc005570 con 0x55bed567eec0 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.060+0000 7fa0c7400700 1 -- 172.21.15.12:0/1870420644 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 9642+0+0 (secure 0 0 0) 0x7fa0d40e2330 con 0x55bed567eec0 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.060+0000 7fa0c7400700 1 -- 172.21.15.12:0/1870420644 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa0d40a9ab0 con 0x55bed567eec0 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.061+0000 7fa0f68b0b80 10 client.26659 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.061+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55bed5882f80 con 0x55bed567eec0 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.061+0000 7fa0c7400700 1 -- 172.21.15.12:0/1870420644 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0cc005570 con 0x55bed567eec0 2024-05-27T07:37:12.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.061+0000 7fa0c7400700 1 -- 172.21.15.12:0/1870420644 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa0d40a96a0 con 0x55bed567eec0 2024-05-27T07:37:12.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.061+0000 7fa0f68b0b80 10 client.26659 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:12.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.061+0000 7fa0f68b0b80 10 client.26659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:12.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.061+0000 7fa0f68b0b80 20 client.26659 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:12.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.062+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5883610 0x55bed5885a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:12.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.062+0000 7fa0f68b0b80 4 client.26659 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:37:12.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.062+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bed57275f0 con 0x55bed5883610 2024-05-27T07:37:12.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.062+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5883610 0x55bed5885a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:12.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.063+0000 7fa0c7400700 10 client.26659 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:12.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.063+0000 7fa0e4a00700 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5883610 0x55bed5885a30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe27cbb3d4f8740 server_cookie=faf34e2a8b582b55 in_seq=0 out_seq=0 2024-05-27T07:37:12.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.066+0000 7fa0c7400700 1 -- 172.21.15.12:0/1870420644 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bed57275f0 con 0x55bed5883610 2024-05-27T07:37:12.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.066+0000 7fa0c7400700 10 client.26659 handle_command_reply: tid=0 2024-05-27T07:37:12.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.066+0000 7fa0f68b0b80 1 client.26659 shutdown 2024-05-27T07:37:12.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.066+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed567eec0 msgr2=0x55bed580fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.066+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed567eec0 0x55bed580fbe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40a6620 tx=0x7fa0d4005910 comp rx=0 tx=0).stop 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 shutdown_connections 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55bed5803db0 0x55bed580c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bed567eec0 0x55bed580fbe0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55bed5883610 0x55bed5885a30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 --2- 172.21.15.12:0/1870420644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bed57fde30 0x55bed5810120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 >> 172.21.15.12:0/1870420644 conn(0x55bed56df040 msgr2=0x55bed5801b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0c7400700 0 client.26659 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 shutdown_connections 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 1 -- 172.21.15.12:0/1870420644 wait complete. 2024-05-27T07:37:12.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.067+0000 7fa0f68b0b80 20 client.26659 trim_cache size 0 max 16384 2024-05-27T07:37:12.065 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:37:12.065 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:37:12.065 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "fd92a7e2-38d1-475b-84a7-ba90ff5306c5", 2024-05-27T07:37:12.065 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:37:12.065 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa0c80b2a70 msgr2=0x7fa0c80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa0c80b2a70 0x7fa0c80b4f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005000 tx=0x7fa0d8028000 comp rx=0 tx=0).stop 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e01045e0 msgr2=0x7fa0e0075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e01045e0 0x7fa0e0075ab0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400d950 tx=0x7fa0d400dc60 comp rx=0 tx=0).stop 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 shutdown_connections 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa0c80b2a70 0x7fa0c80b4f80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e0109860 0x7fa0e0076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0e01045e0 0x7fa0e0075ab0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 --2- 172.21.15.12:0/539659198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0e0108f10 0x7fa0e0075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:12.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.068+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 >> 172.21.15.12:0/539659198 conn(0x7fa0e0100030 msgr2=0x7fa0e01086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:12.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.069+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 shutdown_connections 2024-05-27T07:37:12.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:12.069+0000 7fa0e7e00700 1 -- 172.21.15.12:0/539659198 wait complete. 2024-05-27T07:37:13.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: pgmap v508: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:13 smithi012 ceph-mon[31473]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:37:13.640 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:13.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: pgmap v508: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:13 smithi162 ceph-mon[35676]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:37:13.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:37:13.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:37:13.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:37:13.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:37:13.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: pgmap v508: 129 pgs: 129 active+clean; 283 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2024-05-27T07:37:13.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:13.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:13 smithi062 ceph-mon[35304]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:37:14.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:14 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:14.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:14 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:14 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:37:15.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:15 smithi012 ceph-mon[31473]: pgmap v510: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-05-27T07:37:15.631 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:15 smithi162 ceph-mon[35676]: pgmap v510: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-05-27T07:37:15.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:15 smithi062 ceph-mon[35304]: pgmap v510: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.278+0000 7f4511400700 1 -- 172.21.15.12:0/262109191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 msgr2=0x7f450c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.278+0000 7f4511400700 1 --2- 172.21.15.12:0/262109191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c109320 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f44f80099c0 tx=0x7f44f8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.278+0000 7f4511400700 1 -- 172.21.15.12:0/262109191 shutdown_connections 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.278+0000 7f4511400700 1 --2- 172.21.15.12:0/262109191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f450c109860 0x7f450c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.278+0000 7f4511400700 1 --2- 172.21.15.12:0/262109191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f450c1045e0 0x7f450c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.278+0000 7f4511400700 1 --2- 172.21.15.12:0/262109191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c109320 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.278+0000 7f4511400700 1 -- 172.21.15.12:0/262109191 >> 172.21.15.12:0/262109191 conn(0x7f450c100030 msgr2=0x7f450c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.279+0000 7f4511400700 1 -- 172.21.15.12:0/262109191 shutdown_connections 2024-05-27T07:37:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.279+0000 7f4511400700 1 -- 172.21.15.12:0/262109191 wait complete. 2024-05-27T07:37:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.280+0000 7f4511400700 1 Processor -- start 2024-05-27T07:37:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.280+0000 7f4511400700 1 -- start start 2024-05-27T07:37:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4511400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f450c1045e0 0x7f450c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4511400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4511400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f450c109860 0x7f450c1a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4511400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f450c114370 con 0x7f450c108f10 2024-05-27T07:37:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4511400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f450c1141f0 con 0x7f450c109860 2024-05-27T07:37:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4511400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f450c1144f0 con 0x7f450c1045e0 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4509e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4509e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c19d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44828/0 (socket says 172.21.15.12:44828) 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f4509e00700 1 -- 172.21.15.12:0/2586910053 learned_addr learned my addr 172.21.15.12:0/2586910053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f450a800700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f450c1045e0 0x7f450c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.281+0000 7f450b200700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f450c109860 0x7f450c1a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.282+0000 7f4509e00700 1 -- 172.21.15.12:0/2586910053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f450c1045e0 msgr2=0x7f450c19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.282+0000 7f4509e00700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f450c1045e0 0x7f450c19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.282+0000 7f4509e00700 1 -- 172.21.15.12:0/2586910053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f450c109860 msgr2=0x7f450c1a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.282+0000 7f4509e00700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f450c109860 0x7f450c1a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.282+0000 7f4509e00700 1 -- 172.21.15.12:0/2586910053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c1a1e70 con 0x7f450c108f10 2024-05-27T07:37:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.282+0000 7f4509e00700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c19d540 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f44f800b580 tx=0x7f44f80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.283+0000 7f4503400700 1 -- 172.21.15.12:0/2586910053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8027070 con 0x7f450c108f10 2024-05-27T07:37:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.283+0000 7f4503400700 1 -- 172.21.15.12:0/2586910053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44f800fd30 con 0x7f450c108f10 2024-05-27T07:37:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.283+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450c1a2100 con 0x7f450c108f10 2024-05-27T07:37:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.283+0000 7f4503400700 1 -- 172.21.15.12:0/2586910053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8022e50 con 0x7f450c108f10 2024-05-27T07:37:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.283+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450c1a25e0 con 0x7f450c108f10 2024-05-27T07:37:16.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.287+0000 7f4503400700 1 -- 172.21.15.12:0/2586910053 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f44f802a070 con 0x7f450c108f10 2024-05-27T07:37:16.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.288+0000 7f451ff7fb80 1 Processor -- start 2024-05-27T07:37:16.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.289+0000 7f451ff7fb80 1 -- start start 2024-05-27T07:37:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.289+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c493fc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.289+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493fcd30 0x556c493f6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.289+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c493f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.289+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556c49324e40 con 0x556c49277e40 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.289+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556c493149e0 con 0x556c493f6db0 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.289+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556c49318a20 con 0x556c493fcd30 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.290+0000 7f450b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c493fc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.290+0000 7f4503400700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f44f00b2b90 0x7f44f00b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.290+0000 7f450b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c493fc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44844/0 (socket says 172.21.15.12:44844) 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.290+0000 7f450b200700 1 -- 172.21.15.12:0/3301090604 learned_addr learned my addr 172.21.15.12:0/3301090604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:16.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.290+0000 7f4503400700 1 -- 172.21.15.12:0/2586910053 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f44f80e3a30 con 0x7f450c108f10 2024-05-27T07:37:16.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.290+0000 7f4509e00700 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c493f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.290+0000 7f450a800700 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493fcd30 0x556c493f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.291+0000 7f450b200700 1 -- 172.21.15.12:0/3301090604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493fcd30 msgr2=0x556c493f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.291+0000 7f450b200700 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493fcd30 0x556c493f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.291+0000 7f450b200700 1 -- 172.21.15.12:0/3301090604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 msgr2=0x556c493f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.291+0000 7f450b200700 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c493f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.291+0000 7f450b200700 1 -- 172.21.15.12:0/3301090604 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c492dd420 con 0x556c49277e40 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.291+0000 7f450b200700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f44f00b2b90 0x7f44f00b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.291+0000 7f450b200700 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c493fc7f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4504003f00 tx=0x7f4504009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683ffbcce05a6370 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.292+0000 7f44f7e00700 1 -- 172.21.15.12:0/3301090604 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450401b040 con 0x556c49277e40 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.292+0000 7f44f7e00700 1 -- 172.21.15.12:0/3301090604 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4504007d70 con 0x556c49277e40 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.292+0000 7f450b200700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f44f00b2b90 0x7f44f00b50a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4504021000 tx=0x7f4504022000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.292+0000 7f44f7e00700 1 -- 172.21.15.12:0/3301090604 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504022aa0 con 0x556c49277e40 2024-05-27T07:37:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.293+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3301090604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 msgr2=0x556c493fc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.293+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c493fc7f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4504003f00 tx=0x7f4504009ba0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.293+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3301090604 shutdown_connections 2024-05-27T07:37:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.293+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c493f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.293+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c493fc7f0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.293+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3301090604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493fcd30 0x556c493f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.293+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3301090604 >> 172.21.15.12:0/3301090604 conn(0x556c492d7fc0 msgr2=0x556c493f18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.294+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3301090604 shutdown_connections 2024-05-27T07:37:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.294+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3301090604 wait complete. 2024-05-27T07:37:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.294+0000 7f451ff7fb80 1 -- start start 2024-05-27T07:37:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.295+0000 7f451ff7fb80 1 Processor -- start 2024-05-27T07:37:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.295+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c494005a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.295+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493f6db0 0x556c49400ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.295+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c49401020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.295+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556c4921afa0 con 0x556c49277e40 2024-05-27T07:37:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.295+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556c492b5760 con 0x556c493fcd30 2024-05-27T07:37:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.296+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556c49403800 con 0x556c493f6db0 2024-05-27T07:37:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.296+0000 7f450a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c494005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.296+0000 7f450a800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c494005a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44860/0 (socket says 172.21.15.12:44860) 2024-05-27T07:37:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.296+0000 7f450a800700 1 -- 172.21.15.12:0/456719777 learned_addr learned my addr 172.21.15.12:0/456719777 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.296+0000 7f451ff7fb80 10 client.0 _ll_register_callbacks cb 0x556c492bec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.296+0000 7f4509e00700 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493f6db0 0x556c49400ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.297+0000 7f450b200700 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c49401020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.297+0000 7f450a800700 1 -- 172.21.15.12:0/456719777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493f6db0 msgr2=0x556c49400ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.297+0000 7f450a800700 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493f6db0 0x556c49400ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.297+0000 7f450a800700 1 -- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 msgr2=0x556c49401020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.297+0000 7f450a800700 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c49401020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.297+0000 7f450a800700 1 -- 172.21.15.12:0/456719777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c492648d0 con 0x556c49277e40 2024-05-27T07:37:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.298+0000 7f450a800700 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c494005a0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f44fc012450 tx=0x7f44fc012810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0c30409639526b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.298+0000 7f44f4c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:37:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.300+0000 7f44f4c00700 1 -- 172.21.15.12:0/456719777 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0107e0 con 0x556c49277e40 2024-05-27T07:37:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.301+0000 7f44f4c00700 1 -- 172.21.15.12:0/456719777 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f44fc010980 con 0x556c49277e40 2024-05-27T07:37:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.301+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4504007d70 con 0x556c49277e40 2024-05-27T07:37:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.303+0000 7f44f4c00700 1 -- 172.21.15.12:0/456719777 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f44fc010b80 con 0x556c49277e40 2024-05-27T07:37:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.303+0000 7f451ff7fb80 10 client.17226 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.303+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x556c492e5060 con 0x556c49277e40 2024-05-27T07:37:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f44f4c00700 1 -- 172.21.15.12:0/456719777 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4504007d70 con 0x556c49277e40 2024-05-27T07:37:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f44f4c00700 1 -- 172.21.15.12:0/456719777 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f44fc019070 con 0x556c49277e40 2024-05-27T07:37:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f451ff7fb80 10 client.17226 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:16.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f451ff7fb80 10 client.17226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:16.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f451ff7fb80 20 client.17226 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:16.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c49410bb0 0x556c49413060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f451ff7fb80 4 client.17226 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:16.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.304+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c49326d90 con 0x556c49410bb0 2024-05-27T07:37:16.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.306+0000 7f4509e00700 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c49410bb0 0x556c49413060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.309+0000 7f44f4c00700 10 client.17226 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.309+0000 7f4509e00700 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c49410bb0 0x556c49413060 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=912d23af10513e0b server_cookie=92b41c0b126c1e34 in_seq=0 out_seq=0 2024-05-27T07:37:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.313+0000 7f44f4c00700 1 -- 172.21.15.12:0/456719777 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556c49326d90 con 0x556c49410bb0 2024-05-27T07:37:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.313+0000 7f44f4c00700 10 client.17226 handle_command_reply: tid=0 2024-05-27T07:37:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.313+0000 7f451ff7fb80 1 client.17226 shutdown 2024-05-27T07:37:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.314+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 msgr2=0x556c494005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.314+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c494005a0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f44fc012450 tx=0x7f44fc012810 comp rx=0 tx=0).stop 2024-05-27T07:37:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.315+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 shutdown_connections 2024-05-27T07:37:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.315+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c49410bb0 0x556c49413060 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.315+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c49401020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.315+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c49277e40 0x556c494005a0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.315+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/456719777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c493f6db0 0x556c49400ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.315+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 >> 172.21.15.12:0/456719777 conn(0x556c492d7fc0 msgr2=0x556c493f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.315+0000 7f44f4c00700 0 client.17226 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.316+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 shutdown_connections 2024-05-27T07:37:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.316+0000 7f451ff7fb80 1 -- 172.21.15.12:0/456719777 wait complete. 2024-05-27T07:37:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.317+0000 7f451ff7fb80 20 client.17226 trim_cache size 0 max 16384 2024-05-27T07:37:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.350+0000 7f451ff7fb80 1 Processor -- start 2024-05-27T07:37:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.350+0000 7f451ff7fb80 1 -- start start 2024-05-27T07:37:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 0x556c493f6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493f6db0 0x556c49413290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c493fac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f450c1b1e20 con 0x556c493f6db0 2024-05-27T07:37:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f450c1b1080 con 0x556c493fcd30 2024-05-27T07:37:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f450c1b1750 con 0x556c49277e40 2024-05-27T07:37:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f4509e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493f6db0 0x556c49413290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f4509e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493f6db0 0x556c49413290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44864/0 (socket says 172.21.15.12:44864) 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f4509e00700 1 -- 172.21.15.12:0/3119178834 learned_addr learned my addr 172.21.15.12:0/3119178834 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.351+0000 7f450b200700 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c493fac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f4509e00700 1 -- 172.21.15.12:0/3119178834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 msgr2=0x556c493f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f4509e00700 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 0x556c493f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f4509e00700 1 -- 172.21.15.12:0/3119178834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 msgr2=0x556c493fac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f4509e00700 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c493fac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f4509e00700 1 -- 172.21.15.12:0/3119178834 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c493f7650 con 0x556c493f6db0 2024-05-27T07:37:16.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f4509e00700 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493f6db0 0x556c49413290 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f44f80e0cb0 tx=0x7f44f80e5e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9454ba763c25655 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f44f7400700 1 -- 172.21.15.12:0/3119178834 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80aa0a0 con 0x556c493f6db0 2024-05-27T07:37:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f44f7400700 1 -- 172.21.15.12:0/3119178834 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44f8090e50 con 0x556c493f6db0 2024-05-27T07:37:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.352+0000 7f44f7400700 1 -- 172.21.15.12:0/3119178834 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80a60c0 con 0x556c493f6db0 2024-05-27T07:37:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.353+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3119178834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493f6db0 msgr2=0x556c49413290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.353+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493f6db0 0x556c49413290 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f44f80e0cb0 tx=0x7f44f80e5e80 comp rx=0 tx=0).stop 2024-05-27T07:37:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.353+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3119178834 shutdown_connections 2024-05-27T07:37:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.353+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493fcd30 0x556c493fac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.353+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 0x556c493f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.354+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/3119178834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493f6db0 0x556c49413290 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.354+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3119178834 >> 172.21.15.12:0/3119178834 conn(0x556c492d7fc0 msgr2=0x556c49400380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.354+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3119178834 shutdown_connections 2024-05-27T07:37:16.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.354+0000 7f451ff7fb80 1 -- 172.21.15.12:0/3119178834 wait complete. 2024-05-27T07:37:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.354+0000 7f451ff7fb80 1 -- start start 2024-05-27T07:37:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 1 Processor -- start 2024-05-27T07:37:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 0x556c49418370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c494188b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493fcd30 0x556c49418df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556c493feb10 con 0x556c493fcd30 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556c49402ea0 con 0x556c493f6db0 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556c49401660 con 0x556c49277e40 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.355+0000 7f451ff7fb80 10 client.0 _ll_register_callbacks cb 0x556c492bec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493fcd30 0x556c49418df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493fcd30 0x556c49418df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44874/0 (socket says 172.21.15.12:44874) 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 -- 172.21.15.12:0/1677687674 learned_addr learned my addr 172.21.15.12:0/1677687674 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f4509e00700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c494188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450a800700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 0x556c49418370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 -- 172.21.15.12:0/1677687674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 msgr2=0x556c49418370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 0x556c49418370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 -- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 msgr2=0x556c494188b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c494188b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f450b200700 1 -- 172.21.15.12:0/1677687674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c4947a730 con 0x556c493fcd30 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.356+0000 7f4509e00700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c494188b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f450b200700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493fcd30 0x556c49418df0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f4504004190 tx=0x7f4504007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb92b2b649e6d39 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f44f4c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f44f4c00700 1 -- 172.21.15.12:0/1677687674 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450401b040 con 0x556c493fcd30 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f44f4c00700 1 -- 172.21.15.12:0/1677687674 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f4504022450 con 0x556c493fcd30 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44f8090e50 con 0x556c493fcd30 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f44f4c00700 1 -- 172.21.15.12:0/1677687674 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4504004e10 con 0x556c493fcd30 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f451ff7fb80 10 client.17232 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:16.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.357+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x556c4947b8c0 con 0x556c493fcd30 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f44f4c00700 1 -- 172.21.15.12:0/1677687674 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44f8090e50 con 0x556c493fcd30 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f44f4c00700 1 -- 172.21.15.12:0/1677687674 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f450401f070 con 0x556c493fcd30 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f451ff7fb80 10 client.17232 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f451ff7fb80 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f451ff7fb80 20 client.17232 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c4947bf60 0x556c4947e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f451ff7fb80 4 client.17232 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:37:16.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556c49320570 con 0x556c4947bf60 2024-05-27T07:37:16.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.358+0000 7f450a800700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c4947bf60 0x556c4947e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:16.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.359+0000 7f450a800700 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c4947bf60 0x556c4947e380 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e75026cae11e1ff server_cookie=4abc9eee9fdb4fb3 in_seq=0 out_seq=0 2024-05-27T07:37:16.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.360+0000 7f44f4c00700 10 client.17232 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f44f4c00700 1 -- 172.21.15.12:0/1677687674 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556c49320570 con 0x556c4947bf60 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f44f4c00700 10 client.17232 handle_command_reply: tid=0 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 client.17232 shutdown 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493fcd30 msgr2=0x556c49418df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493fcd30 0x556c49418df0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f4504004190 tx=0x7f4504007eb0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 shutdown_connections 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556c493fcd30 0x556c49418df0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556c49277e40 0x556c49418370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x556c4947bf60 0x556c4947e380 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 --2- 172.21.15.12:0/1677687674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556c493f6db0 0x556c494188b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 >> 172.21.15.12:0/1677687674 conn(0x556c492d7fc0 msgr2=0x556c493fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.364+0000 7f44f4c00700 0 client.17232 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.365+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 shutdown_connections 2024-05-27T07:37:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.365+0000 7f451ff7fb80 1 -- 172.21.15.12:0/1677687674 wait complete. 2024-05-27T07:37:16.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.365+0000 7f451ff7fb80 20 client.17232 trim_cache size 0 max 16384 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: "fd92a7e2-38d1-475b-84a7-ba90ff5306c5": { 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "fd92a7e2-38d1-475b-84a7-ba90ff5306c5", 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:37:16.363 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:37:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f44f00b2b90 msgr2=0x7f44f00b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f44f00b2b90 0x7f44f00b50a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4504021000 tx=0x7f4504022000 comp rx=0 tx=0).stop 2024-05-27T07:37:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 msgr2=0x7f450c19d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c19d540 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f44f800b580 tx=0x7f44f80057c0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 shutdown_connections 2024-05-27T07:37:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f44f00b2b90 0x7f44f00b50a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f450c109860 0x7f450c1a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f450c1045e0 0x7f450c19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 --2- 172.21.15.12:0/2586910053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f450c108f10 0x7f450c19d540 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 >> 172.21.15.12:0/2586910053 conn(0x7f450c100030 msgr2=0x7f450c10a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 shutdown_connections 2024-05-27T07:37:16.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:16.366+0000 7f4511400700 1 -- 172.21.15.12:0/2586910053 wait complete. 2024-05-27T07:37:17.232 INFO:tasks.cephfs.filesystem:scrub status for tag:fd92a7e2-38d1-475b-84a7-ba90ff5306c5 - {'path': '/', 'tag': 'fd92a7e2-38d1-475b-84a7-ba90ff5306c5', 'options': 'recursive,force'} 2024-05-27T07:37:17.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:17 smithi012 ceph-mon[31473]: pgmap v511: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 250 op/s 2024-05-27T07:37:17.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:17 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:17.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:17 smithi162 ceph-mon[35676]: pgmap v511: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 250 op/s 2024-05-27T07:37:17.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:17 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:17.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:17 smithi062 ceph-mon[35304]: pgmap v511: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 250 op/s 2024-05-27T07:37:17.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:17 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:18.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:18 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:18.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:18 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:18.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:18 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:37:18.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:18 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:18.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:18 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:18.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:18 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:37:18.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:18 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:18.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:18 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:18.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:18 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:37:19.082 INFO:tasks.workunit.client.0.smithi012.stdout:copying to largest ever: 0x63ee1da 2024-05-27T07:37:19.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:19 smithi012 ceph-mon[31473]: pgmap v512: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:37:19.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:19 smithi012 ceph-mon[31473]: pgmap v513: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1012 KiB/s wr, 169 op/s 2024-05-27T07:37:19.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:19 smithi062 ceph-mon[35304]: pgmap v512: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:37:19.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:19 smithi062 ceph-mon[35304]: pgmap v513: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1012 KiB/s wr, 169 op/s 2024-05-27T07:37:19.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:19 smithi162 ceph-mon[35676]: pgmap v512: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:37:19.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:19 smithi162 ceph-mon[35676]: pgmap v513: 129 pgs: 129 active+clean; 300 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1012 KiB/s wr, 169 op/s 2024-05-27T07:37:21.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:21 smithi162 ceph-mon[35676]: pgmap v514: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 350 op/s 2024-05-27T07:37:21.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:21 smithi062 ceph-mon[35304]: pgmap v514: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 350 op/s 2024-05-27T07:37:21.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:21 smithi012 ceph-mon[31473]: pgmap v514: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 350 op/s 2024-05-27T07:37:23.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:23 smithi162 ceph-mon[35676]: pgmap v515: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 290 op/s 2024-05-27T07:37:23.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:23 smithi062 ceph-mon[35304]: pgmap v515: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 290 op/s 2024-05-27T07:37:23.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:23 smithi012 ceph-mon[31473]: pgmap v515: 129 pgs: 129 active+clean; 306 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 290 op/s 2024-05-27T07:37:25.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:25 smithi162 ceph-mon[35676]: pgmap v516: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 266 op/s 2024-05-27T07:37:25.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:25 smithi062 ceph-mon[35304]: pgmap v516: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 266 op/s 2024-05-27T07:37:25.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:25 smithi012 ceph-mon[31473]: pgmap v516: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 266 op/s 2024-05-27T07:37:26.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:37:26.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:37:26.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:37:26.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:37:26.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:37:26.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:37:27.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:27 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:37:27.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:27 smithi162 ceph-mon[35676]: pgmap v517: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 266 op/s 2024-05-27T07:37:27.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:27 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:37:27.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:27 smithi062 ceph-mon[35304]: pgmap v517: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 266 op/s 2024-05-27T07:37:27.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:27 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:37:27.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:27 smithi012 ceph-mon[31473]: pgmap v517: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 266 op/s 2024-05-27T07:37:28.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:28 smithi012 ceph-mon[31473]: pgmap v518: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 261 op/s 2024-05-27T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:28 smithi162 ceph-mon[35676]: pgmap v518: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 261 op/s 2024-05-27T07:37:28.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:28 smithi062 ceph-mon[35304]: pgmap v518: 129 pgs: 129 active+clean; 289 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 261 op/s 2024-05-27T07:37:31.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:31 smithi012 ceph-mon[31473]: pgmap v519: 129 pgs: 129 active+clean; 318 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 306 op/s 2024-05-27T07:37:31.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:31 smithi162 ceph-mon[35676]: pgmap v519: 129 pgs: 129 active+clean; 318 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 306 op/s 2024-05-27T07:37:31.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:31 smithi062 ceph-mon[35304]: pgmap v519: 129 pgs: 129 active+clean; 318 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 306 op/s 2024-05-27T07:37:33.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:33 smithi162 ceph-mon[35676]: pgmap v520: 129 pgs: 129 active+clean; 318 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:37:33.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:33 smithi062 ceph-mon[35304]: pgmap v520: 129 pgs: 129 active+clean; 318 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:37:33.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:33 smithi012 ceph-mon[31473]: pgmap v520: 129 pgs: 129 active+clean; 318 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:37:35.484 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:35 smithi162 ceph-mon[35676]: pgmap v521: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 280 op/s 2024-05-27T07:37:35.496 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:35 smithi062 ceph-mon[35304]: pgmap v521: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 280 op/s 2024-05-27T07:37:35.601 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:35 smithi012 ceph-mon[31473]: pgmap v521: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 280 op/s 2024-05-27T07:37:37.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:37 smithi162 ceph-mon[35676]: pgmap v522: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:37.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:37 smithi062 ceph-mon[35304]: pgmap v522: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:37.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:37 smithi012 ceph-mon[31473]: pgmap v522: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:39.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:39 smithi162 ceph-mon[35676]: pgmap v523: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:39.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:39 smithi062 ceph-mon[35304]: pgmap v523: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:39.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:39 smithi012 ceph-mon[31473]: pgmap v523: 129 pgs: 129 active+clean; 314 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:41.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:41 smithi162 ceph-mon[35676]: pgmap v524: 129 pgs: 129 active+clean; 336 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 268 op/s 2024-05-27T07:37:41.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:41 smithi062 ceph-mon[35304]: pgmap v524: 129 pgs: 129 active+clean; 336 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 268 op/s 2024-05-27T07:37:41.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:41 smithi012 ceph-mon[31473]: pgmap v524: 129 pgs: 129 active+clean; 336 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 268 op/s 2024-05-27T07:37:43.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:43 smithi162 ceph-mon[35676]: pgmap v525: 129 pgs: 129 active+clean; 336 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:43.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:43 smithi062 ceph-mon[35304]: pgmap v525: 129 pgs: 129 active+clean; 336 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:43.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:43 smithi012 ceph-mon[31473]: pgmap v525: 129 pgs: 129 active+clean; 336 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:37:45.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:45 smithi162 ceph-mon[35676]: pgmap v526: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 267 op/s 2024-05-27T07:37:45.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:45 smithi062 ceph-mon[35304]: pgmap v526: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 267 op/s 2024-05-27T07:37:45.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:45 smithi012 ceph-mon[31473]: pgmap v526: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 267 op/s 2024-05-27T07:37:47.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:37:47.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:47 smithi162 ceph-mon[35676]: pgmap v527: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 175 op/s 2024-05-27T07:37:47.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:47 smithi062 ceph-mon[35304]: pgmap v527: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 175 op/s 2024-05-27T07:37:47.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:47 smithi012 ceph-mon[31473]: pgmap v527: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 175 op/s 2024-05-27T07:37:47.734 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:49.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:49 smithi162 ceph-mon[35676]: pgmap v528: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 175 op/s 2024-05-27T07:37:49.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:49 smithi062 ceph-mon[35304]: pgmap v528: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 175 op/s 2024-05-27T07:37:49.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:49 smithi012 ceph-mon[31473]: pgmap v528: 129 pgs: 129 active+clean; 312 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 175 op/s 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 -- 172.21.15.12:0/1770488645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4108690 msgr2=0x7f92d4108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1770488645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4108690 0x7f92d4108b60 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009990 tx=0x7f92cc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 -- 172.21.15.12:0/1770488645 shutdown_connections 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1770488645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4108690 0x7f92d4108b60 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1770488645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92d4103240 0x7f92d4103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1770488645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92d4103b90 0x7f92d41080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 -- 172.21.15.12:0/1770488645 >> 172.21.15.12:0/1770488645 conn(0x7f92d40fedb0 msgr2=0x7f92d41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 -- 172.21.15.12:0/1770488645 shutdown_connections 2024-05-27T07:37:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.311+0000 7f92d9a00700 1 -- 172.21.15.12:0/1770488645 wait complete. 2024-05-27T07:37:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.313+0000 7f92d9a00700 1 Processor -- start 2024-05-27T07:37:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.313+0000 7f92d9a00700 1 -- start start 2024-05-27T07:37:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.314+0000 7f92d9a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92d4103240 0x7f92d41a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.314+0000 7f92d9a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4103b90 0x7f92d41a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.314+0000 7f92d9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92d4108690 0x7f92d41a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.314+0000 7f92d9a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f92d411a910 con 0x7f92d4103b90 2024-05-27T07:37:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.314+0000 7f92d9a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f92d411a790 con 0x7f92d4108690 2024-05-27T07:37:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.314+0000 7f92d9a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f92d411aa90 con 0x7f92d4103240 2024-05-27T07:37:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.315+0000 7f92d2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4103b90 0x7f92d41a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.315+0000 7f92d2a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92d4103240 0x7f92d41a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.315+0000 7f92d2a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92d4103240 0x7f92d41a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:36440/0 (socket says 172.21.15.12:36440) 2024-05-27T07:37:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.315+0000 7f92d2a00700 1 -- 172.21.15.12:0/1532858735 learned_addr learned my addr 172.21.15.12:0/1532858735 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.315+0000 7f92d3400700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92d4108690 0x7f92d41a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.316+0000 7f92d2000700 1 -- 172.21.15.12:0/1532858735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92d4103240 msgr2=0x7f92d41a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.316+0000 7f92d2000700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92d4103240 0x7f92d41a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.316+0000 7f92d2000700 1 -- 172.21.15.12:0/1532858735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92d4108690 msgr2=0x7f92d41a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.316+0000 7f92d2000700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92d4108690 0x7f92d41a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.316+0000 7f92d2000700 1 -- 172.21.15.12:0/1532858735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d41a6170 con 0x7f92d4103b90 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.317+0000 7f92d2000700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4103b90 0x7f92d41a1840 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f92c000ef10 tx=0x7f92c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.317+0000 7f92cb400700 1 -- 172.21.15.12:0/1532858735 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c000cad0 con 0x7f92d4103b90 2024-05-27T07:37:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.317+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d41a6400 con 0x7f92d4103b90 2024-05-27T07:37:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.317+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d41add80 con 0x7f92d4103b90 2024-05-27T07:37:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.318+0000 7f92cb400700 1 -- 172.21.15.12:0/1532858735 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92c0019070 con 0x7f92d4103b90 2024-05-27T07:37:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.318+0000 7f92cb400700 1 -- 172.21.15.12:0/1532858735 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0014480 con 0x7f92d4103b90 2024-05-27T07:37:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.328+0000 7f92cb400700 1 -- 172.21.15.12:0/1532858735 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f92c0012050 con 0x7f92d4103b90 2024-05-27T07:37:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.329+0000 7f92e8499b80 1 Processor -- start 2024-05-27T07:37:50.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.329+0000 7f92e8499b80 1 -- start start 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.330+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123ab330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92e8499b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5603122d9e40 con 0x5603123abe70 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92e8499b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5603122c99e0 con 0x5603123189f0 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92e8499b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5603122cda20 con 0x56031222ce40 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92d3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92d3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:46890/0 (socket says 172.21.15.12:46890) 2024-05-27T07:37:50.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.331+0000 7f92d3400700 1 -- 172.21.15.12:0/1622612399 learned_addr learned my addr 172.21.15.12:0/1622612399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:50.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.332+0000 7f92d2a00700 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.332+0000 7f92d2000700 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.332+0000 7f92d3400700 1 -- 172.21.15.12:0/1622612399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 msgr2=0x5603123ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.332+0000 7f92d3400700 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.332+0000 7f92d3400700 1 -- 172.21.15.12:0/1622612399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 msgr2=0x5603123b8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.332+0000 7f92d3400700 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.332+0000 7f92d3400700 1 -- 172.21.15.12:0/1622612399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560312292420 con 0x5603123189f0 2024-05-27T07:37:50.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.333+0000 7f92d3400700 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123ab330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f92cc015000 tx=0x7f92cc0096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=651ce8638547c739 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.333+0000 7f92bfe00700 1 -- 172.21.15.12:0/1622612399 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc005500 con 0x5603123189f0 2024-05-27T07:37:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.333+0000 7f92bfe00700 1 -- 172.21.15.12:0/1622612399 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92cc0056a0 con 0x5603123189f0 2024-05-27T07:37:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.333+0000 7f92bfe00700 1 -- 172.21.15.12:0/1622612399 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc00bcb0 con 0x5603123189f0 2024-05-27T07:37:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.333+0000 7f92d2000700 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.334+0000 7f92cb400700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f92b80b2a70 0x7f92b80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92cb400700 1 -- 172.21.15.12:0/1532858735 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f92c00e29c0 con 0x7f92d4103b90 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 -- 172.21.15.12:0/1622612399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 msgr2=0x5603123ab330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123ab330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f92cc015000 tx=0x7f92cc0096a0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 -- 172.21.15.12:0/1622612399 shutdown_connections 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123ab330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92d2a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f92b80b2a70 0x7f92b80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1622612399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 -- 172.21.15.12:0/1622612399 >> 172.21.15.12:0/1622612399 conn(0x5603122c6390 msgr2=0x5603123a74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 -- 172.21.15.12:0/1622612399 shutdown_connections 2024-05-27T07:37:50.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.335+0000 7f92e8499b80 1 -- 172.21.15.12:0/1622612399 wait complete. 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.336+0000 7f92d2a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f92b80b2a70 0x7f92b80b4f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x5603123a8790 tx=0x7f92c40041a0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.336+0000 7f92e8499b80 1 -- start start 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.336+0000 7f92e8499b80 1 Processor -- start 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92e8499b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5603121cffa0 con 0x5603123abe70 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92e8499b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56031226a760 con 0x5603123189f0 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92e8499b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5603123b8b20 con 0x56031222ce40 2024-05-27T07:37:50.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92e8499b80 10 client.0 _ll_register_callbacks cb 0x560312273c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40058/0 (socket says 172.21.15.12:40058) 2024-05-27T07:37:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92d3400700 1 -- 172.21.15.12:0/547207954 learned_addr learned my addr 172.21.15.12:0/547207954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:50.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.337+0000 7f92d2a00700 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92d2000700 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92d3400700 1 -- 172.21.15.12:0/547207954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 msgr2=0x5603123b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92d3400700 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92d3400700 1 -- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 msgr2=0x5603123b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92d3400700 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92d3400700 1 -- 172.21.15.12:0/547207954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603122198d0 con 0x5603123abe70 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92d3400700 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b6090 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009990 tx=0x7f92cc003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0428b3e4082f09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.338+0000 7f92bcc00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.339+0000 7f92bcc00700 1 -- 172.21.15.12:0/547207954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc02f060 con 0x5603123abe70 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.339+0000 7f92bcc00700 1 -- 172.21.15.12:0/547207954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f92cc027070 con 0x5603123abe70 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.339+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92cc0056a0 con 0x5603123abe70 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.339+0000 7f92bcc00700 1 -- 172.21.15.12:0/547207954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f92cc036050 con 0x5603123abe70 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.339+0000 7f92e8499b80 10 client.17241 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.339+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56031229a060 con 0x5603123abe70 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92bcc00700 1 -- 172.21.15.12:0/547207954 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92cc0056a0 con 0x5603123abe70 2024-05-27T07:37:50.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92bcc00700 1 -- 172.21.15.12:0/547207954 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f92cc00fb30 con 0x5603123abe70 2024-05-27T07:37:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92e8499b80 10 client.17241 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92e8499b80 10 client.17241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92e8499b80 20 client.17241 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92e8499b80 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603123c59e0 0x5603123c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92e8499b80 4 client.17241 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.340+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603122dbd90 con 0x5603123c59e0 2024-05-27T07:37:50.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.341+0000 7f92d2a00700 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603123c59e0 0x5603123c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.342+0000 7f92d2a00700 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603123c59e0 0x5603123c7e90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=819f1d03e3d97e03 server_cookie=1b1085a7c72bf7fa in_seq=0 out_seq=0 2024-05-27T07:37:50.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.342+0000 7f92bcc00700 10 client.17241 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.343+0000 7f92bcc00700 1 -- 172.21.15.12:0/547207954 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5603122dbd90 con 0x5603123c59e0 2024-05-27T07:37:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.343+0000 7f92bcc00700 10 client.17241 handle_command_reply: tid=0 2024-05-27T07:37:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.343+0000 7f92e8499b80 1 client.17241 shutdown 2024-05-27T07:37:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.343+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 msgr2=0x5603123b6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.343+0000 7f92e8499b80 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b6090 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009990 tx=0x7f92cc003eb0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 shutdown_connections 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123abe70 0x5603123b6090 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56031222ce40 0x5603123b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603123c59e0 0x5603123c7e90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 --2- 172.21.15.12:0/547207954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603123189f0 0x5603123b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 >> 172.21.15.12:0/547207954 conn(0x5603122c6390 msgr2=0x5603123ab310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92bcc00700 0 client.17241 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:50.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 shutdown_connections 2024-05-27T07:37:50.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 1 -- 172.21.15.12:0/547207954 wait complete. 2024-05-27T07:37:50.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.344+0000 7f92e8499b80 20 client.17241 trim_cache size 0 max 16384 2024-05-27T07:37:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 Processor -- start 2024-05-27T07:37:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 -- start start 2024-05-27T07:37:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5603123ab420 con 0x5603123189f0 2024-05-27T07:37:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5603123b4790 con 0x56031222ce40 2024-05-27T07:37:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.373+0000 7f92e8499b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f92b400e410 con 0x5603123abe70 2024-05-27T07:37:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123c1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40068/0 (socket says 172.21.15.12:40068) 2024-05-27T07:37:50.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 -- 172.21.15.12:0/167925455 learned_addr learned my addr 172.21.15.12:0/167925455 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2a00700 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d3400700 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 -- 172.21.15.12:0/167925455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 msgr2=0x5603123b5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 -- 172.21.15.12:0/167925455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 msgr2=0x5603123a74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123a74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2000700 1 -- 172.21.15.12:0/167925455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603123abc70 con 0x5603123189f0 2024-05-27T07:37:50.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.374+0000 7f92d2a00700 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:50.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.375+0000 7f92d2000700 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123c1c50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f92c00e0300 tx=0x7f92c00a9660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824b23079ca747dc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.375+0000 7f92bf400700 1 -- 172.21.15.12:0/167925455 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c00a9000 con 0x5603123189f0 2024-05-27T07:37:50.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.375+0000 7f92bf400700 1 -- 172.21.15.12:0/167925455 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92c00a93a0 con 0x5603123189f0 2024-05-27T07:37:50.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.375+0000 7f92bf400700 1 -- 172.21.15.12:0/167925455 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c00a9af0 con 0x5603123189f0 2024-05-27T07:37:50.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.376+0000 7f92e8499b80 1 -- 172.21.15.12:0/167925455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 msgr2=0x5603123c1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.376+0000 7f92e8499b80 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123c1c50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f92c00e0300 tx=0x7f92c00a9660 comp rx=0 tx=0).stop 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.376+0000 7f92e8499b80 1 -- 172.21.15.12:0/167925455 shutdown_connections 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.376+0000 7f92e8499b80 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.376+0000 7f92e8499b80 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.376+0000 7f92e8499b80 1 --2- 172.21.15.12:0/167925455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123c1c50 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.376+0000 7f92e8499b80 1 -- 172.21.15.12:0/167925455 >> 172.21.15.12:0/167925455 conn(0x5603122c6390 msgr2=0x5603123b55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.377+0000 7f92e8499b80 1 -- 172.21.15.12:0/167925455 shutdown_connections 2024-05-27T07:37:50.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.377+0000 7f92e8499b80 1 -- 172.21.15.12:0/167925455 wait complete. 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.377+0000 7f92e8499b80 1 -- start start 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.377+0000 7f92e8499b80 1 Processor -- start 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123be540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92e8499b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92e8499b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f92b400dd40 con 0x5603123189f0 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92e8499b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f92b400d670 con 0x56031222ce40 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92e8499b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5603122d9e40 con 0x5603123abe70 2024-05-27T07:37:50.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92e8499b80 10 client.0 _ll_register_callbacks cb 0x560312273c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92d2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92d2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123b98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40074/0 (socket says 172.21.15.12:40074) 2024-05-27T07:37:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.378+0000 7f92d2000700 1 -- 172.21.15.12:0/1985334317 learned_addr learned my addr 172.21.15.12:0/1985334317 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.379+0000 7f92d3400700 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.379+0000 7f92d2a00700 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.379+0000 7f92d2000700 1 -- 172.21.15.12:0/1985334317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 msgr2=0x5603123b9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.379+0000 7f92d2000700 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.379+0000 7f92d2000700 1 -- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 msgr2=0x5603123be540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.379+0000 7f92d2000700 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123be540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.379+0000 7f92d2000700 1 -- 172.21.15.12:0/1985334317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603123ba560 con 0x5603123189f0 2024-05-27T07:37:50.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92d2000700 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123b98a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f92c00dfdb0 tx=0x7f92c00a9d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e12c80c2dcd4e26 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92bcc00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:37:50.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92bcc00700 1 -- 172.21.15.12:0/1985334317 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c00a9000 con 0x5603123189f0 2024-05-27T07:37:50.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92bcc00700 1 -- 172.21.15.12:0/1985334317 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f92c00b2470 con 0x5603123189f0 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92c00a93a0 con 0x5603123189f0 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92bcc00700 1 -- 172.21.15.12:0/1985334317 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f92c00a9890 con 0x5603123189f0 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92e8499b80 10 client.17247 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.380+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5603124303f0 con 0x5603123189f0 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92bcc00700 1 -- 172.21.15.12:0/1985334317 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92c00a93a0 con 0x5603123189f0 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92bcc00700 1 -- 172.21.15.12:0/1985334317 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f92c00bece0 con 0x5603123189f0 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92e8499b80 10 client.17247 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92e8499b80 10 client.17247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92e8499b80 20 client.17247 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603124309f0 0x560312432e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92e8499b80 4 client.17247 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:37:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.381+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5603122d5570 con 0x5603124309f0 2024-05-27T07:37:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.382+0000 7f92d2a00700 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603124309f0 0x560312432e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.382+0000 7f92d2a00700 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603124309f0 0x560312432e10 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d25e436d0be89a3 server_cookie=f39b1d2b4ac77d17 in_seq=0 out_seq=0 2024-05-27T07:37:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.382+0000 7f92bcc00700 10 client.17247 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92bcc00700 1 -- 172.21.15.12:0/1985334317 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5603122d5570 con 0x5603124309f0 2024-05-27T07:37:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92bcc00700 10 client.17247 handle_command_reply: tid=0 2024-05-27T07:37:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 client.17247 shutdown 2024-05-27T07:37:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 msgr2=0x5603123b98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123b98a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f92c00dfdb0 tx=0x7f92c00a9d80 comp rx=0 tx=0).stop 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 shutdown_connections 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5603124309f0 0x560312432e10 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56031222ce40 0x5603123be540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5603123abe70 0x5603123b9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 --2- 172.21.15.12:0/1985334317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5603123189f0 0x5603123b98a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 >> 172.21.15.12:0/1985334317 conn(0x5603122c6390 msgr2=0x5603123b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.384+0000 7f92bcc00700 0 client.17247 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.385+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 shutdown_connections 2024-05-27T07:37:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.385+0000 7f92e8499b80 1 -- 172.21.15.12:0/1985334317 wait complete. 2024-05-27T07:37:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.385+0000 7f92e8499b80 20 client.17247 trim_cache size 0 max 16384 2024-05-27T07:37:50.383 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:37:50.383 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:37:50.383 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:37:50.383 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f92b80b2a70 msgr2=0x7f92b80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f92b80b2a70 0x7f92b80b4f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x5603123a8790 tx=0x7f92c40041a0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4103b90 msgr2=0x7f92d41a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4103b90 0x7f92d41a1840 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f92c000ef10 tx=0x7f92c000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 shutdown_connections 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f92b80b2a70 0x7f92b80b4f80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92d4108690 0x7f92d41a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92d4103240 0x7f92d41a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 --2- 172.21.15.12:0/1532858735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92d4103b90 0x7f92d41a1840 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 >> 172.21.15.12:0/1532858735 conn(0x7f92d40fedb0 msgr2=0x7f92d4105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.386+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 shutdown_connections 2024-05-27T07:37:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:50.387+0000 7f92d9a00700 1 -- 172.21.15.12:0/1532858735 wait complete. 2024-05-27T07:37:51.377 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:51 smithi012 ceph-mon[31473]: pgmap v529: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 270 op/s 2024-05-27T07:37:51.381 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:37:51.381 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:37:51.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:51 smithi162 ceph-mon[35676]: pgmap v529: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 270 op/s 2024-05-27T07:37:51.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:51 smithi062 ceph-mon[35304]: pgmap v529: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 270 op/s 2024-05-27T07:37:51.866 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:53.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:53 smithi012 ceph-mon[31473]: pgmap v530: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:37:53.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:53 smithi162 ceph-mon[35676]: pgmap v530: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:37:53.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:53 smithi062 ceph-mon[35304]: pgmap v530: 129 pgs: 129 active+clean; 300 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:37:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 -- 172.21.15.12:0/3813942560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4106560 msgr2=0x7f29e4106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 --2- 172.21.15.12:0/3813942560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4106560 0x7f29e4106970 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009990 tx=0x7f29d0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 -- 172.21.15.12:0/3813942560 shutdown_connections 2024-05-27T07:37:54.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 --2- 172.21.15.12:0/3813942560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e4106eb0 0x7f29e4113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 --2- 172.21.15.12:0/3813942560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e410c5a0 0x7f29e410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 --2- 172.21.15.12:0/3813942560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4106560 0x7f29e4106970 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 -- 172.21.15.12:0/3813942560 >> 172.21.15.12:0/3813942560 conn(0x7f29e41020c0 msgr2=0x7f29e4104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 -- 172.21.15.12:0/3813942560 shutdown_connections 2024-05-27T07:37:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.319+0000 7f29ec600700 1 -- 172.21.15.12:0/3813942560 wait complete. 2024-05-27T07:37:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.320+0000 7f29ec600700 1 Processor -- start 2024-05-27T07:37:54.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.321+0000 7f29ec600700 1 -- start start 2024-05-27T07:37:54.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.321+0000 7f29ec600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e4106560 0x7f29e4077cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:54.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.321+0000 7f29ec600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e4106eb0 0x7f29e4078200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.321+0000 7f29ec600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e410c5a0 0x7f29e4078740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.321+0000 7f29ec600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f29e4116680 con 0x7f29e410c5a0 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.321+0000 7f29ec600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f29e4116500 con 0x7f29e4106eb0 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.321+0000 7f29ec600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29e4116800 con 0x7f29e4106560 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.322+0000 7f29ea600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e410c5a0 0x7f29e4078740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.323+0000 7f29e9c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e4106560 0x7f29e4077cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.323+0000 7f29e9c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e4106560 0x7f29e4077cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:48762/0 (socket says 172.21.15.12:48762) 2024-05-27T07:37:54.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.323+0000 7f29e9c00700 1 -- 172.21.15.12:0/1701804979 learned_addr learned my addr 172.21.15.12:0/1701804979 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:54.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.324+0000 7f29ea600700 1 -- 172.21.15.12:0/1701804979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e4106560 msgr2=0x7f29e4077cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:54.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.324+0000 7f29ea600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e4106560 0x7f29e4077cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.324+0000 7f29ea600700 1 -- 172.21.15.12:0/1701804979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e4106eb0 msgr2=0x7f29e4078200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:54.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.324+0000 7f29ea600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e4106eb0 0x7f29e4078200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.324+0000 7f29ea600700 1 -- 172.21.15.12:0/1701804979 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e41ad6b0 con 0x7f29e410c5a0 2024-05-27T07:37:54.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.325+0000 7f29ea600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e410c5a0 0x7f29e4078740 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00be20 tx=0x7f29dc009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:54.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.326+0000 7f29e2000700 1 -- 172.21.15.12:0/1701804979 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc019070 con 0x7f29e410c5a0 2024-05-27T07:37:54.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.326+0000 7f29e2000700 1 -- 172.21.15.12:0/1701804979 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29dc00cc10 con 0x7f29e410c5a0 2024-05-27T07:37:54.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.326+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e41ad940 con 0x7f29e410c5a0 2024-05-27T07:37:54.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.327+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e41adf00 con 0x7f29e410c5a0 2024-05-27T07:37:54.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.327+0000 7f29e2000700 1 -- 172.21.15.12:0/1701804979 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc0125a0 con 0x7f29e410c5a0 2024-05-27T07:37:54.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.328+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c4005330 con 0x7f29e410c5a0 2024-05-27T07:37:54.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.333+0000 7f29e2000700 1 -- 172.21.15.12:0/1701804979 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f29dc01c070 con 0x7f29e410c5a0 2024-05-27T07:37:54.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.335+0000 7f29e2000700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f29cc0b2a70 0x7f29cc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:54.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.335+0000 7f29e2000700 1 -- 172.21.15.12:0/1701804979 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f29dc0e1fb0 con 0x7f29e410c5a0 2024-05-27T07:37:54.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.335+0000 7f29e9c00700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f29cc0b2a70 0x7f29cc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:54.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.336+0000 7f29e9c00700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f29cc0b2a70 0x7f29cc0b4f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29d8009aa0 tx=0x7f29d8009240 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:54.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.340+0000 7f29e2000700 1 -- 172.21.15.12:0/1701804979 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f29dc051df0 con 0x7f29e410c5a0 2024-05-27T07:37:54.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.673+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29c40058f0 con 0x7f29e410c5a0 2024-05-27T07:37:54.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.674+0000 7f29e2000700 1 -- 172.21.15.12:0/1701804979 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f29dc0a9690 con 0x7f29e410c5a0 2024-05-27T07:37:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:37:54.672 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:37:54.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f29cc0b2a70 msgr2=0x7f29cc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f29cc0b2a70 0x7f29cc0b4f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29d8009aa0 tx=0x7f29d8009240 comp rx=0 tx=0).stop 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e410c5a0 msgr2=0x7f29e4078740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e410c5a0 0x7f29e4078740 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00be20 tx=0x7f29dc009ed0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 shutdown_connections 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f29cc0b2a70 0x7f29cc0b4f80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e410c5a0 0x7f29e4078740 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e4106560 0x7f29e4077cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 --2- 172.21.15.12:0/1701804979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e4106eb0 0x7f29e4078200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.677+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 >> 172.21.15.12:0/1701804979 conn(0x7f29e41020c0 msgr2=0x7f29e4104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.678+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 shutdown_connections 2024-05-27T07:37:54.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:54.678+0000 7f29ec600700 1 -- 172.21.15.12:0/1701804979 wait complete. 2024-05-27T07:37:54.676 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:55 smithi012 ceph-mon[31473]: pgmap v531: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 279 op/s 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:55 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1701804979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:55.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:55 smithi162 ceph-mon[35676]: pgmap v531: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 279 op/s 2024-05-27T07:37:55.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:55 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1701804979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:55.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:55 smithi062 ceph-mon[35304]: pgmap v531: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 279 op/s 2024-05-27T07:37:55.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:55 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1701804979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:55.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:37:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:57.479 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:57 smithi162 ceph-mon[35676]: pgmap v532: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 191 op/s 2024-05-27T07:37:57.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:57 smithi062 ceph-mon[35304]: pgmap v532: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 191 op/s 2024-05-27T07:37:57.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:57 smithi012 ceph-mon[31473]: pgmap v532: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 191 op/s 2024-05-27T07:37:57.744 INFO:tasks.workunit.client.0.smithi012.stdout:10000 write 0x4c35901 thru 0x4c3d40f (0x7b0f bytes) 2024-05-27T07:37:59.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 -- 172.21.15.12:0/2458386849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1730105e40 msgr2=0x7f173010e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 --2- 172.21.15.12:0/2458386849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1730105e40 0x7f173010e390 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f171c009990 tx=0x7f171c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 -- 172.21.15.12:0/2458386849 shutdown_connections 2024-05-27T07:37:59.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 --2- 172.21.15.12:0/2458386849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173010e8d0 0x7f1730110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 --2- 172.21.15.12:0/2458386849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17301054f0 0x7f1730105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 --2- 172.21.15.12:0/2458386849 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1730105e40 0x7f173010e390 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 -- 172.21.15.12:0/2458386849 >> 172.21.15.12:0/2458386849 conn(0x7f17300fedb0 msgr2=0x7f17301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:59.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 -- 172.21.15.12:0/2458386849 shutdown_connections 2024-05-27T07:37:59.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.200+0000 7f1737e00700 1 -- 172.21.15.12:0/2458386849 wait complete. 2024-05-27T07:37:59.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 Processor -- start 2024-05-27T07:37:59.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 -- start start 2024-05-27T07:37:59.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17301054f0 0x7f173019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1730105e40 0x7f173019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f173010e8d0 0x7f173019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1730114330 con 0x7f17301054f0 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f17301141b0 con 0x7f1730105e40 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.202+0000 7f1737e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17301144b0 con 0x7f173010e8d0 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17301054f0 0x7f173019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17301054f0 0x7f173019ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40712/0 (socket says 172.21.15.12:40712) 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 -- 172.21.15.12:0/3976842368 learned_addr learned my addr 172.21.15.12:0/3976842368 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1734a00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1730105e40 0x7f173019b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 -- 172.21.15.12:0/3976842368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f173010e8d0 msgr2=0x7f173019f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f173010e8d0 0x7f173019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 -- 172.21.15.12:0/3976842368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1730105e40 msgr2=0x7f173019b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1730105e40 0x7f173019b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.203+0000 7f1735400700 1 -- 172.21.15.12:0/3976842368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173019fc10 con 0x7f17301054f0 2024-05-27T07:37:59.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.204+0000 7f1735400700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17301054f0 0x7f173019ad40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f172400d950 tx=0x7f172400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:59.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.204+0000 7f172e000700 1 -- 172.21.15.12:0/3976842368 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724016070 con 0x7f17301054f0 2024-05-27T07:37:59.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.204+0000 7f172e000700 1 -- 172.21.15.12:0/3976842368 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1724010460 con 0x7f17301054f0 2024-05-27T07:37:59.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.204+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173019ff00 con 0x7f17301054f0 2024-05-27T07:37:59.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.204+0000 7f172e000700 1 -- 172.21.15.12:0/3976842368 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172400f620 con 0x7f17301054f0 2024-05-27T07:37:59.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.205+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17301a7820 con 0x7f17301054f0 2024-05-27T07:37:59.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.208+0000 7f172e000700 1 -- 172.21.15.12:0/3976842368 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1724019070 con 0x7f17301054f0 2024-05-27T07:37:59.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.209+0000 7f1746943b80 1 Processor -- start 2024-05-27T07:37:59.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f1746943b80 1 -- start start 2024-05-27T07:37:59.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f172e000700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f17180b29f0 0x7f17180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f1746943b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c20eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f1746943b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c208c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f1746943b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c2095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f1746943b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56252c132e40 con 0x56252c085e40 2024-05-27T07:37:59.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f1746943b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56252c1229e0 con 0x56252c2091c0 2024-05-27T07:37:59.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f1746943b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56252c126a20 con 0x56252c20f0e0 2024-05-27T07:37:59.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.210+0000 7f172e000700 1 -- 172.21.15.12:0/3976842368 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f17240e26d0 con 0x7f17301054f0 2024-05-27T07:37:59.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f17180b29f0 0x7f17180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c20eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c20eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40728/0 (socket says 172.21.15.12:40728) 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 -- 172.21.15.12:0/1178710226 learned_addr learned my addr 172.21.15.12:0/1178710226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735e00700 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c2095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1734a00700 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c208c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 -- 172.21.15.12:0/1178710226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 msgr2=0x56252c208c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c208c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 -- 172.21.15.12:0/1178710226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 msgr2=0x56252c2095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c2095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.211+0000 7f1735400700 1 -- 172.21.15.12:0/1178710226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56252c0eb420 con 0x56252c085e40 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.212+0000 7f1735400700 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c20eba0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f172400b2f0 tx=0x7f17240a8de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e583c4ed45dff111 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:59.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.212+0000 7f1735e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f17180b29f0 0x7f17180b4f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17280060b0 tx=0x7f1728011a10 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:59.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.212+0000 7f1722a00700 1 -- 172.21.15.12:0/1178710226 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724020030 con 0x56252c085e40 2024-05-27T07:37:59.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.212+0000 7f1722a00700 1 -- 172.21.15.12:0/1178710226 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17240a9540 con 0x56252c085e40 2024-05-27T07:37:59.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.212+0000 7f1722a00700 1 -- 172.21.15.12:0/1178710226 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172405fb30 con 0x56252c085e40 2024-05-27T07:37:59.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 -- 172.21.15.12:0/1178710226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 msgr2=0x56252c20eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c20eba0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f172400b2f0 tx=0x7f17240a8de0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 -- 172.21.15.12:0/1178710226 shutdown_connections 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c2095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c20eba0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 --2- 172.21.15.12:0/1178710226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c208c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 -- 172.21.15.12:0/1178710226 >> 172.21.15.12:0/1178710226 conn(0x56252c0e5fc0 msgr2=0x56252c1ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.213+0000 7f1746943b80 1 -- 172.21.15.12:0/1178710226 shutdown_connections 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.214+0000 7f1746943b80 1 -- 172.21.15.12:0/1178710226 wait complete. 2024-05-27T07:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.214+0000 7f1746943b80 1 -- start start 2024-05-27T07:37:59.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 1 Processor -- start 2024-05-27T07:37:59.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c2128e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c212e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c213360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56252c028fa0 con 0x56252c085e40 2024-05-27T07:37:59.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56252c0c3760 con 0x56252c2091c0 2024-05-27T07:37:59.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56252c215bc0 con 0x56252c20f0e0 2024-05-27T07:37:59.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.215+0000 7f1746943b80 10 client.0 _ll_register_callbacks cb 0x56252c0ccc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c2128e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c2128e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40740/0 (socket says 172.21.15.12:40740) 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 -- 172.21.15.12:0/1329056624 learned_addr learned my addr 172.21.15.12:0/1329056624 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735e00700 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c213360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1734a00700 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c212e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 -- 172.21.15.12:0/1329056624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 msgr2=0x56252c213360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c213360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 -- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 msgr2=0x56252c212e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c212e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.216+0000 7f1735400700 1 -- 172.21.15.12:0/1329056624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56252c0728d0 con 0x56252c085e40 2024-05-27T07:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.217+0000 7f1735400700 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c2128e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f17240a55c0 tx=0x7f1724091210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a987bfeb788b83d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.217+0000 7f1717400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.217+0000 7f1717400700 1 -- 172.21.15.12:0/1329056624 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724020040 con 0x56252c085e40 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.217+0000 7f1717400700 1 -- 172.21.15.12:0/1329056624 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f172405fcf0 con 0x56252c085e40 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.217+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17240a9540 con 0x56252c085e40 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.217+0000 7f1717400700 1 -- 172.21.15.12:0/1329056624 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f17240b23a0 con 0x56252c085e40 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 10 client.17265 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56252c0f3060 con 0x56252c085e40 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1717400700 1 -- 172.21.15.12:0/1329056624 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17240a9540 con 0x56252c085e40 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1717400700 1 -- 172.21.15.12:0/1329056624 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f17240e21f0 con 0x56252c085e40 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 10 client.17265 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 10 client.17265 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 20 client.17265 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c222910 0x56252c224d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 4 client.17265 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.218+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56252c134d90 con 0x56252c222910 2024-05-27T07:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.219+0000 7f1734a00700 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c222910 0x56252c224d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.220+0000 7f1734a00700 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c222910 0x56252c224d30 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=223eac297837a6ec server_cookie=ec00abcc901f12d4 in_seq=0 out_seq=0 2024-05-27T07:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.220+0000 7f1717400700 10 client.17265 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:37:59.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.222+0000 7f1717400700 1 -- 172.21.15.12:0/1329056624 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56252c134d90 con 0x56252c222910 2024-05-27T07:37:59.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1717400700 10 client.17265 handle_command_reply: tid=0 2024-05-27T07:37:59.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 client.17265 shutdown 2024-05-27T07:37:59.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 msgr2=0x56252c2128e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c2128e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f17240a55c0 tx=0x7f1724091210 comp rx=0 tx=0).stop 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 shutdown_connections 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c213360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c085e40 0x56252c2128e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c222910 0x56252c224d30 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 --2- 172.21.15.12:0/1329056624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c2091c0 0x56252c212e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 >> 172.21.15.12:0/1329056624 conn(0x56252c0e5fc0 msgr2=0x56252c1ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1717400700 0 client.17265 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 shutdown_connections 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.223+0000 7f1746943b80 1 -- 172.21.15.12:0/1329056624 wait complete. 2024-05-27T07:37:59.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.224+0000 7f1746943b80 20 client.17265 trim_cache size 0 max 16384 2024-05-27T07:37:59.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.243+0000 7f1746943b80 1 Processor -- start 2024-05-27T07:37:59.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.243+0000 7f1746943b80 1 -- start start 2024-05-27T07:37:59.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.244+0000 7f1746943b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 0x56252c214c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.244+0000 7f1746943b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c2091c0 0x56252c21ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.244+0000 7f1746943b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c20f0e0 0x56252c20cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.244+0000 7f1746943b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f17301adac0 con 0x56252c20f0e0 2024-05-27T07:37:59.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.244+0000 7f1746943b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f17301acd20 con 0x56252c085e40 2024-05-27T07:37:59.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.244+0000 7f1746943b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17301ad3f0 con 0x56252c2091c0 2024-05-27T07:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.244+0000 7f1735e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c20f0e0 0x56252c20cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.245+0000 7f1735e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c20f0e0 0x56252c20cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40752/0 (socket says 172.21.15.12:40752) 2024-05-27T07:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.245+0000 7f1735e00700 1 -- 172.21.15.12:0/3801247806 learned_addr learned my addr 172.21.15.12:0/3801247806 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.246+0000 7f1735e00700 1 -- 172.21.15.12:0/3801247806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c2091c0 msgr2=0x56252c21ec60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.246+0000 7f1735e00700 1 --2- 172.21.15.12:0/3801247806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c2091c0 0x56252c21ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.246+0000 7f1735e00700 1 -- 172.21.15.12:0/3801247806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 msgr2=0x56252c214c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.246+0000 7f1735e00700 1 --2- 172.21.15.12:0/3801247806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 0x56252c214c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.246+0000 7f1735e00700 1 -- 172.21.15.12:0/3801247806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56252c215180 con 0x56252c20f0e0 2024-05-27T07:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.247+0000 7f1735e00700 1 --2- 172.21.15.12:0/3801247806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c20f0e0 0x56252c20cd40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1728009320 tx=0x7f172801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5198c4aec40e42 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:59.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.248+0000 7f1722000700 1 -- 172.21.15.12:0/3801247806 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728028070 con 0x56252c20f0e0 2024-05-27T07:37:59.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.248+0000 7f1722000700 1 -- 172.21.15.12:0/3801247806 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1728009350 con 0x56252c20f0e0 2024-05-27T07:37:59.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.248+0000 7f1722000700 1 -- 172.21.15.12:0/3801247806 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728007a10 con 0x56252c20f0e0 2024-05-27T07:37:59.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.249+0000 7f1746943b80 1 -- 172.21.15.12:0/3801247806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c20f0e0 msgr2=0x56252c20cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.249+0000 7f1746943b80 1 --2- 172.21.15.12:0/3801247806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c20f0e0 0x56252c20cd40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1728009320 tx=0x7f172801a000 comp rx=0 tx=0).stop 2024-05-27T07:37:59.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.250+0000 7f1746943b80 1 -- 172.21.15.12:0/3801247806 shutdown_connections 2024-05-27T07:37:59.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.250+0000 7f1746943b80 1 --2- 172.21.15.12:0/3801247806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 0x56252c214c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.250+0000 7f1746943b80 1 --2- 172.21.15.12:0/3801247806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c20f0e0 0x56252c20cd40 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.250+0000 7f1746943b80 1 --2- 172.21.15.12:0/3801247806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c2091c0 0x56252c21ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.250+0000 7f1746943b80 1 -- 172.21.15.12:0/3801247806 >> 172.21.15.12:0/3801247806 conn(0x56252c0e5fc0 msgr2=0x56252c20c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:59.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.251+0000 7f1746943b80 1 -- 172.21.15.12:0/3801247806 shutdown_connections 2024-05-27T07:37:59.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.252+0000 7f1746943b80 1 -- 172.21.15.12:0/3801247806 wait complete. 2024-05-27T07:37:59.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.253+0000 7f1746943b80 1 -- start start 2024-05-27T07:37:59.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.253+0000 7f1746943b80 1 Processor -- start 2024-05-27T07:37:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.256+0000 7f1746943b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 0x56252c205870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.256+0000 7f1746943b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c2091c0 0x56252c205db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.256+0000 7f1746943b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c2062f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.257+0000 7f1746943b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56252c21f7f0 con 0x56252c2091c0 2024-05-27T07:37:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.257+0000 7f1746943b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56252c210e50 con 0x56252c085e40 2024-05-27T07:37:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.257+0000 7f1746943b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56252c210840 con 0x56252c20f0e0 2024-05-27T07:37:59.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.257+0000 7f1734a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c2091c0 0x56252c205db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.257+0000 7f1746943b80 10 client.0 _ll_register_callbacks cb 0x56252c0ccc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:59.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.257+0000 7f1734a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c2091c0 0x56252c205db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40768/0 (socket says 172.21.15.12:40768) 2024-05-27T07:37:59.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.258+0000 7f1734a00700 1 -- 172.21.15.12:0/276014565 learned_addr learned my addr 172.21.15.12:0/276014565 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:37:59.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.258+0000 7f1735400700 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 0x56252c205870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.258+0000 7f1735e00700 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c2062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.258+0000 7f1734a00700 1 -- 172.21.15.12:0/276014565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 msgr2=0x56252c2062f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.258+0000 7f1734a00700 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c2062f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.258+0000 7f1734a00700 1 -- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 msgr2=0x56252c205870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.258+0000 7f1734a00700 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 0x56252c205870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.259+0000 7f1734a00700 1 -- 172.21.15.12:0/276014565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56252c219e40 con 0x56252c2091c0 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.259+0000 7f1734a00700 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c2091c0 0x56252c205db0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f171c003760 tx=0x7f171c009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16113d54b35e91dc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:59.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.259+0000 7f1717400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:37:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.260+0000 7f1717400700 1 -- 172.21.15.12:0/276014565 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c005680 con 0x56252c2091c0 2024-05-27T07:37:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.260+0000 7f1717400700 1 -- 172.21.15.12:0/276014565 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f171c0037c0 con 0x56252c2091c0 2024-05-27T07:37:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.260+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1728009350 con 0x56252c2091c0 2024-05-27T07:37:59.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.261+0000 7f1717400700 1 -- 172.21.15.12:0/276014565 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f171c0039c0 con 0x56252c2091c0 2024-05-27T07:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.261+0000 7f1746943b80 10 client.17271 fetch_fsmap learned FSMap version 119 2024-05-27T07:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.261+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56252c21b100 con 0x56252c2091c0 2024-05-27T07:37:59.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.265+0000 7f1717400700 1 -- 172.21.15.12:0/276014565 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1728009350 con 0x56252c2091c0 2024-05-27T07:37:59.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.265+0000 7f1717400700 1 -- 172.21.15.12:0/276014565 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f171c00f460 con 0x56252c2091c0 2024-05-27T07:37:59.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.266+0000 7f1746943b80 10 client.17271 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:37:59.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.266+0000 7f1746943b80 10 client.17271 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:37:59.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.266+0000 7f1746943b80 20 client.17271 populate_metadata read hostname 'smithi012' 2024-05-27T07:37:59.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.266+0000 7f1746943b80 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c291b60 0x56252c293f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.266+0000 7f1746943b80 4 client.17271 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:37:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.266+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56252c12e570 con 0x56252c291b60 2024-05-27T07:37:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.268+0000 7f1735400700 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c291b60 0x56252c293f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.269+0000 7f1735400700 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c291b60 0x56252c293f80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f895efe0151ac7df server_cookie=9213bc9ed33c4458 in_seq=0 out_seq=0 2024-05-27T07:37:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.269+0000 7f1717400700 10 client.17271 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:37:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.269+0000 7f1717400700 1 -- 172.21.15.12:0/276014565 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56252c12e570 con 0x56252c291b60 2024-05-27T07:37:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.269+0000 7f1717400700 10 client.17271 handle_command_reply: tid=0 2024-05-27T07:37:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.269+0000 7f1746943b80 1 client.17271 shutdown 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c2091c0 msgr2=0x56252c205db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c2091c0 0x56252c205db0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f171c003760 tx=0x7f171c009510 comp rx=0 tx=0).stop 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 shutdown_connections 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56252c291b60 0x56252c293f80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56252c085e40 0x56252c205870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56252c20f0e0 0x56252c2062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 --2- 172.21.15.12:0/276014565 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56252c2091c0 0x56252c205db0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 >> 172.21.15.12:0/276014565 conn(0x56252c0e5fc0 msgr2=0x56252c2110a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.270+0000 7f1717400700 0 client.17271 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.271+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 shutdown_connections 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.271+0000 7f1746943b80 1 -- 172.21.15.12:0/276014565 wait complete. 2024-05-27T07:37:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.271+0000 7f1746943b80 20 client.17271 trim_cache size 0 max 16384 2024-05-27T07:37:59.269 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:37:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f17180b29f0 msgr2=0x7f17180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f17180b29f0 0x7f17180b4f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17280060b0 tx=0x7f1728011a10 comp rx=0 tx=0).stop 2024-05-27T07:37:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17301054f0 msgr2=0x7f173019ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17301054f0 0x7f173019ad40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f172400d950 tx=0x7f172400dd10 comp rx=0 tx=0).stop 2024-05-27T07:37:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 shutdown_connections 2024-05-27T07:37:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f17180b29f0 0x7f17180b4f00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f173010e8d0 0x7f173019f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17301054f0 0x7f173019ad40 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 --2- 172.21.15.12:0/3976842368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1730105e40 0x7f173019b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 >> 172.21.15.12:0/3976842368 conn(0x7f17300fedb0 msgr2=0x7f17300ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 shutdown_connections 2024-05-27T07:37:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:37:59.272+0000 7f1737e00700 1 -- 172.21.15.12:0/3976842368 wait complete. 2024-05-27T07:37:59.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:37:59 smithi162 ceph-mon[35676]: pgmap v533: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 191 op/s 2024-05-27T07:37:59.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:37:59 smithi062 ceph-mon[35304]: pgmap v533: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 191 op/s 2024-05-27T07:37:59.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:37:59 smithi012 ceph-mon[31473]: pgmap v533: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 191 op/s 2024-05-27T07:38:00.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:38:00.822 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:01.299 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:01 smithi012 ceph-mon[31473]: pgmap v534: 129 pgs: 129 active+clean; 304 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 292 op/s 2024-05-27T07:38:01.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:01 smithi162 ceph-mon[35676]: pgmap v534: 129 pgs: 129 active+clean; 304 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 292 op/s 2024-05-27T07:38:01.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:01 smithi062 ceph-mon[35304]: pgmap v534: 129 pgs: 129 active+clean; 304 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 292 op/s 2024-05-27T07:38:03.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:03 smithi062 ceph-mon[35304]: pgmap v535: 129 pgs: 129 active+clean; 304 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:38:03.511 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:03 smithi012 ceph-mon[31473]: pgmap v535: 129 pgs: 129 active+clean; 304 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:38:03.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:03 smithi162 ceph-mon[35676]: pgmap v535: 129 pgs: 129 active+clean; 304 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:38:04.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 -- 172.21.15.12:0/1365824893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec115360 msgr2=0x7f1fec117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 --2- 172.21.15.12:0/1365824893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec115360 0x7f1fec117780 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b0a0 tx=0x7f1fe400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 -- 172.21.15.12:0/1365824893 shutdown_connections 2024-05-27T07:38:04.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 --2- 172.21.15.12:0/1365824893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fec075b50 0x7f1fec075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 --2- 172.21.15.12:0/1365824893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec115360 0x7f1fec117780 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 --2- 172.21.15.12:0/1365824893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec0764a0 0x7f1fec114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 -- 172.21.15.12:0/1365824893 >> 172.21.15.12:0/1365824893 conn(0x7f1fec100030 msgr2=0x7f1fec102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 -- 172.21.15.12:0/1365824893 shutdown_connections 2024-05-27T07:38:04.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.209+0000 7f1ff4200700 1 -- 172.21.15.12:0/1365824893 wait complete. 2024-05-27T07:38:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.210+0000 7f1ff4200700 1 Processor -- start 2024-05-27T07:38:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.210+0000 7f1ff4200700 1 -- start start 2024-05-27T07:38:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff4200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec075b50 0x7f1fec1a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec0764a0 0x7f1fec1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff4200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fec115360 0x7f1fec1a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec11a960 con 0x7f1fec0764a0 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff4200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec11a7e0 con 0x7f1fec115360 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff4200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec11aae0 con 0x7f1fec075b50 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec0764a0 0x7f1fec1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec0764a0 0x7f1fec1a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38470/0 (socket says 172.21.15.12:38470) 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff0e00700 1 -- 172.21.15.12:0/2298764895 learned_addr learned my addr 172.21.15.12:0/2298764895 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff1800700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec075b50 0x7f1fec1a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.211+0000 7f1ff2200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fec115360 0x7f1fec1a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1ff0e00700 1 -- 172.21.15.12:0/2298764895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec075b50 msgr2=0x7f1fec1a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec075b50 0x7f1fec1a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1ff0e00700 1 -- 172.21.15.12:0/2298764895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fec115360 msgr2=0x7f1fec1a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fec115360 0x7f1fec1a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1ff0e00700 1 -- 172.21.15.12:0/2298764895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec1a6200 con 0x7f1fec0764a0 2024-05-27T07:38:04.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1ff2200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fec115360 0x7f1fec1a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec0764a0 0x7f1fec1a18d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e9a0 tx=0x7f1fd800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.212+0000 7f1fea000700 1 -- 172.21.15.12:0/2298764895 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800cbd0 con 0x7f1fec0764a0 2024-05-27T07:38:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.213+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fec1a64f0 con 0x7f1fec0764a0 2024-05-27T07:38:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.213+0000 7f1fea000700 1 -- 172.21.15.12:0/2298764895 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fd800cd70 con 0x7f1fec0764a0 2024-05-27T07:38:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.213+0000 7f1fea000700 1 -- 172.21.15.12:0/2298764895 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8018680 con 0x7f1fec0764a0 2024-05-27T07:38:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.213+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec1ade10 con 0x7f1fec0764a0 2024-05-27T07:38:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.217+0000 7f1fea000700 1 -- 172.21.15.12:0/2298764895 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1fd8010430 con 0x7f1fec0764a0 2024-05-27T07:38:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 Processor -- start 2024-05-27T07:38:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 -- start start 2024-05-27T07:38:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5f2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5ff110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc520e40 con 0x55e8dc55f9f0 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc5109e0 con 0x55e8dc473e40 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.218+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc514a20 con 0x55e8dc5f2e70 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38474/0 (socket says 172.21.15.12:38474) 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 -- 172.21.15.12:0/250734979 learned_addr learned my addr 172.21.15.12:0/250734979 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff1800700 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 -- 172.21.15.12:0/250734979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc5f2e70 msgr2=0x55e8dc5ff110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5ff110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1fea000700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fd40b29f0 0x7f1fd40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 -- 172.21.15.12:0/250734979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 msgr2=0x55e8dc5f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.219+0000 7f1ff2200700 1 -- 172.21.15.12:0/250734979 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8dc4d9420 con 0x55e8dc55f9f0 2024-05-27T07:38:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.220+0000 7f1fea000700 1 -- 172.21.15.12:0/2298764895 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f1fd80e2c40 con 0x7f1fec0764a0 2024-05-27T07:38:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.220+0000 7f1ff2200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fd40b29f0 0x7f1fd40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.220+0000 7f1ff2200700 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f2330 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009d90 tx=0x7f1fe4009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60ea9ffa13eb0e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.220+0000 7f1fdea00700 1 -- 172.21.15.12:0/250734979 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4004a50 con 0x55e8dc55f9f0 2024-05-27T07:38:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.221+0000 7f1fdea00700 1 -- 172.21.15.12:0/250734979 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fe4022040 con 0x55e8dc55f9f0 2024-05-27T07:38:04.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.221+0000 7f1fdea00700 1 -- 172.21.15.12:0/250734979 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe402f050 con 0x55e8dc55f9f0 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 -- 172.21.15.12:0/250734979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 msgr2=0x55e8dc5f2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f2330 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009d90 tx=0x7f1fe4009dc0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f1ff2200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fd40b29f0 0x7f1fd40b4f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4028000 tx=0x7f1fe402b000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 -- 172.21.15.12:0/250734979 shutdown_connections 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5ff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f2330 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/250734979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 -- 172.21.15.12:0/250734979 >> 172.21.15.12:0/250734979 conn(0x55e8dc50d390 msgr2=0x55e8dc5ee4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 -- 172.21.15.12:0/250734979 shutdown_connections 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.222+0000 7f2002cbdb80 1 -- 172.21.15.12:0/250734979 wait complete. 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.223+0000 7f2002cbdb80 1 -- start start 2024-05-27T07:38:04.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.223+0000 7f2002cbdb80 1 Processor -- start 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.223+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc473e40 0x55e8dc5fc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.223+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5fcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.223+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5fd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.223+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc416fa0 con 0x55e8dc473e40 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.223+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc4b1760 con 0x55e8dc5f2e70 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc5ffb80 con 0x55e8dc55f9f0 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f2002cbdb80 10 client.0 _ll_register_callbacks cb 0x55e8dc4bac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc473e40 0x55e8dc5fc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc473e40 0x55e8dc5fc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38490/0 (socket says 172.21.15.12:38490) 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 -- 172.21.15.12:0/2178376017 learned_addr learned my addr 172.21.15.12:0/2178376017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5fcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff2200700 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 -- 172.21.15.12:0/2178376017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 msgr2=0x55e8dc5fcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5fcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 -- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 msgr2=0x55e8dc5fd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5fd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.224+0000 7f1ff1800700 1 -- 172.21.15.12:0/2178376017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8dc4608d0 con 0x55e8dc473e40 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.225+0000 7f1ff1800700 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc473e40 0x55e8dc5fc5b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000b890 tx=0x7f1fe000bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca6ca75e1d1a99f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.225+0000 7f1fd3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.225+0000 7f1fd3400700 1 -- 172.21.15.12:0/2178376017 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0003d90 con 0x55e8dc473e40 2024-05-27T07:38:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.225+0000 7f1fd3400700 1 -- 172.21.15.12:0/2178376017 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f1fe001b440 con 0x55e8dc473e40 2024-05-27T07:38:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.225+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fe4022040 con 0x55e8dc473e40 2024-05-27T07:38:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.225+0000 7f1fd3400700 1 -- 172.21.15.12:0/2178376017 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f1fe0024050 con 0x55e8dc473e40 2024-05-27T07:38:04.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.226+0000 7f2002cbdb80 10 client.17283 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.226+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55e8dc4e1060 con 0x55e8dc473e40 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.226+0000 7f1fd3400700 1 -- 172.21.15.12:0/2178376017 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fe4022040 con 0x55e8dc473e40 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.226+0000 7f1fd3400700 1 -- 172.21.15.12:0/2178376017 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f1fe0019020 con 0x55e8dc473e40 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.227+0000 7f2002cbdb80 10 client.17283 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.227+0000 7f2002cbdb80 10 client.17283 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.227+0000 7f2002cbdb80 20 client.17283 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.227+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc60ce10 0x55e8dc60f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.227+0000 7f2002cbdb80 4 client.17283 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.227+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8dc522d90 con 0x55e8dc60ce10 2024-05-27T07:38:04.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.227+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc60ce10 0x55e8dc60f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.229+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc60ce10 0x55e8dc60f300 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8666ae8264fd4cfb server_cookie=58dc1b8c51ba3ddc in_seq=0 out_seq=0 2024-05-27T07:38:04.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.229+0000 7f1fd3400700 10 client.17283 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:38:04.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.231+0000 7f1fd3400700 1 -- 172.21.15.12:0/2178376017 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e8dc522d90 con 0x55e8dc60ce10 2024-05-27T07:38:04.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.231+0000 7f1fd3400700 10 client.17283 handle_command_reply: tid=0 2024-05-27T07:38:04.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.231+0000 7f2002cbdb80 1 client.17283 shutdown 2024-05-27T07:38:04.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.231+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc473e40 msgr2=0x55e8dc5fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.231+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc473e40 0x55e8dc5fc5b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000b890 tx=0x7f1fe000bba0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 shutdown_connections 2024-05-27T07:38:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc60ce10 0x55e8dc60f300 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5fd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f1fd3400700 0 client.17283 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:38:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc473e40 0x55e8dc5fc5b0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/2178376017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 >> 172.21.15.12:0/2178376017 conn(0x55e8dc50d390 msgr2=0x55e8dc5f2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 shutdown_connections 2024-05-27T07:38:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 1 -- 172.21.15.12:0/2178376017 wait complete. 2024-05-27T07:38:04.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.232+0000 7f2002cbdb80 20 client.17283 trim_cache size 0 max 16384 2024-05-27T07:38:04.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.252+0000 7f2002cbdb80 1 Processor -- start 2024-05-27T07:38:04.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.252+0000 7f2002cbdb80 1 -- start start 2024-05-27T07:38:04.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5fb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b6290 con 0x55e8dc5f2e70 2024-05-27T07:38:04.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b54f0 con 0x55e8dc473e40 2024-05-27T07:38:04.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b5bc0 con 0x55e8dc55f9f0 2024-05-27T07:38:04.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f1ff2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f1ff2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38506/0 (socket says 172.21.15.12:38506) 2024-05-27T07:38:04.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.253+0000 7f1ff2200700 1 -- 172.21.15.12:0/4124032360 learned_addr learned my addr 172.21.15.12:0/4124032360 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff1800700 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5fb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff2200700 1 -- 172.21.15.12:0/4124032360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 msgr2=0x55e8dc5f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff2200700 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff2200700 1 -- 172.21.15.12:0/4124032360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 msgr2=0x55e8dc5fb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff2200700 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5fb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff2200700 1 -- 172.21.15.12:0/4124032360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8dc5ede80 con 0x55e8dc5f2e70 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.254+0000 7f1ff2200700 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f2720 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b780 tx=0x7f1fe402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763b78579bc75150 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.255+0000 7f1fde000700 1 -- 172.21.15.12:0/4124032360 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4004a50 con 0x55e8dc5f2e70 2024-05-27T07:38:04.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.255+0000 7f1fde000700 1 -- 172.21.15.12:0/4124032360 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fe4022040 con 0x55e8dc5f2e70 2024-05-27T07:38:04.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.255+0000 7f1fde000700 1 -- 172.21.15.12:0/4124032360 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40296c0 con 0x55e8dc5f2e70 2024-05-27T07:38:04.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.256+0000 7f2002cbdb80 1 -- 172.21.15.12:0/4124032360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc5f2e70 msgr2=0x55e8dc5f2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.256+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f2720 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b780 tx=0x7f1fe402d000 comp rx=0 tx=0).stop 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.256+0000 7f2002cbdb80 1 -- 172.21.15.12:0/4124032360 shutdown_connections 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.256+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc473e40 0x55e8dc5fb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.256+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f2720 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.256+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/4124032360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.256+0000 7f2002cbdb80 1 -- 172.21.15.12:0/4124032360 >> 172.21.15.12:0/4124032360 conn(0x55e8dc50d390 msgr2=0x55e8dc5f3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.257+0000 7f2002cbdb80 1 -- 172.21.15.12:0/4124032360 shutdown_connections 2024-05-27T07:38:04.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.257+0000 7f2002cbdb80 1 -- 172.21.15.12:0/4124032360 wait complete. 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.257+0000 7f2002cbdb80 1 -- start start 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.257+0000 7f2002cbdb80 1 Processor -- start 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc473e40 0x55e8dc6052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f2002cbdb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc5ed150 con 0x55e8dc55f9f0 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc4b1760 con 0x55e8dc5f2e70 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f2002cbdb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e8dc605b00 con 0x55e8dc473e40 2024-05-27T07:38:04.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f2002cbdb80 10 client.0 _ll_register_callbacks cb 0x55e8dc4bac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.258+0000 7f1ff0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38520/0 (socket says 172.21.15.12:38520) 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 -- 172.21.15.12:0/3311567425 learned_addr learned my addr 172.21.15.12:0/3311567425 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff2200700 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff1800700 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc473e40 0x55e8dc6052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 -- 172.21.15.12:0/3311567425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc473e40 msgr2=0x55e8dc6052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc473e40 0x55e8dc6052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 -- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 msgr2=0x55e8dc5f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 -- 172.21.15.12:0/3311567425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8dc5f7600 con 0x55e8dc55f9f0 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1ff0e00700 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f6b00 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8000c00 tx=0x7f1fd80a9220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f64b000d92c279d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.259+0000 7f1fd3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:04.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.260+0000 7f1fd3400700 1 -- 172.21.15.12:0/3311567425 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80e6040 con 0x55e8dc55f9f0 2024-05-27T07:38:04.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.260+0000 7f1fd3400700 1 -- 172.21.15.12:0/3311567425 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9554+0+0 (secure 0 0 0) 0x7f1fd80a60a0 con 0x55e8dc55f9f0 2024-05-27T07:38:04.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.260+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fe4022040 con 0x55e8dc55f9f0 2024-05-27T07:38:04.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.260+0000 7f1fd3400700 1 -- 172.21.15.12:0/3311567425 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f1fd808ae00 con 0x55e8dc55f9f0 2024-05-27T07:38:04.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.260+0000 7f2002cbdb80 10 client.17289 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.260+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55e8dc603bd0 con 0x55e8dc55f9f0 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f1fd3400700 1 -- 172.21.15.12:0/3311567425 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1fe4022040 con 0x55e8dc55f9f0 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f1fd3400700 1 -- 172.21.15.12:0/3311567425 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f1fd80e9070 con 0x55e8dc55f9f0 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f2002cbdb80 10 client.17289 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f2002cbdb80 10 client.17289 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f2002cbdb80 20 client.17289 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc677da0 0x55e8dc67a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f2002cbdb80 4 client.17289 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.261+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8dc51c570 con 0x55e8dc677da0 2024-05-27T07:38:04.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.262+0000 7f1ff1800700 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc677da0 0x55e8dc67a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.263+0000 7f1ff1800700 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc677da0 0x55e8dc67a1c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3ac93ebcf947993b server_cookie=1b193122b96d7a91 in_seq=0 out_seq=0 2024-05-27T07:38:04.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.263+0000 7f1fd3400700 10 client.17289 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:38:04.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f1fd3400700 1 -- 172.21.15.12:0/3311567425 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8dc51c570 con 0x55e8dc677da0 2024-05-27T07:38:04.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f1fd3400700 10 client.17289 handle_command_reply: tid=0 2024-05-27T07:38:04.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 client.17289 shutdown 2024-05-27T07:38:04.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 msgr2=0x55e8dc5f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f6b00 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8000c00 tx=0x7f1fd80a9220 comp rx=0 tx=0).stop 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 shutdown_connections 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x55e8dc677da0 0x55e8dc67a1c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8dc5f2e70 0x55e8dc5f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8dc473e40 0x55e8dc6052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 --2- 172.21.15.12:0/3311567425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e8dc55f9f0 0x55e8dc5f6b00 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f1fd3400700 0 client.17289 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.264+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 >> 172.21.15.12:0/3311567425 conn(0x55e8dc50d390 msgr2=0x55e8dc5fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.265+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 shutdown_connections 2024-05-27T07:38:04.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.265+0000 7f2002cbdb80 1 -- 172.21.15.12:0/3311567425 wait complete. 2024-05-27T07:38:04.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.265+0000 7f2002cbdb80 20 client.17289 trim_cache size 0 max 16384 2024-05-27T07:38:04.263 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fd40b29f0 msgr2=0x7f1fd40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fd40b29f0 0x7f1fd40b4f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4028000 tx=0x7f1fe402b000 comp rx=0 tx=0).stop 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec0764a0 msgr2=0x7f1fec1a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec0764a0 0x7f1fec1a18d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e9a0 tx=0x7f1fd800ecb0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 shutdown_connections 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f1fd40b29f0 0x7f1fd40b4f00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fec115360 0x7f1fec1a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec075b50 0x7f1fec1a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 --2- 172.21.15.12:0/2298764895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1fec0764a0 0x7f1fec1a18d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.266+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 >> 172.21.15.12:0/2298764895 conn(0x7f1fec100030 msgr2=0x7f1fec1006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.267+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 shutdown_connections 2024-05-27T07:38:04.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:04.267+0000 7f1ff4200700 1 -- 172.21.15.12:0/2298764895 wait complete. 2024-05-27T07:38:05.244 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:38:05.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:05 smithi062 ceph-mon[35304]: pgmap v536: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 321 op/s 2024-05-27T07:38:05.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:05 smithi012 ceph-mon[31473]: pgmap v536: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 321 op/s 2024-05-27T07:38:05.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:05 smithi162 ceph-mon[35676]: pgmap v536: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 321 op/s 2024-05-27T07:38:05.748 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:07.483 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:07 smithi012 ceph-mon[31473]: pgmap v537: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 224 op/s 2024-05-27T07:38:07.483 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:07 smithi012 ceph-mon[31473]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:38:07.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:07 smithi162 ceph-mon[35676]: pgmap v537: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 224 op/s 2024-05-27T07:38:07.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:07 smithi162 ceph-mon[35676]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:38:07.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:07 smithi062 ceph-mon[35304]: pgmap v537: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 224 op/s 2024-05-27T07:38:07.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:07 smithi062 ceph-mon[35304]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:38:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 -- 172.21.15.12:0/2791939856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f80764a0 msgr2=0x7f69f8114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 --2- 172.21.15.12:0/2791939856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f80764a0 0x7f69f8114e20 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009990 tx=0x7f69e4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 -- 172.21.15.12:0/2791939856 shutdown_connections 2024-05-27T07:38:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 --2- 172.21.15.12:0/2791939856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69f8115360 0x7f69f8117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 --2- 172.21.15.12:0/2791939856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f8075b50 0x7f69f8075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 --2- 172.21.15.12:0/2791939856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f80764a0 0x7f69f8114e20 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 -- 172.21.15.12:0/2791939856 >> 172.21.15.12:0/2791939856 conn(0x7f69f8100030 msgr2=0x7f69f8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.109+0000 7f6a00200700 1 -- 172.21.15.12:0/2791939856 shutdown_connections 2024-05-27T07:38:08.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.110+0000 7f6a00200700 1 -- 172.21.15.12:0/2791939856 wait complete. 2024-05-27T07:38:08.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.111+0000 7f6a00200700 1 Processor -- start 2024-05-27T07:38:08.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.111+0000 7f6a00200700 1 -- start start 2024-05-27T07:38:08.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f6a00200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f8075b50 0x7f69f8114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f6a00200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f80764a0 0x7f69f810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f6a00200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69f8115360 0x7f69f810e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f6a00200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f69f811a960 con 0x7f69f8075b50 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f6a00200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f69f811a7e0 con 0x7f69f8115360 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f6a00200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69f811aae0 con 0x7f69f80764a0 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f69fd800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f8075b50 0x7f69f8114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f69fd800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f8075b50 0x7f69f8114d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38540/0 (socket says 172.21.15.12:38540) 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f69fd800700 1 -- 172.21.15.12:0/237384967 learned_addr learned my addr 172.21.15.12:0/237384967 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f69fe200700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69f8115360 0x7f69f810e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.112+0000 7f69fce00700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f80764a0 0x7f69f810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.113+0000 7f69fd800700 1 -- 172.21.15.12:0/237384967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f80764a0 msgr2=0x7f69f810deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.113+0000 7f69fd800700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f80764a0 0x7f69f810deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.113+0000 7f69fd800700 1 -- 172.21.15.12:0/237384967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69f8115360 msgr2=0x7f69f810e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.113+0000 7f69fd800700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69f8115360 0x7f69f810e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.113+0000 7f69fd800700 1 -- 172.21.15.12:0/237384967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f81128b0 con 0x7f69f8075b50 2024-05-27T07:38:08.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.114+0000 7f69fd800700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f8075b50 0x7f69f8114d80 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00d950 tx=0x7f69ec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:08.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.114+0000 7f69f6000700 1 -- 172.21.15.12:0/237384967 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec016070 con 0x7f69f8075b50 2024-05-27T07:38:08.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.114+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f8112ba0 con 0x7f69f8075b50 2024-05-27T07:38:08.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.114+0000 7f69f6000700 1 -- 172.21.15.12:0/237384967 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69ec009dd0 con 0x7f69f8075b50 2024-05-27T07:38:08.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.114+0000 7f69f6000700 1 -- 172.21.15.12:0/237384967 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec00f3c0 con 0x7f69f8075b50 2024-05-27T07:38:08.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.115+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f8113130 con 0x7f69f8075b50 2024-05-27T07:38:08.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.121+0000 7f69f6000700 1 -- 172.21.15.12:0/237384967 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f69ec019070 con 0x7f69f8075b50 2024-05-27T07:38:08.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.122+0000 7f69f6000700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f69e00b29f0 0x7f69e00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.123+0000 7f6a0eccbb80 1 Processor -- start 2024-05-27T07:38:08.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.123+0000 7f69f6000700 1 -- 172.21.15.12:0/237384967 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 9642+0+0 (secure 0 0 0) 0x7f69ec0e2700 con 0x7f69f8075b50 2024-05-27T07:38:08.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.123+0000 7f6a0eccbb80 1 -- start start 2024-05-27T07:38:08.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.123+0000 7f69fce00700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f69e00b29f0 0x7f69e00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.123+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.123+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7809e40 0x55ffe7988870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.123+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7988e70 0x55ffe7995110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ffe78b6e40 con 0x55ffe7809e40 2024-05-27T07:38:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ffe78a69e0 con 0x55ffe78f59f0 2024-05-27T07:38:08.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ffe78aaa20 con 0x55ffe7988e70 2024-05-27T07:38:08.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f69fce00700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f69e00b29f0 0x7f69e00b4f00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69f81036f0 tx=0x7f69e4005fb0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:08.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f69fe200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7988e70 0x55ffe7995110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f69fce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7809e40 0x55ffe7988870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f69fe200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7988e70 0x55ffe7995110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:50892/0 (socket says 172.21.15.12:50892) 2024-05-27T07:38:08.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f69fe200700 1 -- 172.21.15.12:0/3255978041 learned_addr learned my addr 172.21.15.12:0/3255978041 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:08.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.124+0000 7f69fd800700 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.125+0000 7f69fe200700 1 -- 172.21.15.12:0/3255978041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 msgr2=0x55ffe7988330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.125+0000 7f69fe200700 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.125+0000 7f69fe200700 1 -- 172.21.15.12:0/3255978041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7809e40 msgr2=0x55ffe7988870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.125+0000 7f69fe200700 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7809e40 0x55ffe7988870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.125+0000 7f69fe200700 1 -- 172.21.15.12:0/3255978041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe786f420 con 0x55ffe7988e70 2024-05-27T07:38:08.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.125+0000 7f69fe200700 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7988e70 0x55ffe7995110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009fd0 tx=0x7f69f0009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a8ca60e45a2df95 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.126+0000 7f69eaa00700 1 -- 172.21.15.12:0/3255978041 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f001a070 con 0x55ffe7988e70 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.126+0000 7f69eaa00700 1 -- 172.21.15.12:0/3255978041 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69f00093d0 con 0x55ffe7988e70 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.126+0000 7f69eaa00700 1 -- 172.21.15.12:0/3255978041 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f00153f0 con 0x55ffe7988e70 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3255978041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7988e70 msgr2=0x55ffe7995110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7988e70 0x55ffe7995110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009fd0 tx=0x7f69f0009920 comp rx=0 tx=0).stop 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3255978041 shutdown_connections 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7988e70 0x55ffe7995110 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3255978041 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7809e40 0x55ffe7988870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3255978041 >> 172.21.15.12:0/3255978041 conn(0x55ffe78a3390 msgr2=0x55ffe79844c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:08.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.127+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3255978041 shutdown_connections 2024-05-27T07:38:08.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.128+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3255978041 wait complete. 2024-05-27T07:38:08.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.128+0000 7f6a0eccbb80 1 -- start start 2024-05-27T07:38:08.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.129+0000 7f6a0eccbb80 1 Processor -- start 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.129+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe7992670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.129+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7992bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.129+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe79930f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.129+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ffe77acfa0 con 0x55ffe7988e70 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.129+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ffe7847760 con 0x55ffe78f59f0 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.129+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ffe7995b80 con 0x55ffe7809e40 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.130+0000 7f6a0eccbb80 10 client.0 _ll_register_callbacks cb 0x55ffe7850c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.130+0000 7f69fce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7992bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.130+0000 7f69fce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7992bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:53596/0 (socket says 172.21.15.12:53596) 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.130+0000 7f69fe200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe79930f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.130+0000 7f69fce00700 1 -- 172.21.15.12:0/324879063 learned_addr learned my addr 172.21.15.12:0/324879063 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:08.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.130+0000 7f69fd800700 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe7992670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69fce00700 1 -- 172.21.15.12:0/324879063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 msgr2=0x55ffe7992670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69fce00700 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe7992670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69fce00700 1 -- 172.21.15.12:0/324879063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 msgr2=0x55ffe79930f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69fce00700 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe79930f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69fce00700 1 -- 172.21.15.12:0/324879063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffe77f68d0 con 0x55ffe78f59f0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69fce00700 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7992bb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69e4000c00 tx=0x7f69e4009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d32e70fdcffd40 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69df400700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.131+0000 7f69df400700 1 -- 172.21.15.12:0/324879063 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e400f460 con 0x55ffe78f59f0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.132+0000 7f69df400700 1 -- 172.21.15.12:0/324879063 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9642+0+0 (secure 0 0 0) 0x7f69e400f600 con 0x55ffe78f59f0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.132+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69f00093d0 con 0x55ffe78f59f0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.132+0000 7f69df400700 1 -- 172.21.15.12:0/324879063 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f69e400f800 con 0x55ffe78f59f0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.132+0000 7f6a0eccbb80 10 client.36725 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.132+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55ffe7877060 con 0x55ffe78f59f0 2024-05-27T07:38:08.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.132+0000 7f69df400700 1 -- 172.21.15.12:0/324879063 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69f00093d0 con 0x55ffe78f59f0 2024-05-27T07:38:08.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f69df400700 1 -- 172.21.15.12:0/324879063 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f69e4022070 con 0x55ffe78f59f0 2024-05-27T07:38:08.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f6a0eccbb80 10 client.36725 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:08.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f6a0eccbb80 10 client.36725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:08.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f6a0eccbb80 20 client.36725 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:08.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe79a2f10 0x55ffe79a5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f6a0eccbb80 4 client.36725 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:08.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffe78b8d90 con 0x55ffe79a2f10 2024-05-27T07:38:08.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.133+0000 7f69fd800700 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe79a2f10 0x55ffe79a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.134+0000 7f69fd800700 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe79a2f10 0x55ffe79a5360 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1624c639383c267 server_cookie=6c319125239b9aeb in_seq=0 out_seq=0 2024-05-27T07:38:08.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.134+0000 7f69df400700 10 client.36725 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:08.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f69df400700 1 -- 172.21.15.12:0/324879063 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ffe78b8d90 con 0x55ffe79a2f10 2024-05-27T07:38:08.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f69df400700 10 client.36725 handle_command_reply: tid=0 2024-05-27T07:38:08.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 client.36725 shutdown 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 msgr2=0x55ffe7992bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7992bb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69e4000c00 tx=0x7f69e4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 shutdown_connections 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe79930f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe7992670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe79a2f10 0x55ffe79a5360 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/324879063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7992bb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f69df400700 0 client.36725 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:08.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.136+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 >> 172.21.15.12:0/324879063 conn(0x55ffe78a3390 msgr2=0x55ffe79883b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:08.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.137+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 shutdown_connections 2024-05-27T07:38:08.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.137+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/324879063 wait complete. 2024-05-27T07:38:08.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.137+0000 7f6a0eccbb80 20 client.36725 trim_cache size 0 max 16384 2024-05-27T07:38:08.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.160+0000 7f6a0eccbb80 1 Processor -- start 2024-05-27T07:38:08.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.161+0000 7f6a0eccbb80 1 -- start start 2024-05-27T07:38:08.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.161+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79915d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.161+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.161+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe798de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.161+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81bec70 con 0x55ffe7988e70 2024-05-27T07:38:08.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.161+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81bded0 con 0x55ffe78f59f0 2024-05-27T07:38:08.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.161+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81be5a0 con 0x55ffe7809e40 2024-05-27T07:38:08.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.162+0000 7f69fe200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe798de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.163+0000 7f69fe200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe798de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38568/0 (socket says 172.21.15.12:38568) 2024-05-27T07:38:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.163+0000 7f69fe200700 1 -- 172.21.15.12:0/3208471552 learned_addr learned my addr 172.21.15.12:0/3208471552 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.163+0000 7f69fce00700 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.163+0000 7f69fd800700 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.164+0000 7f69fe200700 1 -- 172.21.15.12:0/3208471552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 msgr2=0x55ffe79915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.164+0000 7f69fe200700 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79915d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.164+0000 7f69fe200700 1 -- 172.21.15.12:0/3208471552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 msgr2=0x55ffe7988830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.164+0000 7f69fe200700 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.164+0000 7f69fe200700 1 -- 172.21.15.12:0/3208471552 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe7985000 con 0x55ffe7988e70 2024-05-27T07:38:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.166+0000 7f69fe200700 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe798de20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009b60 tx=0x7f69f0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a56ebbeb7c34f4b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.166+0000 7f69ea000700 1 -- 172.21.15.12:0/3208471552 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f001a040 con 0x55ffe7988e70 2024-05-27T07:38:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.167+0000 7f69ea000700 1 -- 172.21.15.12:0/3208471552 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69f0004df0 con 0x55ffe7988e70 2024-05-27T07:38:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.167+0000 7f69ea000700 1 -- 172.21.15.12:0/3208471552 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f00042d0 con 0x55ffe7988e70 2024-05-27T07:38:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.168+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3208471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 msgr2=0x55ffe798de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.168+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe798de20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009b60 tx=0x7f69f0009e90 comp rx=0 tx=0).stop 2024-05-27T07:38:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.168+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3208471552 shutdown_connections 2024-05-27T07:38:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.168+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe7988e70 0x55ffe798de20 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.168+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.168+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/3208471552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe78f59f0 0x55ffe7988830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.168+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3208471552 >> 172.21.15.12:0/3208471552 conn(0x55ffe78a3390 msgr2=0x55ffe7988120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.169+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3208471552 shutdown_connections 2024-05-27T07:38:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.169+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/3208471552 wait complete. 2024-05-27T07:38:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.170+0000 7f6a0eccbb80 1 -- start start 2024-05-27T07:38:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.170+0000 7f6a0eccbb80 1 Processor -- start 2024-05-27T07:38:08.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.171+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79965f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.171+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe78f59f0 0x55ffe7996b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.171+0000 7f6a0eccbb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe7988e70 0x55ffe7a059f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.171+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ffe7990be0 con 0x55ffe78f59f0 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.171+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ffe79a5940 con 0x55ffe7988e70 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.171+0000 7f6a0eccbb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ffe7993730 con 0x55ffe7809e40 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.171+0000 7f6a0eccbb80 10 client.0 _ll_register_callbacks cb 0x55ffe7850c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.172+0000 7f69fd800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.172+0000 7f69fce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe78f59f0 0x55ffe7996b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.172+0000 7f69fd800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:50906/0 (socket says 172.21.15.12:50906) 2024-05-27T07:38:08.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.172+0000 7f69fd800700 1 -- 172.21.15.12:0/2628699947 learned_addr learned my addr 172.21.15.12:0/2628699947 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:08.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.172+0000 7f69fe200700 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe7988e70 0x55ffe7a059f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69fd800700 1 -- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe7988e70 msgr2=0x55ffe7a059f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69fd800700 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe7988e70 0x55ffe7a059f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69fd800700 1 -- 172.21.15.12:0/2628699947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe78f59f0 msgr2=0x55ffe7996b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69fd800700 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe78f59f0 0x55ffe7996b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69fd800700 1 -- 172.21.15.12:0/2628699947 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffe7a05fb0 con 0x55ffe7809e40 2024-05-27T07:38:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69fce00700 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe78f59f0 0x55ffe7996b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:38:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69fd800700 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79965f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0a67f0 tx=0x7f69ec0ad930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c7e8d073ec05c05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.173+0000 7f69df400700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:38:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.174+0000 7f69df400700 1 -- 172.21.15.12:0/2628699947 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec020040 con 0x55ffe7809e40 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.174+0000 7f69df400700 1 -- 172.21.15.12:0/2628699947 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9642+0+0 (secure 0 0 0) 0x7f69ec010460 con 0x55ffe7809e40 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.174+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69f0004df0 con 0x55ffe7809e40 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.174+0000 7f69df400700 1 -- 172.21.15.12:0/2628699947 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f69ec0a97e0 con 0x55ffe7809e40 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.174+0000 7f6a0eccbb80 10 client.26740 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.174+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55ffe7a0d340 con 0x55ffe7809e40 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f69df400700 1 -- 172.21.15.12:0/2628699947 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69f0004df0 con 0x55ffe7809e40 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f69df400700 1 -- 172.21.15.12:0/2628699947 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f69ec0a87e0 con 0x55ffe7809e40 2024-05-27T07:38:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f6a0eccbb80 10 client.26740 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f6a0eccbb80 10 client.26740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f6a0eccbb80 20 client.26740 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe7a0da70 0x55ffe7a0fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f6a0eccbb80 4 client.26740 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.175+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffe78b2570 con 0x55ffe7a0da70 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.176+0000 7f69fce00700 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe7a0da70 0x55ffe7a0fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.177+0000 7f69fce00700 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe7a0da70 0x55ffe7a0fe90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d9beeeefa58102b server_cookie=e887824a0edb480b in_seq=0 out_seq=0 2024-05-27T07:38:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.177+0000 7f69df400700 10 client.26740 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.177+0000 7f69df400700 1 -- 172.21.15.12:0/2628699947 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffe78b2570 con 0x55ffe7a0da70 2024-05-27T07:38:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.177+0000 7f69df400700 10 client.26740 handle_command_reply: tid=0 2024-05-27T07:38:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.177+0000 7f6a0eccbb80 1 client.26740 shutdown 2024-05-27T07:38:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 msgr2=0x55ffe79965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79965f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0a67f0 tx=0x7f69ec0ad930 comp rx=0 tx=0).stop 2024-05-27T07:38:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 shutdown_connections 2024-05-27T07:38:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ffe7a0da70 0x55ffe7a0fe90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffe7988e70 0x55ffe7a059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ffe7809e40 0x55ffe79965f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 --2- 172.21.15.12:0/2628699947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ffe78f59f0 0x55ffe7996b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 >> 172.21.15.12:0/2628699947 conn(0x55ffe78a3390 msgr2=0x55ffe7993b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f69df400700 0 client.26740 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.178+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 shutdown_connections 2024-05-27T07:38:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.179+0000 7f6a0eccbb80 1 -- 172.21.15.12:0/2628699947 wait complete. 2024-05-27T07:38:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.179+0000 7f6a0eccbb80 20 client.26740 trim_cache size 0 max 16384 2024-05-27T07:38:08.179 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:38:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f69e00b29f0 msgr2=0x7f69e00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f69e00b29f0 0x7f69e00b4f00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69f81036f0 tx=0x7f69e4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f8075b50 msgr2=0x7f69f8114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f8075b50 0x7f69f8114d80 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00d950 tx=0x7f69ec00dc60 comp rx=0 tx=0).stop 2024-05-27T07:38:08.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 shutdown_connections 2024-05-27T07:38:08.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f69e00b29f0 0x7f69e00b4f00 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69f81036f0 tx=0x7f69e4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69f8115360 0x7f69f810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69f8075b50 0x7f69f8114d80 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 --2- 172.21.15.12:0/237384967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f80764a0 0x7f69f810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:08.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.180+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 >> 172.21.15.12:0/237384967 conn(0x7f69f8100030 msgr2=0x7f69f8100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:08.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.181+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 shutdown_connections 2024-05-27T07:38:08.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:08.181+0000 7f6a00200700 1 -- 172.21.15.12:0/237384967 wait complete. 2024-05-27T07:38:09.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:09 smithi012 ceph-mon[31473]: pgmap v539: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 269 op/s 2024-05-27T07:38:09.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:09 smithi162 ceph-mon[35676]: pgmap v539: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 269 op/s 2024-05-27T07:38:09.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:09 smithi062 ceph-mon[35304]: pgmap v539: 129 pgs: 129 active+clean; 324 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 269 op/s 2024-05-27T07:38:10.160 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:38:10.160 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:38:10.682 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:11.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:11 smithi012 ceph-mon[31473]: pgmap v540: 129 pgs: 129 active+clean; 331 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-27T07:38:11.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:11 smithi162 ceph-mon[35676]: pgmap v540: 129 pgs: 129 active+clean; 331 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-27T07:38:11.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:11 smithi062 ceph-mon[35304]: pgmap v540: 129 pgs: 129 active+clean; 331 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 -- 172.21.15.12:0/2905528121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac0764a0 msgr2=0x7f72ac114c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 --2- 172.21.15.12:0/2905528121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac0764a0 0x7f72ac114c30 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7298009990 tx=0x7f7298009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 -- 172.21.15.12:0/2905528121 shutdown_connections 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 --2- 172.21.15.12:0/2905528121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72ac115170 0x7f72ac117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 --2- 172.21.15.12:0/2905528121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72ac075b50 0x7f72ac075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 --2- 172.21.15.12:0/2905528121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac0764a0 0x7f72ac114c30 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 -- 172.21.15.12:0/2905528121 >> 172.21.15.12:0/2905528121 conn(0x7f72ac079d00 msgr2=0x7f72ac07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 -- 172.21.15.12:0/2905528121 shutdown_connections 2024-05-27T07:38:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.177+0000 7f72b2000700 1 -- 172.21.15.12:0/2905528121 wait complete. 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.178+0000 7f72b2000700 1 Processor -- start 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.179+0000 7f72b2000700 1 -- start start 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.179+0000 7f72b2000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72ac075b50 0x7f72ac1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.179+0000 7f72b2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72ac0764a0 0x7f72ac1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.179+0000 7f72b2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac115170 0x7f72ac1a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.179+0000 7f72b2000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac11a730 con 0x7f72ac115170 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.179+0000 7f72b2000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac11a5b0 con 0x7f72ac0764a0 2024-05-27T07:38:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.179+0000 7f72b2000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac11a8b0 con 0x7f72ac075b50 2024-05-27T07:38:13.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac115170 0x7f72ac1a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac115170 0x7f72ac1a5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52636/0 (socket says 172.21.15.12:52636) 2024-05-27T07:38:13.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 -- 172.21.15.12:0/1558422946 learned_addr learned my addr 172.21.15.12:0/1558422946 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:13.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72abe00700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72ac0764a0 0x7f72ac1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b0c00700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72ac075b50 0x7f72ac1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 -- 172.21.15.12:0/1558422946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72ac075b50 msgr2=0x7f72ac1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72ac075b50 0x7f72ac1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 -- 172.21.15.12:0/1558422946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72ac0764a0 msgr2=0x7f72ac1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72ac0764a0 0x7f72ac1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72b1600700 1 -- 172.21.15.12:0/1558422946 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac1a6130 con 0x7f72ac115170 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.180+0000 7f72abe00700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72ac0764a0 0x7f72ac1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.181+0000 7f72b1600700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac115170 0x7f72ac1a5a90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f72a400ada0 tx=0x7f72a400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.181+0000 7f72a9600700 1 -- 172.21.15.12:0/1558422946 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4017020 con 0x7f72ac115170 2024-05-27T07:38:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.181+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ac1a6420 con 0x7f72ac115170 2024-05-27T07:38:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.181+0000 7f72a9600700 1 -- 172.21.15.12:0/1558422946 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72a400ce10 con 0x7f72ac115170 2024-05-27T07:38:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.181+0000 7f72a9600700 1 -- 172.21.15.12:0/1558422946 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4005440 con 0x7f72ac115170 2024-05-27T07:38:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.181+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ac1add40 con 0x7f72ac115170 2024-05-27T07:38:13.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.185+0000 7f72a9600700 1 -- 172.21.15.12:0/1558422946 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f72a4004ad0 con 0x7f72ac115170 2024-05-27T07:38:13.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 Processor -- start 2024-05-27T07:38:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 -- start start 2024-05-27T07:38:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72a9600700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72940b29f0 0x7f72940b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72a9600700 1 -- 172.21.15.12:0/1558422946 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f72a40e2220 con 0x7f72ac115170 2024-05-27T07:38:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015383ec0 0x55c015501f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015502490 0x55c01550cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c015430ec0 con 0x55c015383ec0 2024-05-27T07:38:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c015420a60 con 0x55c015502490 2024-05-27T07:38:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.187+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c015424aa0 con 0x55c0155028a0 2024-05-27T07:38:13.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72abe00700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72940b29f0 0x7f72940b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015383ec0 0x55c015501f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015383ec0 0x55c015501f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52652/0 (socket says 172.21.15.12:52652) 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 -- 172.21.15.12:0/317669991 learned_addr learned my addr 172.21.15.12:0/317669991 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72abe00700 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015502490 0x55c01550cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b1600700 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 -- 172.21.15.12:0/317669991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 msgr2=0x55c01550f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 -- 172.21.15.12:0/317669991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015502490 msgr2=0x55c01550cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015502490 0x55c01550cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72b0c00700 1 -- 172.21.15.12:0/317669991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0153e94a0 con 0x55c015383ec0 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.188+0000 7f72abe00700 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015502490 0x55c01550cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.189+0000 7f72abe00700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72940b29f0 0x7f72940b4f00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f72ac116ef0 tx=0x7f7298004310 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.189+0000 7f72b0c00700 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015383ec0 0x55c015501f50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f72a000bd70 tx=0x7f72a000f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195e3a9f0d3caaa2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.189+0000 7f729e000700 1 -- 172.21.15.12:0/317669991 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0015070 con 0x55c015383ec0 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.189+0000 7f729e000700 1 -- 172.21.15.12:0/317669991 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72a000fda0 con 0x55c015383ec0 2024-05-27T07:38:13.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.189+0000 7f729e000700 1 -- 172.21.15.12:0/317669991 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00116a0 con 0x55c015383ec0 2024-05-27T07:38:13.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.190+0000 7f72c20f9b80 1 -- 172.21.15.12:0/317669991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015383ec0 msgr2=0x55c015501f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.190+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015383ec0 0x55c015501f50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f72a000bd70 tx=0x7f72a000f640 comp rx=0 tx=0).stop 2024-05-27T07:38:13.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.190+0000 7f72c20f9b80 1 -- 172.21.15.12:0/317669991 shutdown_connections 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.190+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.190+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015383ec0 0x55c015501f50 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.190+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/317669991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015502490 0x55c01550cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.190+0000 7f72c20f9b80 1 -- 172.21.15.12:0/317669991 >> 172.21.15.12:0/317669991 conn(0x55c0153e4040 msgr2=0x55c0154fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.191+0000 7f72c20f9b80 1 -- 172.21.15.12:0/317669991 shutdown_connections 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.191+0000 7f72c20f9b80 1 -- 172.21.15.12:0/317669991 wait complete. 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.191+0000 7f72c20f9b80 1 -- start start 2024-05-27T07:38:13.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.191+0000 7f72c20f9b80 1 Processor -- start 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c01550c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c01550cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c015327020 con 0x55c015502490 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c0153c17e0 con 0x55c015383ec0 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c01550fa10 con 0x55c0155028a0 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72c20f9b80 10 client.0 _ll_register_callbacks cb 0x55c0153cacf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.192+0000 7f72abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c01550cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c01550cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52658/0 (socket says 172.21.15.12:52658) 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 -- 172.21.15.12:0/118277679 learned_addr learned my addr 172.21.15.12:0/118277679 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72b0c00700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c01550c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 -- 172.21.15.12:0/118277679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 msgr2=0x55c01550d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72b1600700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 -- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 msgr2=0x55c01550c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c01550c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 -- 172.21.15.12:0/118277679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c015370950 con 0x55c015502490 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72b1600700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f72abe00700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c01550cb80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f7298005770 tx=0x7f7298024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844c8b03e9454cc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.193+0000 7f7292a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.194+0000 7f7292a00700 1 -- 172.21.15.12:0/118277679 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729800bca0 con 0x55c015502490 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.194+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a000fda0 con 0x55c015502490 2024-05-27T07:38:13.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.194+0000 7f7292a00700 1 -- 172.21.15.12:0/118277679 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f729800be40 con 0x55c015502490 2024-05-27T07:38:13.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.194+0000 7f7292a00700 1 -- 172.21.15.12:0/118277679 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f729803a050 con 0x55c015502490 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.194+0000 7f72c20f9b80 10 client.17319 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.194+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55c0153f10e0 con 0x55c015502490 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f7292a00700 1 -- 172.21.15.12:0/118277679 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72a000fda0 con 0x55c015502490 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f7292a00700 1 -- 172.21.15.12:0/118277679 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f729802f020 con 0x55c015502490 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f72c20f9b80 10 client.17319 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f72c20f9b80 10 client.17319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f72c20f9b80 20 client.17319 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c01551ce20 0x55c01551f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f72c20f9b80 4 client.17319 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.195+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c015432e10 con 0x55c01551ce20 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.196+0000 7f72b0c00700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c01551ce20 0x55c01551f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.197+0000 7f72b0c00700 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c01551ce20 0x55c01551f290 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf54f44a67942232 server_cookie=8cf2acd86c9b1c34 in_seq=0 out_seq=0 2024-05-27T07:38:13.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.197+0000 7f7292a00700 10 client.17319 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f7292a00700 1 -- 172.21.15.12:0/118277679 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c015432e10 con 0x55c01551ce20 2024-05-27T07:38:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f7292a00700 10 client.17319 handle_command_reply: tid=0 2024-05-27T07:38:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 client.17319 shutdown 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 msgr2=0x55c01550cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c01550cb80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f7298005770 tx=0x7f7298024000 comp rx=0 tx=0).stop 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 shutdown_connections 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c01551ce20 0x55c01551f290 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c01550c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c01550d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/118277679 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c01550cb80 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 >> 172.21.15.12:0/118277679 conn(0x55c0153e4040 msgr2=0x55c0154fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f7292a00700 0 client.17319 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 shutdown_connections 2024-05-27T07:38:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.199+0000 7f72c20f9b80 1 -- 172.21.15.12:0/118277679 wait complete. 2024-05-27T07:38:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.200+0000 7f72c20f9b80 20 client.17319 trim_cache size 0 max 16384 2024-05-27T07:38:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 Processor -- start 2024-05-27T07:38:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 -- start start 2024-05-27T07:38:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015507e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c015502490 0x55c015518ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0155028a0 0x55c015524a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1b60d0 con 0x55c0155028a0 2024-05-27T07:38:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1b5330 con 0x55c015383ec0 2024-05-27T07:38:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.226+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1b5a00 con 0x55c015502490 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.227+0000 7f72b1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0155028a0 0x55c015524a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.227+0000 7f72b1600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0155028a0 0x55c015524a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52664/0 (socket says 172.21.15.12:52664) 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.227+0000 7f72b1600700 1 -- 172.21.15.12:0/2133890935 learned_addr learned my addr 172.21.15.12:0/2133890935 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.227+0000 7f72abe00700 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c015502490 0x55c015518ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.227+0000 7f72b0c00700 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015507e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.227+0000 7f72b1600700 1 -- 172.21.15.12:0/2133890935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c015502490 msgr2=0x55c015518ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.227+0000 7f72b1600700 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c015502490 0x55c015518ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.228+0000 7f72b1600700 1 -- 172.21.15.12:0/2133890935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 msgr2=0x55c015507e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.228+0000 7f72b1600700 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015507e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.228+0000 7f72b1600700 1 -- 172.21.15.12:0/2133890935 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c01550d700 con 0x55c0155028a0 2024-05-27T07:38:13.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.228+0000 7f72b1600700 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0155028a0 0x55c015524a90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f72a400a0a0 tx=0x7f72a40a8cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec528fa58f62fae2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.228+0000 7f729d600700 1 -- 172.21.15.12:0/2133890935 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a40a9120 con 0x55c0155028a0 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.228+0000 7f729d600700 1 -- 172.21.15.12:0/2133890935 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72a40ad4d0 con 0x55c0155028a0 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.229+0000 7f729d600700 1 -- 172.21.15.12:0/2133890935 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a40b1a60 con 0x55c0155028a0 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 -- 172.21.15.12:0/2133890935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0155028a0 msgr2=0x55c015524a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0155028a0 0x55c015524a90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f72a400a0a0 tx=0x7f72a40a8cb0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 -- 172.21.15.12:0/2133890935 shutdown_connections 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015507e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0155028a0 0x55c015524a90 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/2133890935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c015502490 0x55c015518ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 -- 172.21.15.12:0/2133890935 >> 172.21.15.12:0/2133890935 conn(0x55c0153e4040 msgr2=0x55c01550ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 -- 172.21.15.12:0/2133890935 shutdown_connections 2024-05-27T07:38:13.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 -- 172.21.15.12:0/2133890935 wait complete. 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.230+0000 7f72c20f9b80 1 -- start start 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 1 Processor -- start 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015515230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c0155115a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c015511ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c01550b970 con 0x55c015502490 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c01550b7f0 con 0x55c015383ec0 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c0154fd430 con 0x55c0155028a0 2024-05-27T07:38:13.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.231+0000 7f72c20f9b80 10 client.0 _ll_register_callbacks cb 0x55c0153cacf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c0155115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c0155115a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52666/0 (socket says 172.21.15.12:52666) 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 -- 172.21.15.12:0/291318781 learned_addr learned my addr 172.21.15.12:0/291318781 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72b0c00700 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015515230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72b1600700 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c015511ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 -- 172.21.15.12:0/291318781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 msgr2=0x55c015511ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c015511ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 -- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 msgr2=0x55c015515230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015515230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.232+0000 7f72abe00700 1 -- 172.21.15.12:0/291318781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c015512120 con 0x55c015502490 2024-05-27T07:38:13.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f72abe00700 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c0155115a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f7298000c00 tx=0x7f7298003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19127366b0636c25 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:13.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f7292a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f7292a00700 1 -- 172.21.15.12:0/291318781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298028630 con 0x55c015502490 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f7292a00700 1 -- 172.21.15.12:0/291318781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f729800baa0 con 0x55c015502490 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a40ad4d0 con 0x55c015502490 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f7292a00700 1 -- 172.21.15.12:0/291318781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f729804d050 con 0x55c015502490 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f72c20f9b80 10 client.17325 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.233+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55c015588740 con 0x55c015502490 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f7292a00700 1 -- 172.21.15.12:0/291318781 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72a40ad4d0 con 0x55c015502490 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f7292a00700 1 -- 172.21.15.12:0/291318781 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f729802f040 con 0x55c015502490 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f72c20f9b80 10 client.17325 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f72c20f9b80 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f72c20f9b80 20 client.17325 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:13.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c015588be0 0x55c01558b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:13.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f72c20f9b80 4 client.17325 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:38:13.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.234+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c01542c5f0 con 0x55c015588be0 2024-05-27T07:38:13.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.235+0000 7f72b0c00700 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c015588be0 0x55c01558b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:13.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.236+0000 7f72b0c00700 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c015588be0 0x55c01558b000 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93760e6909824827 server_cookie=5e00b682899b31e4 in_seq=0 out_seq=0 2024-05-27T07:38:13.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.236+0000 7f7292a00700 10 client.17325 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:13.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f7292a00700 1 -- 172.21.15.12:0/291318781 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c01542c5f0 con 0x55c015588be0 2024-05-27T07:38:13.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f7292a00700 10 client.17325 handle_command_reply: tid=0 2024-05-27T07:38:13.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 client.17325 shutdown 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 msgr2=0x55c0155115a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c0155115a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f7298000c00 tx=0x7f7298003700 comp rx=0 tx=0).stop 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 shutdown_connections 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c015588be0 0x55c01558b000 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c015383ec0 0x55c015515230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0155028a0 0x55c015511ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 --2- 172.21.15.12:0/291318781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c015502490 0x55c0155115a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 >> 172.21.15.12:0/291318781 conn(0x55c0153e4040 msgr2=0x55c01550cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.240+0000 7f7292a00700 0 client.17325 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.241+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 shutdown_connections 2024-05-27T07:38:13.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.241+0000 7f72c20f9b80 1 -- 172.21.15.12:0/291318781 wait complete. 2024-05-27T07:38:13.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.241+0000 7f72c20f9b80 20 client.17325 trim_cache size 0 max 16384 2024-05-27T07:38:13.239 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:38:13.239 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:38:13.239 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "000c87b2-12a5-4555-a728-d5a4e68d914c", 2024-05-27T07:38:13.239 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:38:13.240 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:38:13.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72940b29f0 msgr2=0x7f72940b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72940b29f0 0x7f72940b4f00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f72ac116ef0 tx=0x7f7298004310 comp rx=0 tx=0).stop 2024-05-27T07:38:13.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac115170 msgr2=0x7f72ac1a5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:13.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac115170 0x7f72ac1a5a90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f72a400ada0 tx=0x7f72a400c330 comp rx=0 tx=0).stop 2024-05-27T07:38:13.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 shutdown_connections 2024-05-27T07:38:13.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f72940b29f0 0x7f72940b4f00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac115170 0x7f72ac1a5a90 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72ac075b50 0x7f72ac1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 --2- 172.21.15.12:0/1558422946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72ac0764a0 0x7f72ac1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:13.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 >> 172.21.15.12:0/1558422946 conn(0x7f72ac079d00 msgr2=0x7f72ac1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:13.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 shutdown_connections 2024-05-27T07:38:13.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:13.242+0000 7f72b2000700 1 -- 172.21.15.12:0/1558422946 wait complete. 2024-05-27T07:38:13.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:13 smithi012 ceph-mon[31473]: pgmap v541: 129 pgs: 129 active+clean; 331 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-27T07:38:13.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:13 smithi012 ceph-mon[31473]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:38:13.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:13 smithi162 ceph-mon[35676]: pgmap v541: 129 pgs: 129 active+clean; 331 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-27T07:38:13.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:13 smithi162 ceph-mon[35676]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:38:13.741 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:13 smithi062 ceph-mon[35304]: pgmap v541: 129 pgs: 129 active+clean; 331 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-27T07:38:13.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:13 smithi062 ceph-mon[35304]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:38:14.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:38:14.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:14 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:38:14.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:14 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:14 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:38:14.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:14 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:38:14.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:14 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:38:14.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:14 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:14 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:14.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:14 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:38:14.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:14 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:14 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:38:14.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:14 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:38:14.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:14 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:38:14.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:14 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:14 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:14 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:38:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:14 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:14 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:38:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:14 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:38:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:14 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:38:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:14 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:14.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:14 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:15.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:15 smithi012 ceph-mon[31473]: pgmap v543: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 304 op/s 2024-05-27T07:38:15.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:15 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:15.634 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:15 smithi162 ceph-mon[35676]: pgmap v543: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 304 op/s 2024-05-27T07:38:15.634 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:15 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:15.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:15 smithi062 ceph-mon[35304]: pgmap v543: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 304 op/s 2024-05-27T07:38:15.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:15 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 -- 172.21.15.12:0/1529219173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c34108f10 msgr2=0x7f6c34109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 --2- 172.21.15.12:0/1529219173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c34108f10 0x7f6c34109320 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0099c0 tx=0x7f6c1c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 -- 172.21.15.12:0/1529219173 shutdown_connections 2024-05-27T07:38:17.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 --2- 172.21.15.12:0/1529219173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c341045e0 0x7f6c341049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 --2- 172.21.15.12:0/1529219173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c34109860 0x7f6c341111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 --2- 172.21.15.12:0/1529219173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c34108f10 0x7f6c34109320 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 -- 172.21.15.12:0/1529219173 >> 172.21.15.12:0/1529219173 conn(0x7f6c34100030 msgr2=0x7f6c34102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 -- 172.21.15.12:0/1529219173 shutdown_connections 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.392+0000 7f6c39800700 1 -- 172.21.15.12:0/1529219173 wait complete. 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.393+0000 7f6c39800700 1 Processor -- start 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.394+0000 7f6c39800700 1 -- start start 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.394+0000 7f6c39800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c341045e0 0x7f6c34075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.394+0000 7f6c39800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c34108f10 0x7f6c34075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.394+0000 7f6c39800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c34109860 0x7f6c34076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.394+0000 7f6c39800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6c34114320 con 0x7f6c341045e0 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.394+0000 7f6c39800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6c341141a0 con 0x7f6c34108f10 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.394+0000 7f6c39800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6c341144a0 con 0x7f6c34109860 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c341045e0 0x7f6c34075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c341045e0 0x7f6c34075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52684/0 (socket says 172.21.15.12:52684) 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 -- 172.21.15.12:0/736614277 learned_addr learned my addr 172.21.15.12:0/736614277 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c33400700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c34109860 0x7f6c34076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 -- 172.21.15.12:0/736614277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c34109860 msgr2=0x7f6c34076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c34109860 0x7f6c34076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 -- 172.21.15.12:0/736614277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c34108f10 msgr2=0x7f6c34075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c34108f10 0x7f6c34075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 -- 172.21.15.12:0/736614277 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c341a9080 con 0x7f6c341045e0 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c32a00700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c341045e0 0x7f6c34075ab0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6c24007f10 tx=0x7f6c2400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:17.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c2be00700 1 -- 172.21.15.12:0/736614277 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24003a00 con 0x7f6c341045e0 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c2be00700 1 -- 172.21.15.12:0/736614277 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c24003ba0 con 0x7f6c341045e0 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c341a9370 con 0x7f6c341045e0 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c2be00700 1 -- 172.21.15.12:0/736614277 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24015660 con 0x7f6c341045e0 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.396+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c341a9900 con 0x7f6c341045e0 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.402+0000 7f6c2be00700 1 -- 172.21.15.12:0/736614277 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6c24004000 con 0x7f6c341045e0 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.402+0000 7f6c4827fb80 1 Processor -- start 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c4827fb80 1 -- start start 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455007f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621454fa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 0x5621454fb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562145428e40 con 0x56214537be40 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5621454189e0 con 0x562145500d30 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56214541ca20 con 0x5621454fadb0 2024-05-27T07:38:17.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.403+0000 7f6c33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c33400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52688/0 (socket says 172.21.15.12:52688) 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c33400700 1 -- 172.21.15.12:0/3898396208 learned_addr learned my addr 172.21.15.12:0/3898396208 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c2a000700 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 0x5621454fb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c32a00700 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621454fa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c2be00700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6c180b2a70 0x7f6c180b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c33400700 1 -- 172.21.15.12:0/3898396208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 msgr2=0x5621454fb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c33400700 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 0x5621454fb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c33400700 1 -- 172.21.15.12:0/3898396208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 msgr2=0x5621454fa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c2be00700 1 -- 172.21.15.12:0/736614277 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f6c240e2810 con 0x7f6c341045e0 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c33400700 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621454fa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c33400700 1 -- 172.21.15.12:0/3898396208 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621453e1420 con 0x56214537be40 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.404+0000 7f6c32a00700 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621454fa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.405+0000 7f6c33400700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6c180b2a70 0x7f6c180b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.405+0000 7f6c33400700 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455007f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00cd80 tx=0x7f6c2c00e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380121ac448fbe1c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.405+0000 7f6c23e00700 1 -- 172.21.15.12:0/3898396208 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c022070 con 0x56214537be40 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.405+0000 7f6c23e00700 1 -- 172.21.15.12:0/3898396208 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c2c018070 con 0x56214537be40 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.406+0000 7f6c23e00700 1 -- 172.21.15.12:0/3898396208 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c010420 con 0x56214537be40 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.406+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3898396208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 msgr2=0x5621455007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.406+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455007f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00cd80 tx=0x7f6c2c00e630 comp rx=0 tx=0).stop 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.406+0000 7f6c33400700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6c180b2a70 0x7f6c180b4f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c01f000 tx=0x7f6c2c020000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3898396208 shutdown_connections 2024-05-27T07:38:17.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 0x5621454fb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455007f0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3898396208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621454fa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3898396208 >> 172.21.15.12:0/3898396208 conn(0x5621453dbfc0 msgr2=0x5621454f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3898396208 shutdown_connections 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3898396208 wait complete. 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.407+0000 7f6c4827fb80 1 -- start start 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 1 Processor -- start 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455045c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5621454fadb0 0x562145504b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x562145505040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56214531efa0 con 0x56214537be40 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5621453b9760 con 0x5621454fadb0 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562145507800 con 0x562145500d30 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c4827fb80 10 client.0 _ll_register_callbacks cb 0x5621453c2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52704/0 (socket says 172.21.15.12:52704) 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.408+0000 7f6c32a00700 1 -- 172.21.15.12:0/978513682 learned_addr learned my addr 172.21.15.12:0/978513682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c2a000700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5621454fadb0 0x562145504b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c33400700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x562145505040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c32a00700 1 -- 172.21.15.12:0/978513682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 msgr2=0x562145505040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c32a00700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x562145505040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c32a00700 1 -- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5621454fadb0 msgr2=0x562145504b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c32a00700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5621454fadb0 0x562145504b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c32a00700 1 -- 172.21.15.12:0/978513682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621453688d0 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c33400700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x562145505040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c32a00700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455045c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f6c240dfb50 tx=0x7f6c24004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46a88466a8ae241 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.409+0000 7f6c20c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.410+0000 7f6c20c00700 1 -- 172.21.15.12:0/978513682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b24e0 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.410+0000 7f6c20c00700 1 -- 172.21.15.12:0/978513682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f6c240042f0 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.410+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c2c018070 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.410+0000 7f6c20c00700 1 -- 172.21.15.12:0/978513682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f6c240a9680 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.410+0000 7f6c4827fb80 10 client.17337 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.410+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5621453e9060 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c20c00700 1 -- 172.21.15.12:0/978513682 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c2c018070 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c20c00700 1 -- 172.21.15.12:0/978513682 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f6c240b2050 con 0x56214537be40 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c4827fb80 10 client.17337 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:17.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c4827fb80 10 client.17337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c4827fb80 20 client.17337 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x562145514ab0 0x562145516f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c4827fb80 4 client.17337 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56214542ad90 con 0x562145514ab0 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.411+0000 7f6c2a000700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x562145514ab0 0x562145516f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.413+0000 7f6c2a000700 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x562145514ab0 0x562145516f30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=159e47a2be6975f2 server_cookie=49567f93f64aee9b in_seq=0 out_seq=0 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.413+0000 7f6c20c00700 10 client.17337 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c20c00700 1 -- 172.21.15.12:0/978513682 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56214542ad90 con 0x562145514ab0 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c20c00700 10 client.17337 handle_command_reply: tid=0 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 client.17337 shutdown 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 msgr2=0x5621455045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455045c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f6c240dfb50 tx=0x7f6c24004920 comp rx=0 tx=0).stop 2024-05-27T07:38:17.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 shutdown_connections 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x562145505040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621455045c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x562145514ab0 0x562145516f30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/978513682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5621454fadb0 0x562145504b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 >> 172.21.15.12:0/978513682 conn(0x5621453dbfc0 msgr2=0x5621454fb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c20c00700 0 client.17337 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 shutdown_connections 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.415+0000 7f6c4827fb80 1 -- 172.21.15.12:0/978513682 wait complete. 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.416+0000 7f6c4827fb80 20 client.17337 trim_cache size 0 max 16384 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 Processor -- start 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 -- start start 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621454fa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 0x5621454feb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621455068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6c341b1e40 con 0x56214537be40 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6c341b10a0 con 0x562145500d30 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6c341b1770 con 0x5621454fadb0 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621454fa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c32a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621454fa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52708/0 (socket says 172.21.15.12:52708) 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.441+0000 7f6c32a00700 1 -- 172.21.15.12:0/867164779 learned_addr learned my addr 172.21.15.12:0/867164779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c33400700 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621455068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c32a00700 1 -- 172.21.15.12:0/867164779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 msgr2=0x5621454feb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c32a00700 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 0x5621454feb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c32a00700 1 -- 172.21.15.12:0/867164779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 msgr2=0x5621455068d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c32a00700 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621455068d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c32a00700 1 -- 172.21.15.12:0/867164779 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621455053b0 con 0x56214537be40 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c33400700 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621455068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c32a00700 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621454fa870 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f6c240a68f0 tx=0x7f6c240a9190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9273d7dd9f6133 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c23400700 1 -- 172.21.15.12:0/867164779 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240a86b0 con 0x56214537be40 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c23400700 1 -- 172.21.15.12:0/867164779 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c240a5c70 con 0x56214537be40 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.442+0000 7f6c23400700 1 -- 172.21.15.12:0/867164779 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b2050 con 0x56214537be40 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.443+0000 7f6c4827fb80 1 -- 172.21.15.12:0/867164779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 msgr2=0x5621454fa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.443+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621454fa870 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f6c240a68f0 tx=0x7f6c240a9190 comp rx=0 tx=0).stop 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 -- 172.21.15.12:0/867164779 shutdown_connections 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562145500d30 0x5621455068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56214537be40 0x5621454fa870 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/867164779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621454fadb0 0x5621454feb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 -- 172.21.15.12:0/867164779 >> 172.21.15.12:0/867164779 conn(0x5621453dbfc0 msgr2=0x5621454ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:17.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 -- 172.21.15.12:0/867164779 shutdown_connections 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 -- 172.21.15.12:0/867164779 wait complete. 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 -- start start 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.444+0000 7f6c4827fb80 1 Processor -- start 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56214537be40 0x56214550cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621454fadb0 0x56214550d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c4827fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x56214551be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562145503890 con 0x5621454fadb0 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5621454fbb60 con 0x56214537be40 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c4827fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5621454f6e00 con 0x562145500d30 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c4827fb80 10 client.0 _ll_register_callbacks cb 0x5621453c2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c2a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621454fadb0 0x56214550d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c2a000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621454fadb0 0x56214550d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52718/0 (socket says 172.21.15.12:52718) 2024-05-27T07:38:17.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.445+0000 7f6c2a000700 1 -- 172.21.15.12:0/3075265064 learned_addr learned my addr 172.21.15.12:0/3075265064 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c32a00700 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56214537be40 0x56214550cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c33400700 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x56214551be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c2a000700 1 -- 172.21.15.12:0/3075265064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 msgr2=0x56214551be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c2a000700 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x56214551be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c2a000700 1 -- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56214537be40 msgr2=0x56214550cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c2a000700 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56214537be40 0x56214550cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c2a000700 1 -- 172.21.15.12:0/3075265064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56214551c5a0 con 0x5621454fadb0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c2a000700 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621454fadb0 0x56214550d320 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c009990 tx=0x7f6c1c009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4139fac406161229 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c20c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c20c00700 1 -- 172.21.15.12:0/3075265064 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00b990 con 0x5621454fadb0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c240a5c70 con 0x5621454fadb0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.446+0000 7f6c20c00700 1 -- 172.21.15.12:0/3075265064 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f6c1c00bb30 con 0x5621454fadb0 2024-05-27T07:38:17.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c20c00700 1 -- 172.21.15.12:0/3075265064 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f6c1c032050 con 0x5621454fadb0 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 10 client.17343 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56214557f120 con 0x5621454fadb0 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c20c00700 1 -- 172.21.15.12:0/3075265064 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6c240a5c70 con 0x5621454fadb0 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c20c00700 1 -- 172.21.15.12:0/3075265064 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f6c1c00f460 con 0x5621454fadb0 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 10 client.17343 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 10 client.17343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 20 client.17343 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56214557f860 0x562145581c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 4 client.17343 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.447+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562145424570 con 0x56214557f860 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.448+0000 7f6c32a00700 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56214557f860 0x562145581c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.449+0000 7f6c32a00700 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56214557f860 0x562145581c80 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73dd2c802bb61c17 server_cookie=fde9742d5bf2b43d in_seq=0 out_seq=0 2024-05-27T07:38:17.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.449+0000 7f6c20c00700 10 client.17343 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c20c00700 1 -- 172.21.15.12:0/3075265064 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562145424570 con 0x56214557f860 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c20c00700 10 client.17343 handle_command_reply: tid=0 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 client.17343 shutdown 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621454fadb0 msgr2=0x56214550d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621454fadb0 0x56214550d320 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c009990 tx=0x7f6c1c009710 comp rx=0 tx=0).stop 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 shutdown_connections 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56214557f860 0x562145581c80 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56214537be40 0x56214550cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562145500d30 0x56214551be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 --2- 172.21.15.12:0/3075265064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621454fadb0 0x56214550d320 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c20c00700 0 client.17343 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 >> 172.21.15.12:0/3075265064 conn(0x5621453dbfc0 msgr2=0x5621454fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 shutdown_connections 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 1 -- 172.21.15.12:0/3075265064 wait complete. 2024-05-27T07:38:17.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.450+0000 7f6c4827fb80 20 client.17343 trim_cache size 0 max 16384 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: "000c87b2-12a5-4555-a728-d5a4e68d914c": { 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "000c87b2-12a5-4555-a728-d5a4e68d914c", 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:38:17.474 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6c180b2a70 msgr2=0x7f6c180b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6c180b2a70 0x7f6c180b4f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c01f000 tx=0x7f6c2c020000 comp rx=0 tx=0).stop 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c341045e0 msgr2=0x7f6c34075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c341045e0 0x7f6c34075ab0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6c24007f10 tx=0x7f6c2400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 shutdown_connections 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6c180b2a70 0x7f6c180b4f80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c34109860 0x7f6c34076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c341045e0 0x7f6c34075ab0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 --2- 172.21.15.12:0/736614277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c34108f10 0x7f6c34075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.451+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 >> 172.21.15.12:0/736614277 conn(0x7f6c34100030 msgr2=0x7f6c34102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:17.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.452+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 shutdown_connections 2024-05-27T07:38:17.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:17.452+0000 7f6c39800700 1 -- 172.21.15.12:0/736614277 wait complete. 2024-05-27T07:38:17.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:17 smithi012 ceph-mon[31473]: pgmap v544: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 248 op/s 2024-05-27T07:38:17.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:17 smithi162 ceph-mon[35676]: pgmap v544: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 248 op/s 2024-05-27T07:38:17.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:17 smithi062 ceph-mon[35304]: pgmap v544: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 248 op/s 2024-05-27T07:38:18.375 INFO:tasks.cephfs.filesystem:scrub status for tag:000c87b2-12a5-4555-a728-d5a4e68d914c - {'path': '/', 'tag': '000c87b2-12a5-4555-a728-d5a4e68d914c', 'options': 'recursive,force'} 2024-05-27T07:38:19.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:19 smithi012 ceph-mon[31473]: pgmap v545: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 243 op/s 2024-05-27T07:38:19.596 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:19 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:19.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:19 smithi162 ceph-mon[35676]: pgmap v545: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 243 op/s 2024-05-27T07:38:19.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:19 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:19.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:19 smithi062 ceph-mon[35304]: pgmap v545: 129 pgs: 129 active+clean; 340 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 243 op/s 2024-05-27T07:38:19.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:19 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:20.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:20 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:20.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:20 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:20.747 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:20 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:20.747 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:20 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:20.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:20 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:20.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:20 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:21.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:21 smithi162 ceph-mon[35676]: pgmap v546: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 242 op/s 2024-05-27T07:38:21.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:21 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:38:21.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:21 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:38:21.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:21 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:21.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:21 smithi062 ceph-mon[35304]: pgmap v546: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 242 op/s 2024-05-27T07:38:21.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:21 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:38:21.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:21 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:38:21.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:21 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:21.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:21 smithi012 ceph-mon[31473]: pgmap v546: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 242 op/s 2024-05-27T07:38:21.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:21 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:38:21.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:21 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:38:21.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:21 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:38:22.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:22 smithi162 ceph-mon[35676]: pgmap v547: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 276 op/s 2024-05-27T07:38:22.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:22 smithi162 ceph-mon[35676]: pgmap v548: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:38:22.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:22 smithi062 ceph-mon[35304]: pgmap v547: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 276 op/s 2024-05-27T07:38:22.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:22 smithi062 ceph-mon[35304]: pgmap v548: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:38:22.782 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:22 smithi012 ceph-mon[31473]: pgmap v547: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 276 op/s 2024-05-27T07:38:22.782 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:22 smithi012 ceph-mon[31473]: pgmap v548: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:38:24.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:24 smithi062 ceph-mon[35304]: pgmap v549: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 888 KiB/s wr, 144 op/s 2024-05-27T07:38:24.748 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:24 smithi012 ceph-mon[31473]: pgmap v549: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 888 KiB/s wr, 144 op/s 2024-05-27T07:38:24.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:24 smithi162 ceph-mon[35676]: pgmap v549: 129 pgs: 129 active+clean; 322 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 888 KiB/s wr, 144 op/s 2024-05-27T07:38:26.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:26 smithi012 ceph-mon[31473]: pgmap v550: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:38:26.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:38:26.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:38:26.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:26 smithi162 ceph-mon[35676]: pgmap v550: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:38:26.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:38:26.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:38:26.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:26 smithi062 ceph-mon[35304]: pgmap v550: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:38:26.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:38:26.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:38:27.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:27 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:38:27.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:27 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:38:27.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:27 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:38:28.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:28 smithi012 ceph-mon[31473]: pgmap v551: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:38:28.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:28 smithi162 ceph-mon[35676]: pgmap v551: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:38:28.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:28 smithi062 ceph-mon[35304]: pgmap v551: 129 pgs: 129 active+clean; 369 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:38:30.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:30 smithi012 ceph-mon[31473]: pgmap v552: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 260 op/s 2024-05-27T07:38:30.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:30 smithi162 ceph-mon[35676]: pgmap v552: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 260 op/s 2024-05-27T07:38:30.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:30 smithi062 ceph-mon[35304]: pgmap v552: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 260 op/s 2024-05-27T07:38:32.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:32 smithi012 ceph-mon[31473]: pgmap v553: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:38:32.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:32 smithi162 ceph-mon[35676]: pgmap v553: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:38:32.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:32 smithi062 ceph-mon[35304]: pgmap v553: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:38:34.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:34 smithi012 ceph-mon[31473]: pgmap v554: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-05-27T07:38:34.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:34 smithi162 ceph-mon[35676]: pgmap v554: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-05-27T07:38:34.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:34 smithi062 ceph-mon[35304]: pgmap v554: 129 pgs: 129 active+clean; 335 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-05-27T07:38:36.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:36 smithi012 ceph-mon[31473]: pgmap v555: 129 pgs: 129 active+clean; 345 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 286 op/s 2024-05-27T07:38:36.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:36 smithi162 ceph-mon[35676]: pgmap v555: 129 pgs: 129 active+clean; 345 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 286 op/s 2024-05-27T07:38:36.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:36 smithi062 ceph-mon[35304]: pgmap v555: 129 pgs: 129 active+clean; 345 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 286 op/s 2024-05-27T07:38:37.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:37 smithi162 ceph-mon[35676]: pgmap v556: 129 pgs: 129 active+clean; 345 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2024-05-27T07:38:38.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:37 smithi012 ceph-mon[31473]: pgmap v556: 129 pgs: 129 active+clean; 345 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2024-05-27T07:38:38.199 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:37 smithi062 ceph-mon[35304]: pgmap v556: 129 pgs: 129 active+clean; 345 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2024-05-27T07:38:40.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:39 smithi162 ceph-mon[35676]: pgmap v557: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 266 op/s 2024-05-27T07:38:40.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:39 smithi062 ceph-mon[35304]: pgmap v557: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 266 op/s 2024-05-27T07:38:40.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:39 smithi012 ceph-mon[31473]: pgmap v557: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 266 op/s 2024-05-27T07:38:42.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:41 smithi162 ceph-mon[35676]: pgmap v558: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:38:42.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:41 smithi062 ceph-mon[35304]: pgmap v558: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:38:42.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:41 smithi012 ceph-mon[31473]: pgmap v558: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:38:44.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:43 smithi162 ceph-mon[35676]: pgmap v559: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:38:44.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:43 smithi062 ceph-mon[35304]: pgmap v559: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:38:44.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:43 smithi012 ceph-mon[31473]: pgmap v559: 129 pgs: 129 active+clean; 343 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-05-27T07:38:46.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:45 smithi162 ceph-mon[35676]: pgmap v560: 129 pgs: 129 active+clean; 384 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 261 op/s 2024-05-27T07:38:46.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:45 smithi062 ceph-mon[35304]: pgmap v560: 129 pgs: 129 active+clean; 384 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 261 op/s 2024-05-27T07:38:46.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:45 smithi012 ceph-mon[31473]: pgmap v560: 129 pgs: 129 active+clean; 384 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 261 op/s 2024-05-27T07:38:48.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:47 smithi162 ceph-mon[35676]: pgmap v561: 129 pgs: 129 active+clean; 384 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 165 op/s 2024-05-27T07:38:48.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:47 smithi062 ceph-mon[35304]: pgmap v561: 129 pgs: 129 active+clean; 384 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 165 op/s 2024-05-27T07:38:48.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:47 smithi012 ceph-mon[31473]: pgmap v561: 129 pgs: 129 active+clean; 384 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 165 op/s 2024-05-27T07:38:48.376 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:38:48.883 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:50.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:49 smithi162 ceph-mon[35676]: pgmap v562: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-05-27T07:38:50.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:49 smithi062 ceph-mon[35304]: pgmap v562: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-05-27T07:38:50.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:49 smithi012 ceph-mon[31473]: pgmap v562: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-05-27T07:38:51.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.735+0000 7f868f000700 1 -- 172.21.15.12:0/2280340305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8688108f10 msgr2=0x7f8688109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.735+0000 7f868f000700 1 --2- 172.21.15.12:0/2280340305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8688108f10 0x7f8688109320 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f86740099c0 tx=0x7f8674009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.735+0000 7f868f000700 1 -- 172.21.15.12:0/2280340305 shutdown_connections 2024-05-27T07:38:51.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.735+0000 7f868f000700 1 --2- 172.21.15.12:0/2280340305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86881045e0 0x7f86881049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.735+0000 7f868f000700 1 --2- 172.21.15.12:0/2280340305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8688109860 0x7f86881111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.735+0000 7f868f000700 1 --2- 172.21.15.12:0/2280340305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8688108f10 0x7f8688109320 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.735+0000 7f868f000700 1 -- 172.21.15.12:0/2280340305 >> 172.21.15.12:0/2280340305 conn(0x7f8688100030 msgr2=0x7f8688102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.736+0000 7f868f000700 1 -- 172.21.15.12:0/2280340305 shutdown_connections 2024-05-27T07:38:51.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.736+0000 7f868f000700 1 -- 172.21.15.12:0/2280340305 wait complete. 2024-05-27T07:38:51.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.737+0000 7f868f000700 1 Processor -- start 2024-05-27T07:38:51.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.737+0000 7f868f000700 1 -- start start 2024-05-27T07:38:51.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f868f000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86881045e0 0x7f8688117960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f868f000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8688108f10 0x7f8688113b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f868f000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8688109860 0x7f8688114230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f868f000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86880762e0 con 0x7f86881045e0 2024-05-27T07:38:51.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f868f000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8688076160 con 0x7f8688108f10 2024-05-27T07:38:51.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f868f000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8688076460 con 0x7f8688109860 2024-05-27T07:38:51.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f8687e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86881045e0 0x7f8688117960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f8687e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86881045e0 0x7f8688117960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38678/0 (socket says 172.21.15.12:38678) 2024-05-27T07:38:51.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.738+0000 7f8687e00700 1 -- 172.21.15.12:0/1423706102 learned_addr learned my addr 172.21.15.12:0/1423706102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:51.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f868d000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8688109860 0x7f8688114230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f8687e00700 1 -- 172.21.15.12:0/1423706102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8688109860 msgr2=0x7f8688114230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f8687e00700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8688109860 0x7f8688114230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f8687e00700 1 -- 172.21.15.12:0/1423706102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8688108f10 msgr2=0x7f8688113b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f8687e00700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8688108f10 0x7f8688113b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f8687e00700 1 -- 172.21.15.12:0/1423706102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86881ad630 con 0x7f86881045e0 2024-05-27T07:38:51.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f868d000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8688109860 0x7f8688114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:51.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.739+0000 7f8687e00700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86881045e0 0x7f8688117960 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f867800b530 tx=0x7f867800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.740+0000 7f8684c00700 1 -- 172.21.15.12:0/1423706102 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678015020 con 0x7f86881045e0 2024-05-27T07:38:51.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.740+0000 7f8684c00700 1 -- 172.21.15.12:0/1423706102 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8678011cc0 con 0x7f86881045e0 2024-05-27T07:38:51.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.740+0000 7f8684c00700 1 -- 172.21.15.12:0/1423706102 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678010d30 con 0x7f86881045e0 2024-05-27T07:38:51.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.741+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86881ad800 con 0x7f86881045e0 2024-05-27T07:38:51.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.741+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86881adcf0 con 0x7f86881045e0 2024-05-27T07:38:51.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.746+0000 7f8684c00700 1 -- 172.21.15.12:0/1423706102 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8678018020 con 0x7f86881045e0 2024-05-27T07:38:51.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.747+0000 7f869dbfab80 1 Processor -- start 2024-05-27T07:38:51.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.747+0000 7f869dbfab80 1 -- start start 2024-05-27T07:38:51.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.747+0000 7f8684c00700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f866c0b2a70 0x7f866c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f8684c00700 1 -- 172.21.15.12:0/1423706102 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f86780e2180 con 0x7f86881045e0 2024-05-27T07:38:51.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c689f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c689fd30 0x5593c6899870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c6899db0 0x5593c689a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f869dbfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5593c67c7e40 con 0x5593c671ae40 2024-05-27T07:38:51.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f869dbfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5593c67b79e0 con 0x5593c6899db0 2024-05-27T07:38:51.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f869dbfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593c67bba20 con 0x5593c689fd30 2024-05-27T07:38:51.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f868d000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f866c0b2a70 0x7f866c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f8687e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c689f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f8687e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c689f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38680/0 (socket says 172.21.15.12:38680) 2024-05-27T07:38:51.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.748+0000 7f8687e00700 1 -- 172.21.15.12:0/3357050426 learned_addr learned my addr 172.21.15.12:0/3357050426 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:51.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f868d000700 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c6899db0 0x5593c689a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f8687e00700 1 -- 172.21.15.12:0/3357050426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c689fd30 msgr2=0x5593c6899870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f8687e00700 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c689fd30 0x5593c6899870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f8687e00700 1 -- 172.21.15.12:0/3357050426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c6899db0 msgr2=0x5593c689a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f8687e00700 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c6899db0 0x5593c689a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f8687e00700 1 -- 172.21.15.12:0/3357050426 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593c6780420 con 0x5593c671ae40 2024-05-27T07:38:51.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f868d000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f866c0b2a70 0x7f866c0b4f80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8680009690 tx=0x7f8680007a80 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.749+0000 7f8687e00700 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c689f7f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f867800b500 tx=0x7f8678010ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ab313a5db09612 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.750+0000 7f867d600700 1 -- 172.21.15.12:0/3357050426 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86780c3070 con 0x5593c671ae40 2024-05-27T07:38:51.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.750+0000 7f867d600700 1 -- 172.21.15.12:0/3357050426 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86780a9100 con 0x5593c671ae40 2024-05-27T07:38:51.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.750+0000 7f867d600700 1 -- 172.21.15.12:0/3357050426 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86780a9360 con 0x5593c671ae40 2024-05-27T07:38:51.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.751+0000 7f869dbfab80 1 -- 172.21.15.12:0/3357050426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 msgr2=0x5593c689f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.751+0000 7f869dbfab80 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c689f7f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f867800b500 tx=0x7f8678010ed0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.751+0000 7f869dbfab80 1 -- 172.21.15.12:0/3357050426 shutdown_connections 2024-05-27T07:38:51.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.751+0000 7f869dbfab80 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c6899db0 0x5593c689a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.751+0000 7f869dbfab80 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c689f7f0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.751+0000 7f869dbfab80 1 --2- 172.21.15.12:0/3357050426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c689fd30 0x5593c6899870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.751+0000 7f869dbfab80 1 -- 172.21.15.12:0/3357050426 >> 172.21.15.12:0/3357050426 conn(0x5593c677afc0 msgr2=0x5593c68948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.752+0000 7f869dbfab80 1 -- 172.21.15.12:0/3357050426 shutdown_connections 2024-05-27T07:38:51.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.752+0000 7f869dbfab80 1 -- 172.21.15.12:0/3357050426 wait complete. 2024-05-27T07:38:51.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.752+0000 7f869dbfab80 1 -- start start 2024-05-27T07:38:51.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.753+0000 7f869dbfab80 1 Processor -- start 2024-05-27T07:38:51.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c68a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f869dbfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5593c66bdfa0 con 0x5593c6899db0 2024-05-27T07:38:51.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f869dbfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5593c6758760 con 0x5593c689fd30 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f869dbfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593c68a6800 con 0x5593c671ae40 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f869dbfab80 10 client.0 _ll_register_callbacks cb 0x5593c6761c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.754+0000 7f8687400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c68a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38696/0 (socket says 172.21.15.12:38696) 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687400700 1 -- 172.21.15.12:0/1016761139 learned_addr learned my addr 172.21.15.12:0/1016761139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f868d000700 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687400700 1 -- 172.21.15.12:0/1016761139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 msgr2=0x5593c68a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687400700 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c68a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687400700 1 -- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 msgr2=0x5593c68a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687400700 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.755+0000 7f8687400700 1 -- 172.21.15.12:0/1016761139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593c67078d0 con 0x5593c6899db0 2024-05-27T07:38:51.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.756+0000 7f8687400700 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a3b00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8674015000 tx=0x7f867400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6106150028bdfa9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.756+0000 7f8672000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.756+0000 7f8672000700 1 -- 172.21.15.12:0/1016761139 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674025060 con 0x5593c6899db0 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.757+0000 7f8672000700 1 -- 172.21.15.12:0/1016761139 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f8674004050 con 0x5593c6899db0 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.757+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86780a9100 con 0x5593c6899db0 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.757+0000 7f8672000700 1 -- 172.21.15.12:0/1016761139 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f867402c050 con 0x5593c6899db0 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.757+0000 7f869dbfab80 10 client.17355 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.757+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5593c6788060 con 0x5593c6899db0 2024-05-27T07:38:51.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f8672000700 1 -- 172.21.15.12:0/1016761139 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86780a9100 con 0x5593c6899db0 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f8672000700 1 -- 172.21.15.12:0/1016761139 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f867400f460 con 0x5593c6899db0 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f869dbfab80 10 client.17355 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f869dbfab80 10 client.17355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f869dbfab80 20 client.17355 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c68b3b90 0x5593c68b5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f869dbfab80 4 client.17355 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.758+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593c67c9d90 con 0x5593c68b3b90 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.759+0000 7f8687e00700 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c68b3b90 0x5593c68b5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.760+0000 7f8687e00700 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c68b3b90 0x5593c68b5fe0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf470987e49659e server_cookie=85463502736dee79 in_seq=0 out_seq=0 2024-05-27T07:38:51.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.760+0000 7f8672000700 10 client.17355 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f8672000700 1 -- 172.21.15.12:0/1016761139 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5593c67c9d90 con 0x5593c68b3b90 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f8672000700 10 client.17355 handle_command_reply: tid=0 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 client.17355 shutdown 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 msgr2=0x5593c68a3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a3b00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8674015000 tx=0x7f867400bea0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 shutdown_connections 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c68b3b90 0x5593c68b5fe0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c68a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.762+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1016761139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a3b00 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.763+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 >> 172.21.15.12:0/1016761139 conn(0x5593c677afc0 msgr2=0x5593c6894580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.763+0000 7f8672000700 0 client.17355 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:51.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.763+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 shutdown_connections 2024-05-27T07:38:51.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.763+0000 7f869dbfab80 1 -- 172.21.15.12:0/1016761139 wait complete. 2024-05-27T07:38:51.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.763+0000 7f869dbfab80 20 client.17355 trim_cache size 0 max 16384 2024-05-27T07:38:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.787+0000 7f869dbfab80 1 Processor -- start 2024-05-27T07:38:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.787+0000 7f869dbfab80 1 -- start start 2024-05-27T07:38:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c6899870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68ae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f869dbfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86881b6280 con 0x5593c6899db0 2024-05-27T07:38:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f869dbfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f86881b54e0 con 0x5593c689fd30 2024-05-27T07:38:51.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f869dbfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86881b5bb0 con 0x5593c671ae40 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f8687400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f8687400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38702/0 (socket says 172.21.15.12:38702) 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.788+0000 7f8687400700 1 -- 172.21.15.12:0/443701865 learned_addr learned my addr 172.21.15.12:0/443701865 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f8687e00700 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c6899870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f868d000700 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f8687400700 1 -- 172.21.15.12:0/443701865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 msgr2=0x5593c6899870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f8687400700 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c6899870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f8687400700 1 -- 172.21.15.12:0/443701865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 msgr2=0x5593c68ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f8687400700 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68ae850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f8687400700 1 -- 172.21.15.12:0/443701865 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593c68a43b0 con 0x5593c6899db0 2024-05-27T07:38:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f868d000700 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.789+0000 7f8687400700 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a5a60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f86740058e0 tx=0x7f867400faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7909c6fafda9024 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.790+0000 7f867cc00700 1 -- 172.21.15.12:0/443701865 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674025040 con 0x5593c6899db0 2024-05-27T07:38:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.790+0000 7f867cc00700 1 -- 172.21.15.12:0/443701865 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f867400b540 con 0x5593c6899db0 2024-05-27T07:38:51.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.790+0000 7f867cc00700 1 -- 172.21.15.12:0/443701865 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867400f460 con 0x5593c6899db0 2024-05-27T07:38:51.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.791+0000 7f869dbfab80 1 -- 172.21.15.12:0/443701865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 msgr2=0x5593c68a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.791+0000 7f869dbfab80 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a5a60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f86740058e0 tx=0x7f867400faa0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.791+0000 7f869dbfab80 1 -- 172.21.15.12:0/443701865 shutdown_connections 2024-05-27T07:38:51.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.791+0000 7f869dbfab80 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.791+0000 7f869dbfab80 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c671ae40 0x5593c6899870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.791+0000 7f869dbfab80 1 --2- 172.21.15.12:0/443701865 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c6899db0 0x5593c68a5a60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.791+0000 7f869dbfab80 1 -- 172.21.15.12:0/443701865 >> 172.21.15.12:0/443701865 conn(0x5593c677afc0 msgr2=0x5593c689db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.792+0000 7f869dbfab80 1 -- 172.21.15.12:0/443701865 shutdown_connections 2024-05-27T07:38:51.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.792+0000 7f869dbfab80 1 -- 172.21.15.12:0/443701865 wait complete. 2024-05-27T07:38:51.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 -- start start 2024-05-27T07:38:51.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 Processor -- start 2024-05-27T07:38:51.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c68ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c68a51f0 0x5593c68a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5593c68b65a0 con 0x5593c671ae40 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5593c6895e00 con 0x5593c689fd30 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.793+0000 7f869dbfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593c67c9d90 con 0x5593c68a51f0 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.794+0000 7f869dbfab80 10 client.0 _ll_register_callbacks cb 0x5593c6761c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.794+0000 7f8687e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c68ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.794+0000 7f8687e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c68ac2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38716/0 (socket says 172.21.15.12:38716) 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.794+0000 7f8687e00700 1 -- 172.21.15.12:0/1801723651 learned_addr learned my addr 172.21.15.12:0/1801723651 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.794+0000 7f8687400700 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.794+0000 7f868d000700 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c68a51f0 0x5593c68a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.795+0000 7f8687e00700 1 -- 172.21.15.12:0/1801723651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c68a51f0 msgr2=0x5593c68a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.795+0000 7f8687e00700 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c68a51f0 0x5593c68a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.795+0000 7f8687e00700 1 -- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 msgr2=0x5593c68a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:51.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.795+0000 7f8687e00700 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.795+0000 7f8687e00700 1 -- 172.21.15.12:0/1801723651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593c691d880 con 0x5593c671ae40 2024-05-27T07:38:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.795+0000 7f8687e00700 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c68ac2e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f86780a6370 tx=0x7f86780a8c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206fdd959eba3249 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.795+0000 7f8672000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:38:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.796+0000 7f8672000700 1 -- 172.21.15.12:0/1801723651 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86780c3040 con 0x5593c671ae40 2024-05-27T07:38:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.796+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f867400b540 con 0x5593c671ae40 2024-05-27T07:38:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.796+0000 7f8672000700 1 -- 172.21.15.12:0/1801723651 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f8678011830 con 0x5593c671ae40 2024-05-27T07:38:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.796+0000 7f8672000700 1 -- 172.21.15.12:0/1801723651 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f867808c050 con 0x5593c671ae40 2024-05-27T07:38:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.796+0000 7f869dbfab80 10 client.17361 fetch_fsmap learned FSMap version 119 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.796+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5593c691ea80 con 0x5593c671ae40 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.796+0000 7f8672000700 1 -- 172.21.15.12:0/1801723651 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f867400b540 con 0x5593c671ae40 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f8672000700 1 -- 172.21.15.12:0/1801723651 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f86780b1ad0 con 0x5593c671ae40 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f869dbfab80 10 client.17361 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f869dbfab80 10 client.17361 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f869dbfab80 20 client.17361 populate_metadata read hostname 'smithi012' 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c691ed40 0x5593c6921160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:51.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f869dbfab80 4 client.17361 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:38:51.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5593c67c3570 con 0x5593c691ed40 2024-05-27T07:38:51.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.797+0000 7f8687400700 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c691ed40 0x5593c6921160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:51.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.798+0000 7f8687400700 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c691ed40 0x5593c6921160 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa79a16624c3ee31 server_cookie=b020e02e3c3b1926 in_seq=0 out_seq=0 2024-05-27T07:38:51.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.798+0000 7f8672000700 10 client.17361 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:51.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f8672000700 1 -- 172.21.15.12:0/1801723651 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5593c67c3570 con 0x5593c691ed40 2024-05-27T07:38:51.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f8672000700 10 client.17361 handle_command_reply: tid=0 2024-05-27T07:38:51.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 client.17361 shutdown 2024-05-27T07:38:51.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 msgr2=0x5593c68ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c68ac2e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f86780a6370 tx=0x7f86780a8c10 comp rx=0 tx=0).stop 2024-05-27T07:38:51.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 shutdown_connections 2024-05-27T07:38:51.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593c68a51f0 0x5593c68a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5593c671ae40 0x5593c68ac2e0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5593c691ed40 0x5593c6921160 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 --2- 172.21.15.12:0/1801723651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5593c689fd30 0x5593c68a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 >> 172.21.15.12:0/1801723651 conn(0x5593c677afc0 msgr2=0x5593c689db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.801+0000 7f8672000700 0 client.17361 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.802+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 shutdown_connections 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.802+0000 7f869dbfab80 1 -- 172.21.15.12:0/1801723651 wait complete. 2024-05-27T07:38:51.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.802+0000 7f869dbfab80 20 client.17361 trim_cache size 0 max 16384 2024-05-27T07:38:51.802 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:38:51.802 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:38:51.802 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:38:51.802 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:38:51.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f866c0b2a70 msgr2=0x7f866c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f866c0b2a70 0x7f866c0b4f80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8680009690 tx=0x7f8680007a80 comp rx=0 tx=0).stop 2024-05-27T07:38:51.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86881045e0 msgr2=0x7f8688117960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:51.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86881045e0 0x7f8688117960 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f867800b530 tx=0x7f867800b8f0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 shutdown_connections 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f866c0b2a70 0x7f866c0b4f80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8688109860 0x7f8688114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86881045e0 0x7f8688117960 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 --2- 172.21.15.12:0/1423706102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8688108f10 0x7f8688113b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 >> 172.21.15.12:0/1423706102 conn(0x7f8688100030 msgr2=0x7f8688102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 shutdown_connections 2024-05-27T07:38:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:51.803+0000 7f868f000700 1 -- 172.21.15.12:0/1423706102 wait complete. 2024-05-27T07:38:52.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:51 smithi012 ceph-mon[31473]: pgmap v563: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-05-27T07:38:52.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:51 smithi162 ceph-mon[35676]: pgmap v563: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-05-27T07:38:52.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:51 smithi062 ceph-mon[35304]: pgmap v563: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-05-27T07:38:52.937 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:38:52.937 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:38:53.430 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:54.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:53 smithi162 ceph-mon[35676]: pgmap v564: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 162 op/s 2024-05-27T07:38:54.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:53 smithi062 ceph-mon[35304]: pgmap v564: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 162 op/s 2024-05-27T07:38:54.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:53 smithi012 ceph-mon[31473]: pgmap v564: 129 pgs: 129 active+clean; 349 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 162 op/s 2024-05-27T07:38:56.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:55 smithi162 ceph-mon[35676]: pgmap v565: 129 pgs: 129 active+clean; 359 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 262 op/s 2024-05-27T07:38:56.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:55 smithi062 ceph-mon[35304]: pgmap v565: 129 pgs: 129 active+clean; 359 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 262 op/s 2024-05-27T07:38:56.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.258+0000 7f8b88a00700 1 -- 172.21.15.12:0/1421205699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 msgr2=0x7f8b841198a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:56.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.258+0000 7f8b88a00700 1 --2- 172.21.15.12:0/1421205699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 0x7f8b841198a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009f90 tx=0x7f8b7c00b330 comp rx=0 tx=0).stop 2024-05-27T07:38:56.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.258+0000 7f8b88a00700 1 -- 172.21.15.12:0/1421205699 shutdown_connections 2024-05-27T07:38:56.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.258+0000 7f8b88a00700 1 --2- 172.21.15.12:0/1421205699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b84111bc0 0x7f8b84111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.258+0000 7f8b88a00700 1 --2- 172.21.15.12:0/1421205699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 0x7f8b841198a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.258+0000 7f8b88a00700 1 --2- 172.21.15.12:0/1421205699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b8410a920 0x7f8b8410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.258+0000 7f8b88a00700 1 -- 172.21.15.12:0/1421205699 >> 172.21.15.12:0/1421205699 conn(0x7f8b8406f4c0 msgr2=0x7f8b8406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:56.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.259+0000 7f8b88a00700 1 -- 172.21.15.12:0/1421205699 shutdown_connections 2024-05-27T07:38:56.257 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:55 smithi012 ceph-mon[31473]: pgmap v565: 129 pgs: 129 active+clean; 359 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 262 op/s 2024-05-27T07:38:56.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.259+0000 7f8b88a00700 1 -- 172.21.15.12:0/1421205699 wait complete. 2024-05-27T07:38:56.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.261+0000 7f8b88a00700 1 Processor -- start 2024-05-27T07:38:56.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.261+0000 7f8b88a00700 1 -- start start 2024-05-27T07:38:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.262+0000 7f8b88a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b8410a920 0x7f8b841aa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.262+0000 7f8b88a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 0x7f8b841aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.262+0000 7f8b88a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b84111bc0 0x7f8b841b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.262+0000 7f8b88a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8411caf0 con 0x7f8b8410b270 2024-05-27T07:38:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.262+0000 7f8b88a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8411c970 con 0x7f8b8410a920 2024-05-27T07:38:56.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.262+0000 7f8b88a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8411cc70 con 0x7f8b84111bc0 2024-05-27T07:38:56.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.263+0000 7f8b81400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 0x7f8b841aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:56.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.263+0000 7f8b81e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b8410a920 0x7f8b841aa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:56.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.263+0000 7f8b81400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 0x7f8b841aad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39048/0 (socket says 172.21.15.12:39048) 2024-05-27T07:38:56.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.263+0000 7f8b81400700 1 -- 172.21.15.12:0/27210474 learned_addr learned my addr 172.21.15.12:0/27210474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:38:56.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.263+0000 7f8b82800700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b84111bc0 0x7f8b841b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:56.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.264+0000 7f8b81e00700 1 -- 172.21.15.12:0/27210474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b84111bc0 msgr2=0x7f8b841b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:56.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.264+0000 7f8b81e00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b84111bc0 0x7f8b841b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.264+0000 7f8b81e00700 1 -- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 msgr2=0x7f8b841aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:56.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.264+0000 7f8b81e00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 0x7f8b841aad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.264+0000 7f8b81e00700 1 -- 172.21.15.12:0/27210474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b841ab600 con 0x7f8b8410a920 2024-05-27T07:38:56.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.265+0000 7f8b81e00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b8410a920 0x7f8b841aa7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400d740 tx=0x7f8b7400da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:56.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.267+0000 7f8b7aa00700 1 -- 172.21.15.12:0/27210474 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74017070 con 0x7f8b8410a920 2024-05-27T07:38:56.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.267+0000 7f8b7aa00700 1 -- 172.21.15.12:0/27210474 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b7400f690 con 0x7f8b8410a920 2024-05-27T07:38:56.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.267+0000 7f8b7aa00700 1 -- 172.21.15.12:0/27210474 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74003ae0 con 0x7f8b8410a920 2024-05-27T07:38:56.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.267+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b841b5eb0 con 0x7f8b8410a920 2024-05-27T07:38:56.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.267+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b841b62b0 con 0x7f8b8410a920 2024-05-27T07:38:56.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.271+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b840512b0 con 0x7f8b8410a920 2024-05-27T07:38:56.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.274+0000 7f8b7aa00700 1 -- 172.21.15.12:0/27210474 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8b74003c80 con 0x7f8b8410a920 2024-05-27T07:38:56.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.276+0000 7f8b7aa00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b680b2b40 0x7f8b680b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:56.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.276+0000 7f8b7aa00700 1 -- 172.21.15.12:0/27210474 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9554+0+0 (secure 0 0 0) 0x7f8b740e23b0 con 0x7f8b8410a920 2024-05-27T07:38:56.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.276+0000 7f8b81400700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b680b2b40 0x7f8b680b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:56.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.277+0000 7f8b81400700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b680b2b40 0x7f8b680b5050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8b8410e600 tx=0x7f8b7000b2c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:56.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.283+0000 7f8b7aa00700 1 -- 172.21.15.12:0/27210474 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8b841ab600 con 0x7f8b8410a920 2024-05-27T07:38:56.660 INFO:tasks.workunit.client.0.smithi012.stdout:20000 trunc from 0x62a09d3 to 0x111d204 2024-05-27T07:38:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.674+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b84119370 con 0x7f8b8410a920 2024-05-27T07:38:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.676+0000 7f8b7aa00700 1 -- 172.21.15.12:0/27210474 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f8b740a9980 con 0x7f8b8410a920 2024-05-27T07:38:56.673 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:38:56.673 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:38:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b680b2b40 msgr2=0x7f8b680b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b680b2b40 0x7f8b680b5050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8b8410e600 tx=0x7f8b7000b2c0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b8410a920 msgr2=0x7f8b841aa7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b8410a920 0x7f8b841aa7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400d740 tx=0x7f8b7400da50 comp rx=0 tx=0).stop 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 shutdown_connections 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f8b680b2b40 0x7f8b680b5050 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b8410a920 0x7f8b841aa7d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b84111bc0 0x7f8b841b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 --2- 172.21.15.12:0/27210474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b8410b270 0x7f8b841aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.679+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 >> 172.21.15.12:0/27210474 conn(0x7f8b8406f4c0 msgr2=0x7f8b84110280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.680+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 shutdown_connections 2024-05-27T07:38:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:38:56.681+0000 7f8b88a00700 1 -- 172.21.15.12:0/27210474 wait complete. 2024-05-27T07:38:56.683 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:38:57.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:56 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/27210474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:57.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:56 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/27210474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:57.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:56 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/27210474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:38:57.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:38:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:58.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:57 smithi162 ceph-mon[35676]: pgmap v566: 129 pgs: 129 active+clean; 359 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-27T07:38:58.245 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:57 smithi062 ceph-mon[35304]: pgmap v566: 129 pgs: 129 active+clean; 359 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-27T07:38:58.317 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:57 smithi012 ceph-mon[31473]: pgmap v566: 129 pgs: 129 active+clean; 359 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-27T07:39:00.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:38:59 smithi162 ceph-mon[35676]: pgmap v567: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 293 op/s 2024-05-27T07:39:00.243 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:38:59 smithi062 ceph-mon[35304]: pgmap v567: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 293 op/s 2024-05-27T07:39:00.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:38:59 smithi012 ceph-mon[31473]: pgmap v567: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 293 op/s 2024-05-27T07:39:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 -- 172.21.15.12:0/197741628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab2810e8e0 msgr2=0x7fab28110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 --2- 172.21.15.12:0/197741628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab2810e8e0 0x7fab28110dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab2000b0a0 tx=0x7fab2000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 -- 172.21.15.12:0/197741628 shutdown_connections 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 --2- 172.21.15.12:0/197741628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab2810e8e0 0x7fab28110dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 --2- 172.21.15.12:0/197741628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab281054b0 0x7fab281058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 --2- 172.21.15.12:0/197741628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab28105e00 0x7fab2810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 -- 172.21.15.12:0/197741628 >> 172.21.15.12:0/197741628 conn(0x7fab280fedb0 msgr2=0x7fab281011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 -- 172.21.15.12:0/197741628 shutdown_connections 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.312+0000 7fab2f800700 1 -- 172.21.15.12:0/197741628 wait complete. 2024-05-27T07:39:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.314+0000 7fab2f800700 1 Processor -- start 2024-05-27T07:39:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.314+0000 7fab2f800700 1 -- start start 2024-05-27T07:39:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2f800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab281054b0 0x7fab281031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2f800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab28105e00 0x7fab281036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2f800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab2810e8e0 0x7fab28103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2f800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fab28114340 con 0x7fab28105e00 2024-05-27T07:39:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2f800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fab281141c0 con 0x7fab2810e8e0 2024-05-27T07:39:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2f800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fab281144c0 con 0x7fab281054b0 2024-05-27T07:39:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab27e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab28105e00 0x7fab281036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2ce00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab281054b0 0x7fab281031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab27e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab28105e00 0x7fab281036e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39078/0 (socket says 172.21.15.12:39078) 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab27e00700 1 -- 172.21.15.12:0/885200339 learned_addr learned my addr 172.21.15.12:0/885200339 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.315+0000 7fab2d800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab2810e8e0 0x7fab28103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab2ce00700 1 -- 172.21.15.12:0/885200339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab2810e8e0 msgr2=0x7fab28103c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab2ce00700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab2810e8e0 0x7fab28103c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab2ce00700 1 -- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab28105e00 msgr2=0x7fab281036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab2ce00700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab28105e00 0x7fab281036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab2ce00700 1 -- 172.21.15.12:0/885200339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab281a9050 con 0x7fab281054b0 2024-05-27T07:39:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab2ce00700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab281054b0 0x7fab281031a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab1c00e9f0 tx=0x7fab1c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab25600700 1 -- 172.21.15.12:0/885200339 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c00cb70 con 0x7fab281054b0 2024-05-27T07:39:01.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab25600700 1 -- 172.21.15.12:0/885200339 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab1c00ee90 con 0x7fab281054b0 2024-05-27T07:39:01.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab281a92e0 con 0x7fab281054b0 2024-05-27T07:39:01.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.316+0000 7fab25600700 1 -- 172.21.15.12:0/885200339 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c003c30 con 0x7fab281054b0 2024-05-27T07:39:01.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.317+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab281a9870 con 0x7fab281054b0 2024-05-27T07:39:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.321+0000 7fab25600700 1 -- 172.21.15.12:0/885200339 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fab1c018430 con 0x7fab281054b0 2024-05-27T07:39:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.323+0000 7fab3e3d8b80 1 Processor -- start 2024-05-27T07:39:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.323+0000 7fab3e3d8b80 1 -- start start 2024-05-27T07:39:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.323+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf570de40 0x558cf58927f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf588c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf588d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558cf57bae40 con 0x558cf588cdb0 2024-05-27T07:39:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558cf57aa9e0 con 0x558cf570de40 2024-05-27T07:39:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558cf57aea20 con 0x558cf5892d30 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab25600700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fab100b2a70 0x7fab100b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab2d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf570de40 0x558cf58927f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab2d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf570de40 0x558cf58927f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:56964/0 (socket says 172.21.15.12:56964) 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab25600700 1 -- 172.21.15.12:0/885200339 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 9642+0+0 (secure 0 0 0) 0x7fab1c0e2b30 con 0x7fab281054b0 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab2d800700 1 -- 172.21.15.12:0/257746501 learned_addr learned my addr 172.21.15.12:0/257746501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab27e00700 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf588d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab2ce00700 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf588c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.324+0000 7fab2d800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fab100b2a70 0x7fab100b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.325+0000 7fab27e00700 1 -- 172.21.15.12:0/257746501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 msgr2=0x558cf588c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.325+0000 7fab27e00700 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf588c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.325+0000 7fab27e00700 1 -- 172.21.15.12:0/257746501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf570de40 msgr2=0x558cf58927f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.325+0000 7fab27e00700 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf570de40 0x558cf58927f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.325+0000 7fab27e00700 1 -- 172.21.15.12:0/257746501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cf5773420 con 0x558cf588cdb0 2024-05-27T07:39:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.325+0000 7fab2d800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fab100b2a70 0x7fab100b4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab2000b0a0 tx=0x7fab20009c20 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.326+0000 7fab27e00700 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf588d1c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fab14009c90 tx=0x7fab14011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815e1f9a8f638267 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.326+0000 7fab1a000700 1 -- 172.21.15.12:0/257746501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14011d20 con 0x558cf588cdb0 2024-05-27T07:39:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.326+0000 7fab1a000700 1 -- 172.21.15.12:0/257746501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab140044e0 con 0x558cf588cdb0 2024-05-27T07:39:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.326+0000 7fab1a000700 1 -- 172.21.15.12:0/257746501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14005220 con 0x558cf588cdb0 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/257746501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 msgr2=0x558cf588d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf588d1c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fab14009c90 tx=0x7fab14011440 comp rx=0 tx=0).stop 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/257746501 shutdown_connections 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf570de40 0x558cf58927f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf588d1c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/257746501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf588c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/257746501 >> 172.21.15.12:0/257746501 conn(0x558cf576dfc0 msgr2=0x558cf58878d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/257746501 shutdown_connections 2024-05-27T07:39:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.327+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/257746501 wait complete. 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.328+0000 7fab3e3d8b80 1 -- start start 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.328+0000 7fab3e3d8b80 1 Processor -- start 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf5896530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5896a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5896fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558cf56b0fa0 con 0x558cf588cdb0 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558cf574b760 con 0x558cf5892d30 2024-05-27T07:39:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558cf5899800 con 0x558cf570de40 2024-05-27T07:39:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab3e3d8b80 10 client.0 _ll_register_callbacks cb 0x558cf5754c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab27e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5896a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab27e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5896a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39100/0 (socket says 172.21.15.12:39100) 2024-05-27T07:39:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab27e00700 1 -- 172.21.15.12:0/857094065 learned_addr learned my addr 172.21.15.12:0/857094065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab2d800700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5896fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.329+0000 7fab2ce00700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf5896530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab27e00700 1 -- 172.21.15.12:0/857094065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 msgr2=0x558cf5896530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab27e00700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf5896530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab27e00700 1 -- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 msgr2=0x558cf5896fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab27e00700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5896fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab27e00700 1 -- 172.21.15.12:0/857094065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cf56fa8d0 con 0x558cf588cdb0 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab2d800700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5896fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab27e00700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5896a70 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fab14005e90 tx=0x7fab140041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df140da7bede535 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.330+0000 7fab0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:01.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.331+0000 7fab0ea00700 1 -- 172.21.15.12:0/857094065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14011d30 con 0x558cf588cdb0 2024-05-27T07:39:01.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.331+0000 7fab0ea00700 1 -- 172.21.15.12:0/857094065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9642+0+0 (secure 0 0 0) 0x7fab14003680 con 0x558cf588cdb0 2024-05-27T07:39:01.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.331+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab140044e0 con 0x558cf588cdb0 2024-05-27T07:39:01.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.331+0000 7fab0ea00700 1 -- 172.21.15.12:0/857094065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fab14003880 con 0x558cf588cdb0 2024-05-27T07:39:01.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.331+0000 7fab3e3d8b80 10 client.17379 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:01.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.331+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558cf577b060 con 0x558cf588cdb0 2024-05-27T07:39:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab0ea00700 1 -- 172.21.15.12:0/857094065 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab140044e0 con 0x558cf588cdb0 2024-05-27T07:39:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab0ea00700 1 -- 172.21.15.12:0/857094065 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fab14018070 con 0x558cf588cdb0 2024-05-27T07:39:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab3e3d8b80 10 client.17379 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab3e3d8b80 10 client.17379 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:39:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab3e3d8b80 20 client.17379 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf58a6950 0x558cf58a8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab3e3d8b80 4 client.17379 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:01.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.332+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cf57bcd90 con 0x558cf58a6950 2024-05-27T07:39:01.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.333+0000 7fab2ce00700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf58a6950 0x558cf58a8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.333+0000 7fab2ce00700 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf58a6950 0x558cf58a8da0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62e9772f8f7b6fef server_cookie=cc01feac7de9451d in_seq=0 out_seq=0 2024-05-27T07:39:01.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.333+0000 7fab0ea00700 10 client.17379 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:39:01.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.334+0000 7fab0ea00700 1 -- 172.21.15.12:0/857094065 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558cf57bcd90 con 0x558cf58a6950 2024-05-27T07:39:01.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.334+0000 7fab0ea00700 10 client.17379 handle_command_reply: tid=0 2024-05-27T07:39:01.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 client.17379 shutdown 2024-05-27T07:39:01.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 msgr2=0x558cf5896a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5896a70 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fab14005e90 tx=0x7fab140041e0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 shutdown_connections 2024-05-27T07:39:01.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf58a6950 0x558cf58a8da0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5896fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf5896530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab0ea00700 0 client.17379 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:39:01.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/857094065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5896a70 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 >> 172.21.15.12:0/857094065 conn(0x558cf576dfc0 msgr2=0x558cf5887580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.335+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 shutdown_connections 2024-05-27T07:39:01.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.336+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/857094065 wait complete. 2024-05-27T07:39:01.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.336+0000 7fab3e3d8b80 20 client.17379 trim_cache size 0 max 16384 2024-05-27T07:39:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 Processor -- start 2024-05-27T07:39:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 -- start start 2024-05-27T07:39:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf588c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5895850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5894150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fab281b1e90 con 0x558cf588cdb0 2024-05-27T07:39:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fab281b10f0 con 0x558cf5892d30 2024-05-27T07:39:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.354+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fab281b17c0 con 0x558cf570de40 2024-05-27T07:39:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5895850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5895850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39106/0 (socket says 172.21.15.12:39106) 2024-05-27T07:39:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 -- 172.21.15.12:0/3917261881 learned_addr learned my addr 172.21.15.12:0/3917261881 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab2d800700 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5894150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab2ce00700 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf588c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 -- 172.21.15.12:0/3917261881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 msgr2=0x558cf588c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf588c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 -- 172.21.15.12:0/3917261881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 msgr2=0x558cf5894150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5894150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.355+0000 7fab27e00700 1 -- 172.21.15.12:0/3917261881 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cf5897320 con 0x558cf588cdb0 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.356+0000 7fab27e00700 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5895850 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fab14009be0 tx=0x7fab14009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f263f5693ae51a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.356+0000 7fab19600700 1 -- 172.21.15.12:0/3917261881 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14004250 con 0x558cf588cdb0 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.356+0000 7fab19600700 1 -- 172.21.15.12:0/3917261881 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab14027070 con 0x558cf588cdb0 2024-05-27T07:39:01.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.356+0000 7fab19600700 1 -- 172.21.15.12:0/3917261881 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14018de0 con 0x558cf588cdb0 2024-05-27T07:39:01.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/3917261881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 msgr2=0x558cf5895850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5895850 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fab14009be0 tx=0x7fab14009640 comp rx=0 tx=0).stop 2024-05-27T07:39:01.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/3917261881 shutdown_connections 2024-05-27T07:39:01.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf5892d30 0x558cf5894150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf570de40 0x558cf588c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/3917261881 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf588cdb0 0x558cf5895850 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/3917261881 >> 172.21.15.12:0/3917261881 conn(0x558cf576dfc0 msgr2=0x558cf5890880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/3917261881 shutdown_connections 2024-05-27T07:39:01.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.357+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/3917261881 wait complete. 2024-05-27T07:39:01.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.358+0000 7fab3e3d8b80 1 -- start start 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.358+0000 7fab3e3d8b80 1 Processor -- start 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf570de40 0x558cf589ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf588cdb0 0x558cf589ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab3e3d8b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf58ad590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558cf58a9380 con 0x558cf570de40 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558cf588db60 con 0x558cf588cdb0 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab3e3d8b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558cf5888e00 con 0x558cf5892d30 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab3e3d8b80 10 client.0 _ll_register_callbacks cb 0x558cf5754c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab2ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf570de40 0x558cf589ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab2ce00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf570de40 0x558cf589ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39112/0 (socket says 172.21.15.12:39112) 2024-05-27T07:39:01.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab2ce00700 1 -- 172.21.15.12:0/1439854143 learned_addr learned my addr 172.21.15.12:0/1439854143 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:01.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab27e00700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf588cdb0 0x558cf589ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.359+0000 7fab2d800700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf58ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab2ce00700 1 -- 172.21.15.12:0/1439854143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 msgr2=0x558cf58ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab2ce00700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf58ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab2ce00700 1 -- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf588cdb0 msgr2=0x558cf589ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab2ce00700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf588cdb0 0x558cf589ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab2ce00700 1 -- 172.21.15.12:0/1439854143 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cf589f670 con 0x558cf570de40 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab2d800700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf58ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab2ce00700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf570de40 0x558cf589ea50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0046a0 tx=0x7fab1c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeba111db88ed209 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab0ea00700 1 -- 172.21.15.12:0/1439854143 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c0c2070 con 0x558cf570de40 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.360+0000 7fab0ea00700 1 -- 172.21.15.12:0/1439854143 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9642+0+0 (secure 0 0 0) 0x7fab1c0cc0b0 con 0x558cf570de40 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.361+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab14027070 con 0x558cf570de40 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.361+0000 7fab0ea00700 1 -- 172.21.15.12:0/1439854143 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fab1c0cc2b0 con 0x558cf570de40 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.361+0000 7fab3e3d8b80 10 client.17385 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.361+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558cf59192a0 con 0x558cf570de40 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.361+0000 7fab0ea00700 1 -- 172.21.15.12:0/1439854143 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab14027070 con 0x558cf570de40 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.361+0000 7fab0ea00700 1 -- 172.21.15.12:0/1439854143 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fab1c0a9c80 con 0x558cf570de40 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.362+0000 7fab3e3d8b80 10 client.17385 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.362+0000 7fab3e3d8b80 10 client.17385 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.362+0000 7fab3e3d8b80 20 client.17385 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.362+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf5919900 0x558cf591bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.362+0000 7fab3e3d8b80 4 client.17385 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.362+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558cf57b6570 con 0x558cf5919900 2024-05-27T07:39:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.362+0000 7fab27e00700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf5919900 0x558cf591bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.364+0000 7fab27e00700 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf5919900 0x558cf591bd20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a4c218def2fb673 server_cookie=bfd8e3ccf146923c in_seq=0 out_seq=0 2024-05-27T07:39:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.364+0000 7fab0ea00700 10 client.17385 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:39:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.365+0000 7fab0ea00700 1 -- 172.21.15.12:0/1439854143 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558cf57b6570 con 0x558cf5919900 2024-05-27T07:39:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.365+0000 7fab0ea00700 10 client.17385 handle_command_reply: tid=0 2024-05-27T07:39:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.365+0000 7fab3e3d8b80 1 client.17385 shutdown 2024-05-27T07:39:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.365+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf570de40 msgr2=0x558cf589ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.365+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf570de40 0x558cf589ea50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0046a0 tx=0x7fab1c011000 comp rx=0 tx=0).stop 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 shutdown_connections 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558cf5892d30 0x558cf58ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558cf570de40 0x558cf589ea50 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x558cf5919900 0x558cf591bd20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 --2- 172.21.15.12:0/1439854143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cf588cdb0 0x558cf589ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab0ea00700 0 client.17385 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 >> 172.21.15.12:0/1439854143 conn(0x558cf576dfc0 msgr2=0x558cf5896310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 shutdown_connections 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 1 -- 172.21.15.12:0/1439854143 wait complete. 2024-05-27T07:39:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.366+0000 7fab3e3d8b80 20 client.17385 trim_cache size 0 max 16384 2024-05-27T07:39:01.365 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fab100b2a70 msgr2=0x7fab100b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fab100b2a70 0x7fab100b4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab2000b0a0 tx=0x7fab20009c20 comp rx=0 tx=0).stop 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab281054b0 msgr2=0x7fab281031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab281054b0 0x7fab281031a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab1c00e9f0 tx=0x7fab1c00ed00 comp rx=0 tx=0).stop 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 shutdown_connections 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fab100b2a70 0x7fab100b4f80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab2810e8e0 0x7fab28103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab281054b0 0x7fab281031a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 --2- 172.21.15.12:0/885200339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab28105e00 0x7fab281036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.367+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 >> 172.21.15.12:0/885200339 conn(0x7fab280fedb0 msgr2=0x7fab280ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.368+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 shutdown_connections 2024-05-27T07:39:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:01.368+0000 7fab2f800700 1 -- 172.21.15.12:0/885200339 wait complete. 2024-05-27T07:39:02.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:02 smithi012 ceph-mon[31473]: pgmap v568: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-05-27T07:39:02.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:02 smithi012 ceph-mon[31473]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:39:02.230 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:39:02.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:02 smithi062 ceph-mon[35304]: pgmap v568: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-05-27T07:39:02.243 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:02 smithi062 ceph-mon[35304]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:39:02.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:02 smithi162 ceph-mon[35676]: pgmap v568: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-05-27T07:39:02.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:02 smithi162 ceph-mon[35676]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:39:02.708 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:04.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:04 smithi012 ceph-mon[31473]: pgmap v570: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 253 op/s 2024-05-27T07:39:04.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:04 smithi162 ceph-mon[35676]: pgmap v570: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 253 op/s 2024-05-27T07:39:04.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:04 smithi062 ceph-mon[35304]: pgmap v570: 129 pgs: 129 active+clean; 347 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 253 op/s 2024-05-27T07:39:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 -- 172.21.15.12:0/2023216164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc107de0 msgr2=0x7fdbfc1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 --2- 172.21.15.12:0/2023216164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc107de0 0x7fdbfc1081f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80099c0 tx=0x7fdbe8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 -- 172.21.15.12:0/2023216164 shutdown_connections 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 --2- 172.21.15.12:0/2023216164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc108730 0x7fdbfc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 --2- 172.21.15.12:0/2023216164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc1034b0 0x7fdbfc1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 --2- 172.21.15.12:0/2023216164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc107de0 0x7fdbfc1081f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 -- 172.21.15.12:0/2023216164 >> 172.21.15.12:0/2023216164 conn(0x7fdbfc0fedb0 msgr2=0x7fdbfc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 -- 172.21.15.12:0/2023216164 shutdown_connections 2024-05-27T07:39:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.220+0000 7fdc03800700 1 -- 172.21.15.12:0/2023216164 wait complete. 2024-05-27T07:39:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.222+0000 7fdc03800700 1 Processor -- start 2024-05-27T07:39:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.222+0000 7fdc03800700 1 -- start start 2024-05-27T07:39:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc03800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc1034b0 0x7fdbfc19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc03800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc107de0 0x7fdbfc19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc03800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc108730 0x7fdbfc19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc03800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc113210 con 0x7fdbfc108730 2024-05-27T07:39:05.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc03800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc113090 con 0x7fdbfc107de0 2024-05-27T07:39:05.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc03800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc113390 con 0x7fdbfc1034b0 2024-05-27T07:39:05.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc108730 0x7fdbfc19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc108730 0x7fdbfc19f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45806/0 (socket says 172.21.15.12:45806) 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.223+0000 7fdc01800700 1 -- 172.21.15.12:0/4157071473 learned_addr learned my addr 172.21.15.12:0/4157071473 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.224+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc107de0 0x7fdbfc19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.224+0000 7fdc00e00700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc1034b0 0x7fdbfc19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.224+0000 7fdc01800700 1 -- 172.21.15.12:0/4157071473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc1034b0 msgr2=0x7fdbfc19ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.224+0000 7fdc01800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc1034b0 0x7fdbfc19ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.224+0000 7fdc01800700 1 -- 172.21.15.12:0/4157071473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc107de0 msgr2=0x7fdbfc19b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.224+0000 7fdc01800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc107de0 0x7fdbfc19b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.224+0000 7fdc01800700 1 -- 172.21.15.12:0/4157071473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc19fbe0 con 0x7fdbfc108730 2024-05-27T07:39:05.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.225+0000 7fdc01800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc108730 0x7fdbfc19f540 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400ef30 tx=0x7fdbf400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:05.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.225+0000 7fdbf9600700 1 -- 172.21.15.12:0/4157071473 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40092e0 con 0x7fdbfc108730 2024-05-27T07:39:05.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.225+0000 7fdbf9600700 1 -- 172.21.15.12:0/4157071473 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf4004bb0 con 0x7fdbfc108730 2024-05-27T07:39:05.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.225+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbfc19fe70 con 0x7fdbfc108730 2024-05-27T07:39:05.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.225+0000 7fdbf9600700 1 -- 172.21.15.12:0/4157071473 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4004020 con 0x7fdbfc108730 2024-05-27T07:39:05.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.225+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbfc1a0380 con 0x7fdbfc108730 2024-05-27T07:39:05.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.229+0000 7fdbf9600700 1 -- 172.21.15.12:0/4157071473 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdbf401b030 con 0x7fdbfc108730 2024-05-27T07:39:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.230+0000 7fdc122acb80 1 Processor -- start 2024-05-27T07:39:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc122acb80 1 -- start start 2024-05-27T07:39:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdbf9600700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdbe40b2a70 0x7fdbe40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b371a1e40 0x560b3732abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b3732b110 0x560b37324c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdbf9600700 1 -- 172.21.15.12:0/4157071473 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 9642+0+0 (secure 0 0 0) 0x7fdbf40e2540 con 0x7fdbfc108730 2024-05-27T07:39:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b373251c0 0x560b373255d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc122acb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560b3724ee40 con 0x560b371a1e40 2024-05-27T07:39:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc122acb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560b3723e9e0 con 0x560b373251c0 2024-05-27T07:39:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc122acb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560b37242a20 con 0x560b3732b110 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdbe40b2a70 0x7fdbe40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.231+0000 7fdc00e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b371a1e40 0x560b3732abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc00e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b371a1e40 0x560b3732abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45822/0 (socket says 172.21.15.12:45822) 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc00e00700 1 -- 172.21.15.12:0/1702841994 learned_addr learned my addr 172.21.15.12:0/1702841994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc01800700 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b373251c0 0x560b373255d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b3732b110 0x560b37324c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc00e00700 1 -- 172.21.15.12:0/1702841994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b3732b110 msgr2=0x560b37324c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc00e00700 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b3732b110 0x560b37324c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc00e00700 1 -- 172.21.15.12:0/1702841994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b373251c0 msgr2=0x560b373255d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc00e00700 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b373251c0 0x560b373255d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.232+0000 7fdc00e00700 1 -- 172.21.15.12:0/1702841994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b37207420 con 0x560b371a1e40 2024-05-27T07:39:05.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.233+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdbe40b2a70 0x7fdbe40b4f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8000c00 tx=0x7fdbe80051e0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:05.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.233+0000 7fdc00e00700 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b371a1e40 0x560b3732abd0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000b780 tx=0x7fdbf000ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b9503d39a374f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.233+0000 7fdbee000700 1 -- 172.21.15.12:0/1702841994 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0004a30 con 0x560b371a1e40 2024-05-27T07:39:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.233+0000 7fdbee000700 1 -- 172.21.15.12:0/1702841994 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf0015070 con 0x560b371a1e40 2024-05-27T07:39:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.233+0000 7fdbee000700 1 -- 172.21.15.12:0/1702841994 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0003a90 con 0x560b371a1e40 2024-05-27T07:39:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.234+0000 7fdc122acb80 1 -- 172.21.15.12:0/1702841994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b371a1e40 msgr2=0x560b3732abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.234+0000 7fdc122acb80 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b371a1e40 0x560b3732abd0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000b780 tx=0x7fdbf000ba90 comp rx=0 tx=0).stop 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.235+0000 7fdc122acb80 1 -- 172.21.15.12:0/1702841994 shutdown_connections 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.235+0000 7fdc122acb80 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b373251c0 0x560b373255d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.235+0000 7fdc122acb80 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b371a1e40 0x560b3732abd0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.235+0000 7fdc122acb80 1 --2- 172.21.15.12:0/1702841994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b3732b110 0x560b37324c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.235+0000 7fdc122acb80 1 -- 172.21.15.12:0/1702841994 >> 172.21.15.12:0/1702841994 conn(0x560b37201fc0 msgr2=0x560b3731b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.235+0000 7fdc122acb80 1 -- 172.21.15.12:0/1702841994 shutdown_connections 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.235+0000 7fdc122acb80 1 -- 172.21.15.12:0/1702841994 wait complete. 2024-05-27T07:39:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.236+0000 7fdc122acb80 1 -- start start 2024-05-27T07:39:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.236+0000 7fdc122acb80 1 Processor -- start 2024-05-27T07:39:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.236+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b3732ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.236+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b3732ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.236+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b3732f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc122acb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560b37144fa0 con 0x560b3732b110 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc122acb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560b371df760 con 0x560b371a1e40 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc122acb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560b37331bb0 con 0x560b373251c0 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc122acb80 10 client.0 _ll_register_callbacks cb 0x560b371e8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b3732f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b3732f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45836/0 (socket says 172.21.15.12:45836) 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc01800700 1 -- 172.21.15.12:0/406997555 learned_addr learned my addr 172.21.15.12:0/406997555 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b3732ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc00e00700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b3732ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc01800700 1 -- 172.21.15.12:0/406997555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 msgr2=0x560b3732ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc01800700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b3732ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc01800700 1 -- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 msgr2=0x560b3732ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.237+0000 7fdc01800700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b3732ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdc01800700 1 -- 172.21.15.12:0/406997555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b3718e8d0 con 0x560b3732b110 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdc00e00700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b3732ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdc01800700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b3732f4b0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40a5470 tx=0x7fdbf40e23a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7787f2131e5ad92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdbe2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdbe2a00700 1 -- 172.21.15.12:0/406997555 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40a93f0 con 0x560b3732b110 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdbe2a00700 1 -- 172.21.15.12:0/406997555 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9642+0+0 (secure 0 0 0) 0x7fdbf40b1f40 con 0x560b3732b110 2024-05-27T07:39:05.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf0015070 con 0x560b3732b110 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.238+0000 7fdbe2a00700 1 -- 172.21.15.12:0/406997555 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fdbf40ad7a0 con 0x560b3732b110 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 10 client.17397 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x560b3720f060 con 0x560b3732b110 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdbe2a00700 1 -- 172.21.15.12:0/406997555 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf0015070 con 0x560b3732b110 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdbe2a00700 1 -- 172.21.15.12:0/406997555 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fdbf40a4df0 con 0x560b3732b110 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 10 client.17397 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 10 client.17397 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 20 client.17397 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b3733ef40 0x560b373413b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 4 client.17397 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:05.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.239+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b37250d90 con 0x560b3733ef40 2024-05-27T07:39:05.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.240+0000 7fdc00e00700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b3733ef40 0x560b373413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.241+0000 7fdc00e00700 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b3733ef40 0x560b373413b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d16798140ac79911 server_cookie=2a38c88d430ace5d in_seq=0 out_seq=0 2024-05-27T07:39:05.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.241+0000 7fdbe2a00700 10 client.17397 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:39:05.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdbe2a00700 1 -- 172.21.15.12:0/406997555 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560b37250d90 con 0x560b3733ef40 2024-05-27T07:39:05.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdbe2a00700 10 client.17397 handle_command_reply: tid=0 2024-05-27T07:39:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 client.17397 shutdown 2024-05-27T07:39:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 msgr2=0x560b3732f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b3732f4b0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40a5470 tx=0x7fdbf40e23a0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 shutdown_connections 2024-05-27T07:39:05.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b3733ef40 0x560b373413b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b3732ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b3732f4b0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 --2- 172.21.15.12:0/406997555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b3732ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.243+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 >> 172.21.15.12:0/406997555 conn(0x560b37201fc0 msgr2=0x560b3731b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.244+0000 7fdbe2a00700 0 client.17397 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.244+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 shutdown_connections 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.244+0000 7fdc122acb80 1 -- 172.21.15.12:0/406997555 wait complete. 2024-05-27T07:39:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.244+0000 7fdc122acb80 20 client.17397 trim_cache size 0 max 16384 2024-05-27T07:39:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.261+0000 7fdc122acb80 1 Processor -- start 2024-05-27T07:39:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.261+0000 7fdc122acb80 1 -- start start 2024-05-27T07:39:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.261+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b371a1e40 0x560b3732da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b373251c0 0x560b3731f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b3732b110 0x560b3731fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdc122acb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ada30 con 0x560b373251c0 2024-05-27T07:39:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdc122acb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1acc90 con 0x560b3732b110 2024-05-27T07:39:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdc122acb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ad360 con 0x560b371a1e40 2024-05-27T07:39:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b373251c0 0x560b3731f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b373251c0 0x560b3731f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45838/0 (socket says 172.21.15.12:45838) 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 -- 172.21.15.12:0/2506446521 learned_addr learned my addr 172.21.15.12:0/2506446521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdc01800700 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b3732b110 0x560b3731fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 -- 172.21.15.12:0/2506446521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b371a1e40 msgr2=0x560b3732da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdc00e00700 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b371a1e40 0x560b3732da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b371a1e40 0x560b3732da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 -- 172.21.15.12:0/2506446521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b3732b110 msgr2=0x560b3731fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b3732b110 0x560b3731fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.262+0000 7fdbfbe00700 1 -- 172.21.15.12:0/2506446521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b3732f820 con 0x560b373251c0 2024-05-27T07:39:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.263+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b373251c0 0x560b3731f860 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc076aa0 tx=0x7fdbe8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dafcc1b4a61944 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.263+0000 7fdbed600700 1 -- 172.21.15.12:0/2506446521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8035070 con 0x560b373251c0 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.263+0000 7fdbed600700 1 -- 172.21.15.12:0/2506446521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbe8004d20 con 0x560b373251c0 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.263+0000 7fdbed600700 1 -- 172.21.15.12:0/2506446521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8027830 con 0x560b373251c0 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 -- 172.21.15.12:0/2506446521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b373251c0 msgr2=0x560b3731f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b373251c0 0x560b3731f860 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc076aa0 tx=0x7fdbe8010000 comp rx=0 tx=0).stop 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 -- 172.21.15.12:0/2506446521 shutdown_connections 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b3732b110 0x560b3731fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b371a1e40 0x560b3732da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 --2- 172.21.15.12:0/2506446521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b373251c0 0x560b3731f860 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 -- 172.21.15.12:0/2506446521 >> 172.21.15.12:0/2506446521 conn(0x560b37201fc0 msgr2=0x560b3732d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.264+0000 7fdc122acb80 1 -- 172.21.15.12:0/2506446521 shutdown_connections 2024-05-27T07:39:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.265+0000 7fdc122acb80 1 -- 172.21.15.12:0/2506446521 wait complete. 2024-05-27T07:39:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.265+0000 7fdc122acb80 1 -- start start 2024-05-27T07:39:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.265+0000 7fdc122acb80 1 Processor -- start 2024-05-27T07:39:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b37336e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b37337340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc122acb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b37345ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc122acb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560b37341720 con 0x560b3732b110 2024-05-27T07:39:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc122acb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560b3732a520 con 0x560b371a1e40 2024-05-27T07:39:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc122acb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560b37331af0 con 0x560b373251c0 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc122acb80 10 client.0 _ll_register_callbacks cb 0x560b371e8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b37345ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b37345ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45844/0 (socket says 172.21.15.12:45844) 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.266+0000 7fdc01800700 1 -- 172.21.15.12:0/91061538 learned_addr learned my addr 172.21.15.12:0/91061538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc00e00700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b37336e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdbfbe00700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b37337340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc01800700 1 -- 172.21.15.12:0/91061538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 msgr2=0x560b37337340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc01800700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b37337340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc01800700 1 -- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 msgr2=0x560b37336e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc01800700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b37336e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc01800700 1 -- 172.21.15.12:0/91061538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b373460e0 con 0x560b3732b110 2024-05-27T07:39:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc00e00700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b37336e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.267+0000 7fdc01800700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b37345ba0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400d010 tx=0x7fdbf40a5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3df303bb46c0a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.268+0000 7fdbe2a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.268+0000 7fdbe2a00700 1 -- 172.21.15.12:0/91061538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40a52b0 con 0x560b3732b110 2024-05-27T07:39:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.268+0000 7fdbe2a00700 1 -- 172.21.15.12:0/91061538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(150..150 src has 1..150) ==== 9642+0+0 (secure 0 0 0) 0x7fdbf40a59e0 con 0x560b3732b110 2024-05-27T07:39:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.268+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbe8004d20 con 0x560b3732b110 2024-05-27T07:39:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.268+0000 7fdbe2a00700 1 -- 172.21.15.12:0/91061538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fdbf40133e0 con 0x560b3732b110 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.268+0000 7fdc122acb80 10 client.17403 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.268+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x560b373a9070 con 0x560b3732b110 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdbe2a00700 1 -- 172.21.15.12:0/91061538 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbe8004d20 con 0x560b3732b110 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdbe2a00700 1 -- 172.21.15.12:0/91061538 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fdbf4083ab0 con 0x560b3732b110 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdc122acb80 10 client.17403 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdc122acb80 10 client.17403 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdc122acb80 20 client.17403 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdc122acb80 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b373a96f0 0x560b373abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdc122acb80 4 client.17403 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560b3724a570 con 0x560b373a96f0 2024-05-27T07:39:05.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.269+0000 7fdc00e00700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b373a96f0 0x560b373abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.271+0000 7fdc00e00700 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b373a96f0 0x560b373abb10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d319e81fc748e247 server_cookie=ee4a94f40bcbb118 in_seq=0 out_seq=0 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.271+0000 7fdbe2a00700 10 client.17403 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.271+0000 7fdbe2a00700 1 -- 172.21.15.12:0/91061538 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560b3724a570 con 0x560b373a96f0 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.271+0000 7fdbe2a00700 10 client.17403 handle_command_reply: tid=0 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.271+0000 7fdc122acb80 1 client.17403 shutdown 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 msgr2=0x560b37345ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b37345ba0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400d010 tx=0x7fdbf40a5040 comp rx=0 tx=0).stop 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 shutdown_connections 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x560b373a96f0 0x560b373abb10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560b371a1e40 0x560b37336e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560b3732b110 0x560b37345ba0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 --2- 172.21.15.12:0/91061538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560b373251c0 0x560b37337340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 >> 172.21.15.12:0/91061538 conn(0x560b37201fc0 msgr2=0x560b3731f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdbe2a00700 0 client.17403 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 shutdown_connections 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 1 -- 172.21.15.12:0/91061538 wait complete. 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.272+0000 7fdc122acb80 20 client.17403 trim_cache size 0 max 16384 2024-05-27T07:39:05.273 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:39:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.273+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdbe40b2a70 msgr2=0x7fdbe40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.273+0000 7fdc03800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdbe40b2a70 0x7fdbe40b4f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8000c00 tx=0x7fdbe80051e0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.273+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc108730 msgr2=0x7fdbfc19f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.273+0000 7fdc03800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc108730 0x7fdbfc19f540 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400ef30 tx=0x7fdbf400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 shutdown_connections 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdbe40b2a70 0x7fdbe40b4f80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbfc108730 0x7fdbfc19f540 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbfc1034b0 0x7fdbfc19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 --2- 172.21.15.12:0/4157071473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc107de0 0x7fdbfc19b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 >> 172.21.15.12:0/4157071473 conn(0x7fdbfc0fedb0 msgr2=0x7fdbfc06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 shutdown_connections 2024-05-27T07:39:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:05.274+0000 7fdc03800700 1 -- 172.21.15.12:0/4157071473 wait complete. 2024-05-27T07:39:06.170 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:39:06.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:06 smithi012 ceph-mon[31473]: pgmap v571: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 265 op/s 2024-05-27T07:39:06.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:06 smithi162 ceph-mon[35676]: pgmap v571: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 265 op/s 2024-05-27T07:39:06.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:06 smithi062 ceph-mon[35304]: pgmap v571: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 265 op/s 2024-05-27T07:39:06.687 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:08.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:08 smithi012 ceph-mon[31473]: pgmap v572: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 265 op/s 2024-05-27T07:39:08.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:08 smithi012 ceph-mon[31473]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:39:08.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:08 smithi162 ceph-mon[35676]: pgmap v572: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 265 op/s 2024-05-27T07:39:08.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:08 smithi162 ceph-mon[35676]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:39:08.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:08 smithi062 ceph-mon[35304]: pgmap v572: 129 pgs: 129 active+clean; 394 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 265 op/s 2024-05-27T07:39:08.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:08 smithi062 ceph-mon[35304]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:39:09.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 -- 172.21.15.12:0/211294373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 msgr2=0x7f96e8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 --2- 172.21.15.12:0/211294373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e8109320 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f96d40099c0 tx=0x7f96d4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 -- 172.21.15.12:0/211294373 shutdown_connections 2024-05-27T07:39:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 --2- 172.21.15.12:0/211294373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96e8109860 0x7f96e81111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 --2- 172.21.15.12:0/211294373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e81045e0 0x7f96e81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 --2- 172.21.15.12:0/211294373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e8109320 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 -- 172.21.15.12:0/211294373 >> 172.21.15.12:0/211294373 conn(0x7f96e8100030 msgr2=0x7f96e8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 -- 172.21.15.12:0/211294373 shutdown_connections 2024-05-27T07:39:09.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.365+0000 7f96ee200700 1 -- 172.21.15.12:0/211294373 wait complete. 2024-05-27T07:39:09.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.366+0000 7f96ee200700 1 Processor -- start 2024-05-27T07:39:09.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96ee200700 1 -- start start 2024-05-27T07:39:09.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96ee200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96e81045e0 0x7f96e819d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96ee200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96ee200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e8109860 0x7f96e81a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96ee200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8114370 con 0x7f96e8108f10 2024-05-27T07:39:09.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96ee200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f96e81141f0 con 0x7f96e81045e0 2024-05-27T07:39:09.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96ee200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96e81144f0 con 0x7f96e8109860 2024-05-27T07:39:09.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.367+0000 7f96e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e819d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45864/0 (socket says 172.21.15.12:45864) 2024-05-27T07:39:09.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e6a00700 1 -- 172.21.15.12:0/2122174764 learned_addr learned my addr 172.21.15.12:0/2122174764 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:09.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e7400700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96e81045e0 0x7f96e819d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e7e00700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e8109860 0x7f96e81a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e6a00700 1 -- 172.21.15.12:0/2122174764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e8109860 msgr2=0x7f96e81a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e6a00700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e8109860 0x7f96e81a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e6a00700 1 -- 172.21.15.12:0/2122174764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96e81045e0 msgr2=0x7f96e819d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e6a00700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96e81045e0 0x7f96e819d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.368+0000 7f96e6a00700 1 -- 172.21.15.12:0/2122174764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e81a1e80 con 0x7f96e8108f10 2024-05-27T07:39:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.369+0000 7f96e6a00700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e819d600 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f96d40098e0 tx=0x7f96d400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.369+0000 7f96dfe00700 1 -- 172.21.15.12:0/2122174764 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4027070 con 0x7f96e8108f10 2024-05-27T07:39:09.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.369+0000 7f96dfe00700 1 -- 172.21.15.12:0/2122174764 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96d4022470 con 0x7f96e8108f10 2024-05-27T07:39:09.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.369+0000 7f96dfe00700 1 -- 172.21.15.12:0/2122174764 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4017650 con 0x7f96e8108f10 2024-05-27T07:39:09.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.369+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e81a2110 con 0x7f96e8108f10 2024-05-27T07:39:09.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.369+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e81a2640 con 0x7f96e8108f10 2024-05-27T07:39:09.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.374+0000 7f96dfe00700 1 -- 172.21.15.12:0/2122174764 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f96d402a070 con 0x7f96e8108f10 2024-05-27T07:39:09.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 Processor -- start 2024-05-27T07:39:09.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 -- start start 2024-05-27T07:39:09.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46389f0 0x55b5e46cb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e454ce40 0x55b5e46cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46d8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b5e45f9e40 con 0x55b5e46389f0 2024-05-27T07:39:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b5e45e99e0 con 0x55b5e46cbe70 2024-05-27T07:39:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.375+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5e45eda20 con 0x55b5e454ce40 2024-05-27T07:39:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.376+0000 7f96e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46389f0 0x55b5e46cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.376+0000 7f96e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46389f0 0x55b5e46cb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45874/0 (socket says 172.21.15.12:45874) 2024-05-27T07:39:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.376+0000 7f96e7e00700 1 -- 172.21.15.12:0/1488808323 learned_addr learned my addr 172.21.15.12:0/1488808323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.376+0000 7f96dfe00700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96cc0b2a70 0x7f96cc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.376+0000 7f96e7400700 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e454ce40 0x55b5e46cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.376+0000 7f96dfe00700 1 -- 172.21.15.12:0/2122174764 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f96d40e3950 con 0x7f96e8108f10 2024-05-27T07:39:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.376+0000 7f96e6a00700 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46d8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7e00700 1 -- 172.21.15.12:0/1488808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e454ce40 msgr2=0x55b5e46cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7e00700 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e454ce40 0x55b5e46cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7e00700 1 -- 172.21.15.12:0/1488808323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 msgr2=0x55b5e46d8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7e00700 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46d8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7e00700 1 -- 172.21.15.12:0/1488808323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e45b2420 con 0x55b5e46389f0 2024-05-27T07:39:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7400700 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e454ce40 0x55b5e46cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e6a00700 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46d8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7e00700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96cc0b2a70 0x7f96cc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.377+0000 7f96e7e00700 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46389f0 0x55b5e46cb330 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f96e0008630 tx=0x7f96e0008940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8519cc18ad2a8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.378+0000 7f96dcc00700 1 -- 172.21.15.12:0/1488808323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0022440 con 0x55b5e46389f0 2024-05-27T07:39:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.378+0000 7f96dcc00700 1 -- 172.21.15.12:0/1488808323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96e00207e0 con 0x55b5e46389f0 2024-05-27T07:39:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.378+0000 7f96dcc00700 1 -- 172.21.15.12:0/1488808323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e00165b0 con 0x55b5e46389f0 2024-05-27T07:39:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.378+0000 7f96e7e00700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96cc0b2a70 0x7f96cc0b4f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f96e001f000 tx=0x7f96e0018000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 -- 172.21.15.12:0/1488808323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46389f0 msgr2=0x55b5e46cb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46389f0 0x55b5e46cb330 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f96e0008630 tx=0x7f96e0008940 comp rx=0 tx=0).stop 2024-05-27T07:39:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 -- 172.21.15.12:0/1488808323 shutdown_connections 2024-05-27T07:39:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46d8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46389f0 0x55b5e46cb330 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/1488808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e454ce40 0x55b5e46cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 -- 172.21.15.12:0/1488808323 >> 172.21.15.12:0/1488808323 conn(0x55b5e45e6390 msgr2=0x55b5e46c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 -- 172.21.15.12:0/1488808323 shutdown_connections 2024-05-27T07:39:09.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.379+0000 7f96fcd85b80 1 -- 172.21.15.12:0/1488808323 wait complete. 2024-05-27T07:39:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.380+0000 7f96fcd85b80 1 -- start start 2024-05-27T07:39:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.380+0000 7f96fcd85b80 1 Processor -- start 2024-05-27T07:39:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46d5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46d6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b5e44effa0 con 0x55b5e46cbe70 2024-05-27T07:39:09.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b5e458a760 con 0x55b5e454ce40 2024-05-27T07:39:09.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5e46d8b80 con 0x55b5e46389f0 2024-05-27T07:39:09.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96fcd85b80 10 client.0 _ll_register_callbacks cb 0x55b5e4593c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:09.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46d6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46d6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45888/0 (socket says 172.21.15.12:45888) 2024-05-27T07:39:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96e7e00700 1 -- 172.21.15.12:0/2690758927 learned_addr learned my addr 172.21.15.12:0/2690758927 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96e7400700 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.381+0000 7f96e6a00700 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46d5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.382+0000 7f96e7e00700 1 -- 172.21.15.12:0/2690758927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 msgr2=0x55b5e46d5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.382+0000 7f96e7e00700 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46d5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.382+0000 7f96e7e00700 1 -- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 msgr2=0x55b5e46d56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.382+0000 7f96e7e00700 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.382+0000 7f96e7e00700 1 -- 172.21.15.12:0/2690758927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e45398d0 con 0x55b5e46cbe70 2024-05-27T07:39:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.382+0000 7f96e7e00700 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46d6140 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f96e0020590 tx=0x7f96e0020670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74253f29a2183de0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.382+0000 7f96d1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96d1600700 1 -- 172.21.15.12:0/2690758927 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0030060 con 0x55b5e46cbe70 2024-05-27T07:39:09.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96d1600700 1 -- 172.21.15.12:0/2690758927 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f96e0020b90 con 0x55b5e46cbe70 2024-05-27T07:39:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e00207e0 con 0x55b5e46cbe70 2024-05-27T07:39:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96d1600700 1 -- 172.21.15.12:0/2690758927 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f96e0037050 con 0x55b5e46cbe70 2024-05-27T07:39:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96fcd85b80 10 client.17415 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55b5e45ba060 con 0x55b5e46cbe70 2024-05-27T07:39:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96d1600700 1 -- 172.21.15.12:0/2690758927 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96e00207e0 con 0x55b5e46cbe70 2024-05-27T07:39:09.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96d1600700 1 -- 172.21.15.12:0/2690758927 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f96e002a030 con 0x55b5e46cbe70 2024-05-27T07:39:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96fcd85b80 10 client.17415 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96fcd85b80 10 client.17415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.383+0000 7f96fcd85b80 20 client.17415 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.384+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e46e5e10 0x55b5e46e82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.384+0000 7f96fcd85b80 4 client.17415 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.384+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5e45fbd90 con 0x55b5e46e5e10 2024-05-27T07:39:09.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.384+0000 7f96e7400700 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e46e5e10 0x55b5e46e82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.385+0000 7f96e7400700 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e46e5e10 0x55b5e46e82c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eff570b5f5bd656 server_cookie=ab53075a82c62c96 in_seq=0 out_seq=0 2024-05-27T07:39:09.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.385+0000 7f96d1600700 10 client.17415 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.387+0000 7f96d1600700 1 -- 172.21.15.12:0/2690758927 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b5e45fbd90 con 0x55b5e46e5e10 2024-05-27T07:39:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.387+0000 7f96d1600700 10 client.17415 handle_command_reply: tid=0 2024-05-27T07:39:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.387+0000 7f96fcd85b80 1 client.17415 shutdown 2024-05-27T07:39:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 msgr2=0x55b5e46d6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46d6140 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f96e0020590 tx=0x7f96e0020670 comp rx=0 tx=0).stop 2024-05-27T07:39:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 shutdown_connections 2024-05-27T07:39:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e46e5e10 0x55b5e46e82c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46d6140 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2690758927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46d5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96d1600700 0 client.17415 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:09.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 >> 172.21.15.12:0/2690758927 conn(0x55b5e45e6390 msgr2=0x55b5e46cb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 shutdown_connections 2024-05-27T07:39:09.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2690758927 wait complete. 2024-05-27T07:39:09.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.388+0000 7f96fcd85b80 20 client.17415 trim_cache size 0 max 16384 2024-05-27T07:39:09.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.412+0000 7f96fcd85b80 1 Processor -- start 2024-05-27T07:39:09.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.412+0000 7f96fcd85b80 1 -- start start 2024-05-27T07:39:09.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46eda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b5e46d5140 con 0x55b5e46cbe70 2024-05-27T07:39:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f96e81b1d10 con 0x55b5e454ce40 2024-05-27T07:39:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96e81b0f70 con 0x55b5e46389f0 2024-05-27T07:39:09.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96e7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46eda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45896/0 (socket says 172.21.15.12:45896) 2024-05-27T07:39:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.413+0000 7f96e7e00700 1 -- 172.21.15.12:0/3689765454 learned_addr learned my addr 172.21.15.12:0/3689765454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.414+0000 7f96e7400700 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.414+0000 7f96e7e00700 1 -- 172.21.15.12:0/3689765454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 msgr2=0x55b5e46cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.414+0000 7f96e7e00700 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.414+0000 7f96e7e00700 1 -- 172.21.15.12:0/3689765454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 msgr2=0x55b5e46d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.414+0000 7f96e7e00700 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.414+0000 7f96e7e00700 1 -- 172.21.15.12:0/3689765454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e46d49e0 con 0x55b5e46cbe70 2024-05-27T07:39:09.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.414+0000 7f96e7e00700 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46eda80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f96e0008d40 tx=0x7f96e0004b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995b3070fe283c83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.415+0000 7f96d3e00700 1 -- 172.21.15.12:0/3689765454 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0040070 con 0x55b5e46cbe70 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.415+0000 7f96d3e00700 1 -- 172.21.15.12:0/3689765454 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96e0022e20 con 0x55b5e46cbe70 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.415+0000 7f96d3e00700 1 -- 172.21.15.12:0/3689765454 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0030040 con 0x55b5e46cbe70 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.416+0000 7f96fcd85b80 1 -- 172.21.15.12:0/3689765454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 msgr2=0x55b5e46eda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.416+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46eda80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f96e0008d40 tx=0x7f96e0004b70 comp rx=0 tx=0).stop 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.417+0000 7f96fcd85b80 1 -- 172.21.15.12:0/3689765454 shutdown_connections 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.417+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e454ce40 0x55b5e46d0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.417+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e46cbe70 0x55b5e46eda80 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.417+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/3689765454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.417+0000 7f96fcd85b80 1 -- 172.21.15.12:0/3689765454 >> 172.21.15.12:0/3689765454 conn(0x55b5e45e6390 msgr2=0x55b5e46cb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.418+0000 7f96fcd85b80 1 -- 172.21.15.12:0/3689765454 shutdown_connections 2024-05-27T07:39:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.420+0000 7f96fcd85b80 1 -- 172.21.15.12:0/3689765454 wait complete. 2024-05-27T07:39:09.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.420+0000 7f96fcd85b80 1 -- start start 2024-05-27T07:39:09.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.421+0000 7f96fcd85b80 1 Processor -- start 2024-05-27T07:39:09.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.422+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e454ce40 0x55b5e46cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.422+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.422+0000 7f96fcd85b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46dce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.422+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b5e458a760 con 0x55b5e454ce40 2024-05-27T07:39:09.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.422+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f96e81b1640 con 0x55b5e46cbe70 2024-05-27T07:39:09.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.422+0000 7f96fcd85b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5e46c6150 con 0x55b5e46389f0 2024-05-27T07:39:09.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.422+0000 7f96fcd85b80 10 client.0 _ll_register_callbacks cb 0x55b5e4593c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e454ce40 0x55b5e46cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46dce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e454ce40 0x55b5e46cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45908/0 (socket says 172.21.15.12:45908) 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 -- 172.21.15.12:0/2366876065 learned_addr learned my addr 172.21.15.12:0/2366876065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 -- 172.21.15.12:0/2366876065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 msgr2=0x55b5e46cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 -- 172.21.15.12:0/2366876065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 msgr2=0x55b5e46dce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46dce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.423+0000 7f96e7400700 1 -- 172.21.15.12:0/2366876065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e46dd3b0 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.424+0000 7f96e7400700 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e454ce40 0x55b5e46cfa10 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x55b5e46e8a20 tx=0x7f96d8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c096fdd6f30b2b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.424+0000 7f96d1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.424+0000 7f96d1600700 1 -- 172.21.15.12:0/2366876065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8004320 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.424+0000 7f96d1600700 1 -- 172.21.15.12:0/2366876065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f96d80044c0 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.424+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e0022e20 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96d1600700 1 -- 172.21.15.12:0/2366876065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f96d802c050 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 10 client.17421 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55b5e46de550 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96d1600700 1 -- 172.21.15.12:0/2366876065 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96e0022e20 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96d1600700 1 -- 172.21.15.12:0/2366876065 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f96d801d720 con 0x55b5e454ce40 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 10 client.17421 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:09.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 20 client.17421 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:09.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e47591b0 0x55b5e475b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:09.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 4 client.17421 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:09.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.425+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5e45f5570 con 0x55b5e47591b0 2024-05-27T07:39:09.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.426+0000 7f96e6a00700 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e47591b0 0x55b5e475b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:09.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.429+0000 7f96d1600700 10 client.17421 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:09.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.429+0000 7f96e6a00700 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e47591b0 0x55b5e475b5d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d2df55c93d7f3e1 server_cookie=38565aaf795ec251 in_seq=0 out_seq=0 2024-05-27T07:39:09.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.431+0000 7f96d1600700 1 -- 172.21.15.12:0/2366876065 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5e45f5570 con 0x55b5e47591b0 2024-05-27T07:39:09.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.431+0000 7f96d1600700 10 client.17421 handle_command_reply: tid=0 2024-05-27T07:39:09.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.431+0000 7f96fcd85b80 1 client.17421 shutdown 2024-05-27T07:39:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e454ce40 msgr2=0x55b5e46cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e454ce40 0x55b5e46cfa10 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x55b5e46e8a20 tx=0x7f96d8011a00 comp rx=0 tx=0).stop 2024-05-27T07:39:09.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 shutdown_connections 2024-05-27T07:39:09.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55b5e47591b0 0x55b5e475b5d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e46cbe70 0x55b5e46dce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b5e454ce40 0x55b5e46cfa10 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 --2- 172.21.15.12:0/2366876065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5e46389f0 0x55b5e46cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 >> 172.21.15.12:0/2366876065 conn(0x55b5e45e6390 msgr2=0x55b5e46d5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.433+0000 7f96d1600700 0 client.17421 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.434+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 shutdown_connections 2024-05-27T07:39:09.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.434+0000 7f96fcd85b80 1 -- 172.21.15.12:0/2366876065 wait complete. 2024-05-27T07:39:09.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.434+0000 7f96fcd85b80 20 client.17421 trim_cache size 0 max 16384 2024-05-27T07:39:09.433 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96cc0b2a70 msgr2=0x7f96cc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96cc0b2a70 0x7f96cc0b4f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f96e001f000 tx=0x7f96e0018000 comp rx=0 tx=0).stop 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 msgr2=0x7f96e819d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e819d600 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f96d40098e0 tx=0x7f96d400f690 comp rx=0 tx=0).stop 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 shutdown_connections 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f96cc0b2a70 0x7f96cc0b4f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96e81045e0 0x7f96e819d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e8109860 0x7f96e81a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 --2- 172.21.15.12:0/2122174764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96e8108f10 0x7f96e819d600 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:09.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.437+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 >> 172.21.15.12:0/2122174764 conn(0x7f96e8100030 msgr2=0x7f96e810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:09.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.438+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 shutdown_connections 2024-05-27T07:39:09.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:09.438+0000 7f96ee200700 1 -- 172.21.15.12:0/2122174764 wait complete. 2024-05-27T07:39:10.323 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:10 smithi012 ceph-mon[31473]: pgmap v574: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 310 op/s 2024-05-27T07:39:10.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:10 smithi162 ceph-mon[35676]: pgmap v574: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 310 op/s 2024-05-27T07:39:10.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:10 smithi062 ceph-mon[35304]: pgmap v574: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 310 op/s 2024-05-27T07:39:11.393 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:39:11.393 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:39:11.888 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:12.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:12 smithi012 ceph-mon[31473]: pgmap v575: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 252 op/s 2024-05-27T07:39:12.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:12 smithi162 ceph-mon[35676]: pgmap v575: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 252 op/s 2024-05-27T07:39:12.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:12 smithi062 ceph-mon[35304]: pgmap v575: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 252 op/s 2024-05-27T07:39:14.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:14 smithi012 ceph-mon[31473]: pgmap v576: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 248 op/s 2024-05-27T07:39:14.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 -- 172.21.15.12:0/376210018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 msgr2=0x7fc65010c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 --2- 172.21.15.12:0/376210018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc65010c9b0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc644009990 tx=0x7fc644009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 -- 172.21.15.12:0/376210018 shutdown_connections 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 --2- 172.21.15.12:0/376210018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc650106eb0 0x7fc650113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 --2- 172.21.15.12:0/376210018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc65010c9b0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 --2- 172.21.15.12:0/376210018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc650106560 0x7fc650106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 -- 172.21.15.12:0/376210018 >> 172.21.15.12:0/376210018 conn(0x7fc650102040 msgr2=0x7fc650104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.365+0000 7fc658800700 1 -- 172.21.15.12:0/376210018 shutdown_connections 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.366+0000 7fc658800700 1 -- 172.21.15.12:0/376210018 wait complete. 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 Processor -- start 2024-05-27T07:39:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 -- start start 2024-05-27T07:39:14.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc650106560 0x7fc65019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc650106eb0 0x7fc65019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc6501a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc650116630 con 0x7fc65010c5a0 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc6501164b0 con 0x7fc650106560 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.367+0000 7fc658800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6501167b0 con 0x7fc650106eb0 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.368+0000 7fc656800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc6501a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.368+0000 7fc656800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc6501a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43426/0 (socket says 172.21.15.12:43426) 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.368+0000 7fc656800700 1 -- 172.21.15.12:0/1185906417 learned_addr learned my addr 172.21.15.12:0/1185906417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.368+0000 7fc655e00700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc650106560 0x7fc65019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.368+0000 7fc655400700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc650106eb0 0x7fc65019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc656800700 1 -- 172.21.15.12:0/1185906417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc650106eb0 msgr2=0x7fc65019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc656800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc650106eb0 0x7fc65019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc656800700 1 -- 172.21.15.12:0/1185906417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc650106560 msgr2=0x7fc65019d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc656800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc650106560 0x7fc65019d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc656800700 1 -- 172.21.15.12:0/1185906417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6501a1f20 con 0x7fc65010c5a0 2024-05-27T07:39:14.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc656800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc6501a1880 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00b970 tx=0x7fc64c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:14.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc64aa00700 1 -- 172.21.15.12:0/1185906417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c00d860 con 0x7fc65010c5a0 2024-05-27T07:39:14.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc64aa00700 1 -- 172.21.15.12:0/1185906417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc64c00e430 con 0x7fc65010c5a0 2024-05-27T07:39:14.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.369+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6501a2210 con 0x7fc65010c5a0 2024-05-27T07:39:14.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.370+0000 7fc64aa00700 1 -- 172.21.15.12:0/1185906417 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c017660 con 0x7fc65010c5a0 2024-05-27T07:39:14.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.370+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6501a9b30 con 0x7fc65010c5a0 2024-05-27T07:39:14.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.374+0000 7fc64aa00700 1 -- 172.21.15.12:0/1185906417 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc64c00da00 con 0x7fc65010c5a0 2024-05-27T07:39:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.375+0000 7fc6673efb80 1 Processor -- start 2024-05-27T07:39:14.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc6673efb80 1 -- start start 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc64aa00700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc6380b2a70 0x7fc6380b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e777d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e777ddb0 0x55a1e77778f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7778240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc6673efb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a1e76a5ec0 con 0x55a1e777ddb0 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc6673efb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a1e7695a60 con 0x55a1e75f8ec0 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc6673efb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a1e7699aa0 con 0x55a1e7777e30 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc64aa00700 1 -- 172.21.15.12:0/1185906417 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7fc64c0e2680 con 0x7fc65010c5a0 2024-05-27T07:39:14.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.376+0000 7fc655400700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc6380b2a70 0x7fc6380b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e777ddb0 0x55a1e77778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e777d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e777ddb0 0x55a1e77778f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43432/0 (socket says 172.21.15.12:43432) 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 -- 172.21.15.12:0/4148318742 learned_addr learned my addr 172.21.15.12:0/4148318742 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc656800700 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7778240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 -- 172.21.15.12:0/4148318742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 msgr2=0x55a1e7778240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7778240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 -- 172.21.15.12:0/4148318742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 msgr2=0x55a1e777d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e777d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.377+0000 7fc655400700 1 -- 172.21.15.12:0/4148318742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1e765e4a0 con 0x55a1e777ddb0 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.378+0000 7fc655400700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc6380b2a70 0x7fc6380b4f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc640015740 tx=0x7fc6400099d0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.378+0000 7fc655400700 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e777ddb0 0x55a1e77778f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc6400186f0 tx=0x7fc64001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d579ef891af27cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.378+0000 7fc63f400700 1 -- 172.21.15.12:0/4148318742 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640027070 con 0x55a1e777ddb0 2024-05-27T07:39:14.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.378+0000 7fc63f400700 1 -- 172.21.15.12:0/4148318742 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc640009420 con 0x55a1e777ddb0 2024-05-27T07:39:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.378+0000 7fc63f400700 1 -- 172.21.15.12:0/4148318742 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6400176a0 con 0x55a1e777ddb0 2024-05-27T07:39:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.379+0000 7fc6673efb80 1 -- 172.21.15.12:0/4148318742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e777ddb0 msgr2=0x55a1e77778f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.379+0000 7fc6673efb80 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e777ddb0 0x55a1e77778f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc6400186f0 tx=0x7fc64001a000 comp rx=0 tx=0).stop 2024-05-27T07:39:14.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.379+0000 7fc6673efb80 1 -- 172.21.15.12:0/4148318742 shutdown_connections 2024-05-27T07:39:14.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.379+0000 7fc6673efb80 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e777d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.379+0000 7fc6673efb80 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7778240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.379+0000 7fc6673efb80 1 --2- 172.21.15.12:0/4148318742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e777ddb0 0x55a1e77778f0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.379+0000 7fc6673efb80 1 -- 172.21.15.12:0/4148318742 >> 172.21.15.12:0/4148318742 conn(0x55a1e7659040 msgr2=0x55a1e7772950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:14.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.380+0000 7fc6673efb80 1 -- 172.21.15.12:0/4148318742 shutdown_connections 2024-05-27T07:39:14.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.380+0000 7fc6673efb80 1 -- 172.21.15.12:0/4148318742 wait complete. 2024-05-27T07:39:14.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.380+0000 7fc6673efb80 1 -- start start 2024-05-27T07:39:14.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 1 Processor -- start 2024-05-27T07:39:14.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e77816b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7781bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e777ddb0 0x55a1e7782130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a1e759c020 con 0x55a1e75f8ec0 2024-05-27T07:39:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a1e76367e0 con 0x55a1e777ddb0 2024-05-27T07:39:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a1e7784820 con 0x55a1e7777e30 2024-05-27T07:39:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.381+0000 7fc6673efb80 10 client.0 _ll_register_callbacks cb 0x55a1e763fcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e77816b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e77816b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43438/0 (socket says 172.21.15.12:43438) 2024-05-27T07:39:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 -- 172.21.15.12:0/1131551582 learned_addr learned my addr 172.21.15.12:0/1131551582 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:14.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc656800700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e777ddb0 0x55a1e7782130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655400700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7781bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 -- 172.21.15.12:0/1131551582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 msgr2=0x55a1e7781bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7781bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 -- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e777ddb0 msgr2=0x55a1e7782130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e777ddb0 0x55a1e7782130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655e00700 1 -- 172.21.15.12:0/1131551582 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1e75e5950 con 0x55a1e75f8ec0 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc656800700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e777ddb0 0x55a1e7782130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:14.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.382+0000 7fc655400700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7781bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.383+0000 7fc655e00700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e77816b0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc644009990 tx=0x7fc644003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35a02086711b51e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.383+0000 7fc633e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.383+0000 7fc633e00700 1 -- 172.21.15.12:0/1131551582 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc644028060 con 0x55a1e75f8ec0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.383+0000 7fc633e00700 1 -- 172.21.15.12:0/1131551582 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7fc644005630 con 0x55a1e75f8ec0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.383+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc640009420 con 0x55a1e75f8ec0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc633e00700 1 -- 172.21.15.12:0/1131551582 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc64402f050 con 0x55a1e75f8ec0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 10 client.17433 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a1e76660e0 con 0x55a1e75f8ec0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc633e00700 1 -- 172.21.15.12:0/1131551582 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc640009420 con 0x55a1e75f8ec0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc633e00700 1 -- 172.21.15.12:0/1131551582 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fc644023960 con 0x55a1e75f8ec0 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 10 client.17433 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 10 client.17433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 20 client.17433 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:14.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e7791c20 0x55a1e77940f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 4 client.17433 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:14.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.384+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1e76a7e10 con 0x55a1e7791c20 2024-05-27T07:39:14.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.385+0000 7fc655400700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e7791c20 0x55a1e77940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.386+0000 7fc655400700 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e7791c20 0x55a1e77940f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3294d0f2c5f1766d server_cookie=327a7b20c46a319e in_seq=0 out_seq=0 2024-05-27T07:39:14.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.386+0000 7fc633e00700 10 client.17433 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:14.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.388+0000 7fc633e00700 1 -- 172.21.15.12:0/1131551582 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a1e76a7e10 con 0x55a1e7791c20 2024-05-27T07:39:14.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.388+0000 7fc633e00700 10 client.17433 handle_command_reply: tid=0 2024-05-27T07:39:14.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.388+0000 7fc6673efb80 1 client.17433 shutdown 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.388+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 msgr2=0x55a1e77816b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.388+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e77816b0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc644009990 tx=0x7fc644003730 comp rx=0 tx=0).stop 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 shutdown_connections 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e7791c20 0x55a1e77940f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e777ddb0 0x55a1e7782130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e77816b0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1131551582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e7777e30 0x55a1e7781bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 >> 172.21.15.12:0/1131551582 conn(0x55a1e7659040 msgr2=0x55a1e7772600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc633e00700 0 client.17433 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 shutdown_connections 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 1 -- 172.21.15.12:0/1131551582 wait complete. 2024-05-27T07:39:14.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.389+0000 7fc6673efb80 20 client.17433 trim_cache size 0 max 16384 2024-05-27T07:39:14.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 Processor -- start 2024-05-27T07:39:14.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 -- start start 2024-05-27T07:39:14.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e77778f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e7777e30 0x55a1e77839c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7781ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a1e777cea0 con 0x55a1e7777e30 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc6501b1d40 con 0x55a1e75f8ec0 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.418+0000 7fc6673efb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6501b0fa0 con 0x55a1e777ddb0 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e7777e30 0x55a1e77839c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e7777e30 0x55a1e77839c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43446/0 (socket says 172.21.15.12:43446) 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 -- 172.21.15.12:0/1878061151 learned_addr learned my addr 172.21.15.12:0/1878061151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655e00700 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e77778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc656800700 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7781ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 -- 172.21.15.12:0/1878061151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 msgr2=0x55a1e7781ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7781ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 -- 172.21.15.12:0/1878061151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 msgr2=0x55a1e77778f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e77778f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.419+0000 7fc655400700 1 -- 172.21.15.12:0/1878061151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1e7783f00 con 0x55a1e7777e30 2024-05-27T07:39:14.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.420+0000 7fc655400700 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e7777e30 0x55a1e77839c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc640018ce0 tx=0x7fc640004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9a64ab44259034 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.420+0000 7fc63ea00700 1 -- 172.21.15.12:0/1878061151 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64003e070 con 0x55a1e7777e30 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.420+0000 7fc63ea00700 1 -- 172.21.15.12:0/1878061151 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc640022430 con 0x55a1e7777e30 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.420+0000 7fc63ea00700 1 -- 172.21.15.12:0/1878061151 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640027040 con 0x55a1e7777e30 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 -- 172.21.15.12:0/1878061151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e7777e30 msgr2=0x55a1e77839c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e7777e30 0x55a1e77839c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc640018ce0 tx=0x7fc640004860 comp rx=0 tx=0).stop 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 -- 172.21.15.12:0/1878061151 shutdown_connections 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e75f8ec0 0x55a1e77778f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7781ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1878061151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e7777e30 0x55a1e77839c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 -- 172.21.15.12:0/1878061151 >> 172.21.15.12:0/1878061151 conn(0x55a1e7659040 msgr2=0x55a1e7781490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 -- 172.21.15.12:0/1878061151 shutdown_connections 2024-05-27T07:39:14.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.421+0000 7fc6673efb80 1 -- 172.21.15.12:0/1878061151 wait complete. 2024-05-27T07:39:14.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.422+0000 7fc6673efb80 1 -- start start 2024-05-27T07:39:14.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.422+0000 7fc6673efb80 1 Processor -- start 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e7782d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e7777e30 0x55a1e77832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc6673efb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7783810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc6673efb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a1e777cea0 con 0x55a1e75f8ec0 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc6673efb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc6501b1670 con 0x55a1e7777e30 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc6673efb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a1e77fb7b0 con 0x55a1e777ddb0 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc6673efb80 10 client.0 _ll_register_callbacks cb 0x55a1e763fcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e7782d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e7782d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43458/0 (socket says 172.21.15.12:43458) 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 -- 172.21.15.12:0/1796400108 learned_addr learned my addr 172.21.15.12:0/1796400108 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc656800700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7783810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655400700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e7777e30 0x55a1e77832d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 -- 172.21.15.12:0/1796400108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 msgr2=0x55a1e7783810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7783810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 -- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e7777e30 msgr2=0x55a1e77832d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e7777e30 0x55a1e77832d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655e00700 1 -- 172.21.15.12:0/1796400108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1e77fb9b0 con 0x55a1e75f8ec0 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.423+0000 7fc655400700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e7777e30 0x55a1e77832d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc655e00700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e7782d90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc644023d50 tx=0x7fc644005660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a089fbe7042eae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc633e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc633e00700 1 -- 172.21.15.12:0/1796400108 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc644037060 con 0x55a1e75f8ec0 2024-05-27T07:39:14.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc633e00700 1 -- 172.21.15.12:0/1796400108 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7fc644023670 con 0x55a1e75f8ec0 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc640022430 con 0x55a1e75f8ec0 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc633e00700 1 -- 172.21.15.12:0/1796400108 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc64403e050 con 0x55a1e75f8ec0 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc6673efb80 10 client.17439 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.424+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a1e77fcc30 con 0x55a1e75f8ec0 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc633e00700 1 -- 172.21.15.12:0/1796400108 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc640022430 con 0x55a1e75f8ec0 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc633e00700 1 -- 172.21.15.12:0/1796400108 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fc64400bd00 con 0x55a1e75f8ec0 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc6673efb80 10 client.17439 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc6673efb80 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc6673efb80 20 client.17439 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e77fd2e0 0x55a1e77ff700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc6673efb80 4 client.17439 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:39:14.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.425+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a1e76a15f0 con 0x55a1e77fd2e0 2024-05-27T07:39:14.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.426+0000 7fc655400700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e77fd2e0 0x55a1e77ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:14.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.428+0000 7fc655400700 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e77fd2e0 0x55a1e77ff700 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20cad6bd46856e91 server_cookie=bfff836c401f54ff in_seq=0 out_seq=0 2024-05-27T07:39:14.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.428+0000 7fc633e00700 10 client.17439 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.432+0000 7fc633e00700 1 -- 172.21.15.12:0/1796400108 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a1e76a15f0 con 0x55a1e77fd2e0 2024-05-27T07:39:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.432+0000 7fc633e00700 10 client.17439 handle_command_reply: tid=0 2024-05-27T07:39:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.432+0000 7fc6673efb80 1 client.17439 shutdown 2024-05-27T07:39:14.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.432+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 msgr2=0x55a1e7782d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.432+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e7782d90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc644023d50 tx=0x7fc644005660 comp rx=0 tx=0).stop 2024-05-27T07:39:14.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 shutdown_connections 2024-05-27T07:39:14.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1e777ddb0 0x55a1e7783810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a1e75f8ec0 0x55a1e7782d90 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a1e77fd2e0 0x55a1e77ff700 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 --2- 172.21.15.12:0/1796400108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a1e7777e30 0x55a1e77832d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc633e00700 0 client.17439 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 >> 172.21.15.12:0/1796400108 conn(0x55a1e7659040 msgr2=0x55a1e7781ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 shutdown_connections 2024-05-27T07:39:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 1 -- 172.21.15.12:0/1796400108 wait complete. 2024-05-27T07:39:14.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.433+0000 7fc6673efb80 20 client.17439 trim_cache size 0 max 16384 2024-05-27T07:39:14.432 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:39:14.432 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:39:14.432 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "60c2a9b4-e4f0-4f56-8b29-e8b0a65c0c5d", 2024-05-27T07:39:14.432 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:39:14.432 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.434+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc6380b2a70 msgr2=0x7fc6380b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.434+0000 7fc658800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc6380b2a70 0x7fc6380b4f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc640015740 tx=0x7fc6400099d0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.434+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 msgr2=0x7fc6501a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.434+0000 7fc658800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc6501a1880 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00b970 tx=0x7fc64c00bc80 comp rx=0 tx=0).stop 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 shutdown_connections 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc6380b2a70 0x7fc6380b4f80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc650106560 0x7fc65019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65010c5a0 0x7fc6501a1880 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 --2- 172.21.15.12:0/1185906417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc650106eb0 0x7fc65019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 >> 172.21.15.12:0/1185906417 conn(0x7fc650102040 msgr2=0x7fc6501039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 shutdown_connections 2024-05-27T07:39:14.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:14.435+0000 7fc658800700 1 -- 172.21.15.12:0/1185906417 wait complete. 2024-05-27T07:39:14.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:14 smithi162 ceph-mon[35676]: pgmap v576: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 248 op/s 2024-05-27T07:39:14.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:14 smithi062 ceph-mon[35304]: pgmap v576: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 248 op/s 2024-05-27T07:39:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:15 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:39:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:15 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:15 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:39:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:15 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:39:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:15 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:39:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:15 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:15.289 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:39:15.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:15 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:39:15.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:15 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:15.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:15 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:39:15.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:15 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:39:15.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:15 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:39:15.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:15 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:15.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:15 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:39:15.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:15 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:15.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:15 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:39:15.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:15 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:39:15.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:15 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:39:15.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:15 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:15.817 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:16.307 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:16 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:16.307 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:16 smithi012 ceph-mon[31473]: pgmap v577: 129 pgs: 129 active+clean; 363 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:39:16.307 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:16 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:16.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:16 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:16.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:16 smithi162 ceph-mon[35676]: pgmap v577: 129 pgs: 129 active+clean; 363 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:39:16.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:16 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:16.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:16 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:16.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:16 smithi062 ceph-mon[35304]: pgmap v577: 129 pgs: 129 active+clean; 363 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:39:16.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:16 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:18.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:18 smithi012 ceph-mon[31473]: pgmap v578: 129 pgs: 129 active+clean; 363 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:39:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.152+0000 7f2e0de00700 1 -- 172.21.15.12:0/376553270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 msgr2=0x7f2e08113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.152+0000 7f2e0de00700 1 --2- 172.21.15.12:0/376553270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 0x7f2e08113340 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009990 tx=0x7f2e00009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.152+0000 7f2e0de00700 1 -- 172.21.15.12:0/376553270 shutdown_connections 2024-05-27T07:39:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.152+0000 7f2e0de00700 1 --2- 172.21.15.12:0/376553270 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 0x7f2e08113340 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.152+0000 7f2e0de00700 1 --2- 172.21.15.12:0/376553270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0810c5a0 0x7f2e0810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.152+0000 7f2e0de00700 1 --2- 172.21.15.12:0/376553270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e08106560 0x7f2e08106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.152+0000 7f2e0de00700 1 -- 172.21.15.12:0/376553270 >> 172.21.15.12:0/376553270 conn(0x7f2e08102040 msgr2=0x7f2e08104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.153+0000 7f2e0de00700 1 -- 172.21.15.12:0/376553270 shutdown_connections 2024-05-27T07:39:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.153+0000 7f2e0de00700 1 -- 172.21.15.12:0/376553270 wait complete. 2024-05-27T07:39:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.154+0000 7f2e0de00700 1 Processor -- start 2024-05-27T07:39:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.155+0000 7f2e0de00700 1 -- start start 2024-05-27T07:39:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.155+0000 7f2e0de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e08106560 0x7f2e0819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.155+0000 7f2e0de00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 0x7f2e0819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.156+0000 7f2e0de00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0810c5a0 0x7f2e081a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.156+0000 7f2e0de00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081165d0 con 0x7f2e08106eb0 2024-05-27T07:39:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.156+0000 7f2e0de00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08116450 con 0x7f2e08106560 2024-05-27T07:39:18.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.156+0000 7f2e0de00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08116750 con 0x7f2e0810c5a0 2024-05-27T07:39:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.157+0000 7f2e06a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e08106560 0x7f2e0819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.157+0000 7f2e06a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e08106560 0x7f2e0819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:51544/0 (socket says 172.21.15.12:51544) 2024-05-27T07:39:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.157+0000 7f2e06a00700 1 -- 172.21.15.12:0/2413592806 learned_addr learned my addr 172.21.15.12:0/2413592806 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.157+0000 7f2dfe000700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 0x7f2e0819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.157+0000 7f2e07400700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0810c5a0 0x7f2e081a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.158+0000 7f2dfe000700 1 -- 172.21.15.12:0/2413592806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0810c5a0 msgr2=0x7f2e081a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.158+0000 7f2dfe000700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0810c5a0 0x7f2e081a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.158+0000 7f2dfe000700 1 -- 172.21.15.12:0/2413592806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e08106560 msgr2=0x7f2e0819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.158+0000 7f2dfe000700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e08106560 0x7f2e0819cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.158+0000 7f2dfe000700 1 -- 172.21.15.12:0/2413592806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e081a1df0 con 0x7f2e08106eb0 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.158+0000 7f2dfe000700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 0x7f2e0819d4c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2e0806c5d0 tx=0x7f2df000b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.160+0000 7f2e0ca00700 1 -- 172.21.15.12:0/2413592806 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0015020 con 0x7f2e08106eb0 2024-05-27T07:39:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.160+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e081a20e0 con 0x7f2e08106eb0 2024-05-27T07:39:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.161+0000 7f2e0ca00700 1 -- 172.21.15.12:0/2413592806 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2df0011cc0 con 0x7f2e08106eb0 2024-05-27T07:39:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.161+0000 7f2e0ca00700 1 -- 172.21.15.12:0/2413592806 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0010c50 con 0x7f2e08106eb0 2024-05-27T07:39:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.161+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e081a9a00 con 0x7f2e08106eb0 2024-05-27T07:39:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.168+0000 7f2e1c89eb80 1 Processor -- start 2024-05-27T07:39:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.168+0000 7f2e1c89eb80 1 -- start start 2024-05-27T07:39:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f6099f0 0x56118f69c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f69c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f69ce70 0x56118f6a9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56118f5cae40 con 0x56118f51de40 2024-05-27T07:39:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56118f5ba9e0 con 0x56118f6099f0 2024-05-27T07:39:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56118f5bea20 con 0x56118f69ce70 2024-05-27T07:39:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e07400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f6099f0 0x56118f69c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e07400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f6099f0 0x56118f69c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:51548/0 (socket says 172.21.15.12:51548) 2024-05-27T07:39:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.169+0000 7f2e07400700 1 -- 172.21.15.12:0/1527841740 learned_addr learned my addr 172.21.15.12:0/1527841740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.170+0000 7f2e0ca00700 1 -- 172.21.15.12:0/2413592806 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2df0018020 con 0x7f2e08106eb0 2024-05-27T07:39:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.170+0000 7f2e06a00700 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f69c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.170+0000 7f2e07400700 1 -- 172.21.15.12:0/1527841740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f69ce70 msgr2=0x56118f6a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.170+0000 7f2e07400700 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f69ce70 0x56118f6a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.170+0000 7f2e07400700 1 -- 172.21.15.12:0/1527841740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 msgr2=0x56118f69c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.170+0000 7f2e07400700 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f69c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.170+0000 7f2e07400700 1 -- 172.21.15.12:0/1527841740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56118f583420 con 0x56118f6099f0 2024-05-27T07:39:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.171+0000 7f2e07400700 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f6099f0 0x56118f69c330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000b5c0 tx=0x7f2e00005980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d6c260a18bc276f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.172+0000 7f2dfcc00700 1 -- 172.21.15.12:0/1527841740 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00027070 con 0x56118f6099f0 2024-05-27T07:39:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.172+0000 7f2dfcc00700 1 -- 172.21.15.12:0/1527841740 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e00004ce0 con 0x56118f6099f0 2024-05-27T07:39:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.172+0000 7f2dfcc00700 1 -- 172.21.15.12:0/1527841740 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0000b810 con 0x56118f6099f0 2024-05-27T07:39:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.172+0000 7f2e0ca00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2dec0b2af0 0x7f2dec0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e0ca00700 1 -- 172.21.15.12:0/2413592806 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f2df00e2020 con 0x7f2e08106eb0 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e06a00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2dec0b2af0 0x7f2dec0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/1527841740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f6099f0 msgr2=0x56118f69c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f6099f0 0x56118f69c330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000b5c0 tx=0x7f2e00005980 comp rx=0 tx=0).stop 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/1527841740 shutdown_connections 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f6099f0 0x56118f69c330 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f69ce70 0x56118f6a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/1527841740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f69c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/1527841740 >> 172.21.15.12:0/1527841740 conn(0x56118f5b7390 msgr2=0x56118f6984c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/1527841740 shutdown_connections 2024-05-27T07:39:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.173+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/1527841740 wait complete. 2024-05-27T07:39:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.174+0000 7f2e06a00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2dec0b2af0 0x7f2dec0b5000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56118f699790 tx=0x7f2df8009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.174+0000 7f2e1c89eb80 1 -- start start 2024-05-27T07:39:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.174+0000 7f2e1c89eb80 1 Processor -- start 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f6b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f69ce70 0x56118f6af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56118f4c0fa0 con 0x56118f51de40 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56118f55b760 con 0x56118f69ce70 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56118f6a9b20 con 0x56118f6099f0 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e1c89eb80 10 client.0 _ll_register_callbacks cb 0x56118f564c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2dfe000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2dfe000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:34374/0 (socket says 172.21.15.12:34374) 2024-05-27T07:39:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2dfe000700 1 -- 172.21.15.12:0/4088941244 learned_addr learned my addr 172.21.15.12:0/4088941244 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e06a00700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f6b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.175+0000 7f2e07400700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f69ce70 0x56118f6af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2dfe000700 1 -- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f69ce70 msgr2=0x56118f6af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2dfe000700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f69ce70 0x56118f6af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2dfe000700 1 -- 172.21.15.12:0/4088941244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 msgr2=0x56118f6b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2dfe000700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f6b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2dfe000700 1 -- 172.21.15.12:0/4088941244 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56118f50a8d0 con 0x56118f6099f0 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2e06a00700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f6b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2dfe000700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b57b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2df00a5020 tx=0x7f2df00a8970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba34b3e0cb4aee3b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.176+0000 7f2df5600700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.177+0000 7f2df5600700 1 -- 172.21.15.12:0/4088941244 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df00a96a0 con 0x56118f6099f0 2024-05-27T07:39:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.177+0000 7f2df5600700 1 -- 172.21.15.12:0/4088941244 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f2df00a9200 con 0x56118f6099f0 2024-05-27T07:39:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.177+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e00004ce0 con 0x56118f6099f0 2024-05-27T07:39:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.177+0000 7f2df5600700 1 -- 172.21.15.12:0/4088941244 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f2df00a8fa0 con 0x56118f6099f0 2024-05-27T07:39:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.177+0000 7f2e1c89eb80 10 client.26842 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.177+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56118f58b060 con 0x56118f6099f0 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.178+0000 7f2df5600700 1 -- 172.21.15.12:0/4088941244 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e00004ce0 con 0x56118f6099f0 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.178+0000 7f2df5600700 1 -- 172.21.15.12:0/4088941244 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f2df00b1a00 con 0x56118f6099f0 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.178+0000 7f2e1c89eb80 10 client.26842 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.178+0000 7f2e1c89eb80 10 client.26842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.178+0000 7f2e1c89eb80 20 client.26842 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.179+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f6bf640 0x56118f6c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.179+0000 7f2e1c89eb80 4 client.26842 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.179+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56118f5ccd90 con 0x56118f6bf640 2024-05-27T07:39:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.179+0000 7f2e06a00700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f6bf640 0x56118f6c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.181+0000 7f2e06a00700 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f6bf640 0x56118f6c1a90 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=637cbd4a96ed8554 server_cookie=a264a57ff1cac91e in_seq=0 out_seq=0 2024-05-27T07:39:18.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.181+0000 7f2df5600700 10 client.26842 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:18.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2df5600700 1 -- 172.21.15.12:0/4088941244 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56118f5ccd90 con 0x56118f6bf640 2024-05-27T07:39:18.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2df5600700 10 client.26842 handle_command_reply: tid=0 2024-05-27T07:39:18.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 client.26842 shutdown 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 msgr2=0x56118f6b57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b57b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2df00a5020 tx=0x7f2df00a8970 comp rx=0 tx=0).stop 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 shutdown_connections 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f6bf640 0x56118f6c1a90 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f69ce70 0x56118f6af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f51de40 0x56118f6b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/4088941244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b57b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 >> 172.21.15.12:0/4088941244 conn(0x56118f5b7390 msgr2=0x56118f69c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2df5600700 0 client.26842 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 shutdown_connections 2024-05-27T07:39:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/4088941244 wait complete. 2024-05-27T07:39:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.183+0000 7f2e1c89eb80 20 client.26842 trim_cache size 0 max 16384 2024-05-27T07:39:18.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.215+0000 7f2e1c89eb80 1 Processor -- start 2024-05-27T07:39:18.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.215+0000 7f2e1c89eb80 1 -- start start 2024-05-27T07:39:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.215+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f69c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f697f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f698490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081adab0 con 0x56118f69ce70 2024-05-27T07:39:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081acd10 con 0x56118f51de40 2024-05-27T07:39:18.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081ad3e0 con 0x56118f6099f0 2024-05-27T07:39:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f698490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f698490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43510/0 (socket says 172.21.15.12:43510) 2024-05-27T07:39:18.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e07400700 1 -- 172.21.15.12:0/3908297231 learned_addr learned my addr 172.21.15.12:0/3908297231 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2dfe000700 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f697f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.216+0000 7f2e06a00700 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f69c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.217+0000 7f2e07400700 1 -- 172.21.15.12:0/3908297231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 msgr2=0x56118f697f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.217+0000 7f2e07400700 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f697f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.217+0000 7f2e07400700 1 -- 172.21.15.12:0/3908297231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 msgr2=0x56118f69c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.217+0000 7f2e07400700 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f69c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.217+0000 7f2e07400700 1 -- 172.21.15.12:0/3908297231 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56118f6ae990 con 0x56118f69ce70 2024-05-27T07:39:18.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.217+0000 7f2e07400700 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f698490 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009350 tx=0x7f2e0000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2105da640a72f214 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.217+0000 7f2df7e00700 1 -- 172.21.15.12:0/3908297231 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00027040 con 0x56118f69ce70 2024-05-27T07:39:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.218+0000 7f2df7e00700 1 -- 172.21.15.12:0/3908297231 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e00032070 con 0x56118f69ce70 2024-05-27T07:39:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.218+0000 7f2df7e00700 1 -- 172.21.15.12:0/3908297231 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00022dd0 con 0x56118f69ce70 2024-05-27T07:39:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/3908297231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 msgr2=0x56118f698490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f698490 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009350 tx=0x7f2e0000b960 comp rx=0 tx=0).stop 2024-05-27T07:39:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/3908297231 shutdown_connections 2024-05-27T07:39:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f69c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f698490 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/3908297231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f697f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/3908297231 >> 172.21.15.12:0/3908297231 conn(0x56118f5b7390 msgr2=0x56118f69c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/3908297231 shutdown_connections 2024-05-27T07:39:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.219+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/3908297231 wait complete. 2024-05-27T07:39:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 -- start start 2024-05-27T07:39:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 Processor -- start 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f6bb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f6b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56118f5ccd90 con 0x56118f69ce70 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56118f697150 con 0x56118f51de40 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.220+0000 7f2e1c89eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56118f694950 con 0x56118f6099f0 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e1c89eb80 10 client.0 _ll_register_callbacks cb 0x56118f564c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f6b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f6b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43526/0 (socket says 172.21.15.12:43526) 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 -- 172.21.15.12:0/2570312963 learned_addr learned my addr 172.21.15.12:0/2570312963 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e06a00700 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f6bb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2dfe000700 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 -- 172.21.15.12:0/2570312963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 msgr2=0x56118f6b6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 -- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 msgr2=0x56118f6bb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f6bb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.221+0000 7f2e07400700 1 -- 172.21.15.12:0/2570312963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56118f6b7810 con 0x56118f69ce70 2024-05-27T07:39:18.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2e07400700 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f6b72d0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f2e00000c00 tx=0x7f2e0000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa25ad5e969f4b8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:18.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2df5600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:18.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2df5600700 1 -- 172.21.15.12:0/2570312963 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0003c020 con 0x56118f69ce70 2024-05-27T07:39:18.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e00032070 con 0x56118f69ce70 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2df5600700 1 -- 172.21.15.12:0/2570312963 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f2e00022680 con 0x56118f69ce70 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2df5600700 1 -- 172.21.15.12:0/2570312963 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f2e00047050 con 0x56118f69ce70 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2e1c89eb80 10 client.17457 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.222+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56118f729dd0 con 0x56118f69ce70 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2df5600700 1 -- 172.21.15.12:0/2570312963 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e00032070 con 0x56118f69ce70 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2df5600700 1 -- 172.21.15.12:0/2570312963 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f2e0000f7d0 con 0x56118f69ce70 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2e1c89eb80 10 client.17457 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2e1c89eb80 10 client.17457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2e1c89eb80 20 client.17457 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f729fa0 0x56118f72c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2e1c89eb80 4 client.17457 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.223+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56118f5c6570 con 0x56118f729fa0 2024-05-27T07:39:18.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.224+0000 7f2e06a00700 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f729fa0 0x56118f72c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:18.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2df5600700 10 client.17457 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:18.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e06a00700 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f729fa0 0x56118f72c3c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=643e1372bde6e5aa server_cookie=6239a9c7668d9585 in_seq=0 out_seq=0 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2df5600700 1 -- 172.21.15.12:0/2570312963 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56118f5c6570 con 0x56118f729fa0 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2df5600700 10 client.17457 handle_command_reply: tid=0 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 client.17457 shutdown 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 msgr2=0x56118f6b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f6b72d0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f2e00000c00 tx=0x7f2e0000b8e0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 shutdown_connections 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x56118f729fa0 0x56118f72c3c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56118f51de40 0x56118f6bb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56118f69ce70 0x56118f6b72d0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 --2- 172.21.15.12:0/2570312963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56118f6099f0 0x56118f6b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.225+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 >> 172.21.15.12:0/2570312963 conn(0x56118f5b7390 msgr2=0x56118f6adba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.226+0000 7f2df5600700 0 client.17457 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:18.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.226+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 shutdown_connections 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.226+0000 7f2e1c89eb80 1 -- 172.21.15.12:0/2570312963 wait complete. 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.226+0000 7f2e1c89eb80 20 client.17457 trim_cache size 0 max 16384 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout: "60c2a9b4-e4f0-4f56-8b29-e8b0a65c0c5d": { 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "60c2a9b4-e4f0-4f56-8b29-e8b0a65c0c5d", 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:39:18.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2dec0b2af0 msgr2=0x7f2dec0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2dec0b2af0 0x7f2dec0b5000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56118f699790 tx=0x7f2df8009040 comp rx=0 tx=0).stop 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 msgr2=0x7f2e0819d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 0x7f2e0819d4c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2e0806c5d0 tx=0x7f2df000b7e0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 shutdown_connections 2024-05-27T07:39:18.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f2dec0b2af0 0x7f2dec0b5000 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e08106560 0x7f2e0819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0810c5a0 0x7f2e081a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 --2- 172.21.15.12:0/2413592806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e08106eb0 0x7f2e0819d4c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:18.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 >> 172.21.15.12:0/2413592806 conn(0x7f2e08102040 msgr2=0x7f2e081037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:18.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 shutdown_connections 2024-05-27T07:39:18.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:18.227+0000 7f2e0de00700 1 -- 172.21.15.12:0/2413592806 wait complete. 2024-05-27T07:39:18.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:18 smithi162 ceph-mon[35676]: pgmap v578: 129 pgs: 129 active+clean; 363 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:39:18.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:18 smithi062 ceph-mon[35304]: pgmap v578: 129 pgs: 129 active+clean; 363 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 228 op/s 2024-05-27T07:39:19.253 INFO:tasks.cephfs.filesystem:scrub status for tag:60c2a9b4-e4f0-4f56-8b29-e8b0a65c0c5d - {'path': '/', 'tag': '60c2a9b4-e4f0-4f56-8b29-e8b0a65c0c5d', 'options': 'recursive,force'} 2024-05-27T07:39:20.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:20 smithi012 ceph-mon[31473]: pgmap v579: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 314 op/s 2024-05-27T07:39:20.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:20 smithi162 ceph-mon[35676]: pgmap v579: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 314 op/s 2024-05-27T07:39:20.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:20 smithi062 ceph-mon[35304]: pgmap v579: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 314 op/s 2024-05-27T07:39:22.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:22 smithi012 ceph-mon[31473]: pgmap v580: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 196 op/s 2024-05-27T07:39:22.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:22 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:22.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:22 smithi162 ceph-mon[35676]: pgmap v580: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 196 op/s 2024-05-27T07:39:22.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:22 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:22.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:22 smithi062 ceph-mon[35304]: pgmap v580: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 196 op/s 2024-05-27T07:39:22.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:22 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:39:23.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:23 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.483 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.484 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.484 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.484 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:23 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:23.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:23 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:24.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:24 smithi162 ceph-mon[35676]: pgmap v581: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 196 op/s 2024-05-27T07:39:24.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:24 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:24.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:24 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:24.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:24 smithi162 ceph-mon[35676]: pgmap v582: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 221 op/s 2024-05-27T07:39:24.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:24 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:24.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:24 smithi062 ceph-mon[35304]: pgmap v581: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 196 op/s 2024-05-27T07:39:24.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:24 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:24.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:24 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:24.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:24 smithi062 ceph-mon[35304]: pgmap v582: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 221 op/s 2024-05-27T07:39:24.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:24 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:24.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:24 smithi012 ceph-mon[31473]: pgmap v581: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 196 op/s 2024-05-27T07:39:24.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:24 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:24.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:24 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:24.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:24 smithi012 ceph-mon[31473]: pgmap v582: 129 pgs: 129 active+clean; 393 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 221 op/s 2024-05-27T07:39:24.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:24 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:39:25.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:25 smithi162 ceph-mon[35676]: pgmap v583: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 246 op/s 2024-05-27T07:39:25.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:25 smithi062 ceph-mon[35304]: pgmap v583: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 246 op/s 2024-05-27T07:39:26.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:25 smithi012 ceph-mon[31473]: pgmap v583: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 246 op/s 2024-05-27T07:39:26.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:39:26.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:39:26.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:39:26.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:39:26.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:39:26.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:39:27.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:39:27.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:39:27.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:39:27.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:27 smithi162 ceph-mon[35676]: pgmap v584: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 246 op/s 2024-05-27T07:39:27.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:27 smithi062 ceph-mon[35304]: pgmap v584: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 246 op/s 2024-05-27T07:39:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:27 smithi012 ceph-mon[31473]: pgmap v584: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 246 op/s 2024-05-27T07:39:29.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:29 smithi162 ceph-mon[35676]: pgmap v585: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:29.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:29 smithi062 ceph-mon[35304]: pgmap v585: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:30.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:29 smithi012 ceph-mon[31473]: pgmap v585: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:31.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:31 smithi162 ceph-mon[35676]: pgmap v586: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:31.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:31 smithi062 ceph-mon[35304]: pgmap v586: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:32.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:31 smithi012 ceph-mon[31473]: pgmap v586: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:33.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:33 smithi012 ceph-mon[31473]: pgmap v587: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:33.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:33 smithi162 ceph-mon[35676]: pgmap v587: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:33.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:33 smithi062 ceph-mon[35304]: pgmap v587: 129 pgs: 129 active+clean; 403 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 237 op/s 2024-05-27T07:39:35.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:35 smithi162 ceph-mon[35676]: pgmap v588: 129 pgs: 129 active+clean; 383 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-05-27T07:39:35.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:35 smithi062 ceph-mon[35304]: pgmap v588: 129 pgs: 129 active+clean; 383 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-05-27T07:39:36.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:35 smithi012 ceph-mon[31473]: pgmap v588: 129 pgs: 129 active+clean; 383 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-05-27T07:39:37.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:37 smithi162 ceph-mon[35676]: pgmap v589: 129 pgs: 129 active+clean; 383 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:39:37.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:37 smithi062 ceph-mon[35304]: pgmap v589: 129 pgs: 129 active+clean; 383 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:39:38.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:37 smithi012 ceph-mon[31473]: pgmap v589: 129 pgs: 129 active+clean; 383 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:39:39.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:39 smithi162 ceph-mon[35676]: pgmap v590: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 278 op/s 2024-05-27T07:39:39.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:39 smithi062 ceph-mon[35304]: pgmap v590: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 278 op/s 2024-05-27T07:39:40.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:39 smithi012 ceph-mon[31473]: pgmap v590: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 278 op/s 2024-05-27T07:39:41.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:41 smithi162 ceph-mon[35676]: pgmap v591: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:39:41.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:41 smithi062 ceph-mon[35304]: pgmap v591: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:39:42.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:41 smithi012 ceph-mon[31473]: pgmap v591: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:39:43.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:43 smithi162 ceph-mon[35676]: pgmap v592: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:39:43.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:43 smithi062 ceph-mon[35304]: pgmap v592: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:39:44.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:43 smithi012 ceph-mon[31473]: pgmap v592: 129 pgs: 129 active+clean; 403 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-05-27T07:39:45.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:45 smithi162 ceph-mon[35676]: pgmap v593: 129 pgs: 129 active+clean; 390 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 277 op/s 2024-05-27T07:39:45.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:45 smithi062 ceph-mon[35304]: pgmap v593: 129 pgs: 129 active+clean; 390 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 277 op/s 2024-05-27T07:39:46.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:45 smithi012 ceph-mon[31473]: pgmap v593: 129 pgs: 129 active+clean; 390 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 277 op/s 2024-05-27T07:39:47.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:47 smithi162 ceph-mon[35676]: pgmap v594: 129 pgs: 129 active+clean; 390 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 184 op/s 2024-05-27T07:39:47.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:47 smithi062 ceph-mon[35304]: pgmap v594: 129 pgs: 129 active+clean; 390 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 184 op/s 2024-05-27T07:39:48.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:47 smithi012 ceph-mon[31473]: pgmap v594: 129 pgs: 129 active+clean; 390 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 184 op/s 2024-05-27T07:39:49.254 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:39:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:49.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:49 smithi012 ceph-mon[31473]: pgmap v595: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-05-27T07:39:49.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:49 smithi162 ceph-mon[35676]: pgmap v595: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-05-27T07:39:49.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:49 smithi062 ceph-mon[35304]: pgmap v595: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-05-27T07:39:51.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:51 smithi162 ceph-mon[35676]: pgmap v596: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1011 KiB/s wr, 172 op/s 2024-05-27T07:39:51.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:51 smithi062 ceph-mon[35304]: pgmap v596: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1011 KiB/s wr, 172 op/s 2024-05-27T07:39:52.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:51 smithi012 ceph-mon[31473]: pgmap v596: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1011 KiB/s wr, 172 op/s 2024-05-27T07:39:52.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 -- 172.21.15.12:0/4126072309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105280 msgr2=0x7f3bc4105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 --2- 172.21.15.12:0/4126072309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105280 0x7f3bc4105690 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009990 tx=0x7f3bb4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 -- 172.21.15.12:0/4126072309 shutdown_connections 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 --2- 172.21.15.12:0/4126072309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bc410e6f0 0x7f3bc4110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 --2- 172.21.15.12:0/4126072309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105280 0x7f3bc4105690 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 --2- 172.21.15.12:0/4126072309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bc4105bd0 0x7f3bc410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 -- 172.21.15.12:0/4126072309 >> 172.21.15.12:0/4126072309 conn(0x7f3bc4078ab0 msgr2=0x7f3bc4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 -- 172.21.15.12:0/4126072309 shutdown_connections 2024-05-27T07:39:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.604+0000 7f3bc3200700 1 -- 172.21.15.12:0/4126072309 wait complete. 2024-05-27T07:39:52.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.605+0000 7f3bc3200700 1 Processor -- start 2024-05-27T07:39:52.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.606+0000 7f3bc3200700 1 -- start start 2024-05-27T07:39:52.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.606+0000 7f3bc3200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105bd0 0x7f3bc419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.606+0000 7f3bc3200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bc410e6f0 0x7f3bc419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bc41a1780 0x7f3bc41a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc3200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4114160 con 0x7f3bc4105bd0 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4113fe0 con 0x7f3bc41a1780 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc3200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41142e0 con 0x7f3bc410e6f0 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105bd0 0x7f3bc419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105bd0 0x7f3bc419d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56306/0 (socket says 172.21.15.12:56306) 2024-05-27T07:39:52.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc1e00700 1 -- 172.21.15.12:0/231648781 learned_addr learned my addr 172.21.15.12:0/231648781 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:52.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.607+0000 7f3bc1400700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bc410e6f0 0x7f3bc419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.608+0000 7f3bc2800700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bc41a1780 0x7f3bc41a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.608+0000 7f3bc1e00700 1 -- 172.21.15.12:0/231648781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bc410e6f0 msgr2=0x7f3bc419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.608+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bc410e6f0 0x7f3bc419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.608+0000 7f3bc1e00700 1 -- 172.21.15.12:0/231648781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bc41a1780 msgr2=0x7f3bc41a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.608+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bc41a1780 0x7f3bc41a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.608+0000 7f3bc1e00700 1 -- 172.21.15.12:0/231648781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc41a22d0 con 0x7f3bc4105bd0 2024-05-27T07:39:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.608+0000 7f3bc2800700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bc41a1780 0x7f3bc41a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.609+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105bd0 0x7f3bc419d010 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4000c00 tx=0x7f3bb40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.609+0000 7f3bbaa00700 1 -- 172.21.15.12:0/231648781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4026070 con 0x7f3bc4105bd0 2024-05-27T07:39:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.609+0000 7f3bbaa00700 1 -- 172.21.15.12:0/231648781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb4021410 con 0x7f3bc4105bd0 2024-05-27T07:39:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.609+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc41a2560 con 0x7f3bc4105bd0 2024-05-27T07:39:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.609+0000 7f3bbaa00700 1 -- 172.21.15.12:0/231648781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4017650 con 0x7f3bc4105bd0 2024-05-27T07:39:52.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.609+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc41a9d10 con 0x7f3bc4105bd0 2024-05-27T07:39:52.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bbaa00700 1 -- 172.21.15.12:0/231648781 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3bb4029070 con 0x7f3bc4105bd0 2024-05-27T07:39:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 Processor -- start 2024-05-27T07:39:52.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 -- start start 2024-05-27T07:39:52.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fc0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561102eefe40 con 0x561102fc1820 2024-05-27T07:39:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561102edf9e0 con 0x561102e42e40 2024-05-27T07:39:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.615+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561102ee3a20 con 0x561102fc1410 2024-05-27T07:39:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.616+0000 7f3bc1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.616+0000 7f3bc1400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56308/0 (socket says 172.21.15.12:56308) 2024-05-27T07:39:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.616+0000 7f3bc1400700 1 -- 172.21.15.12:0/1182544547 learned_addr learned my addr 172.21.15.12:0/1182544547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:52.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.616+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.616+0000 7f3bc2800700 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fc0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.617+0000 7f3bc1400700 1 -- 172.21.15.12:0/1182544547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 msgr2=0x561102fcbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.617+0000 7f3bc1400700 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.617+0000 7f3bc1400700 1 -- 172.21.15.12:0/1182544547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 msgr2=0x561102fc0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.617+0000 7f3bc1400700 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fc0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.617+0000 7f3bc1400700 1 -- 172.21.15.12:0/1182544547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561102ea8420 con 0x561102fc1820 2024-05-27T07:39:52.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.617+0000 7f3bc1400700 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fce020 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009d20 tx=0x7f3bb0011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e21ece09205fc9a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.617+0000 7f3bbaa00700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3ba80b2a70 0x7f3ba80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.618+0000 7f3baf400700 1 -- 172.21.15.12:0/1182544547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb001e070 con 0x561102fc1820 2024-05-27T07:39:52.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.618+0000 7f3bbaa00700 1 -- 172.21.15.12:0/231648781 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f3bb40e36f0 con 0x7f3bc4105bd0 2024-05-27T07:39:52.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.618+0000 7f3baf400700 1 -- 172.21.15.12:0/1182544547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb0019ac0 con 0x561102fc1820 2024-05-27T07:39:52.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.618+0000 7f3baf400700 1 -- 172.21.15.12:0/1182544547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb000e980 con 0x561102fc1820 2024-05-27T07:39:52.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.618+0000 7f3bc2800700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3ba80b2a70 0x7f3ba80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/1182544547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 msgr2=0x561102fce020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bc2800700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3ba80b2a70 0x7f3ba80b4f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009fd0 tx=0x7f3bbc009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fce020 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009d20 tx=0x7f3bb0011720 comp rx=0 tx=0).stop 2024-05-27T07:39:52.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/1182544547 shutdown_connections 2024-05-27T07:39:52.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fc0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fce020 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/1182544547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/1182544547 >> 172.21.15.12:0/1182544547 conn(0x561102ea2fc0 msgr2=0x561102fbcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/1182544547 shutdown_connections 2024-05-27T07:39:52.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.619+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/1182544547 wait complete. 2024-05-27T07:39:52.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.620+0000 7f3bd7bbcb80 1 -- start start 2024-05-27T07:39:52.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.620+0000 7f3bd7bbcb80 1 Processor -- start 2024-05-27T07:39:52.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fcb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1410 0x561102fcbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1820 0x561102fcbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561102de5fa0 con 0x561102e42e40 2024-05-27T07:39:52.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561102e80760 con 0x561102fc1410 2024-05-27T07:39:52.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561102fce990 con 0x561102fc1820 2024-05-27T07:39:52.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bd7bbcb80 10 client.0 _ll_register_callbacks cb 0x561102e89c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:52.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fcb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fcb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56324/0 (socket says 172.21.15.12:56324) 2024-05-27T07:39:52.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bc1e00700 1 -- 172.21.15.12:0/4058446435 learned_addr learned my addr 172.21.15.12:0/4058446435 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:52.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.621+0000 7f3bc2800700 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1820 0x561102fcbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.622+0000 7f3bc1400700 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1410 0x561102fcbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.622+0000 7f3bc1e00700 1 -- 172.21.15.12:0/4058446435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1820 msgr2=0x561102fcbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.622+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1820 0x561102fcbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.622+0000 7f3bc1e00700 1 -- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1410 msgr2=0x561102fcbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.622+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1410 0x561102fcbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.622+0000 7f3bc1e00700 1 -- 172.21.15.12:0/4058446435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561102e2f8d0 con 0x561102e42e40 2024-05-27T07:39:52.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fcb570 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40a7860 tx=0x7f3bb4021be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f381f551e48cb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3ba3e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:52.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3ba3e00700 1 -- 172.21.15.12:0/4058446435 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40a64b0 con 0x561102e42e40 2024-05-27T07:39:52.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3ba3e00700 1 -- 172.21.15.12:0/4058446435 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f3bb40aa850 con 0x561102e42e40 2024-05-27T07:39:52.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb0019ac0 con 0x561102e42e40 2024-05-27T07:39:52.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3ba3e00700 1 -- 172.21.15.12:0/4058446435 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f3bb40a9d60 con 0x561102e42e40 2024-05-27T07:39:52.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3bd7bbcb80 10 client.17469 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:52.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.623+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x561102eb0060 con 0x561102e42e40 2024-05-27T07:39:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3ba3e00700 1 -- 172.21.15.12:0/4058446435 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb40e3070 con 0x561102e42e40 2024-05-27T07:39:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3ba3e00700 1 -- 172.21.15.12:0/4058446435 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f3bb40b2f30 con 0x561102e42e40 2024-05-27T07:39:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3bd7bbcb80 10 client.17469 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3bd7bbcb80 10 client.17469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3bd7bbcb80 20 client.17469 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561102fdba90 0x561102fddef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3bd7bbcb80 4 client.17469 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.624+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561102ef1d90 con 0x561102fdba90 2024-05-27T07:39:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.625+0000 7f3bc1400700 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561102fdba90 0x561102fddef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.627+0000 7f3bc1400700 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561102fdba90 0x561102fddef0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac1b20102142325a server_cookie=ba3700347903b8e0 in_seq=0 out_seq=0 2024-05-27T07:39:52.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.627+0000 7f3ba3e00700 10 client.17469 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:52.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3ba3e00700 1 -- 172.21.15.12:0/4058446435 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561102ef1d90 con 0x561102fdba90 2024-05-27T07:39:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3ba3e00700 10 client.17469 handle_command_reply: tid=0 2024-05-27T07:39:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 client.17469 shutdown 2024-05-27T07:39:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 msgr2=0x561102fcb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fcb570 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40a7860 tx=0x7f3bb4021be0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 shutdown_connections 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1820 0x561102fcbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fcb570 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561102fdba90 0x561102fddef0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/4058446435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1410 0x561102fcbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3ba3e00700 0 client.17469 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 >> 172.21.15.12:0/4058446435 conn(0x561102ea2fc0 msgr2=0x561102fbbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.629+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 shutdown_connections 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.630+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/4058446435 wait complete. 2024-05-27T07:39:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.630+0000 7f3bd7bbcb80 20 client.17469 trim_cache size 0 max 16384 2024-05-27T07:39:52.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.677+0000 7f3bd7bbcb80 1 Processor -- start 2024-05-27T07:39:52.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.678+0000 7f3bd7bbcb80 1 -- start start 2024-05-27T07:39:52.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.678+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fc6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.678+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fbce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.678+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1820 0x561102fcd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561102fb5f70 con 0x561102e42e40 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41b20d0 con 0x561102fc1820 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41b1330 con 0x561102fc1410 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fc6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fc6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56336/0 (socket says 172.21.15.12:56336) 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1e00700 1 -- 172.21.15.12:0/3475161843 learned_addr learned my addr 172.21.15.12:0/3475161843 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1400700 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fbce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc2800700 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1820 0x561102fcd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1e00700 1 -- 172.21.15.12:0/3475161843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 msgr2=0x561102fbce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fbce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1e00700 1 -- 172.21.15.12:0/3475161843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1820 msgr2=0x561102fcd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.679+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1820 0x561102fcd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.680+0000 7f3bc1e00700 1 -- 172.21.15.12:0/3475161843 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561102fbe350 con 0x561102e42e40 2024-05-27T07:39:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.680+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fc6d90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40aa580 tx=0x7f3bb4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14239ead0c7ebb51 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.680+0000 7f3baea00700 1 -- 172.21.15.12:0/3475161843 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40a9f40 con 0x561102e42e40 2024-05-27T07:39:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.680+0000 7f3baea00700 1 -- 172.21.15.12:0/3475161843 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb40e3050 con 0x561102e42e40 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.680+0000 7f3baea00700 1 -- 172.21.15.12:0/3475161843 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40aa3c0 con 0x561102e42e40 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.681+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3475161843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 msgr2=0x561102fc6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.681+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fc6d90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40aa580 tx=0x7f3bb4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.681+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3475161843 shutdown_connections 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.681+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102fc1820 0x561102fcd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.681+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102e42e40 0x561102fc6d90 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.681+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3475161843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fbce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.681+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3475161843 >> 172.21.15.12:0/3475161843 conn(0x561102ea2fc0 msgr2=0x561102fcbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.682+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3475161843 shutdown_connections 2024-05-27T07:39:52.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.682+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3475161843 wait complete. 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.682+0000 7f3bd7bbcb80 1 -- start start 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 1 Processor -- start 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fcf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fd03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561102fcde70 con 0x561102fc1820 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41b1a00 con 0x561102e42e40 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561102fc2460 con 0x561102fc1410 2024-05-27T07:39:52.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.683+0000 7f3bd7bbcb80 10 client.0 _ll_register_callbacks cb 0x561102e89c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:52.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fd03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fd03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56350/0 (socket says 172.21.15.12:56350) 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 -- 172.21.15.12:0/3706171615 learned_addr learned my addr 172.21.15.12:0/3706171615 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fcf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc1400700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 -- 172.21.15.12:0/3706171615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 msgr2=0x561102fcfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 -- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 msgr2=0x561102fcf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fcf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc2800700 1 -- 172.21.15.12:0/3706171615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611030455c0 con 0x561102fc1820 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.684+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fcf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.685+0000 7f3bc2800700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fd03d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc016e30 tx=0x7f3bbc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88dfdcc21ee1f2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:52.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.685+0000 7f3ba3e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.685+0000 7f3ba3e00700 1 -- 172.21.15.12:0/3706171615 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc028070 con 0x561102fc1820 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.685+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb40e3050 con 0x561102fc1820 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.685+0000 7f3ba3e00700 1 -- 172.21.15.12:0/3706171615 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f3bbc004bb0 con 0x561102fc1820 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.686+0000 7f3ba3e00700 1 -- 172.21.15.12:0/3706171615 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f3bbc004df0 con 0x561102fc1820 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.686+0000 7f3bd7bbcb80 10 client.17475 fetch_fsmap learned FSMap version 119 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.686+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5611030467d0 con 0x561102fc1820 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.686+0000 7f3ba3e00700 1 -- 172.21.15.12:0/3706171615 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bb40e3050 con 0x561102fc1820 2024-05-27T07:39:52.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.686+0000 7f3ba3e00700 1 -- 172.21.15.12:0/3706171615 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f3bbc0243f0 con 0x561102fc1820 2024-05-27T07:39:52.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.687+0000 7f3bd7bbcb80 10 client.17475 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:39:52.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.687+0000 7f3bd7bbcb80 10 client.17475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:52.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.687+0000 7f3bd7bbcb80 20 client.17475 populate_metadata read hostname 'smithi012' 2024-05-27T07:39:52.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.687+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561103046ee0 0x561103049300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:52.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.687+0000 7f3bd7bbcb80 4 client.17475 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:39:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.687+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561102eeb570 con 0x561103046ee0 2024-05-27T07:39:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.688+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561103046ee0 0x561103049300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.690+0000 7f3ba3e00700 10 client.17475 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:52.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.690+0000 7f3bc1e00700 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561103046ee0 0x561103049300 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0c3031cc2fd484 server_cookie=71dd1c248f528d17 in_seq=0 out_seq=0 2024-05-27T07:39:52.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3ba3e00700 1 -- 172.21.15.12:0/3706171615 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561102eeb570 con 0x561103046ee0 2024-05-27T07:39:52.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3ba3e00700 10 client.17475 handle_command_reply: tid=0 2024-05-27T07:39:52.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 client.17475 shutdown 2024-05-27T07:39:52.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 msgr2=0x561102fd03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fd03d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc016e30 tx=0x7f3bbc00b040 comp rx=0 tx=0).stop 2024-05-27T07:39:52.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 shutdown_connections 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x561103046ee0 0x561103049300 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561102e42e40 0x561102fcf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561102fc1820 0x561102fd03d0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 --2- 172.21.15.12:0/3706171615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561102fc1410 0x561102fcfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3ba3e00700 0 client.17475 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.691+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 >> 172.21.15.12:0/3706171615 conn(0x561102ea2fc0 msgr2=0x561102fcb350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.692+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 shutdown_connections 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.692+0000 7f3bd7bbcb80 1 -- 172.21.15.12:0/3706171615 wait complete. 2024-05-27T07:39:52.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.692+0000 7f3bd7bbcb80 20 client.17475 trim_cache size 0 max 16384 2024-05-27T07:39:52.691 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:39:52.691 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:39:52.691 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:39:52.691 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3ba80b2a70 msgr2=0x7f3ba80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3ba80b2a70 0x7f3ba80b4f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009fd0 tx=0x7f3bbc009040 comp rx=0 tx=0).stop 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105bd0 msgr2=0x7f3bc419d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105bd0 0x7f3bc419d010 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4000c00 tx=0x7f3bb40057c0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 shutdown_connections 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f3ba80b2a70 0x7f3ba80b4f80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bc41a1780 0x7f3bc41a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4105bd0 0x7f3bc419d010 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 --2- 172.21.15.12:0/231648781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bc410e6f0 0x7f3bc419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:52.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.693+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 >> 172.21.15.12:0/231648781 conn(0x7f3bc4078ab0 msgr2=0x7f3bc4103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:52.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.694+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 shutdown_connections 2024-05-27T07:39:52.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:52.694+0000 7f3bc3200700 1 -- 172.21.15.12:0/231648781 wait complete. 2024-05-27T07:39:53.694 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:39:53.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:39:53.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:53 smithi162 ceph-mon[35676]: pgmap v597: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1011 KiB/s wr, 172 op/s 2024-05-27T07:39:53.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:53 smithi062 ceph-mon[35304]: pgmap v597: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1011 KiB/s wr, 172 op/s 2024-05-27T07:39:54.051 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:53 smithi012 ceph-mon[31473]: pgmap v597: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1011 KiB/s wr, 172 op/s 2024-05-27T07:39:54.184 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:55.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:55 smithi012 ceph-mon[31473]: pgmap v598: 129 pgs: 129 active+clean; 389 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 267 op/s 2024-05-27T07:39:55.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:55 smithi162 ceph-mon[35676]: pgmap v598: 129 pgs: 129 active+clean; 389 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 267 op/s 2024-05-27T07:39:55.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:55 smithi062 ceph-mon[35304]: pgmap v598: 129 pgs: 129 active+clean; 389 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 267 op/s 2024-05-27T07:39:56.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 -- 172.21.15.12:0/3961267336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 msgr2=0x7fb960075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:56.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 --2- 172.21.15.12:0/3961267336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb960075f00 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb954009990 tx=0x7fb954009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 -- 172.21.15.12:0/3961267336 shutdown_connections 2024-05-27T07:39:56.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 --2- 172.21.15.12:0/3961267336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb960110c10 0x7fb960113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 --2- 172.21.15.12:0/3961267336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb960075f00 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 --2- 172.21.15.12:0/3961267336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb960076440 0x7fb9601106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 -- 172.21.15.12:0/3961267336 >> 172.21.15.12:0/3961267336 conn(0x7fb96007af60 msgr2=0x7fb96007b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:56.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 -- 172.21.15.12:0/3961267336 shutdown_connections 2024-05-27T07:39:56.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.855+0000 7fb967600700 1 -- 172.21.15.12:0/3961267336 wait complete. 2024-05-27T07:39:56.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.856+0000 7fb967600700 1 Processor -- start 2024-05-27T07:39:56.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.856+0000 7fb967600700 1 -- start start 2024-05-27T07:39:56.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb967600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb96019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:56.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb967600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb960076440 0x7fb96019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:56.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb967600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb960110c10 0x7fb9601a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:56.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb967600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb960116550 con 0x7fb960075af0 2024-05-27T07:39:56.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb967600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601163d0 con 0x7fb960076440 2024-05-27T07:39:56.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb967600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601166d0 con 0x7fb960110c10 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb964c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb96019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb964c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb96019cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56366/0 (socket says 172.21.15.12:56366) 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb964c00700 1 -- 172.21.15.12:0/1447615234 learned_addr learned my addr 172.21.15.12:0/1447615234 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb965600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb960110c10 0x7fb9601a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.857+0000 7fb95fe00700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb960076440 0x7fb96019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.858+0000 7fb964c00700 1 -- 172.21.15.12:0/1447615234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb960110c10 msgr2=0x7fb9601a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.858+0000 7fb964c00700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb960110c10 0x7fb9601a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.858+0000 7fb964c00700 1 -- 172.21.15.12:0/1447615234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb960076440 msgr2=0x7fb96019d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.858+0000 7fb964c00700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb960076440 0x7fb96019d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.858+0000 7fb964c00700 1 -- 172.21.15.12:0/1447615234 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9601a1eb0 con 0x7fb960075af0 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.858+0000 7fb964c00700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb96019cfe0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb9540057c0 tx=0x7fb9540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.859+0000 7fb95d600700 1 -- 172.21.15.12:0/1447615234 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954027070 con 0x7fb960075af0 2024-05-27T07:39:56.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.859+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9601a2140 con 0x7fb960075af0 2024-05-27T07:39:56.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.859+0000 7fb95d600700 1 -- 172.21.15.12:0/1447615234 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb95400fb00 con 0x7fb960075af0 2024-05-27T07:39:56.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.859+0000 7fb95d600700 1 -- 172.21.15.12:0/1447615234 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954022a80 con 0x7fb960075af0 2024-05-27T07:39:56.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.859+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9601a9ac0 con 0x7fb960075af0 2024-05-27T07:39:56.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.860+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9600512b0 con 0x7fb960075af0 2024-05-27T07:39:56.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.863+0000 7fb95d600700 1 -- 172.21.15.12:0/1447615234 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb95402a070 con 0x7fb960075af0 2024-05-27T07:39:56.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.865+0000 7fb95d600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb9480b29f0 0x7fb9480b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:56.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.866+0000 7fb95d600700 1 -- 172.21.15.12:0/1447615234 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7fb9540e3da0 con 0x7fb960075af0 2024-05-27T07:39:56.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.866+0000 7fb95fe00700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb9480b29f0 0x7fb9480b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:56.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.867+0000 7fb95fe00700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb9480b29f0 0x7fb9480b4f00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb960103820 tx=0x7fb94c00b410 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:56.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:56.869+0000 7fb95d600700 1 -- 172.21.15.12:0/1447615234 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb954013070 con 0x7fb960075af0 2024-05-27T07:39:57.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.246+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb96019e340 con 0x7fb960075af0 2024-05-27T07:39:57.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.248+0000 7fb95d600700 1 -- 172.21.15.12:0/1447615234 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7fb9540ab430 con 0x7fb960075af0 2024-05-27T07:39:57.250 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:39:57.250 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:39:57.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.253+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb9480b29f0 msgr2=0x7fb9480b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:57.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.253+0000 7fb967600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb9480b29f0 0x7fb9480b4f00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb960103820 tx=0x7fb94c00b410 comp rx=0 tx=0).stop 2024-05-27T07:39:57.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.253+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 msgr2=0x7fb96019cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.253+0000 7fb967600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb96019cfe0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb9540057c0 tx=0x7fb9540057f0 comp rx=0 tx=0).stop 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.254+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 shutdown_connections 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.254+0000 7fb967600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb9480b29f0 0x7fb9480b4f00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.254+0000 7fb967600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb960110c10 0x7fb9601a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.254+0000 7fb967600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb960075af0 0x7fb96019cfe0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.254+0000 7fb967600700 1 --2- 172.21.15.12:0/1447615234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb960076440 0x7fb96019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.254+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 >> 172.21.15.12:0/1447615234 conn(0x7fb96007af60 msgr2=0x7fb960102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.255+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 shutdown_connections 2024-05-27T07:39:57.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:39:57.255+0000 7fb967600700 1 -- 172.21.15.12:0/1447615234 wait complete. 2024-05-27T07:39:57.256 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:39:57.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:57 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1447615234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:57.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:57 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1447615234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:57.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:57 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1447615234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:58.195 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:39:58.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:58 smithi012 ceph-mon[31473]: pgmap v599: 129 pgs: 129 active+clean; 389 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 172 op/s 2024-05-27T07:39:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:58.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:58 smithi162 ceph-mon[35676]: pgmap v599: 129 pgs: 129 active+clean; 389 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 172 op/s 2024-05-27T07:39:58.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:58 smithi062 ceph-mon[35304]: pgmap v599: 129 pgs: 129 active+clean; 389 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 172 op/s 2024-05-27T07:39:59.925 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:39:59 smithi012 ceph-mon[31473]: pgmap v600: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 275 op/s 2024-05-27T07:39:59.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:39:59 smithi162 ceph-mon[35676]: pgmap v600: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 275 op/s 2024-05-27T07:39:59.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:39:59 smithi062 ceph-mon[35304]: pgmap v600: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 275 op/s 2024-05-27T07:40:00.962 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: Health detail: HEALTH_WARN 9 failed cephadm daemon(s) 2024-05-27T07:40:00.962 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: [WRN] CEPHADM_FAILED_DAEMON: 9 failed cephadm daemon(s) 2024-05-27T07:40:00.962 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.j on smithi012 is in error state 2024-05-27T07:40:00.962 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.d on smithi012 is in error state 2024-05-27T07:40:00.963 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.g on smithi012 is in error state 2024-05-27T07:40:00.963 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.a on smithi012 is in error state 2024-05-27T07:40:00.963 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.k on smithi062 is in error state 2024-05-27T07:40:00.963 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.l on smithi162 is in error state 2024-05-27T07:40:00.963 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.f on smithi162 is in error state 2024-05-27T07:40:00.963 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.c on smithi162 is in error state 2024-05-27T07:40:00.963 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:00 smithi012 ceph-mon[31473]: daemon mds.i on smithi162 is in error state 2024-05-27T07:40:00.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: Health detail: HEALTH_WARN 9 failed cephadm daemon(s) 2024-05-27T07:40:00.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: [WRN] CEPHADM_FAILED_DAEMON: 9 failed cephadm daemon(s) 2024-05-27T07:40:00.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.j on smithi012 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.d on smithi012 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.g on smithi012 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.a on smithi012 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.k on smithi062 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.l on smithi162 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.f on smithi162 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.c on smithi162 is in error state 2024-05-27T07:40:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:00 smithi162 ceph-mon[35676]: daemon mds.i on smithi162 is in error state 2024-05-27T07:40:00.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: Health detail: HEALTH_WARN 9 failed cephadm daemon(s) 2024-05-27T07:40:00.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: [WRN] CEPHADM_FAILED_DAEMON: 9 failed cephadm daemon(s) 2024-05-27T07:40:00.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.j on smithi012 is in error state 2024-05-27T07:40:00.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.d on smithi012 is in error state 2024-05-27T07:40:00.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.g on smithi012 is in error state 2024-05-27T07:40:00.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.a on smithi012 is in error state 2024-05-27T07:40:00.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.k on smithi062 is in error state 2024-05-27T07:40:00.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.l on smithi162 is in error state 2024-05-27T07:40:00.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.f on smithi162 is in error state 2024-05-27T07:40:00.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.c on smithi162 is in error state 2024-05-27T07:40:00.997 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:00 smithi062 ceph-mon[35304]: daemon mds.i on smithi162 is in error state 2024-05-27T07:40:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 -- 172.21.15.12:0/2214153088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc10af40 msgr2=0x7f5dfc10b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 --2- 172.21.15.12:0/2214153088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc10af40 0x7f5dfc10b350 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f5de80099c0 tx=0x7f5de8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 -- 172.21.15.12:0/2214153088 shutdown_connections 2024-05-27T07:40:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 --2- 172.21.15.12:0/2214153088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dfc106610 0x7f5dfc106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 --2- 172.21.15.12:0/2214153088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dfc10b890 0x7f5dfc1131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 --2- 172.21.15.12:0/2214153088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc10af40 0x7f5dfc10b350 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 -- 172.21.15.12:0/2214153088 >> 172.21.15.12:0/2214153088 conn(0x7f5dfc102020 msgr2=0x7f5dfc104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 -- 172.21.15.12:0/2214153088 shutdown_connections 2024-05-27T07:40:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.256+0000 7f5e01800700 1 -- 172.21.15.12:0/2214153088 wait complete. 2024-05-27T07:40:01.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.257+0000 7f5e01800700 1 Processor -- start 2024-05-27T07:40:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5e01800700 1 -- start start 2024-05-27T07:40:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5e01800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc106610 0x7f5dfc1106b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5e01800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dfc10af40 0x7f5dfc110bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5e01800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dfc10b890 0x7f5dfc111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5e01800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1163a0 con 0x7f5dfc106610 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5e01800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc116220 con 0x7f5dfc10b890 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5e01800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc116520 con 0x7f5dfc10af40 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5dfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc106610 0x7f5dfc1106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5dfbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc106610 0x7f5dfc1106b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56380/0 (socket says 172.21.15.12:56380) 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.258+0000 7f5dfbe00700 1 -- 172.21.15.12:0/2390767618 learned_addr learned my addr 172.21.15.12:0/2390767618 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dfc10b890 0x7f5dfc111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5dfb400700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dfc10af40 0x7f5dfc110bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5dfbe00700 1 -- 172.21.15.12:0/2390767618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dfc10af40 msgr2=0x7f5dfc110bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5dfbe00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dfc10af40 0x7f5dfc110bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5dfbe00700 1 -- 172.21.15.12:0/2390767618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dfc10b890 msgr2=0x7f5dfc111130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5dfbe00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dfc10b890 0x7f5dfc111130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5dfbe00700 1 -- 172.21.15.12:0/2390767618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc1ad2a0 con 0x7f5dfc106610 2024-05-27T07:40:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dfc10b890 0x7f5dfc111130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.259+0000 7f5dfbe00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc106610 0x7f5dfc1106b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c920 tx=0x7f5dec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.260+0000 7f5df8c00700 1 -- 172.21.15.12:0/2390767618 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec007b00 con 0x7f5dfc106610 2024-05-27T07:40:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.260+0000 7f5df8c00700 1 -- 172.21.15.12:0/2390767618 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dec004cc0 con 0x7f5dfc106610 2024-05-27T07:40:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.260+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dfc1ad590 con 0x7f5dfc106610 2024-05-27T07:40:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.260+0000 7f5df8c00700 1 -- 172.21.15.12:0/2390767618 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec005440 con 0x7f5dfc106610 2024-05-27T07:40:01.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.260+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dfc1adaf0 con 0x7f5dfc106610 2024-05-27T07:40:01.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.264+0000 7f5df8c00700 1 -- 172.21.15.12:0/2390767618 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5dec004750 con 0x7f5dfc106610 2024-05-27T07:40:01.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.265+0000 7f5e11979b80 1 Processor -- start 2024-05-27T07:40:01.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.265+0000 7f5e11979b80 1 -- start start 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ad3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5df8c00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5de00b2a70 0x7f5de00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4410 0x557068adead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4820 0x557068ae1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e11979b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557068a02e40 con 0x557068ad4410 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e11979b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5570689f29e0 con 0x557068955e40 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e11979b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5570689f6a20 con 0x557068ad4820 2024-05-27T07:40:01.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5df8c00700 1 -- 172.21.15.12:0/2390767618 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f5dec0e2580 con 0x7f5dfc106610 2024-05-27T07:40:01.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5dfb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4410 0x557068adead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e00e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ad3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e00e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ad3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:48812/0 (socket says 172.21.15.12:48812) 2024-05-27T07:40:01.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5e00e00700 1 -- 172.21.15.12:0/2708979790 learned_addr learned my addr 172.21.15.12:0/2708979790 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:01.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.266+0000 7f5dfbe00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5de00b2a70 0x7f5de00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4820 0x557068ae1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5e00e00700 1 -- 172.21.15.12:0/2708979790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4820 msgr2=0x557068ae1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4820 0x557068ae1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5e00e00700 1 -- 172.21.15.12:0/2708979790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4410 msgr2=0x557068adead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4410 0x557068adead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5e00e00700 1 -- 172.21.15.12:0/2708979790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570689bb420 con 0x557068955e40 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5dfb400700 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4410 0x557068adead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.267+0000 7f5dfbe00700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5de00b2a70 0x7f5de00b4f80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0a5470 tx=0x7f5dec00a5a0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:01.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.268+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ad3ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5df40127e0 tx=0x7f5df4007cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2673d206645b64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:01.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.268+0000 7f5df1600700 1 -- 172.21.15.12:0/2708979790 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4004bb0 con 0x557068955e40 2024-05-27T07:40:01.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.268+0000 7f5df1600700 1 -- 172.21.15.12:0/2708979790 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5df4004d50 con 0x557068955e40 2024-05-27T07:40:01.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.268+0000 7f5df1600700 1 -- 172.21.15.12:0/2708979790 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4004610 con 0x557068955e40 2024-05-27T07:40:01.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.269+0000 7f5e11979b80 1 -- 172.21.15.12:0/2708979790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 msgr2=0x557068ad3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.269+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ad3ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5df40127e0 tx=0x7f5df4007cb0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.269+0000 7f5e11979b80 1 -- 172.21.15.12:0/2708979790 shutdown_connections 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.269+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ad3ed0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.269+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4820 0x557068ae1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.269+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2708979790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4410 0x557068adead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.269+0000 7f5e11979b80 1 -- 172.21.15.12:0/2708979790 >> 172.21.15.12:0/2708979790 conn(0x5570689b5fc0 msgr2=0x557068acfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.270+0000 7f5e11979b80 1 -- 172.21.15.12:0/2708979790 shutdown_connections 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.270+0000 7f5e11979b80 1 -- 172.21.15.12:0/2708979790 wait complete. 2024-05-27T07:40:01.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.270+0000 7f5e11979b80 1 -- start start 2024-05-27T07:40:01.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.270+0000 7f5e11979b80 1 Processor -- start 2024-05-27T07:40:01.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.271+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ade5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.271+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 0x557068adeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.271+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4820 0x557068adf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.271+0000 7f5e11979b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5570688f8fa0 con 0x557068ad4820 2024-05-27T07:40:01.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.271+0000 7f5e11979b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557068993760 con 0x557068955e40 2024-05-27T07:40:01.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.271+0000 7f5e11979b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557068ae1990 con 0x557068ad4410 2024-05-27T07:40:01.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.271+0000 7f5e11979b80 10 client.0 _ll_register_callbacks cb 0x55706899cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:01.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4820 0x557068adf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5dfb400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 0x557068adeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4820 0x557068adf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56388/0 (socket says 172.21.15.12:56388) 2024-05-27T07:40:01.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 -- 172.21.15.12:0/1139221771 learned_addr learned my addr 172.21.15.12:0/1139221771 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:01.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ade5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 -- 172.21.15.12:0/1139221771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 msgr2=0x557068adeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 0x557068adeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 -- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 msgr2=0x557068ade5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ade5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.272+0000 7f5e00e00700 1 -- 172.21.15.12:0/1139221771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570689428d0 con 0x557068ad4820 2024-05-27T07:40:01.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.273+0000 7f5e00e00700 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4820 0x557068adf070 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5df40049c0 tx=0x7f5df4007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88641056a1aaba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:01.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.273+0000 7f5de6000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:01.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.273+0000 7f5de6000700 1 -- 172.21.15.12:0/1139221771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4003c10 con 0x557068ad4820 2024-05-27T07:40:01.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.273+0000 7f5de6000700 1 -- 172.21.15.12:0/1139221771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f5df4003db0 con 0x557068ad4820 2024-05-27T07:40:01.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.273+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5df4004d50 con 0x557068ad4820 2024-05-27T07:40:01.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.274+0000 7f5de6000700 1 -- 172.21.15.12:0/1139221771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5df4003fb0 con 0x557068ad4820 2024-05-27T07:40:01.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.275+0000 7f5e11979b80 10 client.17493 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:01.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.275+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5570689c3060 con 0x557068ad4820 2024-05-27T07:40:01.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.277+0000 7f5de6000700 1 -- 172.21.15.12:0/1139221771 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5df4004d50 con 0x557068ad4820 2024-05-27T07:40:01.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.277+0000 7f5de6000700 1 -- 172.21.15.12:0/1139221771 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5df402d030 con 0x557068ad4820 2024-05-27T07:40:01.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.278+0000 7f5e11979b80 10 client.17493 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:01.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.278+0000 7f5e11979b80 10 client.17493 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:40:01.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.278+0000 7f5e11979b80 20 client.17493 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:01.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.278+0000 7f5e11979b80 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068aeeab0 0x557068af0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.278+0000 7f5e11979b80 4 client.17493 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:01.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.278+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557068a04d90 con 0x557068aeeab0 2024-05-27T07:40:01.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.280+0000 7f5dfb400700 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068aeeab0 0x557068af0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.283+0000 7f5dfb400700 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068aeeab0 0x557068af0f40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bb6541437bad356c server_cookie=63467e2a44f1a437 in_seq=0 out_seq=0 2024-05-27T07:40:01.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.283+0000 7f5de6000700 10 client.17493 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:40:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.286+0000 7f5de6000700 1 -- 172.21.15.12:0/1139221771 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557068a04d90 con 0x557068aeeab0 2024-05-27T07:40:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.286+0000 7f5de6000700 10 client.17493 handle_command_reply: tid=0 2024-05-27T07:40:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.286+0000 7f5e11979b80 1 client.17493 shutdown 2024-05-27T07:40:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.286+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4820 msgr2=0x557068adf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.286+0000 7f5e11979b80 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4820 0x557068adf070 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5df40049c0 tx=0x7f5df4007a30 comp rx=0 tx=0).stop 2024-05-27T07:40:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 shutdown_connections 2024-05-27T07:40:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5e11979b80 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068aeeab0 0x557068af0f40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5e11979b80 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068955e40 0x557068ade5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5e11979b80 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068ad4820 0x557068adf070 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5e11979b80 1 --2- 172.21.15.12:0/1139221771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 0x557068adeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 >> 172.21.15.12:0/1139221771 conn(0x5570689b5fc0 msgr2=0x557068aceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:01.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5de6000700 0 client.17493 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:40:01.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.287+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 shutdown_connections 2024-05-27T07:40:01.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.288+0000 7f5e11979b80 1 -- 172.21.15.12:0/1139221771 wait complete. 2024-05-27T07:40:01.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.288+0000 7f5e11979b80 20 client.17493 trim_cache size 0 max 16384 2024-05-27T07:40:01.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.311+0000 7f5e11979b80 1 Processor -- start 2024-05-27T07:40:01.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.311+0000 7f5e11979b80 1 -- start start 2024-05-27T07:40:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.312+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570689b5fc0 0x557068ad87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.312+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068aeaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.312+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068ad4410 0x557068acf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.312+0000 7f5e11979b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557068ade070 con 0x557068955e40 2024-05-27T07:40:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.312+0000 7f5e11979b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1b61c0 con 0x557068ad4410 2024-05-27T07:40:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.312+0000 7f5e11979b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1b5420 con 0x5570689b5fc0 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068aeaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068aeaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56398/0 (socket says 172.21.15.12:56398) 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 -- 172.21.15.12:0/2134537603 learned_addr learned my addr 172.21.15.12:0/2134537603 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570689b5fc0 0x557068ad87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 -- 172.21.15.12:0/2134537603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570689b5fc0 msgr2=0x557068ad87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570689b5fc0 0x557068ad87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5e00e00700 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068ad4410 0x557068acf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 -- 172.21.15.12:0/2134537603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068ad4410 msgr2=0x557068acf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068ad4410 0x557068acf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 -- 172.21.15.12:0/2134537603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557068ad1350 con 0x557068955e40 2024-05-27T07:40:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.313+0000 7f5dfb400700 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068aeaa30 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009990 tx=0x7f5de8005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ef22149f9102a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.314+0000 7f5df0c00700 1 -- 172.21.15.12:0/2134537603 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de800b990 con 0x557068955e40 2024-05-27T07:40:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.314+0000 7f5df0c00700 1 -- 172.21.15.12:0/2134537603 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5de8003680 con 0x557068955e40 2024-05-27T07:40:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.314+0000 7f5df0c00700 1 -- 172.21.15.12:0/2134537603 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de800f6c0 con 0x557068955e40 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 -- 172.21.15.12:0/2134537603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 msgr2=0x557068aeaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068aeaa30 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009990 tx=0x7f5de8005040 comp rx=0 tx=0).stop 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 -- 172.21.15.12:0/2134537603 shutdown_connections 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557068ad4410 0x557068acf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570689b5fc0 0x557068ad87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 --2- 172.21.15.12:0/2134537603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068aeaa30 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 -- 172.21.15.12:0/2134537603 >> 172.21.15.12:0/2134537603 conn(0x557068ac8f60 msgr2=0x557068aceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 -- 172.21.15.12:0/2134537603 shutdown_connections 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.315+0000 7f5e11979b80 1 -- 172.21.15.12:0/2134537603 wait complete. 2024-05-27T07:40:01.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.316+0000 7f5e11979b80 1 -- start start 2024-05-27T07:40:01.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.316+0000 7f5e11979b80 1 Processor -- start 2024-05-27T07:40:01.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068ae72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570689b5fc0 0x557068ae2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e11979b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 0x557068ae2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e11979b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557068ad08d0 con 0x557068955e40 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e11979b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1b5af0 con 0x5570689b5fc0 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e11979b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557068aca220 con 0x557068ad4410 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e11979b80 10 client.0 _ll_register_callbacks cb 0x55706899cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e00e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068ae72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e00e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068ae72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56406/0 (socket says 172.21.15.12:56406) 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.317+0000 7f5e00e00700 1 -- 172.21.15.12:0/3738808298 learned_addr learned my addr 172.21.15.12:0/3738808298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.318+0000 7f5dfb400700 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570689b5fc0 0x557068ae2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.318+0000 7f5e00e00700 1 -- 172.21.15.12:0/3738808298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 msgr2=0x557068ae2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.318+0000 7f5e00e00700 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 0x557068ae2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.318+0000 7f5e00e00700 1 -- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570689b5fc0 msgr2=0x557068ae2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:01.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.318+0000 7f5e00e00700 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570689b5fc0 0x557068ae2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.318+0000 7f5e00e00700 1 -- 172.21.15.12:0/3738808298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557068ae3290 con 0x557068955e40 2024-05-27T07:40:01.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.318+0000 7f5dfb400700 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570689b5fc0 0x557068ae2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.319+0000 7f5e00e00700 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068ae72c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f5df4017540 tx=0x7f5df40021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16790d9070ef35f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.319+0000 7f5de6000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.319+0000 7f5de6000700 1 -- 172.21.15.12:0/3738808298 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df403c060 con 0x557068955e40 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.319+0000 7f5de6000700 1 -- 172.21.15.12:0/3738808298 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7f5df4004c30 con 0x557068955e40 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.319+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de8003680 con 0x557068955e40 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.320+0000 7f5de6000700 1 -- 172.21.15.12:0/3738808298 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f5df4043050 con 0x557068955e40 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.320+0000 7f5e11979b80 10 client.17499 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.320+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x557068b59470 con 0x557068955e40 2024-05-27T07:40:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.320+0000 7f5de6000700 1 -- 172.21.15.12:0/3738808298 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5de8003680 con 0x557068955e40 2024-05-27T07:40:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.321+0000 7f5de6000700 1 -- 172.21.15.12:0/3738808298 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f5df402d040 con 0x557068955e40 2024-05-27T07:40:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.321+0000 7f5e11979b80 10 client.17499 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.321+0000 7f5e11979b80 10 client.17499 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:40:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.321+0000 7f5e11979b80 20 client.17499 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.321+0000 7f5e11979b80 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068ad4820 0x557068b5bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:01.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.321+0000 7f5e11979b80 4 client.17499 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.321+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570689fe570 con 0x557068ad4820 2024-05-27T07:40:01.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.323+0000 7f5dfb400700 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068ad4820 0x557068b5bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:01.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.325+0000 7f5dfb400700 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068ad4820 0x557068b5bb30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f6640e386c36d4b4 server_cookie=7b66646d90fed6f9 in_seq=0 out_seq=0 2024-05-27T07:40:01.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.325+0000 7f5de6000700 10 client.17499 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:40:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.326+0000 7f5de6000700 1 -- 172.21.15.12:0/3738808298 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570689fe570 con 0x557068ad4820 2024-05-27T07:40:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.326+0000 7f5de6000700 10 client.17499 handle_command_reply: tid=0 2024-05-27T07:40:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.326+0000 7f5e11979b80 1 client.17499 shutdown 2024-05-27T07:40:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.326+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 msgr2=0x557068ae72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.326+0000 7f5e11979b80 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068ae72c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f5df4017540 tx=0x7f5df40021f0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.326+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 shutdown_connections 2024-05-27T07:40:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.326+0000 7f5e11979b80 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557068ad4410 0x557068ae2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5e11979b80 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557068955e40 0x557068ae72c0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5e11979b80 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x557068ad4820 0x557068b5bb30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5e11979b80 1 --2- 172.21.15.12:0/3738808298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570689b5fc0 0x557068ae2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5de6000700 0 client.17499 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:40:01.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 >> 172.21.15.12:0/3738808298 conn(0x557068ac8f60 msgr2=0x557068acfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 shutdown_connections 2024-05-27T07:40:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5e11979b80 1 -- 172.21.15.12:0/3738808298 wait complete. 2024-05-27T07:40:01.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.327+0000 7f5e11979b80 20 client.17499 trim_cache size 0 max 16384 2024-05-27T07:40:01.326 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5de00b2a70 msgr2=0x7f5de00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5de00b2a70 0x7f5de00b4f80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0a5470 tx=0x7f5dec00a5a0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc106610 msgr2=0x7f5dfc1106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc106610 0x7f5dfc1106b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c920 tx=0x7f5dec00cc30 comp rx=0 tx=0).stop 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 shutdown_connections 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f5de00b2a70 0x7f5de00b4f80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dfc10b890 0x7f5dfc111130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dfc106610 0x7f5dfc1106b0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 --2- 172.21.15.12:0/2390767618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dfc10af40 0x7f5dfc110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 >> 172.21.15.12:0/2390767618 conn(0x7f5dfc102020 msgr2=0x7f5dfc10a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 shutdown_connections 2024-05-27T07:40:01.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:01.328+0000 7f5e01800700 1 -- 172.21.15.12:0/2390767618 wait complete. 2024-05-27T07:40:01.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:01 smithi162 ceph-mon[35676]: pgmap v601: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:40:01.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:01 smithi062 ceph-mon[35304]: pgmap v601: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:40:02.067 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:01 smithi012 ceph-mon[31473]: pgmap v601: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:40:02.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:40:02.626 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:03.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:03 smithi162 ceph-mon[35676]: pgmap v602: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:40:03.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:03 smithi062 ceph-mon[35304]: pgmap v602: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:40:04.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:03 smithi012 ceph-mon[31473]: pgmap v602: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 -- 172.21.15.12:0/3525672478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 msgr2=0x7ff880075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 --2- 172.21.15.12:0/3525672478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff880075f60 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff874009990 tx=0x7ff874009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 -- 172.21.15.12:0/3525672478 shutdown_connections 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 --2- 172.21.15.12:0/3525672478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff880115360 0x7ff880117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 --2- 172.21.15.12:0/3525672478 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff880075f60 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 --2- 172.21.15.12:0/3525672478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8800764a0 0x7ff880114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 -- 172.21.15.12:0/3525672478 >> 172.21.15.12:0/3525672478 conn(0x7ff880100030 msgr2=0x7ff880102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 -- 172.21.15.12:0/3525672478 shutdown_connections 2024-05-27T07:40:05.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.093+0000 7ff887800700 1 -- 172.21.15.12:0/3525672478 wait complete. 2024-05-27T07:40:05.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.094+0000 7ff887800700 1 Processor -- start 2024-05-27T07:40:05.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.095+0000 7ff887800700 1 -- start start 2024-05-27T07:40:05.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.095+0000 7ff887800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff8801a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.095+0000 7ff887800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8800764a0 0x7ff8801a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.095+0000 7ff887800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff880115360 0x7ff8801a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.095+0000 7ff887800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff88011a910 con 0x7ff880075b50 2024-05-27T07:40:05.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.095+0000 7ff887800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff88011a790 con 0x7ff8800764a0 2024-05-27T07:40:05.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.095+0000 7ff887800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff88011aa90 con 0x7ff880115360 2024-05-27T07:40:05.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.096+0000 7ff884e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff8801a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.096+0000 7ff87fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8800764a0 0x7ff8801a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.096+0000 7ff884e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff8801a1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34758/0 (socket says 172.21.15.12:34758) 2024-05-27T07:40:05.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.096+0000 7ff884e00700 1 -- 172.21.15.12:0/2713855312 learned_addr learned my addr 172.21.15.12:0/2713855312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:05.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.097+0000 7ff884e00700 1 -- 172.21.15.12:0/2713855312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff880115360 msgr2=0x7ff8801a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.097+0000 7ff884e00700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff880115360 0x7ff8801a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.097+0000 7ff884e00700 1 -- 172.21.15.12:0/2713855312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8800764a0 msgr2=0x7ff8801a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.097+0000 7ff884e00700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8800764a0 0x7ff8801a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.097+0000 7ff884e00700 1 -- 172.21.15.12:0/2713855312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8801a6160 con 0x7ff880075b50 2024-05-27T07:40:05.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.097+0000 7ff87fe00700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8800764a0 0x7ff8801a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:05.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.097+0000 7ff884e00700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff8801a1340 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff874000c00 tx=0x7ff874009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:05.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.098+0000 7ff87d600700 1 -- 172.21.15.12:0/2713855312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874027070 con 0x7ff880075b50 2024-05-27T07:40:05.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.098+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8801a63f0 con 0x7ff880075b50 2024-05-27T07:40:05.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.098+0000 7ff87d600700 1 -- 172.21.15.12:0/2713855312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff87400fc70 con 0x7ff880075b50 2024-05-27T07:40:05.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.098+0000 7ff87d600700 1 -- 172.21.15.12:0/2713855312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874022d40 con 0x7ff880075b50 2024-05-27T07:40:05.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.100+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8801a6920 con 0x7ff880075b50 2024-05-27T07:40:05.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.107+0000 7ff87d600700 1 -- 172.21.15.12:0/2713855312 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff87402a070 con 0x7ff880075b50 2024-05-27T07:40:05.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.108+0000 7ff896335b80 1 Processor -- start 2024-05-27T07:40:05.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.108+0000 7ff896335b80 1 -- start start 2024-05-27T07:40:05.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.109+0000 7ff896335b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051eebf9f0 0x56051ef52330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.109+0000 7ff896335b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef52870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.109+0000 7ff896335b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051ef52e70 0x56051ef5f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.109+0000 7ff896335b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56051ee80e40 con 0x56051edd3e40 2024-05-27T07:40:05.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.109+0000 7ff896335b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56051ee709e0 con 0x56051ef52e70 2024-05-27T07:40:05.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.109+0000 7ff896335b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56051ee74a20 con 0x56051eebf9f0 2024-05-27T07:40:05.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.109+0000 7ff884e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef52870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.110+0000 7ff884e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef52870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34770/0 (socket says 172.21.15.12:34770) 2024-05-27T07:40:05.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.110+0000 7ff884e00700 1 -- 172.21.15.12:0/3640325728 learned_addr learned my addr 172.21.15.12:0/3640325728 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:05.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.110+0000 7ff885800700 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051eebf9f0 0x56051ef52330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.110+0000 7ff87fe00700 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051ef52e70 0x56051ef5f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.111+0000 7ff87d600700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff8680b2a70 0x7ff8680b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.112+0000 7ff884e00700 1 -- 172.21.15.12:0/3640325728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051eebf9f0 msgr2=0x56051ef52330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.112+0000 7ff884e00700 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051eebf9f0 0x56051ef52330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.112+0000 7ff884e00700 1 -- 172.21.15.12:0/3640325728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051ef52e70 msgr2=0x56051ef5f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.112+0000 7ff884e00700 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051ef52e70 0x56051ef5f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.112+0000 7ff884e00700 1 -- 172.21.15.12:0/3640325728 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56051ee39420 con 0x56051edd3e40 2024-05-27T07:40:05.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.112+0000 7ff885800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff8680b2a70 0x7ff8680b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.113+0000 7ff87d600700 1 -- 172.21.15.12:0/2713855312 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7ff8740e3780 con 0x7ff880075b50 2024-05-27T07:40:05.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.113+0000 7ff87fe00700 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051ef52e70 0x56051ef5f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:05.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.113+0000 7ff884e00700 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef52870 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff8740e0a30 tx=0x7ff87401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75488454031135 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:05.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.114+0000 7ff872000700 1 -- 172.21.15.12:0/3640325728 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8740aa030 con 0x56051edd3e40 2024-05-27T07:40:05.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.114+0000 7ff872000700 1 -- 172.21.15.12:0/3640325728 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8740aa1f0 con 0x56051edd3e40 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.114+0000 7ff872000700 1 -- 172.21.15.12:0/3640325728 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8740a9e50 con 0x56051edd3e40 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff885800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff8680b2a70 0x7ff8680b4f80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff878007fd0 tx=0x7ff878009040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 -- 172.21.15.12:0/3640325728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 msgr2=0x56051ef52870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef52870 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff8740e0a30 tx=0x7ff87401a000 comp rx=0 tx=0).stop 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 -- 172.21.15.12:0/3640325728 shutdown_connections 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051ef52e70 0x56051ef5f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051eebf9f0 0x56051ef52330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 --2- 172.21.15.12:0/3640325728 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef52870 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 -- 172.21.15.12:0/3640325728 >> 172.21.15.12:0/3640325728 conn(0x56051ee6d390 msgr2=0x56051ef4e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:05.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.116+0000 7ff896335b80 1 -- 172.21.15.12:0/3640325728 shutdown_connections 2024-05-27T07:40:05.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.117+0000 7ff896335b80 1 -- 172.21.15.12:0/3640325728 wait complete. 2024-05-27T07:40:05.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.117+0000 7ff896335b80 1 -- start start 2024-05-27T07:40:05.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.118+0000 7ff896335b80 1 Processor -- start 2024-05-27T07:40:05.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.118+0000 7ff896335b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef5c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff896335b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef5cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff896335b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef5d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff896335b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56051ed76fa0 con 0x56051eebf9f0 2024-05-27T07:40:05.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff896335b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56051ee11760 con 0x56051edd3e40 2024-05-27T07:40:05.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff896335b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56051ef5f940 con 0x56051ef52e70 2024-05-27T07:40:05.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff896335b80 10 client.0 _ll_register_callbacks cb 0x56051ee1ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff87fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef5cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff87fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef5cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34786/0 (socket says 172.21.15.12:34786) 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff87fe00700 1 -- 172.21.15.12:0/2815504912 learned_addr learned my addr 172.21.15.12:0/2815504912 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff884e00700 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef5c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.119+0000 7ff885800700 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef5d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.120+0000 7ff87fe00700 1 -- 172.21.15.12:0/2815504912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 msgr2=0x56051ef5d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.120+0000 7ff87fe00700 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef5d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.120+0000 7ff87fe00700 1 -- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 msgr2=0x56051ef5c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.120+0000 7ff87fe00700 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef5c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.120+0000 7ff87fe00700 1 -- 172.21.15.12:0/2815504912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56051edc08d0 con 0x56051eebf9f0 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.120+0000 7ff87fe00700 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef5cbc0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff86c011490 tx=0x7ff86c0117a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e2afdaf387a43f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.120+0000 7ff866a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.121+0000 7ff866a00700 1 -- 172.21.15.12:0/2815504912 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c01d680 con 0x56051eebf9f0 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.121+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8740aa1f0 con 0x56051eebf9f0 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.121+0000 7ff866a00700 1 -- 172.21.15.12:0/2815504912 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7ff86c012450 con 0x56051eebf9f0 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.121+0000 7ff866a00700 1 -- 172.21.15.12:0/2815504912 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7ff86c01d820 con 0x56051eebf9f0 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.121+0000 7ff896335b80 10 client.17511 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:05.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.121+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56051ee41060 con 0x56051eebf9f0 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff866a00700 1 -- 172.21.15.12:0/2815504912 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8740aa1f0 con 0x56051eebf9f0 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff866a00700 1 -- 172.21.15.12:0/2815504912 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7ff86c01b380 con 0x56051eebf9f0 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff896335b80 10 client.17511 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff896335b80 10 client.17511 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff896335b80 20 client.17511 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff896335b80 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051ef6cb40 0x56051ef6efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff896335b80 4 client.17511 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:05.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.122+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56051ee82d90 con 0x56051ef6cb40 2024-05-27T07:40:05.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.123+0000 7ff884e00700 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051ef6cb40 0x56051ef6efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.124+0000 7ff884e00700 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051ef6cb40 0x56051ef6efd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95022ed5309b2e9b server_cookie=9b13ae8165ff208c in_seq=0 out_seq=0 2024-05-27T07:40:05.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.124+0000 7ff866a00700 10 client.17511 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:40:05.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.125+0000 7ff866a00700 1 -- 172.21.15.12:0/2815504912 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56051ee82d90 con 0x56051ef6cb40 2024-05-27T07:40:05.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.125+0000 7ff866a00700 10 client.17511 handle_command_reply: tid=0 2024-05-27T07:40:05.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.125+0000 7ff896335b80 1 client.17511 shutdown 2024-05-27T07:40:05.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 msgr2=0x56051ef5cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef5cbc0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff86c011490 tx=0x7ff86c0117a0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 shutdown_connections 2024-05-27T07:40:05.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051ef6cb40 0x56051ef6efd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef5c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef5d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff866a00700 0 client.17511 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:40:05.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 --2- 172.21.15.12:0/2815504912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef5cbc0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 >> 172.21.15.12:0/2815504912 conn(0x56051ee6d390 msgr2=0x56051ef52310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:05.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 shutdown_connections 2024-05-27T07:40:05.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 1 -- 172.21.15.12:0/2815504912 wait complete. 2024-05-27T07:40:05.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.126+0000 7ff896335b80 20 client.17511 trim_cache size 0 max 16384 2024-05-27T07:40:05.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.149+0000 7ff896335b80 1 Processor -- start 2024-05-27T07:40:05.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff896335b80 1 -- start start 2024-05-27T07:40:05.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff896335b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef5b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff896335b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051eebf9f0 0x56051ef57e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff896335b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff896335b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801b6140 con 0x56051edd3e40 2024-05-27T07:40:05.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff896335b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801b53a0 con 0x56051eebf9f0 2024-05-27T07:40:05.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff896335b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801b5a70 con 0x56051ef52e70 2024-05-27T07:40:05.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff884e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef5b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff884e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef5b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34790/0 (socket says 172.21.15.12:34790) 2024-05-27T07:40:05.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.150+0000 7ff884e00700 1 -- 172.21.15.12:0/3950035992 learned_addr learned my addr 172.21.15.12:0/3950035992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:05.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff87fe00700 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051eebf9f0 0x56051ef57e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff885800700 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff884e00700 1 -- 172.21.15.12:0/3950035992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 msgr2=0x56051ef527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff884e00700 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff884e00700 1 -- 172.21.15.12:0/3950035992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051eebf9f0 msgr2=0x56051ef57e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff884e00700 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051eebf9f0 0x56051ef57e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff884e00700 1 -- 172.21.15.12:0/3950035992 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56051ef4f000 con 0x56051edd3e40 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff87fe00700 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051eebf9f0 0x56051ef57e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.151+0000 7ff884e00700 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef5b5e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff8740a6760 tx=0x7ff87405a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4fd88d00e78d15 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.152+0000 7ff871600700 1 -- 172.21.15.12:0/3950035992 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8740aa1f0 con 0x56051edd3e40 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.152+0000 7ff871600700 1 -- 172.21.15.12:0/3950035992 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8740c4070 con 0x56051edd3e40 2024-05-27T07:40:05.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.152+0000 7ff871600700 1 -- 172.21.15.12:0/3950035992 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8740aa6e0 con 0x56051edd3e40 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 -- 172.21.15.12:0/3950035992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 msgr2=0x56051ef5b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef5b5e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff8740a6760 tx=0x7ff87405a040 comp rx=0 tx=0).stop 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 -- 172.21.15.12:0/3950035992 shutdown_connections 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051edd3e40 0x56051ef5b5e0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 --2- 172.21.15.12:0/3950035992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051eebf9f0 0x56051ef57e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 -- 172.21.15.12:0/3950035992 >> 172.21.15.12:0/3950035992 conn(0x56051ee6d390 msgr2=0x56051ef520b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 -- 172.21.15.12:0/3950035992 shutdown_connections 2024-05-27T07:40:05.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.153+0000 7ff896335b80 1 -- 172.21.15.12:0/3950035992 wait complete. 2024-05-27T07:40:05.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.154+0000 7ff896335b80 1 -- start start 2024-05-27T07:40:05.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.154+0000 7ff896335b80 1 Processor -- start 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.154+0000 7ff896335b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef56780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff896335b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef56cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff896335b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef63790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff896335b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56051ef4d150 con 0x56051eebf9f0 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff896335b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56051ee11760 con 0x56051edd3e40 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff896335b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56051ee82d90 con 0x56051ef52e70 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff896335b80 10 client.0 _ll_register_callbacks cb 0x56051ee1ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff87fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef56cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff87fe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef56cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34794/0 (socket says 172.21.15.12:34794) 2024-05-27T07:40:05.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff87fe00700 1 -- 172.21.15.12:0/751348323 learned_addr learned my addr 172.21.15.12:0/751348323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:05.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff884e00700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef56780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff885800700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef63790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff87fe00700 1 -- 172.21.15.12:0/751348323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 msgr2=0x56051ef63790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff87fe00700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef63790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff87fe00700 1 -- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 msgr2=0x56051ef56780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.155+0000 7ff87fe00700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef56780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.156+0000 7ff87fe00700 1 -- 172.21.15.12:0/751348323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56051ef575c0 con 0x56051eebf9f0 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.156+0000 7ff885800700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef63790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.156+0000 7ff87fe00700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef56cc0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff86c011460 tx=0x7ff86c01ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42777a68da2e267b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.156+0000 7ff866a00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.156+0000 7ff866a00700 1 -- 172.21.15.12:0/751348323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c02f070 con 0x56051eebf9f0 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.156+0000 7ff866a00700 1 -- 172.21.15.12:0/751348323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9554+0+0 (secure 0 0 0) 0x7ff86c00e550 con 0x56051eebf9f0 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.156+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8740c4070 con 0x56051eebf9f0 2024-05-27T07:40:05.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff866a00700 1 -- 172.21.15.12:0/751348323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7ff86c03b050 con 0x56051eebf9f0 2024-05-27T07:40:05.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 10 client.17517 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:05.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56051ef64d60 con 0x56051eebf9f0 2024-05-27T07:40:05.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff866a00700 1 -- 172.21.15.12:0/751348323 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8740c4070 con 0x56051eebf9f0 2024-05-27T07:40:05.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff866a00700 1 -- 172.21.15.12:0/751348323 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7ff86c005040 con 0x56051eebf9f0 2024-05-27T07:40:05.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 10 client.17517 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:05.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 10 client.17517 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:40:05.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 20 client.17517 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:05.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051efd76a0 0x56051efd9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:05.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 4 client.17517 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:05.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.157+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56051ee7c570 con 0x56051efd76a0 2024-05-27T07:40:05.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.158+0000 7ff884e00700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051efd76a0 0x56051efd9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:05.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.159+0000 7ff884e00700 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051efd76a0 0x56051efd9ac0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d362a8bc96f6717 server_cookie=dfd36316ded2dfba in_seq=0 out_seq=0 2024-05-27T07:40:05.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.159+0000 7ff866a00700 10 client.17517 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:40:05.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff866a00700 1 -- 172.21.15.12:0/751348323 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56051ee7c570 con 0x56051efd76a0 2024-05-27T07:40:05.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff866a00700 10 client.17517 handle_command_reply: tid=0 2024-05-27T07:40:05.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 client.17517 shutdown 2024-05-27T07:40:05.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 msgr2=0x56051ef56cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef56cc0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff86c011460 tx=0x7ff86c01ef10 comp rx=0 tx=0).stop 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 shutdown_connections 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56051efd76a0 0x56051efd9ac0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051edd3e40 0x56051ef56780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56051ef52e70 0x56051ef63790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 --2- 172.21.15.12:0/751348323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56051eebf9f0 0x56051ef56cc0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 >> 172.21.15.12:0/751348323 conn(0x56051ee6d390 msgr2=0x56051ef53ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.160+0000 7ff866a00700 0 client.17517 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.161+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 shutdown_connections 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.161+0000 7ff896335b80 1 -- 172.21.15.12:0/751348323 wait complete. 2024-05-27T07:40:05.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.161+0000 7ff896335b80 20 client.17517 trim_cache size 0 max 16384 2024-05-27T07:40:05.160 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:40:05.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff8680b2a70 msgr2=0x7ff8680b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff8680b2a70 0x7ff8680b4f80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff878007fd0 tx=0x7ff878009040 comp rx=0 tx=0).stop 2024-05-27T07:40:05.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 msgr2=0x7ff8801a1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff8801a1340 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff874000c00 tx=0x7ff874009f90 comp rx=0 tx=0).stop 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 shutdown_connections 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7ff8680b2a70 0x7ff8680b4f80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff880115360 0x7ff8801a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff880075b50 0x7ff8801a1340 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 --2- 172.21.15.12:0/2713855312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8800764a0 0x7ff8801a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 >> 172.21.15.12:0/2713855312 conn(0x7ff880100030 msgr2=0x7ff880100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 shutdown_connections 2024-05-27T07:40:05.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:05.162+0000 7ff887800700 1 -- 172.21.15.12:0/2713855312 wait complete. 2024-05-27T07:40:05.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:05 smithi162 ceph-mon[35676]: pgmap v603: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 312 op/s 2024-05-27T07:40:05.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:05 smithi062 ceph-mon[35304]: pgmap v603: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 312 op/s 2024-05-27T07:40:06.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:05 smithi012 ceph-mon[31473]: pgmap v603: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 312 op/s 2024-05-27T07:40:06.196 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:40:06.720 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:07.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:07 smithi012 ceph-mon[31473]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:40:07.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:07 smithi012 ceph-mon[31473]: pgmap v605: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 260 op/s 2024-05-27T07:40:07.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:07 smithi162 ceph-mon[35676]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:40:07.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:07 smithi162 ceph-mon[35676]: pgmap v605: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 260 op/s 2024-05-27T07:40:07.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:07 smithi062 ceph-mon[35304]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:40:07.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:07 smithi062 ceph-mon[35304]: pgmap v605: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 260 op/s 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 -- 172.21.15.12:0/3285215203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 msgr2=0x7fa5c010c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/3285215203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 0x7fa5c010c9b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009990 tx=0x7fa5b0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 -- 172.21.15.12:0/3285215203 shutdown_connections 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/3285215203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 0x7fa5c0113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/3285215203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 0x7fa5c010c9b0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/3285215203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0106560 0x7fa5c0106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 -- 172.21.15.12:0/3285215203 >> 172.21.15.12:0/3285215203 conn(0x7fa5c0102040 msgr2=0x7fa5c0104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.193+0000 7fa5c5c00700 1 -- 172.21.15.12:0/3285215203 shutdown_connections 2024-05-27T07:40:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.194+0000 7fa5c5c00700 1 -- 172.21.15.12:0/3285215203 wait complete. 2024-05-27T07:40:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 Processor -- start 2024-05-27T07:40:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 -- start start 2024-05-27T07:40:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0106560 0x7fa5c019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 0x7fa5c019d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 0x7fa5c01a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c01165b0 con 0x7fa5c010c5a0 2024-05-27T07:40:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0116430 con 0x7fa5c0106560 2024-05-27T07:40:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.195+0000 7fa5c5c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0116730 con 0x7fa5c0106eb0 2024-05-27T07:40:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 0x7fa5c01a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 0x7fa5c019d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 0x7fa5c019d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:36448/0 (socket says 172.21.15.12:36448) 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 -- 172.21.15.12:0/1928427247 learned_addr learned my addr 172.21.15.12:0/1928427247 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5bea00700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0106560 0x7fa5c019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 -- 172.21.15.12:0/1928427247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0106560 msgr2=0x7fa5c019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0106560 0x7fa5c019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 -- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 msgr2=0x7fa5c01a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 0x7fa5c01a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.196+0000 7fa5be000700 1 -- 172.21.15.12:0/1928427247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c01a1e50 con 0x7fa5c0106eb0 2024-05-27T07:40:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.197+0000 7fa5be000700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 0x7fa5c019d520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00e940 tx=0x7fa5ac00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.197+0000 7fa5b7400700 1 -- 172.21.15.12:0/1928427247 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac00c9e0 con 0x7fa5c0106eb0 2024-05-27T07:40:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.197+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c01a2140 con 0x7fa5c0106eb0 2024-05-27T07:40:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.197+0000 7fa5b7400700 1 -- 172.21.15.12:0/1928427247 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5ac00cb80 con 0x7fa5c0106eb0 2024-05-27T07:40:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.197+0000 7fa5b7400700 1 -- 172.21.15.12:0/1928427247 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac018420 con 0x7fa5c0106eb0 2024-05-27T07:40:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.197+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c01a9a60 con 0x7fa5c0106eb0 2024-05-27T07:40:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.206+0000 7fa5d4692b80 1 Processor -- start 2024-05-27T07:40:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.206+0000 7fa5b7400700 1 -- 172.21.15.12:0/1928427247 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa5ac010430 con 0x7fa5c0106eb0 2024-05-27T07:40:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.207+0000 7fa5d4692b80 1 -- start start 2024-05-27T07:40:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.208+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e12c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.208+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32dfade40 0x55c32e12c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.208+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e12ce70 0x55c32e139110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.208+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c32e05ae40 con 0x55c32e0999f0 2024-05-27T07:40:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.208+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c32e04a9e0 con 0x55c32e12ce70 2024-05-27T07:40:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.208+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c32e04ea20 con 0x55c32dfade40 2024-05-27T07:40:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.209+0000 7fa5bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e12c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.209+0000 7fa5bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e12c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34808/0 (socket says 172.21.15.12:34808) 2024-05-27T07:40:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.209+0000 7fa5bf400700 1 -- 172.21.15.12:0/2445381355 learned_addr learned my addr 172.21.15.12:0/2445381355 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.209+0000 7fa5bea00700 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32dfade40 0x55c32e12c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.209+0000 7fa5be000700 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e12ce70 0x55c32e139110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.209+0000 7fa5bea00700 1 -- 172.21.15.12:0/2445381355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e12ce70 msgr2=0x55c32e139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5bea00700 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e12ce70 0x55c32e139110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5bea00700 1 -- 172.21.15.12:0/2445381355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 msgr2=0x55c32e12c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5bea00700 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e12c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5bea00700 1 -- 172.21.15.12:0/2445381355 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c32e013420 con 0x55c32dfade40 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5bf400700 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e12c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5b7400700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a40b29f0 0x7fa5a40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5b7400700 1 -- 172.21.15.12:0/1928427247 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 9642+0+0 (secure 0 0 0) 0x7fa5ac0e2cb0 con 0x7fa5c0106eb0 2024-05-27T07:40:09.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.210+0000 7fa5bea00700 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32dfade40 0x55c32e12c870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098e0 tx=0x7fa5b00096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=912e7764be89511 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.212+0000 7fa5bf400700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a40b29f0 0x7fa5a40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.213+0000 7fa5abe00700 1 -- 172.21.15.12:0/2445381355 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0027070 con 0x55c32dfade40 2024-05-27T07:40:09.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.213+0000 7fa5abe00700 1 -- 172.21.15.12:0/2445381355 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b0005630 con 0x55c32dfade40 2024-05-27T07:40:09.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.213+0000 7fa5abe00700 1 -- 172.21.15.12:0/2445381355 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000bb10 con 0x55c32dfade40 2024-05-27T07:40:09.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.213+0000 7fa5bf400700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a40b29f0 0x7fa5a40b4f00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55c32e12eff0 tx=0x7fa5b800b040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.214+0000 7fa5d4692b80 1 -- 172.21.15.12:0/2445381355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32dfade40 msgr2=0x55c32e12c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.214+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32dfade40 0x55c32e12c870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098e0 tx=0x7fa5b00096a0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.215+0000 7fa5d4692b80 1 -- 172.21.15.12:0/2445381355 shutdown_connections 2024-05-27T07:40:09.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.215+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e12ce70 0x55c32e139110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.215+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e12c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.215+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/2445381355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32dfade40 0x55c32e12c870 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098e0 tx=0x7fa5b00096a0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.215+0000 7fa5d4692b80 1 -- 172.21.15.12:0/2445381355 >> 172.21.15.12:0/2445381355 conn(0x55c32e047390 msgr2=0x55c32e1284c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.216+0000 7fa5d4692b80 1 -- 172.21.15.12:0/2445381355 shutdown_connections 2024-05-27T07:40:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.216+0000 7fa5d4692b80 1 -- 172.21.15.12:0/2445381355 wait complete. 2024-05-27T07:40:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.217+0000 7fa5d4692b80 1 -- start start 2024-05-27T07:40:09.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.218+0000 7fa5d4692b80 1 Processor -- start 2024-05-27T07:40:09.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e0999f0 0x55c32e136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e12ce70 0x55c32e136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e14b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c32df50fa0 con 0x55c32e12ce70 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c32dfeb760 con 0x55c32e1371a0 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c32e142330 con 0x55c32e0999f0 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5d4692b80 10 client.0 _ll_register_callbacks cb 0x55c32dff4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5bea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e0999f0 0x55c32e136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5bea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e0999f0 0x55c32e136720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:36462/0 (socket says 172.21.15.12:36462) 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.219+0000 7fa5bea00700 1 -- 172.21.15.12:0/741450240 learned_addr learned my addr 172.21.15.12:0/741450240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:09.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.220+0000 7fa5be000700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e12ce70 0x55c32e136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.220+0000 7fa5bf400700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e14b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.221+0000 7fa5bea00700 1 -- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 msgr2=0x55c32e14b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.221+0000 7fa5bea00700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e14b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.221+0000 7fa5bea00700 1 -- 172.21.15.12:0/741450240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e12ce70 msgr2=0x55c32e136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.221+0000 7fa5bea00700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e12ce70 0x55c32e136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.221+0000 7fa5bea00700 1 -- 172.21.15.12:0/741450240 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c32df9a8d0 con 0x55c32e0999f0 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.221+0000 7fa5bf400700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e14b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:09.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.221+0000 7fa5be000700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e12ce70 0x55c32e136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:09.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.222+0000 7fa5bea00700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e0999f0 0x55c32e136720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0005ae0 tx=0x7fa5b0003960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd0e421791f4841e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.222+0000 7fa5a8c00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:40:09.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.223+0000 7fa5a8c00700 1 -- 172.21.15.12:0/741450240 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0031060 con 0x55c32e0999f0 2024-05-27T07:40:09.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.223+0000 7fa5a8c00700 1 -- 172.21.15.12:0/741450240 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 9642+0+0 (secure 0 0 0) 0x7fa5b0003d10 con 0x55c32e0999f0 2024-05-27T07:40:09.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.223+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b0005630 con 0x55c32e0999f0 2024-05-27T07:40:09.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.224+0000 7fa5a8c00700 1 -- 172.21.15.12:0/741450240 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa5b0038050 con 0x55c32e0999f0 2024-05-27T07:40:09.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.224+0000 7fa5d4692b80 10 client.26911 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:09.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.224+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55c32e01b060 con 0x55c32e0999f0 2024-05-27T07:40:09.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.224+0000 7fa5a8c00700 1 -- 172.21.15.12:0/741450240 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b0005630 con 0x55c32e0999f0 2024-05-27T07:40:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.225+0000 7fa5a8c00700 1 -- 172.21.15.12:0/741450240 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa5b0022710 con 0x55c32e0999f0 2024-05-27T07:40:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.225+0000 7fa5d4692b80 10 client.26911 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.225+0000 7fa5d4692b80 10 client.26911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.225+0000 7fa5d4692b80 20 client.26911 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.225+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e14f6c0 0x55c32e151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.225+0000 7fa5d4692b80 4 client.26911 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:09.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.225+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c32e05cd90 con 0x55c32e14f6c0 2024-05-27T07:40:09.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.226+0000 7fa5be000700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e14f6c0 0x55c32e151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.227+0000 7fa5be000700 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e14f6c0 0x55c32e151ae0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69ced3e26c37dbc0 server_cookie=69414a0a55c43919 in_seq=0 out_seq=0 2024-05-27T07:40:09.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.227+0000 7fa5a8c00700 10 client.26911 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:09.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5a8c00700 1 -- 172.21.15.12:0/741450240 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c32e05cd90 con 0x55c32e14f6c0 2024-05-27T07:40:09.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5a8c00700 10 client.26911 handle_command_reply: tid=0 2024-05-27T07:40:09.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 client.26911 shutdown 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e0999f0 msgr2=0x55c32e136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e0999f0 0x55c32e136720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0005ae0 tx=0x7fa5b0003960 comp rx=0 tx=0).stop 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 shutdown_connections 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e14f6c0 0x55c32e151ae0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e14b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e0999f0 0x55c32e136720 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/741450240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e12ce70 0x55c32e136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 >> 172.21.15.12:0/741450240 conn(0x55c32e047390 msgr2=0x55c32e12c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.228+0000 7fa5a8c00700 0 client.26911 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.229+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 shutdown_connections 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.229+0000 7fa5d4692b80 1 -- 172.21.15.12:0/741450240 wait complete. 2024-05-27T07:40:09.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.229+0000 7fa5d4692b80 20 client.26911 trim_cache size 0 max 16384 2024-05-27T07:40:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.247+0000 7fa5d4692b80 1 Processor -- start 2024-05-27T07:40:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.247+0000 7fa5d4692b80 1 -- start start 2024-05-27T07:40:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.247+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e136500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e136a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e12c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c01adc10 con 0x55c32e0999f0 2024-05-27T07:40:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c01ace70 con 0x55c32e1371a0 2024-05-27T07:40:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c01ad540 con 0x55c32e12ce70 2024-05-27T07:40:09.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e136500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e136500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34830/0 (socket says 172.21.15.12:34830) 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5bea00700 1 -- 172.21.15.12:0/472379287 learned_addr learned my addr 172.21.15.12:0/472379287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5bf400700 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e12c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.248+0000 7fa5be000700 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e136a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.249+0000 7fa5bea00700 1 -- 172.21.15.12:0/472379287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 msgr2=0x55c32e136a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.249+0000 7fa5bea00700 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e136a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.249+0000 7fa5bea00700 1 -- 172.21.15.12:0/472379287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 msgr2=0x55c32e12c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.249+0000 7fa5bea00700 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e12c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.249+0000 7fa5bea00700 1 -- 172.21.15.12:0/472379287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c32e124240 con 0x55c32e0999f0 2024-05-27T07:40:09.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.249+0000 7fa5bea00700 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e136500 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000b2e0 tx=0x7fa5b0004fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62b98aff7c00185 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.249+0000 7fa5ab400700 1 -- 172.21.15.12:0/472379287 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0031070 con 0x55c32e0999f0 2024-05-27T07:40:09.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.250+0000 7fa5ab400700 1 -- 172.21.15.12:0/472379287 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b0028040 con 0x55c32e0999f0 2024-05-27T07:40:09.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.250+0000 7fa5ab400700 1 -- 172.21.15.12:0/472379287 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0027070 con 0x55c32e0999f0 2024-05-27T07:40:09.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 -- 172.21.15.12:0/472379287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 msgr2=0x55c32e136500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e136500 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000b2e0 tx=0x7fa5b0004fe0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 -- 172.21.15.12:0/472379287 shutdown_connections 2024-05-27T07:40:09.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e12c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e136500 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/472379287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e136a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 -- 172.21.15.12:0/472379287 >> 172.21.15.12:0/472379287 conn(0x55c32e047390 msgr2=0x55c32e135010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 -- 172.21.15.12:0/472379287 shutdown_connections 2024-05-27T07:40:09.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.251+0000 7fa5d4692b80 1 -- 172.21.15.12:0/472379287 wait complete. 2024-05-27T07:40:09.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.252+0000 7fa5d4692b80 1 -- start start 2024-05-27T07:40:09.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.252+0000 7fa5d4692b80 1 Processor -- start 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e131870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e12fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5d4692b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e130140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c32e12da50 con 0x55c32e0999f0 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c32e135720 con 0x55c32e1371a0 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5d4692b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c32e130700 con 0x55c32e12ce70 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5d4692b80 10 client.0 _ll_register_callbacks cb 0x55c32dff4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e131870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5bea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e131870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34836/0 (socket says 172.21.15.12:34836) 2024-05-27T07:40:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5bea00700 1 -- 172.21.15.12:0/1280522489 learned_addr learned my addr 172.21.15.12:0/1280522489 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5be000700 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e12fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.253+0000 7fa5bf400700 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e130140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5bea00700 1 -- 172.21.15.12:0/1280522489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 msgr2=0x55c32e12fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5bea00700 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e12fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5bea00700 1 -- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 msgr2=0x55c32e130140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5bea00700 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e130140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5bea00700 1 -- 172.21.15.12:0/1280522489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c32e130900 con 0x55c32e0999f0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5bea00700 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e131870 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0022bd0 tx=0x7fa5b0005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125dbcb246a83797 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5a8c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.254+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1280522489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0027040 con 0x55c32e0999f0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.255+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1280522489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 9642+0+0 (secure 0 0 0) 0x7fa5b0022680 con 0x55c32e0999f0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.255+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b0028040 con 0x55c32e0999f0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.255+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1280522489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa5b004e050 con 0x55c32e0999f0 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.255+0000 7fa5d4692b80 10 client.17532 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:09.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.255+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55c32e147510 con 0x55c32e0999f0 2024-05-27T07:40:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1280522489 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5b0028040 con 0x55c32e0999f0 2024-05-27T07:40:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1280522489 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fa5b000b7c0 con 0x55c32e0999f0 2024-05-27T07:40:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5d4692b80 10 client.17532 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5d4692b80 10 client.17532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5d4692b80 20 client.17532 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:09.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e147830 0x55c32e147c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5d4692b80 4 client.17532 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:09.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c32e056570 con 0x55c32e147830 2024-05-27T07:40:09.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.256+0000 7fa5be000700 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e147830 0x55c32e147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.257+0000 7fa5be000700 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e147830 0x55c32e147c40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93dfa2c12ef03203 server_cookie=848fab985658b574 in_seq=0 out_seq=0 2024-05-27T07:40:09.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.257+0000 7fa5a8c00700 10 client.17532 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5a8c00700 1 -- 172.21.15.12:0/1280522489 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c32e056570 con 0x55c32e147830 2024-05-27T07:40:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5a8c00700 10 client.17532 handle_command_reply: tid=0 2024-05-27T07:40:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 client.17532 shutdown 2024-05-27T07:40:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 msgr2=0x55c32e131870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e131870 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0022bd0 tx=0x7fa5b0005d00 comp rx=0 tx=0).stop 2024-05-27T07:40:09.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 shutdown_connections 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55c32e147830 0x55c32e147c40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5a8c00700 0 client.17532 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c32e1371a0 0x55c32e130140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c32e0999f0 0x55c32e131870 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 --2- 172.21.15.12:0/1280522489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c32e12ce70 0x55c32e12fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 >> 172.21.15.12:0/1280522489 conn(0x55c32e047390 msgr2=0x55c32e12c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.258+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 shutdown_connections 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.259+0000 7fa5d4692b80 1 -- 172.21.15.12:0/1280522489 wait complete. 2024-05-27T07:40:09.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.259+0000 7fa5d4692b80 20 client.17532 trim_cache size 0 max 16384 2024-05-27T07:40:09.258 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:40:09.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a40b29f0 msgr2=0x7fa5a40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a40b29f0 0x7fa5a40b4f00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55c32e12eff0 tx=0x7fa5b800b040 comp rx=0 tx=0).stop 2024-05-27T07:40:09.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 msgr2=0x7fa5c019d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 0x7fa5c019d520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00e940 tx=0x7fa5ac00ec50 comp rx=0 tx=0).stop 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 shutdown_connections 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa5a40b29f0 0x7fa5a40b4f00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0106560 0x7fa5c019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c010c5a0 0x7fa5c01a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 --2- 172.21.15.12:0/1928427247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5c0106eb0 0x7fa5c019d520 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 >> 172.21.15.12:0/1928427247 conn(0x7fa5c0102040 msgr2=0x7fa5c01039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 shutdown_connections 2024-05-27T07:40:09.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:09.260+0000 7fa5c5c00700 1 -- 172.21.15.12:0/1928427247 wait complete. 2024-05-27T07:40:09.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:09 smithi162 ceph-mon[35676]: pgmap v606: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:40:09.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:09 smithi062 ceph-mon[35304]: pgmap v606: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:40:10.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:09 smithi012 ceph-mon[31473]: pgmap v606: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:40:11.326 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:40:11.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:40:11.815 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:11.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:11 smithi012 ceph-mon[31473]: pgmap v607: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:40:11.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:11 smithi162 ceph-mon[35676]: pgmap v607: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:40:11.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:11 smithi062 ceph-mon[35304]: pgmap v607: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-05-27T07:40:13.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:13 smithi162 ceph-mon[35676]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:40:13.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:13 smithi162 ceph-mon[35676]: pgmap v609: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 893 KiB/s wr, 143 op/s 2024-05-27T07:40:13.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:13 smithi062 ceph-mon[35304]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:40:13.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:13 smithi062 ceph-mon[35304]: pgmap v609: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 893 KiB/s wr, 143 op/s 2024-05-27T07:40:14.048 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:13 smithi012 ceph-mon[31473]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:40:14.048 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:13 smithi012 ceph-mon[31473]: pgmap v609: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 893 KiB/s wr, 143 op/s 2024-05-27T07:40:14.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 -- 172.21.15.12:0/2590865133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a8075d20 msgr2=0x7fd3a8076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/2590865133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a8075d20 0x7fd3a8076130 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fd39800b0a0 tx=0x7fd39800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 -- 172.21.15.12:0/2590865133 shutdown_connections 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/2590865133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3a81109a0 0x7fd3a8112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/2590865133 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a8075d20 0x7fd3a8076130 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/2590865133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a8076670 0x7fd3a8110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 -- 172.21.15.12:0/2590865133 >> 172.21.15.12:0/2590865133 conn(0x7fd3a8079d00 msgr2=0x7fd3a807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 -- 172.21.15.12:0/2590865133 shutdown_connections 2024-05-27T07:40:14.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.669+0000 7fd3a7e00700 1 -- 172.21.15.12:0/2590865133 wait complete. 2024-05-27T07:40:14.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 Processor -- start 2024-05-27T07:40:14.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 -- start start 2024-05-27T07:40:14.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3a8076670 0x7fd3a819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a81109a0 0x7fd3a819d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a81a16a0 0x7fd3a81a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8116390 con 0x7fd3a81109a0 2024-05-27T07:40:14.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8116210 con 0x7fd3a81a16a0 2024-05-27T07:40:14.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.671+0000 7fd3a7e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8116510 con 0x7fd3a8076670 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a81109a0 0x7fd3a819d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a81109a0 0x7fd3a819d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39792/0 (socket says 172.21.15.12:39792) 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 -- 172.21.15.12:0/157049719 learned_addr learned my addr 172.21.15.12:0/157049719 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3a8076670 0x7fd3a819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd3a7400700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a81a16a0 0x7fd3a81a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 -- 172.21.15.12:0/157049719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3a8076670 msgr2=0x7fd3a819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3a8076670 0x7fd3a819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 -- 172.21.15.12:0/157049719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a81a16a0 msgr2=0x7fd3a81a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a81a16a0 0x7fd3a81a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.672+0000 7fd39e000700 1 -- 172.21.15.12:0/157049719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a81a2160 con 0x7fd3a81109a0 2024-05-27T07:40:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.673+0000 7fd39e000700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a81109a0 0x7fd3a819d520 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c9b0 tx=0x7fd39000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.673+0000 7fd3aca00700 1 -- 172.21.15.12:0/157049719 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390007b00 con 0x7fd3a81109a0 2024-05-27T07:40:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.673+0000 7fd3aca00700 1 -- 172.21.15.12:0/157049719 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd390007ca0 con 0x7fd3a81109a0 2024-05-27T07:40:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.673+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a81a2450 con 0x7fd3a81109a0 2024-05-27T07:40:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.673+0000 7fd3aca00700 1 -- 172.21.15.12:0/157049719 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390018660 con 0x7fd3a81109a0 2024-05-27T07:40:14.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.673+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a81a9c30 con 0x7fd3a81109a0 2024-05-27T07:40:14.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.677+0000 7fd3aca00700 1 -- 172.21.15.12:0/157049719 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd39000f450 con 0x7fd3a81109a0 2024-05-27T07:40:14.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.679+0000 7fd3bc8bab80 1 Processor -- start 2024-05-27T07:40:14.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.679+0000 7fd3bc8bab80 1 -- start start 2024-05-27T07:40:14.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.679+0000 7fd3aca00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd38c0b29f0 0x7fd38c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.679+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.679+0000 7fd3aca00700 1 -- 172.21.15.12:0/157049719 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7fd3900e2b40 con 0x7fd3a81109a0 2024-05-27T07:40:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.679+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 0x55d9843c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.679+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d9842eaec0 con 0x55d9843bc8a0 2024-05-27T07:40:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9842daa60 con 0x55d98423dec0 2024-05-27T07:40:14.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d9842deaa0 con 0x55d9843bc490 2024-05-27T07:40:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3a7400700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd38c0b29f0 0x7fd38c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3a6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd39e000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 0x55d9843c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39808/0 (socket says 172.21.15.12:39808) 2024-05-27T07:40:14.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.680+0000 7fd3a7400700 1 -- 172.21.15.12:0/1211392905 learned_addr learned my addr 172.21.15.12:0/1211392905 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd3a7400700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd38c0b29f0 0x7fd38c0b4f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0012440 tx=0x7fd3a0012800 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd3a7400700 1 -- 172.21.15.12:0/1211392905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 msgr2=0x55d9843c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd3a7400700 1 --2- 172.21.15.12:0/1211392905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 0x55d9843c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd3a7400700 1 -- 172.21.15.12:0/1211392905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 msgr2=0x55d9843bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd3a7400700 1 --2- 172.21.15.12:0/1211392905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd3a7400700 1 -- 172.21.15.12:0/1211392905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9842a34a0 con 0x55d9843bc8a0 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd3a7400700 1 --2- 172.21.15.12:0/1211392905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c90a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0012c00 tx=0x7fd3a001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79136857ee37049 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.681+0000 7fd39cc00700 1 -- 172.21.15.12:0/1211392905 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a001dcb0 con 0x55d9843bc8a0 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.682+0000 7fd39cc00700 1 -- 172.21.15.12:0/1211392905 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3a001de50 con 0x55d9843bc8a0 2024-05-27T07:40:14.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.682+0000 7fd39cc00700 1 -- 172.21.15.12:0/1211392905 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00057b0 con 0x55d9843bc8a0 2024-05-27T07:40:14.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.682+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/1211392905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 msgr2=0x55d9843c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.682+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/1211392905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c90a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0012c00 tx=0x7fd3a001d000 comp rx=0 tx=0).stop 2024-05-27T07:40:14.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/1211392905 shutdown_connections 2024-05-27T07:40:14.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/1211392905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/1211392905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c90a0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/1211392905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 0x55d9843c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/1211392905 >> 172.21.15.12:0/1211392905 conn(0x55d98429e040 msgr2=0x55d9843b7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:14.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/1211392905 shutdown_connections 2024-05-27T07:40:14.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/1211392905 wait complete. 2024-05-27T07:40:14.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.683+0000 7fd3bc8bab80 1 -- start start 2024-05-27T07:40:14.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 1 Processor -- start 2024-05-27T07:40:14.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843c6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 0x55d9843c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d9841e1020 con 0x55d9843bc490 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d98427b7e0 con 0x55d9843bc8a0 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d9843c99b0 con 0x55d98423dec0 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.684+0000 7fd3bc8bab80 10 client.0 _ll_register_callbacks cb 0x55d984284cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39818/0 (socket says 172.21.15.12:39818) 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 -- 172.21.15.12:0/2340499445 learned_addr learned my addr 172.21.15.12:0/2340499445 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 -- 172.21.15.12:0/2340499445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 msgr2=0x55d9843c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 -- 172.21.15.12:0/2340499445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 msgr2=0x55d9843c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 0x55d9843c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 -- 172.21.15.12:0/2340499445 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d98422a950 con 0x55d9843bc490 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd39e000700 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843c6b60 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd390000c00 tx=0x7fd3900b2270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79fb1064bc248ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.685+0000 7fd395600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd395600700 1 -- 172.21.15.12:0/2340499445 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3900cc2d0 con 0x55d9843bc490 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd395600700 1 -- 172.21.15.12:0/2340499445 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7fd3900bee00 con 0x55d9843bc490 2024-05-27T07:40:14.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3a001de50 con 0x55d9843bc490 2024-05-27T07:40:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd395600700 1 -- 172.21.15.12:0/2340499445 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fd3900a99b0 con 0x55d9843bc490 2024-05-27T07:40:14.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd3bc8bab80 10 client.17544 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55d9842ab0e0 con 0x55d9843bc490 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd395600700 1 -- 172.21.15.12:0/2340499445 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3a001de50 con 0x55d9843bc490 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.686+0000 7fd395600700 1 -- 172.21.15.12:0/2340499445 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fd3900c6050 con 0x55d9843bc490 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.687+0000 7fd3bc8bab80 10 client.17544 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.687+0000 7fd3bc8bab80 10 client.17544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.687+0000 7fd3bc8bab80 20 client.17544 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.687+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9843d6c70 0x55d9843d9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.687+0000 7fd3bc8bab80 4 client.17544 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.687+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9842ece10 con 0x55d9843d6c70 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.687+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9843d6c70 0x55d9843d9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.688+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9843d6c70 0x55d9843d9160 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e7b7111ba8caf30 server_cookie=456769eb7b196e5d in_seq=0 out_seq=0 2024-05-27T07:40:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.688+0000 7fd395600700 10 client.17544 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd395600700 1 -- 172.21.15.12:0/2340499445 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d9842ece10 con 0x55d9843d6c70 2024-05-27T07:40:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd395600700 10 client.17544 handle_command_reply: tid=0 2024-05-27T07:40:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 client.17544 shutdown 2024-05-27T07:40:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 msgr2=0x55d9843c6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843c6b60 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd390000c00 tx=0x7fd3900b2270 comp rx=0 tx=0).stop 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 shutdown_connections 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9843d6c70 0x55d9843d9160 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 0x55d9843c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2340499445 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843c6b60 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 >> 172.21.15.12:0/2340499445 conn(0x55d98429e040 msgr2=0x55d9843b7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.691+0000 7fd395600700 0 client.17544 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.692+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 shutdown_connections 2024-05-27T07:40:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.692+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2340499445 wait complete. 2024-05-27T07:40:14.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.692+0000 7fd3bc8bab80 20 client.17544 trim_cache size 0 max 16384 2024-05-27T07:40:14.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.721+0000 7fd3bc8bab80 1 Processor -- start 2024-05-27T07:40:14.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.721+0000 7fd3bc8bab80 1 -- start start 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843c0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 0x55d9843c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81b21e0 con 0x55d9843bc8a0 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81b1440 con 0x55d98423dec0 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81b1b10 con 0x55d9843bc490 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3a7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39834/0 (socket says 172.21.15.12:39834) 2024-05-27T07:40:14.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.722+0000 7fd3a7400700 1 -- 172.21.15.12:0/3585860943 learned_addr learned my addr 172.21.15.12:0/3585860943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.723+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.723+0000 7fd3a7400700 1 -- 172.21.15.12:0/3585860943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 msgr2=0x55d9843c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.723+0000 7fd3a7400700 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 0x55d9843c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.723+0000 7fd3a7400700 1 -- 172.21.15.12:0/3585860943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 msgr2=0x55d9843c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.723+0000 7fd3a7400700 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843c0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.723+0000 7fd3a7400700 1 -- 172.21.15.12:0/3585860943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9843b2250 con 0x55d9843bc8a0 2024-05-27T07:40:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.724+0000 7fd3a7400700 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c6c50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fd3a001d610 tx=0x7fd3a000c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad6010ae4659d6b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.724+0000 7fd397e00700 1 -- 172.21.15.12:0/3585860943 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a002e030 con 0x55d9843bc8a0 2024-05-27T07:40:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.724+0000 7fd397e00700 1 -- 172.21.15.12:0/3585860943 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3a00054b0 con 0x55d9843bc8a0 2024-05-27T07:40:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.724+0000 7fd397e00700 1 -- 172.21.15.12:0/3585860943 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00049e0 con 0x55d9843bc8a0 2024-05-27T07:40:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.725+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/3585860943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 msgr2=0x55d9843c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.725+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c6c50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fd3a001d610 tx=0x7fd3a000c570 comp rx=0 tx=0).stop 2024-05-27T07:40:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.726+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/3585860943 shutdown_connections 2024-05-27T07:40:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.726+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d98423dec0 0x55d9843c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.726+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc8a0 0x55d9843c6c50 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.726+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/3585860943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d9843bc490 0x55d9843c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.726+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/3585860943 >> 172.21.15.12:0/3585860943 conn(0x55d98429e040 msgr2=0x55d9843d2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.726+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/3585860943 shutdown_connections 2024-05-27T07:40:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.726+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/3585860943 wait complete. 2024-05-27T07:40:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.727+0000 7fd3bc8bab80 1 -- start start 2024-05-27T07:40:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.727+0000 7fd3bc8bab80 1 Processor -- start 2024-05-27T07:40:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd3bc8bab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 0x55d9843cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55d9842ece10 con 0x55d9843bc490 2024-05-27T07:40:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9843c6400 con 0x55d9843bc8a0 2024-05-27T07:40:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd3bc8bab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d984441a20 con 0x55d98423dec0 2024-05-27T07:40:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd3bc8bab80 10 client.0 _ll_register_callbacks cb 0x55d9843d3230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:14.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.728+0000 7fd39e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39848/0 (socket says 172.21.15.12:39848) 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 -- 172.21.15.12:0/2483094692 learned_addr learned my addr 172.21.15.12:0/2483094692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd3a7400700 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 0x55d9843cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 -- 172.21.15.12:0/2483094692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 msgr2=0x55d9843cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 -- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 msgr2=0x55d9843cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 0x55d9843cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 -- 172.21.15.12:0/2483094692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9843b88f0 con 0x55d9843bc490 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd39e000700 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843cc190 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fd39000afd0 tx=0x7fd39006f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a82b8068e31ea9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.729+0000 7fd395600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.730+0000 7fd395600700 1 -- 172.21.15.12:0/2483094692 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3900cc2d0 con 0x55d9843bc490 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.730+0000 7fd395600700 1 -- 172.21.15.12:0/2483094692 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7fd3900bee00 con 0x55d9843bc490 2024-05-27T07:40:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.730+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3a00054b0 con 0x55d9843bc490 2024-05-27T07:40:14.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.730+0000 7fd395600700 1 -- 172.21.15.12:0/2483094692 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fd3900b2380 con 0x55d9843bc490 2024-05-27T07:40:14.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.730+0000 7fd3bc8bab80 10 client.17550 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.730+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55d9843d6a50 con 0x55d9843bc490 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.730+0000 7fd395600700 1 -- 172.21.15.12:0/2483094692 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd3a00054b0 con 0x55d9843bc490 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd395600700 1 -- 172.21.15.12:0/2483094692 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fd3900c6030 con 0x55d9843bc490 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd3bc8bab80 10 client.17550 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd3bc8bab80 10 client.17550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd3bc8bab80 20 client.17550 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9844540f0 0x55d984456510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd3bc8bab80 4 client.17550 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d9842e65f0 con 0x55d9844540f0 2024-05-27T07:40:14.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.731+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9844540f0 0x55d984456510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:14.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.734+0000 7fd3a6a00700 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9844540f0 0x55d984456510 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd6347208c4e087 server_cookie=cd0c034f6b2e064c in_seq=0 out_seq=0 2024-05-27T07:40:14.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.734+0000 7fd395600700 10 client.17550 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:14.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd395600700 1 -- 172.21.15.12:0/2483094692 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d9842e65f0 con 0x55d9844540f0 2024-05-27T07:40:14.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd395600700 10 client.17550 handle_command_reply: tid=0 2024-05-27T07:40:14.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 client.17550 shutdown 2024-05-27T07:40:14.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 msgr2=0x55d9843cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843cc190 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fd39000afd0 tx=0x7fd39006f350 comp rx=0 tx=0).stop 2024-05-27T07:40:14.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 shutdown_connections 2024-05-27T07:40:14.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55d9844540f0 0x55d984456510 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9843bc8a0 0x55d9843cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98423dec0 0x55d9843cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 --2- 172.21.15.12:0/2483094692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55d9843bc490 0x55d9843cc190 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 >> 172.21.15.12:0/2483094692 conn(0x55d98429e040 msgr2=0x55d9843d2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd395600700 0 client.17550 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 shutdown_connections 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.739+0000 7fd3bc8bab80 1 -- 172.21.15.12:0/2483094692 wait complete. 2024-05-27T07:40:14.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.740+0000 7fd3bc8bab80 20 client.17550 trim_cache size 0 max 16384 2024-05-27T07:40:14.739 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:40:14.739 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:40:14.739 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "fe6a6aa0-94a7-44dc-b9d0-671736eb91f3", 2024-05-27T07:40:14.739 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:40:14.739 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.740+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd38c0b29f0 msgr2=0x7fd38c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.740+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd38c0b29f0 0x7fd38c0b4f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0012440 tx=0x7fd3a0012800 comp rx=0 tx=0).stop 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.740+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a81109a0 msgr2=0x7fd3a819d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.740+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a81109a0 0x7fd3a819d520 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c9b0 tx=0x7fd39000cd70 comp rx=0 tx=0).stop 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 shutdown_connections 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd38c0b29f0 0x7fd38c0b4f00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a81a16a0 0x7fd3a81a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3a8076670 0x7fd3a819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 --2- 172.21.15.12:0/157049719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3a81109a0 0x7fd3a819d520 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 >> 172.21.15.12:0/157049719 conn(0x7fd3a8079d00 msgr2=0x7fd3a8101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:14.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 shutdown_connections 2024-05-27T07:40:14.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:14.741+0000 7fd3a7e00700 1 -- 172.21.15.12:0/157049719 wait complete. 2024-05-27T07:40:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:14 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:40:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:14 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:14 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:40:15.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:14 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:15.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:14 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:40:15.230 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:14 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:15.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:14 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:40:15.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:14 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:15.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:14 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:40:15.242 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:14 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:15.243 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:14 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:40:15.243 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:14 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:15.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:40:16.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:15 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:40:16.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:15 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:40:16.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:15 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:16.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:15 smithi162 ceph-mon[35676]: pgmap v610: 129 pgs: 129 active+clean; 410 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 273 op/s 2024-05-27T07:40:16.243 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:15 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:40:16.244 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:15 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:40:16.244 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:15 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:16.244 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:15 smithi062 ceph-mon[35304]: pgmap v610: 129 pgs: 129 active+clean; 410 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 273 op/s 2024-05-27T07:40:16.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:15 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:40:16.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:15 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:40:16.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:15 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:16.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:15 smithi012 ceph-mon[31473]: pgmap v610: 129 pgs: 129 active+clean; 410 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 273 op/s 2024-05-27T07:40:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:17.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:16 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:17.243 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:16 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:17.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:16 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:40:18.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:17 smithi162 ceph-mon[35676]: pgmap v611: 129 pgs: 129 active+clean; 410 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 244 op/s 2024-05-27T07:40:18.243 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:17 smithi062 ceph-mon[35304]: pgmap v611: 129 pgs: 129 active+clean; 410 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 244 op/s 2024-05-27T07:40:18.347 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:17 smithi012 ceph-mon[31473]: pgmap v611: 129 pgs: 129 active+clean; 410 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 244 op/s 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 -- 172.21.15.12:0/1012811919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105c90 msgr2=0x7f51f410e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1012811919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105c90 0x7f51f410e1e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009990 tx=0x7f51e0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 -- 172.21.15.12:0/1012811919 shutdown_connections 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1012811919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51f410e720 0x7f51f4110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1012811919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105340 0x7f51f4105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1012811919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105c90 0x7f51f410e1e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 -- 172.21.15.12:0/1012811919 >> 172.21.15.12:0/1012811919 conn(0x7f51f4078ab0 msgr2=0x7f51f4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 -- 172.21.15.12:0/1012811919 shutdown_connections 2024-05-27T07:40:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.339+0000 7f51f3e00700 1 -- 172.21.15.12:0/1012811919 wait complete. 2024-05-27T07:40:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 Processor -- start 2024-05-27T07:40:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 -- start start 2024-05-27T07:40:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105340 0x7f51f419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105c90 0x7f51f419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51f410e720 0x7f51f41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4114150 con 0x7f51f4105c90 2024-05-27T07:40:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4113fd0 con 0x7f51f410e720 2024-05-27T07:40:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.341+0000 7f51f3e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41142d0 con 0x7f51f4105340 2024-05-27T07:40:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105c90 0x7f51f419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105c90 0x7f51f419d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39880/0 (socket says 172.21.15.12:39880) 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 -- 172.21.15.12:0/1719040024 learned_addr learned my addr 172.21.15.12:0/1719040024 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2a00700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105340 0x7f51f419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 -- 172.21.15.12:0/1719040024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105340 msgr2=0x7f51f419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105340 0x7f51f419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 -- 172.21.15.12:0/1719040024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51f410e720 msgr2=0x7f51f41a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51f410e720 0x7f51f41a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.342+0000 7f51f2000700 1 -- 172.21.15.12:0/1719040024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f41a1c70 con 0x7f51f4105c90 2024-05-27T07:40:19.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.343+0000 7f51f2000700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105c90 0x7f51f419d340 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f51e00057f0 tx=0x7f51e0009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:19.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.343+0000 7f51eb400700 1 -- 172.21.15.12:0/1719040024 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0027070 con 0x7f51f4105c90 2024-05-27T07:40:19.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.344+0000 7f51eb400700 1 -- 172.21.15.12:0/1719040024 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51e000fbb0 con 0x7f51f4105c90 2024-05-27T07:40:19.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.344+0000 7f51eb400700 1 -- 172.21.15.12:0/1719040024 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0017650 con 0x7f51f4105c90 2024-05-27T07:40:19.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.344+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f41a1f00 con 0x7f51f4105c90 2024-05-27T07:40:19.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.344+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f41a23e0 con 0x7f51f4105c90 2024-05-27T07:40:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.349+0000 7f51eb400700 1 -- 172.21.15.12:0/1719040024 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f51e002a070 con 0x7f51f4105c90 2024-05-27T07:40:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.349+0000 7f52084f3b80 1 Processor -- start 2024-05-27T07:40:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.349+0000 7f52084f3b80 1 -- start start 2024-05-27T07:40:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b81ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f52084f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564e63ab0e40 con 0x564e63b82820 2024-05-27T07:40:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f52084f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564e63aa09e0 con 0x564e63a03e40 2024-05-27T07:40:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f52084f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564e63aa4a20 con 0x564e63b82410 2024-05-27T07:40:19.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f51f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f51f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39886/0 (socket says 172.21.15.12:39886) 2024-05-27T07:40:19.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f51f2000700 1 -- 172.21.15.12:0/284536018 learned_addr learned my addr 172.21.15.12:0/284536018 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:19.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.350+0000 7f51f3400700 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b81ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51f2a00700 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51f2000700 1 -- 172.21.15.12:0/284536018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 msgr2=0x564e63b8cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51f2000700 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51f2000700 1 -- 172.21.15.12:0/284536018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 msgr2=0x564e63b81ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51f2000700 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b81ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51eb400700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f51d80b29f0 0x7f51d80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51f2000700 1 -- 172.21.15.12:0/284536018 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e63a69420 con 0x564e63b82820 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.351+0000 7f51eb400700 1 -- 172.21.15.12:0/1719040024 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7f51e00e3ca0 con 0x7f51f4105c90 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.352+0000 7f51f3400700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f51d80b29f0 0x7f51d80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.352+0000 7f51f2000700 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8f020 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f51e00e0ec0 tx=0x7f51e00a71f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ca5d09f01d56a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:19.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.352+0000 7f51dfe00700 1 -- 172.21.15.12:0/284536018 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00aa970 con 0x564e63b82820 2024-05-27T07:40:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.352+0000 7f51dfe00700 1 -- 172.21.15.12:0/284536018 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51e00aa6b0 con 0x564e63b82820 2024-05-27T07:40:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.352+0000 7f51dfe00700 1 -- 172.21.15.12:0/284536018 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00b82f0 con 0x564e63b82820 2024-05-27T07:40:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 -- 172.21.15.12:0/284536018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 msgr2=0x564e63b8f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f51f3400700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f51d80b29f0 0x7f51d80b4f00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f51e4005950 tx=0x7f51e400b310 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8f020 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f51e00e0ec0 tx=0x7f51e00a71f0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 -- 172.21.15.12:0/284536018 shutdown_connections 2024-05-27T07:40:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b81ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8f020 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 --2- 172.21.15.12:0/284536018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 -- 172.21.15.12:0/284536018 >> 172.21.15.12:0/284536018 conn(0x564e63a63fc0 msgr2=0x564e63b7dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 -- 172.21.15.12:0/284536018 shutdown_connections 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.353+0000 7f52084f3b80 1 -- 172.21.15.12:0/284536018 wait complete. 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.354+0000 7f52084f3b80 1 -- start start 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 1 Processor -- start 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b8c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564e639a6fa0 con 0x564e63b82820 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564e63a41760 con 0x564e63a03e40 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564e63b8f750 con 0x564e63b82410 2024-05-27T07:40:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.355+0000 7f52084f3b80 10 client.0 _ll_register_callbacks cb 0x564e63a4ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39888/0 (socket says 172.21.15.12:39888) 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 -- 172.21.15.12:0/2149477707 learned_addr learned my addr 172.21.15.12:0/2149477707 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f2000700 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f2a00700 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b8c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 -- 172.21.15.12:0/2149477707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 msgr2=0x564e63b8cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 -- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 msgr2=0x564e63b8c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b8c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.356+0000 7f51f3400700 1 -- 172.21.15.12:0/2149477707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e639f08d0 con 0x564e63b82820 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f51f3400700 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8d020 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f51e4005c50 tx=0x7f51e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199304ad9fe4c02f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f51dcc00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f51dcc00700 1 -- 172.21.15.12:0/2149477707 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4011aa0 con 0x564e63b82820 2024-05-27T07:40:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f51dcc00700 1 -- 172.21.15.12:0/2149477707 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7f51e4011c40 con 0x564e63b82820 2024-05-27T07:40:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e00aa6b0 con 0x564e63b82820 2024-05-27T07:40:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f51dcc00700 1 -- 172.21.15.12:0/2149477707 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f51e402e050 con 0x564e63b82820 2024-05-27T07:40:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f52084f3b80 10 client.17562 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.357+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x564e63a71060 con 0x564e63b82820 2024-05-27T07:40:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f51dcc00700 1 -- 172.21.15.12:0/2149477707 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51e00aa6b0 con 0x564e63b82820 2024-05-27T07:40:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f51dcc00700 1 -- 172.21.15.12:0/2149477707 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f51e401c3b0 con 0x564e63b82820 2024-05-27T07:40:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f52084f3b80 10 client.17562 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:19.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f52084f3b80 10 client.17562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f52084f3b80 20 client.17562 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f52084f3b80 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b9cae0 0x564e63b9ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f52084f3b80 4 client.17562 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.358+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e63ab2d90 con 0x564e63b9cae0 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.359+0000 7f51f2a00700 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b9cae0 0x564e63b9ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.360+0000 7f51f2a00700 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b9cae0 0x564e63b9ef90 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264cb12350d0c7b3 server_cookie=cdc2572aeccb96fa in_seq=0 out_seq=0 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.360+0000 7f51dcc00700 10 client.17562 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f51dcc00700 1 -- 172.21.15.12:0/2149477707 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564e63ab2d90 con 0x564e63b9cae0 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f51dcc00700 10 client.17562 handle_command_reply: tid=0 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 client.17562 shutdown 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 msgr2=0x564e63b8d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8d020 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f51e4005c50 tx=0x7f51e4004750 comp rx=0 tx=0).stop 2024-05-27T07:40:19.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 shutdown_connections 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b9cae0 0x564e63b9ef90 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a03e40 0x564e63b8c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82820 0x564e63b8d020 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 --2- 172.21.15.12:0/2149477707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63b82410 0x564e63b8cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 >> 172.21.15.12:0/2149477707 conn(0x564e63a63fc0 msgr2=0x564e63b7cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f51dcc00700 0 client.17562 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 shutdown_connections 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 1 -- 172.21.15.12:0/2149477707 wait complete. 2024-05-27T07:40:19.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.363+0000 7f52084f3b80 20 client.17562 trim_cache size 0 max 16384 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 Processor -- start 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 -- start start 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a63fc0 0x564e63b867c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63a03e40 0x564e63b8cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63b82410 0x564e63b8ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41b1c00 con 0x564e63a03e40 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41b0e60 con 0x564e63b82410 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.386+0000 7f52084f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41b1530 con 0x564e63a63fc0 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63a03e40 0x564e63b8cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63a03e40 0x564e63b8cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39890/0 (socket says 172.21.15.12:39890) 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 -- 172.21.15.12:0/1702359706 learned_addr learned my addr 172.21.15.12:0/1702359706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2a00700 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a63fc0 0x564e63b867c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f3400700 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63b82410 0x564e63b8ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 -- 172.21.15.12:0/1702359706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a63fc0 msgr2=0x564e63b867c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a63fc0 0x564e63b867c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 -- 172.21.15.12:0/1702359706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63b82410 msgr2=0x564e63b8ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63b82410 0x564e63b8ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.387+0000 7f51f2000700 1 -- 172.21.15.12:0/1702359706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e63b7f350 con 0x564e63a03e40 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.388+0000 7f51f2000700 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63a03e40 0x564e63b8cf90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f51e00a7360 tx=0x7f51e0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7579b3f73e0187 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.388+0000 7f51df400700 1 -- 172.21.15.12:0/1702359706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00b84f0 con 0x564e63a03e40 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.388+0000 7f51df400700 1 -- 172.21.15.12:0/1702359706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51e00aa6b0 con 0x564e63a03e40 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.388+0000 7f51df400700 1 -- 172.21.15.12:0/1702359706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00b8950 con 0x564e63a03e40 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 -- 172.21.15.12:0/1702359706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63a03e40 msgr2=0x564e63b8cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63a03e40 0x564e63b8cf90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f51e00a7360 tx=0x7f51e0009310 comp rx=0 tx=0).stop 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 -- 172.21.15.12:0/1702359706 shutdown_connections 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63b82410 0x564e63b8ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a63fc0 0x564e63b867c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 --2- 172.21.15.12:0/1702359706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63a03e40 0x564e63b8cf90 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 -- 172.21.15.12:0/1702359706 >> 172.21.15.12:0/1702359706 conn(0x564e63b76f60 msgr2=0x564e63b9eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 -- 172.21.15.12:0/1702359706 shutdown_connections 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.389+0000 7f52084f3b80 1 -- 172.21.15.12:0/1702359706 wait complete. 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.390+0000 7f52084f3b80 1 -- start start 2024-05-27T07:40:19.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.390+0000 7f52084f3b80 1 Processor -- start 2024-05-27T07:40:19.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a03e40 0x564e63b90710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a63fc0 0x564e63b90c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f52084f3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82410 0x564e63b91190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f52084f3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564e63b9f200 con 0x564e63b82410 2024-05-27T07:40:19.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f52084f3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564e63ab2d90 con 0x564e63a63fc0 2024-05-27T07:40:19.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f52084f3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564e63c06620 con 0x564e63a03e40 2024-05-27T07:40:19.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f52084f3b80 10 client.0 _ll_register_callbacks cb 0x564e63a4ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:19.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f51f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82410 0x564e63b91190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f51f3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82410 0x564e63b91190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39906/0 (socket says 172.21.15.12:39906) 2024-05-27T07:40:19.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f51f2a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a03e40 0x564e63b90710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.391+0000 7f51f3400700 1 -- 172.21.15.12:0/792745333 learned_addr learned my addr 172.21.15.12:0/792745333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.392+0000 7f51f2000700 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a63fc0 0x564e63b90c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.392+0000 7f51f3400700 1 -- 172.21.15.12:0/792745333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a03e40 msgr2=0x564e63b90710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.392+0000 7f51f3400700 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a03e40 0x564e63b90710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.392+0000 7f51f3400700 1 -- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a63fc0 msgr2=0x564e63b90c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.392+0000 7f51f3400700 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a63fc0 0x564e63b90c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.392+0000 7f51f3400700 1 -- 172.21.15.12:0/792745333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e63c067a0 con 0x564e63b82410 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f51f3400700 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82410 0x564e63b91190 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f51e401ca10 tx=0x7f51e401a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727c960d58ce0615 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f51dcc00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f51dcc00700 1 -- 172.21.15.12:0/792745333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e401cc90 con 0x564e63b82410 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f51dcc00700 1 -- 172.21.15.12:0/792745333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7f51e401a950 con 0x564e63b82410 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e00aa6b0 con 0x564e63b82410 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f51dcc00700 1 -- 172.21.15.12:0/792745333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f51e4043050 con 0x564e63b82410 2024-05-27T07:40:19.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f52084f3b80 10 client.17568 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.393+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x564e63c079c0 con 0x564e63b82410 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f51dcc00700 1 -- 172.21.15.12:0/792745333 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51e00aa6b0 con 0x564e63b82410 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f51dcc00700 1 -- 172.21.15.12:0/792745333 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f51e4011720 con 0x564e63b82410 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f52084f3b80 10 client.17568 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f52084f3b80 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f52084f3b80 20 client.17568 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f52084f3b80 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b82820 0x564e63c0a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f52084f3b80 4 client.17568 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:40:19.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.394+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e63aac570 con 0x564e63b82820 2024-05-27T07:40:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.395+0000 7f51f2a00700 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b82820 0x564e63c0a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.397+0000 7f51f2a00700 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b82820 0x564e63c0a0e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59fddfebecf9e120 server_cookie=6d2c01451360a6e in_seq=0 out_seq=0 2024-05-27T07:40:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.397+0000 7f51dcc00700 10 client.17568 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.397+0000 7f51dcc00700 1 -- 172.21.15.12:0/792745333 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564e63aac570 con 0x564e63b82820 2024-05-27T07:40:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.397+0000 7f51dcc00700 10 client.17568 handle_command_reply: tid=0 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.397+0000 7f52084f3b80 1 client.17568 shutdown 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.397+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82410 msgr2=0x564e63b91190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.397+0000 7f52084f3b80 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82410 0x564e63b91190 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f51e401ca10 tx=0x7f51e401a420 comp rx=0 tx=0).stop 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 shutdown_connections 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e63b82410 0x564e63b91190 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e63a03e40 0x564e63b90710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564e63b82820 0x564e63c0a0e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 --2- 172.21.15.12:0/792745333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e63a63fc0 0x564e63b90c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 >> 172.21.15.12:0/792745333 conn(0x564e63b76f60 msgr2=0x564e63b9eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f51dcc00700 0 client.17568 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 shutdown_connections 2024-05-27T07:40:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 1 -- 172.21.15.12:0/792745333 wait complete. 2024-05-27T07:40:19.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.398+0000 7f52084f3b80 20 client.17568 trim_cache size 0 max 16384 2024-05-27T07:40:19.398 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:40:19.398 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout: "fe6a6aa0-94a7-44dc-b9d0-671736eb91f3": { 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "fe6a6aa0-94a7-44dc-b9d0-671736eb91f3", 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:40:19.399 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.399+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f51d80b29f0 msgr2=0x7f51d80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.399+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f51d80b29f0 0x7f51d80b4f00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f51e4005950 tx=0x7f51e400b310 comp rx=0 tx=0).stop 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.399+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105c90 msgr2=0x7f51f419d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.399+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105c90 0x7f51f419d340 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f51e00057f0 tx=0x7f51e0009570 comp rx=0 tx=0).stop 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 shutdown_connections 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f51d80b29f0 0x7f51d80b4f00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51f410e720 0x7f51f41a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4105340 0x7f51f419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 --2- 172.21.15.12:0/1719040024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f51f4105c90 0x7f51f419d340 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 >> 172.21.15.12:0/1719040024 conn(0x7f51f4078ab0 msgr2=0x7f51f4103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 shutdown_connections 2024-05-27T07:40:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:19.400+0000 7f51f3e00700 1 -- 172.21.15.12:0/1719040024 wait complete. 2024-05-27T07:40:19.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:19 smithi162 ceph-mon[35676]: pgmap v612: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 241 op/s 2024-05-27T07:40:19.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:19 smithi062 ceph-mon[35304]: pgmap v612: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 241 op/s 2024-05-27T07:40:20.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:19 smithi012 ceph-mon[31473]: pgmap v612: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 241 op/s 2024-05-27T07:40:21.215 INFO:tasks.cephfs.filesystem:scrub status for tag:fe6a6aa0-94a7-44dc-b9d0-671736eb91f3 - {'path': '/', 'tag': 'fe6a6aa0-94a7-44dc-b9d0-671736eb91f3', 'options': 'recursive,force'} 2024-05-27T07:40:21.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:21 smithi162 ceph-mon[35676]: pgmap v613: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 241 op/s 2024-05-27T07:40:21.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:21 smithi062 ceph-mon[35304]: pgmap v613: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 241 op/s 2024-05-27T07:40:22.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:21 smithi012 ceph-mon[31473]: pgmap v613: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 241 op/s 2024-05-27T07:40:23.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:23 smithi162 ceph-mon[35676]: pgmap v614: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 222 op/s 2024-05-27T07:40:23.992 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:23 smithi062 ceph-mon[35304]: pgmap v614: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 222 op/s 2024-05-27T07:40:24.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:23 smithi012 ceph-mon[31473]: pgmap v614: 129 pgs: 129 active+clean; 426 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 222 op/s 2024-05-27T07:40:24.160 INFO:tasks.workunit.client.0.smithi012.stdout:fallocating to largest ever: 0x6400000 2024-05-27T07:40:24.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:24 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:24.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:24 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:25.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:24 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:25.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:25 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:25.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:25 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:25.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:25 smithi162 ceph-mon[35676]: pgmap v615: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 332 op/s 2024-05-27T07:40:25.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:25 smithi162 ceph-mon[35676]: pgmap v616: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-05-27T07:40:25.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:25 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:40:25.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:25 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:25.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:25 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:25.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:25 smithi062 ceph-mon[35304]: pgmap v615: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 332 op/s 2024-05-27T07:40:25.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:25 smithi062 ceph-mon[35304]: pgmap v616: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-05-27T07:40:25.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:25 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:40:26.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:25 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:26.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:25 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:26.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:25 smithi012 ceph-mon[31473]: pgmap v615: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 332 op/s 2024-05-27T07:40:26.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:25 smithi012 ceph-mon[31473]: pgmap v616: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-05-27T07:40:26.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:25 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:40:26.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:40:26.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:40:26.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:26 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:40:26.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:40:26.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:40:26.996 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:26 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:40:27.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:40:27.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:40:27.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:26 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:40:27.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:27 smithi162 ceph-mon[35676]: pgmap v617: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-05-27T07:40:27.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:27 smithi062 ceph-mon[35304]: pgmap v617: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-05-27T07:40:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:27 smithi012 ceph-mon[31473]: pgmap v617: 129 pgs: 129 active+clean; 427 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-05-27T07:40:30.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:30 smithi162 ceph-mon[35676]: pgmap v618: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:30.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:30 smithi062 ceph-mon[35304]: pgmap v618: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:30.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:30 smithi012 ceph-mon[31473]: pgmap v618: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:32.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:32 smithi162 ceph-mon[35676]: pgmap v619: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:32.742 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:32 smithi062 ceph-mon[35304]: pgmap v619: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:32.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:32 smithi012 ceph-mon[31473]: pgmap v619: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:34.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:34 smithi162 ceph-mon[35676]: pgmap v620: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:34.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:34 smithi062 ceph-mon[35304]: pgmap v620: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:34.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:34 smithi012 ceph-mon[31473]: pgmap v620: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:36.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:36 smithi162 ceph-mon[35676]: pgmap v621: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 231 op/s 2024-05-27T07:40:36.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:36 smithi062 ceph-mon[35304]: pgmap v621: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 231 op/s 2024-05-27T07:40:36.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:36 smithi012 ceph-mon[31473]: pgmap v621: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 231 op/s 2024-05-27T07:40:38.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:38 smithi162 ceph-mon[35676]: pgmap v622: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-05-27T07:40:38.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:38 smithi062 ceph-mon[35304]: pgmap v622: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-05-27T07:40:38.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:38 smithi012 ceph-mon[31473]: pgmap v622: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-05-27T07:40:40.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:40 smithi162 ceph-mon[35676]: pgmap v623: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 269 op/s 2024-05-27T07:40:40.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:40 smithi062 ceph-mon[35304]: pgmap v623: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 269 op/s 2024-05-27T07:40:40.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:40 smithi012 ceph-mon[31473]: pgmap v623: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 269 op/s 2024-05-27T07:40:42.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:42 smithi162 ceph-mon[35676]: pgmap v624: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:42.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:42 smithi062 ceph-mon[35304]: pgmap v624: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:42.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:42 smithi012 ceph-mon[31473]: pgmap v624: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:44 smithi162 ceph-mon[35676]: pgmap v625: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:44.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:44 smithi062 ceph-mon[35304]: pgmap v625: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:44.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:44 smithi012 ceph-mon[31473]: pgmap v625: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:46.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:46 smithi162 ceph-mon[35676]: pgmap v626: 129 pgs: 129 active+clean; 441 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:46.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:46 smithi062 ceph-mon[35304]: pgmap v626: 129 pgs: 129 active+clean; 441 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:46.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:46 smithi012 ceph-mon[31473]: pgmap v626: 129 pgs: 129 active+clean; 441 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 270 op/s 2024-05-27T07:40:48.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:48 smithi162 ceph-mon[35676]: pgmap v627: 129 pgs: 129 active+clean; 441 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:48.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:48 smithi062 ceph-mon[35304]: pgmap v627: 129 pgs: 129 active+clean; 441 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:48.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:48 smithi012 ceph-mon[31473]: pgmap v627: 129 pgs: 129 active+clean; 441 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-27T07:40:50.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:50 smithi162 ceph-mon[35676]: pgmap v628: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 257 op/s 2024-05-27T07:40:50.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:50 smithi062 ceph-mon[35304]: pgmap v628: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 257 op/s 2024-05-27T07:40:50.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:50 smithi012 ceph-mon[31473]: pgmap v628: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 257 op/s 2024-05-27T07:40:51.216 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:40:51.726 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:52.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:52 smithi162 ceph-mon[35676]: pgmap v629: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 180 op/s 2024-05-27T07:40:52.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:52 smithi062 ceph-mon[35304]: pgmap v629: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 180 op/s 2024-05-27T07:40:52.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:52 smithi012 ceph-mon[31473]: pgmap v629: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 180 op/s 2024-05-27T07:40:54.507 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:54 smithi012 ceph-mon[31473]: pgmap v630: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 180 op/s 2024-05-27T07:40:54.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 -- 172.21.15.12:0/4251513709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a0105da0 msgr2=0x7f46a010e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 --2- 172.21.15.12:0/4251513709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a0105da0 0x7f46a010e2f0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f468c009990 tx=0x7f468c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 -- 172.21.15.12:0/4251513709 shutdown_connections 2024-05-27T07:40:54.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 --2- 172.21.15.12:0/4251513709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46a010e8c0 0x7f46a0110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 --2- 172.21.15.12:0/4251513709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46a0105450 0x7f46a0105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 --2- 172.21.15.12:0/4251513709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a0105da0 0x7f46a010e2f0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 -- 172.21.15.12:0/4251513709 >> 172.21.15.12:0/4251513709 conn(0x7f46a00fedb0 msgr2=0x7f46a01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 -- 172.21.15.12:0/4251513709 shutdown_connections 2024-05-27T07:40:54.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.508+0000 7f46a5a00700 1 -- 172.21.15.12:0/4251513709 wait complete. 2024-05-27T07:40:54.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.509+0000 7f46a5a00700 1 Processor -- start 2024-05-27T07:40:54.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f46a5a00700 1 -- start start 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f46a5a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46a0105450 0x7f46a019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f46a5a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46a0105da0 0x7f46a019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f46a5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a010e8c0 0x7f46a019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f46a5a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0114390 con 0x7f46a010e8c0 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f46a5a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0114210 con 0x7f46a0105da0 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f46a5a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0114510 con 0x7f46a0105450 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f469f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a010e8c0 0x7f46a019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f469f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a010e8c0 0x7f46a019f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34908/0 (socket says 172.21.15.12:34908) 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.510+0000 7f469f400700 1 -- 172.21.15.12:0/1510659226 learned_addr learned my addr 172.21.15.12:0/1510659226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:54.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469e000700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46a0105da0 0x7f46a019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469ea00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46a0105450 0x7f46a019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469f400700 1 -- 172.21.15.12:0/1510659226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46a0105450 msgr2=0x7f46a019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469f400700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46a0105450 0x7f46a019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469f400700 1 -- 172.21.15.12:0/1510659226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46a0105da0 msgr2=0x7f46a019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469f400700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46a0105da0 0x7f46a019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469f400700 1 -- 172.21.15.12:0/1510659226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a019fc90 con 0x7f46a010e8c0 2024-05-27T07:40:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.511+0000 7f469f400700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a010e8c0 0x7f46a019f5f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f469800ef30 tx=0x7f469800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.512+0000 7f4697400700 1 -- 172.21.15.12:0/1510659226 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46980092e0 con 0x7f46a010e8c0 2024-05-27T07:40:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.512+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a019ff20 con 0x7f46a010e8c0 2024-05-27T07:40:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.512+0000 7f4697400700 1 -- 172.21.15.12:0/1510659226 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4698004bb0 con 0x7f46a010e8c0 2024-05-27T07:40:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.512+0000 7f4697400700 1 -- 172.21.15.12:0/1510659226 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698004080 con 0x7f46a010e8c0 2024-05-27T07:40:54.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.512+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a01a78a0 con 0x7f46a010e8c0 2024-05-27T07:40:54.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.518+0000 7f46b44d3b80 1 Processor -- start 2024-05-27T07:40:54.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.519+0000 7f4697400700 1 -- 172.21.15.12:0/1510659226 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f469801b030 con 0x7f46a010e8c0 2024-05-27T07:40:54.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.519+0000 7f46b44d3b80 1 -- start start 2024-05-27T07:40:54.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.519+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5299f0 0x5556af5cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.519+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af43de40 0x5556af5cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5bcf30 0x5556af5bd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5556af4eae40 con 0x5556af5bcf30 2024-05-27T07:40:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5556af4da9e0 con 0x5556af43de40 2024-05-27T07:40:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556af4dea20 con 0x5556af5299f0 2024-05-27T07:40:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f469e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5bcf30 0x5556af5bd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f469ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af43de40 0x5556af5cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f469e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5bcf30 0x5556af5bd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34922/0 (socket says 172.21.15.12:34922) 2024-05-27T07:40:54.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.520+0000 7f469e000700 1 -- 172.21.15.12:0/2078832753 learned_addr learned my addr 172.21.15.12:0/2078832753 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:54.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.521+0000 7f469e000700 1 -- 172.21.15.12:0/2078832753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5299f0 msgr2=0x5556af5cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.521+0000 7f469e000700 1 --2- 172.21.15.12:0/2078832753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5299f0 0x5556af5cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.521+0000 7f469e000700 1 -- 172.21.15.12:0/2078832753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af43de40 msgr2=0x5556af5cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.521+0000 7f469e000700 1 --2- 172.21.15.12:0/2078832753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af43de40 0x5556af5cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.521+0000 7f469e000700 1 -- 172.21.15.12:0/2078832753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556af4a3420 con 0x5556af5bcf30 2024-05-27T07:40:54.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.522+0000 7f469e000700 1 --2- 172.21.15.12:0/2078832753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5bcf30 0x5556af5bd3e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f468c005f50 tx=0x7f468c009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a17bba2efe16501 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.523+0000 7f4697400700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f46840b2a70 0x7f46840b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.523+0000 7f4697400700 1 -- 172.21.15.12:0/1510659226 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7f46980e2600 con 0x7f46a010e8c0 2024-05-27T07:40:54.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.523+0000 7f468be00700 1 -- 172.21.15.12:0/2078832753 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c003680 con 0x5556af5bcf30 2024-05-27T07:40:54.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.523+0000 7f468be00700 1 -- 172.21.15.12:0/2078832753 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f468c01f070 con 0x5556af5bcf30 2024-05-27T07:40:54.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.523+0000 7f468be00700 1 -- 172.21.15.12:0/2078832753 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c003680 con 0x5556af5bcf30 2024-05-27T07:40:54.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.524+0000 7f469ea00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f46840b2a70 0x7f46840b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.524+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2078832753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5bcf30 msgr2=0x5556af5bd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.524+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2078832753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5bcf30 0x5556af5bd3e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f468c005f50 tx=0x7f468c009650 comp rx=0 tx=0).stop 2024-05-27T07:40:54.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.525+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2078832753 shutdown_connections 2024-05-27T07:40:54.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.525+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2078832753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5bcf30 0x5556af5bd3e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.525+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2078832753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5299f0 0x5556af5cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.525+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2078832753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af43de40 0x5556af5cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.525+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2078832753 >> 172.21.15.12:0/2078832753 conn(0x5556af4d7390 msgr2=0x5556af5b80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.526+0000 7f469ea00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f46840b2a70 0x7f46840b4f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4690006fd0 tx=0x7f4690008040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.526+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2078832753 shutdown_connections 2024-05-27T07:40:54.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.526+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2078832753 wait complete. 2024-05-27T07:40:54.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.527+0000 7f46b44d3b80 1 -- start start 2024-05-27T07:40:54.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.527+0000 7f46b44d3b80 1 Processor -- start 2024-05-27T07:40:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5299f0 0x5556af5cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5556af3e0fa0 con 0x5556af5299f0 2024-05-27T07:40:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5556af47b760 con 0x5556af5bcf30 2024-05-27T07:40:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556af5d2500 con 0x5556af43de40 2024-05-27T07:40:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f46b44d3b80 10 client.0 _ll_register_callbacks cb 0x5556af484c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f469e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5299f0 0x5556af5cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f469e000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5299f0 0x5556af5cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34928/0 (socket says 172.21.15.12:34928) 2024-05-27T07:40:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f469e000700 1 -- 172.21.15.12:0/3267391771 learned_addr learned my addr 172.21.15.12:0/3267391771 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.528+0000 7f469ea00700 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.529+0000 7f469f400700 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.529+0000 7f469e000700 1 -- 172.21.15.12:0/3267391771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 msgr2=0x5556af5cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.529+0000 7f469e000700 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.529+0000 7f469e000700 1 -- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 msgr2=0x5556af5cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.529+0000 7f469e000700 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.529+0000 7f469e000700 1 -- 172.21.15.12:0/3267391771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556af42a8d0 con 0x5556af5299f0 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.529+0000 7f469e000700 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5299f0 0x5556af5cfa10 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f468c0098e0 tx=0x7f468c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d248d5c28e73b9e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.530+0000 7f4688c00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.530+0000 7f4688c00700 1 -- 172.21.15.12:0/3267391771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c003680 con 0x5556af5299f0 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.530+0000 7f4688c00700 1 -- 172.21.15.12:0/3267391771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7f468c003820 con 0x5556af5299f0 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.530+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f468c01f070 con 0x5556af5299f0 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.531+0000 7f4688c00700 1 -- 172.21.15.12:0/3267391771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f468c003a20 con 0x5556af5299f0 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.531+0000 7f46b44d3b80 10 client.17580 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:54.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.531+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5556af4ab060 con 0x5556af5299f0 2024-05-27T07:40:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f4688c00700 1 -- 172.21.15.12:0/3267391771 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f468c01f070 con 0x5556af5299f0 2024-05-27T07:40:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f4688c00700 1 -- 172.21.15.12:0/3267391771 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f468c02c460 con 0x5556af5299f0 2024-05-27T07:40:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f46b44d3b80 10 client.17580 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f46b44d3b80 10 client.17580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f46b44d3b80 20 client.17580 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af5df310 0x5556af5e1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f46b44d3b80 4 client.17580 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.532+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556af4ecd90 con 0x5556af5df310 2024-05-27T07:40:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.533+0000 7f469ea00700 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af5df310 0x5556af5e1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.533+0000 7f469ea00700 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af5df310 0x5556af5e1750 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f2430d0457191d6 server_cookie=2581f1f45660764d in_seq=0 out_seq=0 2024-05-27T07:40:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.533+0000 7f4688c00700 10 client.17580 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f4688c00700 1 -- 172.21.15.12:0/3267391771 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5556af4ecd90 con 0x5556af5df310 2024-05-27T07:40:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f4688c00700 10 client.17580 handle_command_reply: tid=0 2024-05-27T07:40:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 client.17580 shutdown 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5299f0 msgr2=0x5556af5cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5299f0 0x5556af5cfa10 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f468c0098e0 tx=0x7f468c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 shutdown_connections 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af5df310 0x5556af5e1750 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/3267391771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5299f0 0x5556af5cfa10 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 >> 172.21.15.12:0/3267391771 conn(0x5556af4d7390 msgr2=0x5556af5b7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.535+0000 7f4688c00700 0 client.17580 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.536+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 shutdown_connections 2024-05-27T07:40:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.536+0000 7f46b44d3b80 1 -- 172.21.15.12:0/3267391771 wait complete. 2024-05-27T07:40:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.536+0000 7f46b44d3b80 20 client.17580 trim_cache size 0 max 16384 2024-05-27T07:40:54.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.562+0000 7f46b44d3b80 1 Processor -- start 2024-05-27T07:40:54.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.562+0000 7f46b44d3b80 1 -- start start 2024-05-27T07:40:54.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af43de40 0x5556af5cb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5bcf30 0x5556af5db560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5cd260 0x5556af5bbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01ada60 con 0x5556af43de40 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01accc0 con 0x5556af5cd260 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01ad390 con 0x5556af5bcf30 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f469ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af43de40 0x5556af5cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f469ea00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af43de40 0x5556af5cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34938/0 (socket says 172.21.15.12:34938) 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f469ea00700 1 -- 172.21.15.12:0/1710666915 learned_addr learned my addr 172.21.15.12:0/1710666915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:54.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.563+0000 7f469f400700 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5cd260 0x5556af5bbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.564+0000 7f469ea00700 1 -- 172.21.15.12:0/1710666915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5bcf30 msgr2=0x5556af5db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.564+0000 7f469ea00700 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5bcf30 0x5556af5db560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.564+0000 7f469ea00700 1 -- 172.21.15.12:0/1710666915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5cd260 msgr2=0x5556af5bbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.564+0000 7f469ea00700 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5cd260 0x5556af5bbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.564+0000 7f469ea00700 1 -- 172.21.15.12:0/1710666915 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556af5d02c0 con 0x5556af43de40 2024-05-27T07:40:54.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.564+0000 7f469ea00700 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af43de40 0x5556af5cb020 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x5556af5b9720 tx=0x7f4690012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4294d6f669075000 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.565+0000 7f468b400700 1 -- 172.21.15.12:0/1710666915 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690030070 con 0x5556af43de40 2024-05-27T07:40:54.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.565+0000 7f468b400700 1 -- 172.21.15.12:0/1710666915 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4690014bb0 con 0x5556af43de40 2024-05-27T07:40:54.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.565+0000 7f468b400700 1 -- 172.21.15.12:0/1710666915 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469001b040 con 0x5556af43de40 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 -- 172.21.15.12:0/1710666915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af43de40 msgr2=0x5556af5cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af43de40 0x5556af5cb020 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x5556af5b9720 tx=0x7f4690012000 comp rx=0 tx=0).stop 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 -- 172.21.15.12:0/1710666915 shutdown_connections 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5cd260 0x5556af5bbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af43de40 0x5556af5cb020 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/1710666915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af5bcf30 0x5556af5db560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 -- 172.21.15.12:0/1710666915 >> 172.21.15.12:0/1710666915 conn(0x5556af4d7390 msgr2=0x5556af5cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.566+0000 7f46b44d3b80 1 -- 172.21.15.12:0/1710666915 shutdown_connections 2024-05-27T07:40:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.567+0000 7f46b44d3b80 1 -- 172.21.15.12:0/1710666915 wait complete. 2024-05-27T07:40:54.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.567+0000 7f46b44d3b80 1 -- start start 2024-05-27T07:40:54.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 1 Processor -- start 2024-05-27T07:40:54.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5d7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5d7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5d4720 0x5556af5d4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5556af4ecd90 con 0x5556af5d4720 2024-05-27T07:40:54.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5556af5ce4d0 con 0x5556af5bcf30 2024-05-27T07:40:54.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556af5b4950 con 0x5556af43de40 2024-05-27T07:40:54.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.568+0000 7f46b44d3b80 10 client.0 _ll_register_callbacks cb 0x5556af484c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:54.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5d7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5d7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:58222/0 (socket says 172.21.15.12:58222) 2024-05-27T07:40:54.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 -- 172.21.15.12:0/2073927779 learned_addr learned my addr 172.21.15.12:0/2073927779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:54.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469e000700 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5d7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469f400700 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5d4720 0x5556af5d4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 -- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 msgr2=0x5556af5d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5d7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 -- 172.21.15.12:0/2073927779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5d4720 msgr2=0x5556af5d4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5d4720 0x5556af5d4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 -- 172.21.15.12:0/2073927779 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556af5d8370 con 0x5556af43de40 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f469ea00700 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5d7750 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4690004790 tx=0x7f4690014df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbbb418816ac98b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.569+0000 7f4688c00700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.570+0000 7f4688c00700 1 -- 172.21.15.12:0/2073927779 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690039060 con 0x5556af43de40 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.570+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4690014bb0 con 0x5556af43de40 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.570+0000 7f4688c00700 1 -- 172.21.15.12:0/2073927779 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7f4690014520 con 0x5556af43de40 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.570+0000 7f4688c00700 1 -- 172.21.15.12:0/2073927779 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4690040050 con 0x5556af43de40 2024-05-27T07:40:54.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.570+0000 7f46b44d3b80 10 client.26947 fetch_fsmap learned FSMap version 119 2024-05-27T07:40:54.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.570+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5556af64a800 con 0x5556af43de40 2024-05-27T07:40:54.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f4688c00700 1 -- 172.21.15.12:0/2073927779 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4690014bb0 con 0x5556af43de40 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f4688c00700 1 -- 172.21.15.12:0/2073927779 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f469002a030 con 0x5556af43de40 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f46b44d3b80 10 client.26947 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f46b44d3b80 10 client.26947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f46b44d3b80 20 client.26947 populate_metadata read hostname 'smithi012' 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af64acb0 0x5556af64d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f46b44d3b80 4 client.26947 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5556af5b7120 con 0x5556af64acb0 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.571+0000 7f469e000700 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af64acb0 0x5556af64d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.572+0000 7f469e000700 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af64acb0 0x5556af64d0d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d81a961e92846e79 server_cookie=4c30fc9097684805 in_seq=0 out_seq=0 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.572+0000 7f4688c00700 10 client.26947 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f4688c00700 1 -- 172.21.15.12:0/2073927779 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5556af5b7120 con 0x5556af64acb0 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f4688c00700 10 client.26947 handle_command_reply: tid=0 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 client.26947 shutdown 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 msgr2=0x5556af5d7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5d7750 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4690004790 tx=0x7f4690014df0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 shutdown_connections 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5556af5d4720 0x5556af5d4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556af43de40 0x5556af5d7750 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5556af64acb0 0x5556af64d0d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 --2- 172.21.15.12:0/2073927779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556af5bcf30 0x5556af5d7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 >> 172.21.15.12:0/2073927779 conn(0x5556af4d7390 msgr2=0x5556af5cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.573+0000 7f4688c00700 0 client.26947 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:40:54.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.574+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 shutdown_connections 2024-05-27T07:40:54.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.574+0000 7f46b44d3b80 1 -- 172.21.15.12:0/2073927779 wait complete. 2024-05-27T07:40:54.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.574+0000 7f46b44d3b80 20 client.26947 trim_cache size 0 max 16384 2024-05-27T07:40:54.574 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:40:54.575 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:40:54.575 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:40:54.575 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:40:54.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.574+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f46840b2a70 msgr2=0x7f46840b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.574+0000 7f46a5a00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f46840b2a70 0x7f46840b4f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4690006fd0 tx=0x7f4690008040 comp rx=0 tx=0).stop 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.574+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a010e8c0 msgr2=0x7f46a019f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.574+0000 7f46a5a00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a010e8c0 0x7f46a019f5f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f469800ef30 tx=0x7f469800c450 comp rx=0 tx=0).stop 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 shutdown_connections 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f46840b2a70 0x7f46840b4f80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a010e8c0 0x7f46a019f5f0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46a0105450 0x7f46a019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 --2- 172.21.15.12:0/1510659226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46a0105da0 0x7f46a019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 >> 172.21.15.12:0/1510659226 conn(0x7f46a00fedb0 msgr2=0x7f46a00ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 shutdown_connections 2024-05-27T07:40:54.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:54.575+0000 7f46a5a00700 1 -- 172.21.15.12:0/1510659226 wait complete. 2024-05-27T07:40:54.621 INFO:tasks.workunit.client.0.smithi012.stdout:40000 falloc from 0x4cbeead to 0x4cc78f9 (0x8a4c bytes) 2024-05-27T07:40:54.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:54 smithi162 ceph-mon[35676]: pgmap v630: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 180 op/s 2024-05-27T07:40:54.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:54 smithi062 ceph-mon[35304]: pgmap v630: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 180 op/s 2024-05-27T07:40:55.659 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:40:55.659 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:40:56.142 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:56.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:56 smithi012 ceph-mon[31473]: pgmap v631: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-05-27T07:40:56.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:56 smithi062 ceph-mon[35304]: pgmap v631: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-05-27T07:40:56.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:56 smithi162 ceph-mon[35676]: pgmap v631: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.650+0000 7f6f61c00700 1 -- 172.21.15.12:0/1267747028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106ec0 msgr2=0x7f6f5c113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.650+0000 7f6f61c00700 1 --2- 172.21.15.12:0/1267747028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c113350 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009f90 tx=0x7f6f5400b330 comp rx=0 tx=0).stop 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.651+0000 7f6f61c00700 1 -- 172.21.15.12:0/1267747028 shutdown_connections 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.651+0000 7f6f61c00700 1 --2- 172.21.15.12:0/1267747028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c113350 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.651+0000 7f6f61c00700 1 --2- 172.21.15.12:0/1267747028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f5c10c5b0 0x7f6f5c10c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.651+0000 7f6f61c00700 1 --2- 172.21.15.12:0/1267747028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106570 0x7f6f5c106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.651+0000 7f6f61c00700 1 -- 172.21.15.12:0/1267747028 >> 172.21.15.12:0/1267747028 conn(0x7f6f5c102070 msgr2=0x7f6f5c1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.651+0000 7f6f61c00700 1 -- 172.21.15.12:0/1267747028 shutdown_connections 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.651+0000 7f6f61c00700 1 -- 172.21.15.12:0/1267747028 wait complete. 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.653+0000 7f6f61c00700 1 Processor -- start 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.653+0000 7f6f61c00700 1 -- start start 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.653+0000 7f6f61c00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106570 0x7f6f5c19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.653+0000 7f6f61c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f61c00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f5c10c5b0 0x7f6f5c1a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f61c00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1165e0 con 0x7f6f5c10c5b0 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f61c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c116460 con 0x7f6f5c106ec0 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f61c00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c116760 con 0x7f6f5c106570 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f5a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f5a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:41980/0 (socket says 172.21.15.12:41980) 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f5a000700 1 -- 172.21.15.12:0/3717007768 learned_addr learned my addr 172.21.15.12:0/3717007768 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f5b400700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f5c10c5b0 0x7f6f5c1a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.654+0000 7f6f5aa00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106570 0x7f6f5c19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.655+0000 7f6f5a000700 1 -- 172.21.15.12:0/3717007768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106570 msgr2=0x7f6f5c19cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.655+0000 7f6f5a000700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106570 0x7f6f5c19cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.655+0000 7f6f5a000700 1 -- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f5c10c5b0 msgr2=0x7f6f5c1a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.655+0000 7f6f5a000700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f5c10c5b0 0x7f6f5c1a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.655+0000 7f6f5a000700 1 -- 172.21.15.12:0/3717007768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5c1a1de0 con 0x7f6f5c106ec0 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.655+0000 7f6f5aa00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106570 0x7f6f5c19cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.655+0000 7f6f5a000700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c19d4b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800d950 tx=0x7f6f4800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.656+0000 7f6f53400700 1 -- 172.21.15.12:0/3717007768 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48017070 con 0x7f6f5c106ec0 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.656+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f5c1a20d0 con 0x7f6f5c106ec0 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.656+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f5c1a99f0 con 0x7f6f5c106ec0 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.656+0000 7f6f53400700 1 -- 172.21.15.12:0/3717007768 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f4800fd20 con 0x7f6f5c106ec0 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.656+0000 7f6f53400700 1 -- 172.21.15.12:0/3717007768 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48021060 con 0x7f6f5c106ec0 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.657+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f3c005330 con 0x7f6f5c106ec0 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.661+0000 7f6f53400700 1 -- 172.21.15.12:0/3717007768 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6f4801a070 con 0x7f6f5c106ec0 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.663+0000 7f6f53400700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6f400b29f0 0x7f6f400b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.663+0000 7f6f53400700 1 -- 172.21.15.12:0/3717007768 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7f6f480e2100 con 0x7f6f5c106ec0 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.663+0000 7f6f5aa00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6f400b29f0 0x7f6f400b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.664+0000 7f6f5aa00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6f400b29f0 0x7f6f400b4f00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009750 tx=0x7f6f4c009310 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:58.669+0000 7f6f53400700 1 -- 172.21.15.12:0/3717007768 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6f480a9670 con 0x7f6f5c106ec0 2024-05-27T07:40:58.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:58 smithi012 ceph-mon[31473]: pgmap v632: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 174 op/s 2024-05-27T07:40:58.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:58 smithi162 ceph-mon[35676]: pgmap v632: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 174 op/s 2024-05-27T07:40:58.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:58 smithi062 ceph-mon[35304]: pgmap v632: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 174 op/s 2024-05-27T07:40:59.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.040+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f3c005620 con 0x7f6f5c106ec0 2024-05-27T07:40:59.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.041+0000 7f6f53400700 1 -- 172.21.15.12:0/3717007768 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4410 (secure 0 0 0) 0x7f6f480a8f70 con 0x7f6f5c106ec0 2024-05-27T07:40:59.040 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:40:59.040 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:30:55.313625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34439,34475,34415]},"id":1}]} 2024-05-27T07:40:59.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.044+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6f400b29f0 msgr2=0x7f6f400b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:59.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.044+0000 7f6f61c00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6f400b29f0 0x7f6f400b4f00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009750 tx=0x7f6f4c009310 comp rx=0 tx=0).stop 2024-05-27T07:40:59.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.044+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106ec0 msgr2=0x7f6f5c19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:59.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.044+0000 7f6f61c00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c19d4b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800d950 tx=0x7f6f4800dc60 comp rx=0 tx=0).stop 2024-05-27T07:40:59.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 shutdown_connections 2024-05-27T07:40:59.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f6f400b29f0 0x7f6f400b4f00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:59.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f5c10c5b0 0x7f6f5c1a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:59.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f5c106570 0x7f6f5c19cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:59.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 --2- 172.21.15.12:0/3717007768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f5c106ec0 0x7f6f5c19d4b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:59.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 >> 172.21.15.12:0/3717007768 conn(0x7f6f5c102070 msgr2=0x7f6f5c1047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:59.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 shutdown_connections 2024-05-27T07:40:59.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:40:59.045+0000 7f6f61c00700 1 -- 172.21.15.12:0/3717007768 wait complete. 2024-05-27T07:40:59.044 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 119 2024-05-27T07:40:59.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:40:59 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3717007768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:59.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:40:59 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3717007768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:59.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:40:59 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3717007768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:41:00.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:41:00.506 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:00.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:00 smithi012 ceph-mon[31473]: pgmap v633: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:41:00.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:00 smithi162 ceph-mon[35676]: pgmap v633: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:41:00.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:00 smithi062 ceph-mon[35304]: pgmap v633: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 287 op/s 2024-05-27T07:41:02.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:02 smithi012 ceph-mon[31473]: pgmap v634: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:02.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:02 smithi162 ceph-mon[35676]: pgmap v634: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:02.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:02 smithi062 ceph-mon[35304]: pgmap v634: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.078+0000 7efd9d200700 1 -- 172.21.15.12:0/2445502747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105da0 msgr2=0x7efd9810e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.078+0000 7efd9d200700 1 --2- 172.21.15.12:0/2445502747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105da0 0x7efd9810e2f0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7efd84009990 tx=0x7efd84009ca0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.079+0000 7efd9d200700 1 -- 172.21.15.12:0/2445502747 shutdown_connections 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.079+0000 7efd9d200700 1 --2- 172.21.15.12:0/2445502747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd9810e8c0 0x7efd98110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.079+0000 7efd9d200700 1 --2- 172.21.15.12:0/2445502747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd98105450 0x7efd98105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.079+0000 7efd9d200700 1 --2- 172.21.15.12:0/2445502747 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105da0 0x7efd9810e2f0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.079+0000 7efd9d200700 1 -- 172.21.15.12:0/2445502747 >> 172.21.15.12:0/2445502747 conn(0x7efd980fedb0 msgr2=0x7efd981011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.079+0000 7efd9d200700 1 -- 172.21.15.12:0/2445502747 shutdown_connections 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.079+0000 7efd9d200700 1 -- 172.21.15.12:0/2445502747 wait complete. 2024-05-27T07:41:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.080+0000 7efd9d200700 1 Processor -- start 2024-05-27T07:41:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.080+0000 7efd9d200700 1 -- start start 2024-05-27T07:41:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.081+0000 7efd9d200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105450 0x7efd981a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.081+0000 7efd9d200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd98105da0 0x7efd981a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.081+0000 7efd9d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd9810e8c0 0x7efd9819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.081+0000 7efd9d200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efd981142d0 con 0x7efd98105450 2024-05-27T07:41:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.081+0000 7efd9d200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efd98114150 con 0x7efd9810e8c0 2024-05-27T07:41:03.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.081+0000 7efd9d200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efd98114450 con 0x7efd98105da0 2024-05-27T07:41:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.081+0000 7efd95e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105450 0x7efd981a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105450 0x7efd981a09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42744/0 (socket says 172.21.15.12:42744) 2024-05-27T07:41:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95e00700 1 -- 172.21.15.12:0/1067942159 learned_addr learned my addr 172.21.15.12:0/1067942159 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95400700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd98105da0 0x7efd981a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd96800700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd9810e8c0 0x7efd9819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95e00700 1 -- 172.21.15.12:0/1067942159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd98105da0 msgr2=0x7efd981a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95e00700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd98105da0 0x7efd981a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95e00700 1 -- 172.21.15.12:0/1067942159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd9810e8c0 msgr2=0x7efd9819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95e00700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd9810e8c0 0x7efd9819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd95e00700 1 -- 172.21.15.12:0/1067942159 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9819b280 con 0x7efd98105450 2024-05-27T07:41:03.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.082+0000 7efd96800700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd9810e8c0 0x7efd9819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.083+0000 7efd95e00700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105450 0x7efd981a09c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7efd8800d950 tx=0x7efd8800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.083+0000 7efd8ea00700 1 -- 172.21.15.12:0/1067942159 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88016070 con 0x7efd98105450 2024-05-27T07:41:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.083+0000 7efd8ea00700 1 -- 172.21.15.12:0/1067942159 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd88010460 con 0x7efd98105450 2024-05-27T07:41:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.083+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd9819b570 con 0x7efd98105450 2024-05-27T07:41:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.083+0000 7efd8ea00700 1 -- 172.21.15.12:0/1067942159 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8800f620 con 0x7efd98105450 2024-05-27T07:41:03.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.083+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd981a7770 con 0x7efd98105450 2024-05-27T07:41:03.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.087+0000 7efd8ea00700 1 -- 172.21.15.12:0/1067942159 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7efd88019070 con 0x7efd98105450 2024-05-27T07:41:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 Processor -- start 2024-05-27T07:41:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 -- start start 2024-05-27T07:41:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efd8ea00700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7efd7c0b2a70 0x7efd7c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fa83330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa83870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa90110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56396f9b1e40 con 0x56396f9f09f0 2024-05-27T07:41:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56396f9a19e0 con 0x56396f904e40 2024-05-27T07:41:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efdabc88b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56396f9a5a20 con 0x56396fa83e70 2024-05-27T07:41:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.089+0000 7efd8ea00700 1 -- 172.21.15.12:0/1067942159 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7efd880e2620 con 0x7efd98105450 2024-05-27T07:41:03.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.090+0000 7efd95e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fa83330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.090+0000 7efd95e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fa83330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42760/0 (socket says 172.21.15.12:42760) 2024-05-27T07:41:03.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.090+0000 7efd95e00700 1 -- 172.21.15.12:0/903501172 learned_addr learned my addr 172.21.15.12:0/903501172 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:03.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.090+0000 7efd95400700 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa83870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.090+0000 7efd96800700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7efd7c0b2a70 0x7efd7c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.090+0000 7efd96800700 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa90110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd95e00700 1 -- 172.21.15.12:0/903501172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 msgr2=0x56396fa90110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd95e00700 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa90110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd95e00700 1 -- 172.21.15.12:0/903501172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 msgr2=0x56396fa83870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd95e00700 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa83870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd95e00700 1 -- 172.21.15.12:0/903501172 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56396f96a420 con 0x56396f9f09f0 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd96800700 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa90110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd95e00700 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fa83330 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7efd8800e010 tx=0x7efd880a9020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430fc722757782dc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd96800700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7efd7c0b2a70 0x7efd7c0b4f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efd900094f0 tx=0x7efd90009130 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:03.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd83400700 1 -- 172.21.15.12:0/903501172 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88020070 con 0x56396f9f09f0 2024-05-27T07:41:03.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.091+0000 7efd83400700 1 -- 172.21.15.12:0/903501172 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd880a9490 con 0x56396f9f09f0 2024-05-27T07:41:03.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.092+0000 7efd83400700 1 -- 172.21.15.12:0/903501172 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd880a98f0 con 0x56396f9f09f0 2024-05-27T07:41:03.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.092+0000 7efdabc88b80 1 -- 172.21.15.12:0/903501172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 msgr2=0x56396fa83330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.092+0000 7efdabc88b80 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fa83330 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7efd8800e010 tx=0x7efd880a9020 comp rx=0 tx=0).stop 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 -- 172.21.15.12:0/903501172 shutdown_connections 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa90110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fa83330 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 --2- 172.21.15.12:0/903501172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa83870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 -- 172.21.15.12:0/903501172 >> 172.21.15.12:0/903501172 conn(0x56396f99e390 msgr2=0x56396fa7f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 -- 172.21.15.12:0/903501172 shutdown_connections 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 -- 172.21.15.12:0/903501172 wait complete. 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.093+0000 7efdabc88b80 1 -- start start 2024-05-27T07:41:03.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.094+0000 7efdabc88b80 1 Processor -- start 2024-05-27T07:41:03.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.094+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f904e40 0x56396fa8d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.094+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f9f09f0 0x56396fa8dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.094+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa8e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.094+0000 7efdabc88b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56396f8a7fa0 con 0x56396fa83e70 2024-05-27T07:41:03.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.094+0000 7efdabc88b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56396f942760 con 0x56396f9f09f0 2024-05-27T07:41:03.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.094+0000 7efdabc88b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56396fa90b20 con 0x56396f904e40 2024-05-27T07:41:03.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.095+0000 7efdabc88b80 10 client.0 _ll_register_callbacks cb 0x56396f94bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:03.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.095+0000 7efd96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa8e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.095+0000 7efd96800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa8e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42768/0 (socket says 172.21.15.12:42768) 2024-05-27T07:41:03.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.095+0000 7efd96800700 1 -- 172.21.15.12:0/3423466082 learned_addr learned my addr 172.21.15.12:0/3423466082 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.095+0000 7efd95e00700 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f904e40 0x56396fa8d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.095+0000 7efd95400700 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f9f09f0 0x56396fa8dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.096+0000 7efd96800700 1 -- 172.21.15.12:0/3423466082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f904e40 msgr2=0x56396fa8d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.096+0000 7efd96800700 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f904e40 0x56396fa8d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.096+0000 7efd96800700 1 -- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f9f09f0 msgr2=0x56396fa8dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.096+0000 7efd96800700 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f9f09f0 0x56396fa8dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.096+0000 7efd96800700 1 -- 172.21.15.12:0/3423466082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56396f8f18d0 con 0x56396fa83e70 2024-05-27T07:41:03.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.097+0000 7efd96800700 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa8e100 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7efd90019ad0 tx=0x7efd90004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d968463934940c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:03.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.097+0000 7efd77e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.097+0000 7efd77e00700 1 -- 172.21.15.12:0/3423466082 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd90017790 con 0x56396fa83e70 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.097+0000 7efd77e00700 1 -- 172.21.15.12:0/3423466082 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7efd90012070 con 0x56396fa83e70 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.097+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd880a9490 con 0x56396fa83e70 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efd77e00700 1 -- 172.21.15.12:0/3423466082 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7efd9002f050 con 0x56396fa83e70 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 10 client.17604 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56396f972060 con 0x56396fa83e70 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efd77e00700 1 -- 172.21.15.12:0/3423466082 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd880a9490 con 0x56396fa83e70 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efd77e00700 1 -- 172.21.15.12:0/3423466082 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7efd90024870 con 0x56396fa83e70 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 10 client.17604 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 10 client.17604 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 20 client.17604 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fa9deb0 0x56396faa0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 4 client.17604 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:03.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.098+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56396f9b3d90 con 0x56396fa9deb0 2024-05-27T07:41:03.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.099+0000 7efd95e00700 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fa9deb0 0x56396faa0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.106+0000 7efd95e00700 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fa9deb0 0x56396faa0340 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c91525fcf290d0e2 server_cookie=8e94414694444fac in_seq=0 out_seq=0 2024-05-27T07:41:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.107+0000 7efd77e00700 10 client.17604 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:41:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.112+0000 7efd77e00700 1 -- 172.21.15.12:0/3423466082 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56396f9b3d90 con 0x56396fa9deb0 2024-05-27T07:41:03.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.112+0000 7efd77e00700 10 client.17604 handle_command_reply: tid=0 2024-05-27T07:41:03.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.112+0000 7efdabc88b80 1 client.17604 shutdown 2024-05-27T07:41:03.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 msgr2=0x56396fa8e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa8e100 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7efd90019ad0 tx=0x7efd90004b40 comp rx=0 tx=0).stop 2024-05-27T07:41:03.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 shutdown_connections 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa8e100 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f904e40 0x56396fa8d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fa9deb0 0x56396faa0340 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 --2- 172.21.15.12:0/3423466082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f9f09f0 0x56396fa8dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 >> 172.21.15.12:0/3423466082 conn(0x56396f99e390 msgr2=0x56396fa833b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efd77e00700 0 client.17604 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 shutdown_connections 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.113+0000 7efdabc88b80 1 -- 172.21.15.12:0/3423466082 wait complete. 2024-05-27T07:41:03.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.114+0000 7efdabc88b80 20 client.17604 trim_cache size 0 max 16384 2024-05-27T07:41:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.142+0000 7efdabc88b80 1 Processor -- start 2024-05-27T07:41:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.142+0000 7efdabc88b80 1 -- start start 2024-05-27T07:41:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.143+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa838c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.143+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f9f09f0 0x56396fa8c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.143+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa88e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.143+0000 7efdabc88b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efd981adb10 con 0x56396fa83e70 2024-05-27T07:41:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.143+0000 7efdabc88b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efd981acd70 con 0x56396f904e40 2024-05-27T07:41:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.143+0000 7efdabc88b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efd981ad440 con 0x56396f9f09f0 2024-05-27T07:41:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.143+0000 7efd95e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa838c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd95400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f9f09f0 0x56396fa8c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd95e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa838c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:58596/0 (socket says 172.21.15.12:58596) 2024-05-27T07:41:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd95e00700 1 -- 172.21.15.12:0/863647923 learned_addr learned my addr 172.21.15.12:0/863647923 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd96800700 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa88e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd96800700 1 -- 172.21.15.12:0/863647923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f9f09f0 msgr2=0x56396fa8c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd96800700 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f9f09f0 0x56396fa8c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd96800700 1 -- 172.21.15.12:0/863647923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 msgr2=0x56396fa838c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd96800700 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa838c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.144+0000 7efd96800700 1 -- 172.21.15.12:0/863647923 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56396fa90550 con 0x56396fa83e70 2024-05-27T07:41:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.145+0000 7efd96800700 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa88e30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7efd90024c90 tx=0x7efd90004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6f48753441e1f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.146+0000 7efd82a00700 1 -- 172.21.15.12:0/863647923 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd90037070 con 0x56396fa83e70 2024-05-27T07:41:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.146+0000 7efd82a00700 1 -- 172.21.15.12:0/863647923 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd90013e00 con 0x56396fa83e70 2024-05-27T07:41:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.146+0000 7efd82a00700 1 -- 172.21.15.12:0/863647923 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd90013530 con 0x56396fa83e70 2024-05-27T07:41:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 -- 172.21.15.12:0/863647923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 msgr2=0x56396fa88e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa88e30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7efd90024c90 tx=0x7efd90004ab0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 -- 172.21.15.12:0/863647923 shutdown_connections 2024-05-27T07:41:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fa838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396fa83e70 0x56396fa88e30 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 --2- 172.21.15.12:0/863647923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396f9f09f0 0x56396fa8c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 -- 172.21.15.12:0/863647923 >> 172.21.15.12:0/863647923 conn(0x56396f99e390 msgr2=0x56396fa831b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 -- 172.21.15.12:0/863647923 shutdown_connections 2024-05-27T07:41:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.147+0000 7efdabc88b80 1 -- 172.21.15.12:0/863647923 wait complete. 2024-05-27T07:41:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.148+0000 7efdabc88b80 1 -- start start 2024-05-27T07:41:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.148+0000 7efdabc88b80 1 Processor -- start 2024-05-27T07:41:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fb05e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fb06380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efdabc88b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa95dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efdabc88b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56396fa8e740 con 0x56396f9f09f0 2024-05-27T07:41:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efdabc88b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56396fa8bbf0 con 0x56396f904e40 2024-05-27T07:41:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efdabc88b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56396fa9dc40 con 0x56396fa83e70 2024-05-27T07:41:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efdabc88b80 10 client.0 _ll_register_callbacks cb 0x56396f94bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efd95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fb06380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efd95400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fb06380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42784/0 (socket says 172.21.15.12:42784) 2024-05-27T07:41:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efd95400700 1 -- 172.21.15.12:0/641275264 learned_addr learned my addr 172.21.15.12:0/641275264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efd95e00700 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fb05e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efd96800700 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa95dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.149+0000 7efd95400700 1 -- 172.21.15.12:0/641275264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 msgr2=0x56396fa95dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.150+0000 7efd95400700 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa95dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.150+0000 7efd95400700 1 -- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 msgr2=0x56396fb05e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.150+0000 7efd95400700 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fb05e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.150+0000 7efd95400700 1 -- 172.21.15.12:0/641275264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56396fa96560 con 0x56396f9f09f0 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.150+0000 7efd95400700 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fb06380 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7efd84000c00 tx=0x7efd84005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986fbea3bedd6f50 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.150+0000 7efd77e00700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.150+0000 7efd77e00700 1 -- 172.21.15.12:0/641275264 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8402d060 con 0x56396f9f09f0 2024-05-27T07:41:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.151+0000 7efd77e00700 1 -- 172.21.15.12:0/641275264 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9554+0+0 (secure 0 0 0) 0x7efd84021040 con 0x56396f9f09f0 2024-05-27T07:41:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.151+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd90013e00 con 0x56396f9f09f0 2024-05-27T07:41:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.151+0000 7efd77e00700 1 -- 172.21.15.12:0/641275264 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7efd84034050 con 0x56396f9f09f0 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.151+0000 7efdabc88b80 10 client.17610 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.151+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x56396fa88830 con 0x56396f9f09f0 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.151+0000 7efd77e00700 1 -- 172.21.15.12:0/641275264 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd90013e00 con 0x56396f9f09f0 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efd77e00700 1 -- 172.21.15.12:0/641275264 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7efd8400bd00 con 0x56396f9f09f0 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efdabc88b80 10 client.17610 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efdabc88b80 10 client.17610 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efdabc88b80 20 client.17610 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efdabc88b80 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fb115d0 0x56396fb139f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efdabc88b80 4 client.17610 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56396f9ad570 con 0x56396fb115d0 2024-05-27T07:41:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.152+0000 7efd95e00700 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fb115d0 0x56396fb139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.153+0000 7efd95e00700 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fb115d0 0x56396fb139f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=132fc9cb00138b03 server_cookie=4f2ad92dcf86a91a in_seq=0 out_seq=0 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.153+0000 7efd77e00700 10 client.17610 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.153+0000 7efd77e00700 1 -- 172.21.15.12:0/641275264 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56396f9ad570 con 0x56396fb115d0 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.153+0000 7efd77e00700 10 client.17610 handle_command_reply: tid=0 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.153+0000 7efdabc88b80 1 client.17610 shutdown 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 msgr2=0x56396fb06380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fb06380 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7efd84000c00 tx=0x7efd84005790 comp rx=0 tx=0).stop 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 shutdown_connections 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x56396fb115d0 0x56396fb139f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56396f904e40 0x56396fb05e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56396fa83e70 0x56396fa95dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efd77e00700 0 client.17610 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:41:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 --2- 172.21.15.12:0/641275264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56396f9f09f0 0x56396fb06380 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 >> 172.21.15.12:0/641275264 conn(0x56396f99e390 msgr2=0x56396fa84ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 shutdown_connections 2024-05-27T07:41:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 1 -- 172.21.15.12:0/641275264 wait complete. 2024-05-27T07:41:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.154+0000 7efdabc88b80 20 client.17610 trim_cache size 0 max 16384 2024-05-27T07:41:03.154 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:41:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7efd7c0b2a70 msgr2=0x7efd7c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7efd7c0b2a70 0x7efd7c0b4f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efd900094f0 tx=0x7efd90009130 comp rx=0 tx=0).stop 2024-05-27T07:41:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105450 msgr2=0x7efd981a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105450 0x7efd981a09c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7efd8800d950 tx=0x7efd8800dd10 comp rx=0 tx=0).stop 2024-05-27T07:41:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 shutdown_connections 2024-05-27T07:41:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7efd7c0b2a70 0x7efd7c0b4f80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd9810e8c0 0x7efd9819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd98105450 0x7efd981a09c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.155+0000 7efd9d200700 1 --2- 172.21.15.12:0/1067942159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd98105da0 0x7efd981a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.156+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 >> 172.21.15.12:0/1067942159 conn(0x7efd980fedb0 msgr2=0x7efd980ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.156+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 shutdown_connections 2024-05-27T07:41:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:03.156+0000 7efd9d200700 1 -- 172.21.15.12:0/1067942159 wait complete. 2024-05-27T07:41:04.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:41:04.612 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:04.812 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:04 smithi012 ceph-mon[31473]: pgmap v635: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:04.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:04 smithi162 ceph-mon[35676]: pgmap v635: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:04.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:04 smithi062 ceph-mon[35304]: pgmap v635: 129 pgs: 129 active+clean; 426 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:06.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:06 smithi012 ceph-mon[31473]: pgmap v636: 129 pgs: 129 active+clean; 483 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 324 op/s 2024-05-27T07:41:06.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:06 smithi162 ceph-mon[35676]: pgmap v636: 129 pgs: 129 active+clean; 483 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 324 op/s 2024-05-27T07:41:06.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:06 smithi062 ceph-mon[35304]: pgmap v636: 129 pgs: 129 active+clean; 483 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 324 op/s 2024-05-27T07:41:07.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.178+0000 7fc18e400700 1 -- 172.21.15.12:0/2178743026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106570 msgr2=0x7fc188106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.178+0000 7fc18e400700 1 --2- 172.21.15.12:0/2178743026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106570 0x7fc188106980 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc174009990 tx=0x7fc174009ca0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.178+0000 7fc18e400700 1 -- 172.21.15.12:0/2178743026 shutdown_connections 2024-05-27T07:41:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.179+0000 7fc18e400700 1 --2- 172.21.15.12:0/2178743026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188106ec0 0x7fc188113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.179+0000 7fc18e400700 1 --2- 172.21.15.12:0/2178743026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18810c5b0 0x7fc18810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.179+0000 7fc18e400700 1 --2- 172.21.15.12:0/2178743026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106570 0x7fc188106980 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.179+0000 7fc18e400700 1 -- 172.21.15.12:0/2178743026 >> 172.21.15.12:0/2178743026 conn(0x7fc188102070 msgr2=0x7fc1881044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.179+0000 7fc18e400700 1 -- 172.21.15.12:0/2178743026 shutdown_connections 2024-05-27T07:41:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.179+0000 7fc18e400700 1 -- 172.21.15.12:0/2178743026 wait complete. 2024-05-27T07:41:07.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.180+0000 7fc18e400700 1 Processor -- start 2024-05-27T07:41:07.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.180+0000 7fc18e400700 1 -- start start 2024-05-27T07:41:07.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc18e400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188106570 0x7fc18819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc18e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106ec0 0x7fc18819d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc18e400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18810c5b0 0x7fc1881a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc18e400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc188116590 con 0x7fc188106ec0 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc18e400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc188116410 con 0x7fc188106570 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc18e400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc188116710 con 0x7fc18810c5b0 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc186a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106ec0 0x7fc18819d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc186a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106ec0 0x7fc18819d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42796/0 (socket says 172.21.15.12:42796) 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.181+0000 7fc186a00700 1 -- 172.21.15.12:0/1207824576 learned_addr learned my addr 172.21.15.12:0/1207824576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc187400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188106570 0x7fc18819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc187e00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18810c5b0 0x7fc1881a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc186a00700 1 -- 172.21.15.12:0/1207824576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18810c5b0 msgr2=0x7fc1881a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc186a00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18810c5b0 0x7fc1881a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc186a00700 1 -- 172.21.15.12:0/1207824576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188106570 msgr2=0x7fc18819d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc186a00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188106570 0x7fc18819d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc186a00700 1 -- 172.21.15.12:0/1207824576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1881a1ed0 con 0x7fc188106ec0 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc187400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188106570 0x7fc18819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc187e00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18810c5b0 0x7fc1881a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.182+0000 7fc186a00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106ec0 0x7fc18819d5a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc1740098e0 tx=0x7fc17400bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.183+0000 7fc17fe00700 1 -- 172.21.15.12:0/1207824576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174027070 con 0x7fc188106ec0 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.183+0000 7fc17fe00700 1 -- 172.21.15.12:0/1207824576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc17400fb80 con 0x7fc188106ec0 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.183+0000 7fc17fe00700 1 -- 172.21.15.12:0/1207824576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174022ca0 con 0x7fc188106ec0 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.183+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1881a2160 con 0x7fc188106ec0 2024-05-27T07:41:07.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.183+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1881a25f0 con 0x7fc188106ec0 2024-05-27T07:41:07.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.187+0000 7fc17fe00700 1 -- 172.21.15.12:0/1207824576 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc174022470 con 0x7fc188106ec0 2024-05-27T07:41:07.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.188+0000 7fc19cfe0b80 1 Processor -- start 2024-05-27T07:41:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc19cfe0b80 1 -- start start 2024-05-27T07:41:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc17fe00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc16c0b29f0 0x7fc16c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558f458f5e40 con 0x558f459c7e70 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558f458e59e0 con 0x558f45848e40 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558f458e9a20 con 0x558f459349f0 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.189+0000 7fc17fe00700 1 -- 172.21.15.12:0/1207824576 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 9642+0+0 (secure 0 0 0) 0x7fc1740e3980 con 0x7fc188106ec0 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc187e00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc16c0b29f0 0x7fc16c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc187400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc187400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:37234/0 (socket says 172.21.15.12:37234) 2024-05-27T07:41:07.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc187400700 1 -- 172.21.15.12:0/2577876525 learned_addr learned my addr 172.21.15.12:0/2577876525 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:07.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc186a00700 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc187e00700 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc187400700 1 -- 172.21.15.12:0/2577876525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 msgr2=0x558f459c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.190+0000 7fc187400700 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.191+0000 7fc187400700 1 -- 172.21.15.12:0/2577876525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 msgr2=0x558f459d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.191+0000 7fc187400700 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.191+0000 7fc187400700 1 -- 172.21.15.12:0/2577876525 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f458ae420 con 0x558f459349f0 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.191+0000 7fc186a00700 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.191+0000 7fc187e00700 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.191+0000 7fc187e00700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc16c0b29f0 0x7fc16c0b4f00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc180009fd0 tx=0x7fc180009ce0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.191+0000 7fc187400700 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459c7330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc17800bcc0 tx=0x7fc17800f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=677d7d27a1531651 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.192+0000 7fc17cc00700 1 -- 172.21.15.12:0/2577876525 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17800fbb0 con 0x558f459349f0 2024-05-27T07:41:07.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.192+0000 7fc17cc00700 1 -- 172.21.15.12:0/2577876525 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc17800fd50 con 0x558f459349f0 2024-05-27T07:41:07.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.192+0000 7fc17cc00700 1 -- 172.21.15.12:0/2577876525 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17801b670 con 0x558f459349f0 2024-05-27T07:41:07.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/2577876525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 msgr2=0x558f459c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459c7330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc17800bcc0 tx=0x7fc17800f360 comp rx=0 tx=0).stop 2024-05-27T07:41:07.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/2577876525 shutdown_connections 2024-05-27T07:41:07.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459c7330 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/2577876525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/2577876525 >> 172.21.15.12:0/2577876525 conn(0x558f458e2390 msgr2=0x558f459c34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:07.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.193+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/2577876525 shutdown_connections 2024-05-27T07:41:07.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.194+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/2577876525 wait complete. 2024-05-27T07:41:07.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.194+0000 7fc19cfe0b80 1 -- start start 2024-05-27T07:41:07.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.194+0000 7fc19cfe0b80 1 Processor -- start 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459d1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558f457ebfa0 con 0x558f459c7e70 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558f45886760 con 0x558f45848e40 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558f459d4b80 con 0x558f459349f0 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc19cfe0b80 10 client.0 _ll_register_callbacks cb 0x558f4588fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc187e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc187e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42820/0 (socket says 172.21.15.12:42820) 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.195+0000 7fc187e00700 1 -- 172.21.15.12:0/614739557 learned_addr learned my addr 172.21.15.12:0/614739557 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:07.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.196+0000 7fc186a00700 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459d1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.196+0000 7fc187400700 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.196+0000 7fc187e00700 1 -- 172.21.15.12:0/614739557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 msgr2=0x558f459d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.196+0000 7fc187e00700 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459d1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.196+0000 7fc187e00700 1 -- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 msgr2=0x558f459d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.196+0000 7fc187e00700 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459d1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.196+0000 7fc187e00700 1 -- 172.21.15.12:0/614739557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f458358d0 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc187e00700 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d20e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc180018de0 tx=0x7fc180009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f991fe0c5937a73f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc171600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc171600700 1 -- 172.21.15.12:0/614739557 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18002a060 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc171600700 1 -- 172.21.15.12:0/614739557 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9642+0+0 (secure 0 0 0) 0x7fc180012040 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc17800fd50 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc171600700 1 -- 172.21.15.12:0/614739557 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc180031050 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc19cfe0b80 10 client.17622 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.197+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558f458b6060 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc171600700 1 -- 172.21.15.12:0/614739557 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc17800fd50 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc171600700 1 -- 172.21.15.12:0/614739557 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fc180025ae0 con 0x558f459c7e70 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc19cfe0b80 10 client.17622 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:07.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc19cfe0b80 10 client.17622 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:41:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc19cfe0b80 20 client.17622 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f459e1ef0 0x558f459e43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc19cfe0b80 4 client.17622 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.198+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f458f7d90 con 0x558f459e1ef0 2024-05-27T07:41:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.199+0000 7fc187400700 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f459e1ef0 0x558f459e43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.200+0000 7fc187400700 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f459e1ef0 0x558f459e43a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54448af3eeec6391 server_cookie=42e98b8759e9b541 in_seq=0 out_seq=0 2024-05-27T07:41:07.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.200+0000 7fc171600700 10 client.17622 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:41:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc171600700 1 -- 172.21.15.12:0/614739557 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558f458f7d90 con 0x558f459e1ef0 2024-05-27T07:41:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc171600700 10 client.17622 handle_command_reply: tid=0 2024-05-27T07:41:07.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 client.17622 shutdown 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 msgr2=0x558f459d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d20e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc180018de0 tx=0x7fc180009050 comp rx=0 tx=0).stop 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 shutdown_connections 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f459e1ef0 0x558f459e43a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f45848e40 0x558f459d1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459d20e0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/614739557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459d1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc171600700 0 client.17622 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 >> 172.21.15.12:0/614739557 conn(0x558f458e2390 msgr2=0x558f459c7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:07.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.203+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 shutdown_connections 2024-05-27T07:41:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.204+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/614739557 wait complete. 2024-05-27T07:41:07.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.204+0000 7fc19cfe0b80 20 client.17622 trim_cache size 0 max 16384 2024-05-27T07:41:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.224+0000 7fc19cfe0b80 1 Processor -- start 2024-05-27T07:41:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.225+0000 7fc19cfe0b80 1 -- start start 2024-05-27T07:41:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.225+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f45848e40 0x558f459d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.225+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459349f0 0x558f459cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.225+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459ddc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.225+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558f459c3680 con 0x558f459c7e70 2024-05-27T07:41:07.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.225+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b1ea0 con 0x558f459349f0 2024-05-27T07:41:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.225+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b1100 con 0x558f45848e40 2024-05-27T07:41:07.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.226+0000 7fc187e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459ddc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.226+0000 7fc187400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f45848e40 0x558f459d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.226+0000 7fc187e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459ddc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42822/0 (socket says 172.21.15.12:42822) 2024-05-27T07:41:07.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.226+0000 7fc187e00700 1 -- 172.21.15.12:0/3171755563 learned_addr learned my addr 172.21.15.12:0/3171755563 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:07.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.227+0000 7fc187e00700 1 -- 172.21.15.12:0/3171755563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f45848e40 msgr2=0x558f459d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.227+0000 7fc187e00700 1 --2- 172.21.15.12:0/3171755563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f45848e40 0x558f459d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.227+0000 7fc187e00700 1 -- 172.21.15.12:0/3171755563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459349f0 msgr2=0x558f459cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.227+0000 7fc187e00700 1 --2- 172.21.15.12:0/3171755563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459349f0 0x558f459cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.227+0000 7fc187e00700 1 -- 172.21.15.12:0/3171755563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f459c4000 con 0x558f459c7e70 2024-05-27T07:41:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.228+0000 7fc187e00700 1 --2- 172.21.15.12:0/3171755563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459ddc20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc180007fd0 tx=0x7fc180009160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7b1f0dba7b9667 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.228+0000 7fc173e00700 1 -- 172.21.15.12:0/3171755563 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18003a070 con 0x558f459c7e70 2024-05-27T07:41:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.229+0000 7fc173e00700 1 -- 172.21.15.12:0/3171755563 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc18001aa00 con 0x558f459c7e70 2024-05-27T07:41:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.229+0000 7fc173e00700 1 -- 172.21.15.12:0/3171755563 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18002a070 con 0x558f459c7e70 2024-05-27T07:41:07.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.230+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3171755563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 msgr2=0x558f459ddc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.230+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3171755563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459ddc20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc180007fd0 tx=0x7fc180009160 comp rx=0 tx=0).stop 2024-05-27T07:41:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.231+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3171755563 shutdown_connections 2024-05-27T07:41:07.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.231+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3171755563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f459c7e70 0x558f459ddc20 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.231+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3171755563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f45848e40 0x558f459d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.231+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3171755563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459349f0 0x558f459cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.231+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3171755563 >> 172.21.15.12:0/3171755563 conn(0x558f458e2390 msgr2=0x558f459d1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.231+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3171755563 shutdown_connections 2024-05-27T07:41:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.231+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3171755563 wait complete. 2024-05-27T07:41:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.232+0000 7fc19cfe0b80 1 -- start start 2024-05-27T07:41:07.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.232+0000 7fc19cfe0b80 1 Processor -- start 2024-05-27T07:41:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.233+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f45848e40 0x558f459da100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.233+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459da640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.233+0000 7fc19cfe0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459c7e70 0x558f459cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.233+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b17d0 con 0x558f45848e40 2024-05-27T07:41:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.233+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558f459cc190 con 0x558f459c7e70 2024-05-27T07:41:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.233+0000 7fc19cfe0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558f459cc390 con 0x558f459349f0 2024-05-27T07:41:07.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc19cfe0b80 10 client.0 _ll_register_callbacks cb 0x558f4588fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f45848e40 0x558f459da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f45848e40 0x558f459da100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42838/0 (socket says 172.21.15.12:42838) 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187400700 1 -- 172.21.15.12:0/3268284692 learned_addr learned my addr 172.21.15.12:0/3268284692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc186a00700 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459da640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187e00700 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459c7e70 0x558f459cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187400700 1 -- 172.21.15.12:0/3268284692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 msgr2=0x558f459da640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187400700 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459da640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187400700 1 -- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459c7e70 msgr2=0x558f459cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.234+0000 7fc187400700 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459c7e70 0x558f459cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.235+0000 7fc187400700 1 -- 172.21.15.12:0/3268284692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f459cc590 con 0x558f45848e40 2024-05-27T07:41:07.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.235+0000 7fc187400700 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f45848e40 0x558f459da100 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc178012da0 tx=0x7fc17800bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0198dcd7af6ee52 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:07.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.235+0000 7fc171600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:07.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.236+0000 7fc171600700 1 -- 172.21.15.12:0/3268284692 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17802d070 con 0x558f45848e40 2024-05-27T07:41:07.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.236+0000 7fc171600700 1 -- 172.21.15.12:0/3268284692 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9642+0+0 (secure 0 0 0) 0x7fc178004040 con 0x558f45848e40 2024-05-27T07:41:07.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.236+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc18001aa00 con 0x558f45848e40 2024-05-27T07:41:07.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.237+0000 7fc171600700 1 -- 172.21.15.12:0/3268284692 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc178039050 con 0x558f45848e40 2024-05-27T07:41:07.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.237+0000 7fc19cfe0b80 10 client.17628 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:07.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.237+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x558f459d9cb0 con 0x558f45848e40 2024-05-27T07:41:07.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.238+0000 7fc171600700 1 -- 172.21.15.12:0/3268284692 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc18001aa00 con 0x558f45848e40 2024-05-27T07:41:07.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.238+0000 7fc171600700 1 -- 172.21.15.12:0/3268284692 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fc178004710 con 0x558f45848e40 2024-05-27T07:41:07.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.238+0000 7fc19cfe0b80 10 client.17628 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:07.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.238+0000 7fc19cfe0b80 10 client.17628 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:41:07.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.238+0000 7fc19cfe0b80 20 client.17628 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:07.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.239+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f45a55670 0x558f45a57a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:07.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.239+0000 7fc19cfe0b80 4 client.17628 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:41:07.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.239+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f458f1570 con 0x558f45a55670 2024-05-27T07:41:07.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.240+0000 7fc186a00700 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f45a55670 0x558f45a57a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:07.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.241+0000 7fc186a00700 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f45a55670 0x558f45a57a90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=876fc5a5d87528ce server_cookie=b3358003adfb9086 in_seq=0 out_seq=0 2024-05-27T07:41:07.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.241+0000 7fc171600700 10 client.17628 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:41:07.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.241+0000 7fc171600700 1 -- 172.21.15.12:0/3268284692 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f458f1570 con 0x558f45a55670 2024-05-27T07:41:07.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.241+0000 7fc171600700 10 client.17628 handle_command_reply: tid=0 2024-05-27T07:41:07.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.241+0000 7fc19cfe0b80 1 client.17628 shutdown 2024-05-27T07:41:07.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f45848e40 msgr2=0x558f459da100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f45848e40 0x558f459da100 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc178012da0 tx=0x7fc17800bbf0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 shutdown_connections 2024-05-27T07:41:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x558f45a55670 0x558f45a57a90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f459c7e70 0x558f459cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f45848e40 0x558f459da100 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 --2- 172.21.15.12:0/3268284692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558f459349f0 0x558f459da640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 >> 172.21.15.12:0/3268284692 conn(0x558f458e2390 msgr2=0x558f459d1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.242+0000 7fc171600700 0 client.17628 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:41:07.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.243+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 shutdown_connections 2024-05-27T07:41:07.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.243+0000 7fc19cfe0b80 1 -- 172.21.15.12:0/3268284692 wait complete. 2024-05-27T07:41:07.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.243+0000 7fc19cfe0b80 20 client.17628 trim_cache size 0 max 16384 2024-05-27T07:41:07.242 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.244+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc16c0b29f0 msgr2=0x7fc16c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.244+0000 7fc18e400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc16c0b29f0 0x7fc16c0b4f00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc180009fd0 tx=0x7fc180009ce0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.244+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106ec0 msgr2=0x7fc18819d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.244+0000 7fc18e400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106ec0 0x7fc18819d5a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc1740098e0 tx=0x7fc17400bfa0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 shutdown_connections 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fc16c0b29f0 0x7fc16c0b4f00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188106570 0x7fc18819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18810c5b0 0x7fc1881a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 --2- 172.21.15.12:0/1207824576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc188106ec0 0x7fc18819d5a0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:07.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 >> 172.21.15.12:0/1207824576 conn(0x7fc188102070 msgr2=0x7fc188104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:07.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 shutdown_connections 2024-05-27T07:41:07.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:07.245+0000 7fc18e400700 1 -- 172.21.15.12:0/1207824576 wait complete. 2024-05-27T07:41:07.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:07 smithi012 ceph-mon[31473]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:41:07.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:07 smithi162 ceph-mon[35676]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:41:07.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:07 smithi062 ceph-mon[35304]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:41:08.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:41:08.548 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:08.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:08 smithi012 ceph-mon[31473]: pgmap v638: 129 pgs: 129 active+clean; 483 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 279 op/s 2024-05-27T07:41:08.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:08 smithi162 ceph-mon[35676]: pgmap v638: 129 pgs: 129 active+clean; 483 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 279 op/s 2024-05-27T07:41:08.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:08 smithi062 ceph-mon[35304]: pgmap v638: 129 pgs: 129 active+clean; 483 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 279 op/s 2024-05-27T07:41:10.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:10 smithi012 ceph-mon[31473]: pgmap v639: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 284 op/s 2024-05-27T07:41:10.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:10 smithi162 ceph-mon[35676]: pgmap v639: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 284 op/s 2024-05-27T07:41:10.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:10 smithi062 ceph-mon[35304]: pgmap v639: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 284 op/s 2024-05-27T07:41:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.672+0000 7fdc72e00700 1 -- 172.21.15.12:0/3248700554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 msgr2=0x7fdc6c076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.672+0000 7fdc72e00700 1 --2- 172.21.15.12:0/3248700554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c076130 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009990 tx=0x7fdc60009ca0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.673+0000 7fdc72e00700 1 -- 172.21.15.12:0/3248700554 shutdown_connections 2024-05-27T07:41:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.673+0000 7fdc72e00700 1 --2- 172.21.15.12:0/3248700554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c1109a0 0x7fdc6c112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.673+0000 7fdc72e00700 1 --2- 172.21.15.12:0/3248700554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c076130 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.673+0000 7fdc72e00700 1 --2- 172.21.15.12:0/3248700554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c076670 0x7fdc6c110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.673+0000 7fdc72e00700 1 -- 172.21.15.12:0/3248700554 >> 172.21.15.12:0/3248700554 conn(0x7fdc6c079d00 msgr2=0x7fdc6c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.673+0000 7fdc72e00700 1 -- 172.21.15.12:0/3248700554 shutdown_connections 2024-05-27T07:41:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.673+0000 7fdc72e00700 1 -- 172.21.15.12:0/3248700554 wait complete. 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.674+0000 7fdc72e00700 1 Processor -- start 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.674+0000 7fdc72e00700 1 -- start start 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.674+0000 7fdc72e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc72e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076670 0x7fdc6c19d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc72e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c1109a0 0x7fdc6c1a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc72e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c116340 con 0x7fdc6c075d20 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc72e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1161c0 con 0x7fdc6c076670 2024-05-27T07:41:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc72e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1164c0 con 0x7fdc6c1109a0 2024-05-27T07:41:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c19cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42870/0 (socket says 172.21.15.12:42870) 2024-05-27T07:41:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71a00700 1 -- 172.21.15.12:0/1617246483 learned_addr learned my addr 172.21.15.12:0/1617246483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71000700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076670 0x7fdc6c19d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71a00700 1 -- 172.21.15.12:0/1617246483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c1109a0 msgr2=0x7fdc6c1a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc72400700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c1109a0 0x7fdc6c1a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71a00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c1109a0 0x7fdc6c1a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71a00700 1 -- 172.21.15.12:0/1617246483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076670 msgr2=0x7fdc6c19d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.675+0000 7fdc71a00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076670 0x7fdc6c19d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc71a00700 1 -- 172.21.15.12:0/1617246483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c1a1c20 con 0x7fdc6c075d20 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc71000700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076670 0x7fdc6c19d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc72400700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c1109a0 0x7fdc6c1a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc71a00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c19cd50 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fdc600057f0 tx=0x7fdc60009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc6a000700 1 -- 172.21.15.12:0/1617246483 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60027070 con 0x7fdc6c075d20 2024-05-27T07:41:11.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc6a000700 1 -- 172.21.15.12:0/1617246483 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc6000fd60 con 0x7fdc6c075d20 2024-05-27T07:41:11.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc6a000700 1 -- 172.21.15.12:0/1617246483 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60022de0 con 0x7fdc6c075d20 2024-05-27T07:41:11.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6c1a1eb0 con 0x7fdc6c075d20 2024-05-27T07:41:11.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.676+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6c1a2340 con 0x7fdc6c075d20 2024-05-27T07:41:11.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.680+0000 7fdc6a000700 1 -- 172.21.15.12:0/1617246483 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdc6002a070 con 0x7fdc6c075d20 2024-05-27T07:41:11.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.682+0000 7fdc6a000700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc540b29f0 0x7fdc540b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.682+0000 7fdc82feab80 1 Processor -- start 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.682+0000 7fdc6a000700 1 -- 172.21.15.12:0/1617246483 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 9642+0+0 (secure 0 0 0) 0x7fdc600e3a80 con 0x7fdc6c075d20 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.682+0000 7fdc82feab80 1 -- start start 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.682+0000 7fdc71000700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc540b29f0 0x7fdc540b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.683+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f5919f0 0x559a8f625750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.683+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f4a5e40 0x559a8f625c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.683+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6316e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.683+0000 7fdc82feab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559a8f552e40 con 0x559a8f4a5e40 2024-05-27T07:41:11.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.683+0000 7fdc82feab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559a8f5429e0 con 0x559a8f62f2c0 2024-05-27T07:41:11.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.683+0000 7fdc82feab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559a8f546a20 con 0x559a8f5919f0 2024-05-27T07:41:11.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.683+0000 7fdc71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f4a5e40 0x559a8f625c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f4a5e40 0x559a8f625c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42886/0 (socket says 172.21.15.12:42886) 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 -- 172.21.15.12:0/1339689597 learned_addr learned my addr 172.21.15.12:0/1339689597 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71a00700 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f5919f0 0x559a8f625750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc540b29f0 0x7fdc540b4f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdc58005fd0 tx=0x7fdc58005dc0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc72400700 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 -- 172.21.15.12:0/1339689597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f5919f0 msgr2=0x559a8f625750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f5919f0 0x559a8f625750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 -- 172.21.15.12:0/1339689597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 msgr2=0x559a8f6316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6316e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71000700 1 -- 172.21.15.12:0/1339689597 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a8f50b420 con 0x559a8f4a5e40 2024-05-27T07:41:11.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.684+0000 7fdc71a00700 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f5919f0 0x559a8f625750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:11.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.685+0000 7fdc71000700 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f4a5e40 0x559a8f625c90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800db70 tx=0x7fdc58010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab04fe50fd9f648 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:11.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.685+0000 7fdc5ea00700 1 -- 172.21.15.12:0/1339689597 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58020070 con 0x559a8f4a5e40 2024-05-27T07:41:11.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.685+0000 7fdc5ea00700 1 -- 172.21.15.12:0/1339689597 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc5800fdc0 con 0x559a8f4a5e40 2024-05-27T07:41:11.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.685+0000 7fdc5ea00700 1 -- 172.21.15.12:0/1339689597 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5801b3b0 con 0x559a8f4a5e40 2024-05-27T07:41:11.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.686+0000 7fdc82feab80 1 -- 172.21.15.12:0/1339689597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f4a5e40 msgr2=0x559a8f625c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.686+0000 7fdc82feab80 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f4a5e40 0x559a8f625c90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800db70 tx=0x7fdc58010000 comp rx=0 tx=0).stop 2024-05-27T07:41:11.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 -- 172.21.15.12:0/1339689597 shutdown_connections 2024-05-27T07:41:11.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6316e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f5919f0 0x559a8f625750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 --2- 172.21.15.12:0/1339689597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f4a5e40 0x559a8f625c90 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 -- 172.21.15.12:0/1339689597 >> 172.21.15.12:0/1339689597 conn(0x559a8f53f390 msgr2=0x559a8f620710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:11.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 -- 172.21.15.12:0/1339689597 shutdown_connections 2024-05-27T07:41:11.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 -- 172.21.15.12:0/1339689597 wait complete. 2024-05-27T07:41:11.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.687+0000 7fdc82feab80 1 -- start start 2024-05-27T07:41:11.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.688+0000 7fdc82feab80 1 Processor -- start 2024-05-27T07:41:11.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.688+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.688+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f5919f0 0x559a8f62c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.688+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f62f2c0 0x559a8f62cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.688+0000 7fdc82feab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559a8f448fa0 con 0x559a8f62f2c0 2024-05-27T07:41:11.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.688+0000 7fdc82feab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559a8f4e3760 con 0x559a8f5919f0 2024-05-27T07:41:11.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.688+0000 7fdc82feab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559a8f631e00 con 0x559a8f4a5e40 2024-05-27T07:41:11.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.689+0000 7fdc82feab80 10 client.0 _ll_register_callbacks cb 0x559a8f4ecc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.689+0000 7fdc72400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f62f2c0 0x559a8f62cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.689+0000 7fdc72400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f62f2c0 0x559a8f62cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42898/0 (socket says 172.21.15.12:42898) 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.689+0000 7fdc72400700 1 -- 172.21.15.12:0/677967583 learned_addr learned my addr 172.21.15.12:0/677967583 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.689+0000 7fdc71000700 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f5919f0 0x559a8f62c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.689+0000 7fdc71a00700 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.690+0000 7fdc72400700 1 -- 172.21.15.12:0/677967583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 msgr2=0x559a8f62c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.690+0000 7fdc72400700 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.690+0000 7fdc72400700 1 -- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f5919f0 msgr2=0x559a8f62c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.690+0000 7fdc72400700 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f5919f0 0x559a8f62c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.690+0000 7fdc72400700 1 -- 172.21.15.12:0/677967583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a8f4928d0 con 0x559a8f62f2c0 2024-05-27T07:41:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.690+0000 7fdc72400700 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f62f2c0 0x559a8f62cd00 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fdc64008bb0 tx=0x7fdc64008ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92e47df962be66d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.690+0000 7fdc53400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.691+0000 7fdc53400700 1 -- 172.21.15.12:0/677967583 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400f920 con 0x559a8f62f2c0 2024-05-27T07:41:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.691+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5800fdc0 con 0x559a8f62f2c0 2024-05-27T07:41:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.691+0000 7fdc53400700 1 -- 172.21.15.12:0/677967583 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9642+0+0 (secure 0 0 0) 0x7fdc6400fac0 con 0x559a8f62f2c0 2024-05-27T07:41:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.691+0000 7fdc53400700 1 -- 172.21.15.12:0/677967583 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fdc64023050 con 0x559a8f62f2c0 2024-05-27T07:41:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.691+0000 7fdc82feab80 10 client.17640 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.691+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x559a8f513060 con 0x559a8f62f2c0 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.691+0000 7fdc53400700 1 -- 172.21.15.12:0/677967583 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc5800fdc0 con 0x559a8f62f2c0 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc53400700 1 -- 172.21.15.12:0/677967583 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fdc6401e3b0 con 0x559a8f62f2c0 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc82feab80 10 client.17640 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc82feab80 10 client.17640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc82feab80 20 client.17640 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc82feab80 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f63f190 0x559a8f641640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc82feab80 4 client.17640 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a8f554d90 con 0x559a8f63f190 2024-05-27T07:41:11.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.692+0000 7fdc71a00700 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f63f190 0x559a8f641640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.693+0000 7fdc71a00700 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f63f190 0x559a8f641640 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81214e8f5ed63255 server_cookie=f584188190bd27e8 in_seq=0 out_seq=0 2024-05-27T07:41:11.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.693+0000 7fdc53400700 10 client.17640 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:11.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc53400700 1 -- 172.21.15.12:0/677967583 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559a8f554d90 con 0x559a8f63f190 2024-05-27T07:41:11.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc53400700 10 client.17640 handle_command_reply: tid=0 2024-05-27T07:41:11.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 client.17640 shutdown 2024-05-27T07:41:11.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f62f2c0 msgr2=0x559a8f62cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f62f2c0 0x559a8f62cd00 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fdc64008bb0 tx=0x7fdc64008ec0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 shutdown_connections 2024-05-27T07:41:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f62f2c0 0x559a8f62cd00 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f63f190 0x559a8f641640 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 --2- 172.21.15.12:0/677967583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f5919f0 0x559a8f62c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 >> 172.21.15.12:0/677967583 conn(0x559a8f53f390 msgr2=0x559a8f620300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.696+0000 7fdc53400700 0 client.17640 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.697+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 shutdown_connections 2024-05-27T07:41:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.697+0000 7fdc82feab80 1 -- 172.21.15.12:0/677967583 wait complete. 2024-05-27T07:41:11.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.697+0000 7fdc82feab80 20 client.17640 trim_cache size 0 max 16384 2024-05-27T07:41:11.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 Processor -- start 2024-05-27T07:41:11.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 -- start start 2024-05-27T07:41:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f63aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6277c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1b1d50 con 0x559a8f5919f0 2024-05-27T07:41:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1b0fb0 con 0x559a8f62f2c0 2024-05-27T07:41:11.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.715+0000 7fdc82feab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1b1680 con 0x559a8f4a5e40 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.716+0000 7fdc71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f63aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.716+0000 7fdc71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f63aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42904/0 (socket says 172.21.15.12:42904) 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.716+0000 7fdc71000700 1 -- 172.21.15.12:0/4127985771 learned_addr learned my addr 172.21.15.12:0/4127985771 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.716+0000 7fdc72400700 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6277c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.716+0000 7fdc71a00700 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.716+0000 7fdc71000700 1 -- 172.21.15.12:0/4127985771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 msgr2=0x559a8f62b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.716+0000 7fdc71000700 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.717+0000 7fdc71000700 1 -- 172.21.15.12:0/4127985771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 msgr2=0x559a8f6277c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.717+0000 7fdc71000700 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6277c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.717+0000 7fdc71000700 1 -- 172.21.15.12:0/4127985771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a8f621bf0 con 0x559a8f5919f0 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.717+0000 7fdc72400700 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6277c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:11.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.717+0000 7fdc71a00700 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.717+0000 7fdc71000700 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f63aed0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800df30 tx=0x7fdc5800f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c321b3552685ae37 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.717+0000 7fdc5e000700 1 -- 172.21.15.12:0/4127985771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58013050 con 0x559a8f5919f0 2024-05-27T07:41:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.718+0000 7fdc5e000700 1 -- 172.21.15.12:0/4127985771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc5800fd90 con 0x559a8f5919f0 2024-05-27T07:41:11.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.718+0000 7fdc5e000700 1 -- 172.21.15.12:0/4127985771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58020040 con 0x559a8f5919f0 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.718+0000 7fdc82feab80 1 -- 172.21.15.12:0/4127985771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 msgr2=0x559a8f63aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.718+0000 7fdc82feab80 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f63aed0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800df30 tx=0x7fdc5800f970 comp rx=0 tx=0).stop 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 -- 172.21.15.12:0/4127985771 shutdown_connections 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6277c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f62b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 --2- 172.21.15.12:0/4127985771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f63aed0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 -- 172.21.15.12:0/4127985771 >> 172.21.15.12:0/4127985771 conn(0x559a8f53f390 msgr2=0x559a8f6270b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 -- 172.21.15.12:0/4127985771 shutdown_connections 2024-05-27T07:41:11.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 -- 172.21.15.12:0/4127985771 wait complete. 2024-05-27T07:41:11.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.719+0000 7fdc82feab80 1 -- start start 2024-05-27T07:41:11.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 1 Processor -- start 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f6320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f632620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559a8f62a7f0 con 0x559a8f5919f0 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559a8f61cba0 con 0x559a8f62f2c0 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559a8f62d340 con 0x559a8f4a5e40 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.720+0000 7fdc82feab80 10 client.0 _ll_register_callbacks cb 0x559a8f4ecc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f632620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f632620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42920/0 (socket says 172.21.15.12:42920) 2024-05-27T07:41:11.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 -- 172.21.15.12:0/3200923944 learned_addr learned my addr 172.21.15.12:0/3200923944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:11.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc72400700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71a00700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f6320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 -- 172.21.15.12:0/3200923944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 msgr2=0x559a8f6320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f6320e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 -- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 msgr2=0x559a8f6a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6a7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71000700 1 -- 172.21.15.12:0/3200923944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a8f632ed0 con 0x559a8f5919f0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc72400700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6a7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.721+0000 7fdc71a00700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f6320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc71000700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f632620 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fdc58025ab0 tx=0x7fdc58025b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fbb88bf233aad5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc53400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc53400700 1 -- 172.21.15.12:0/3200923944 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5803d020 con 0x559a8f5919f0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5800fd90 con 0x559a8f5919f0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc53400700 1 -- 172.21.15.12:0/3200923944 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9642+0+0 (secure 0 0 0) 0x7fdc58038660 con 0x559a8f5919f0 2024-05-27T07:41:11.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc53400700 1 -- 172.21.15.12:0/3200923944 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fdc58013020 con 0x559a8f5919f0 2024-05-27T07:41:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc82feab80 10 client.17646 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.722+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x559a8f6a9100 con 0x559a8f5919f0 2024-05-27T07:41:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc53400700 1 -- 172.21.15.12:0/3200923944 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc5800fd90 con 0x559a8f5919f0 2024-05-27T07:41:11.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc53400700 1 -- 172.21.15.12:0/3200923944 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fdc5801ca00 con 0x559a8f5919f0 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc82feab80 10 client.17646 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc82feab80 10 client.17646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc82feab80 20 client.17646 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc82feab80 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f6a9410 0x559a8f6ab830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc82feab80 4 client.17646 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.723+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a8f54e570 con 0x559a8f6a9410 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.724+0000 7fdc71a00700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f6a9410 0x559a8f6ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.724+0000 7fdc53400700 10 client.17646 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:11.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.724+0000 7fdc71a00700 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f6a9410 0x559a8f6ab830 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1251cf1aa8a0c19a server_cookie=bf77df34afe175e9 in_seq=0 out_seq=0 2024-05-27T07:41:11.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.725+0000 7fdc53400700 1 -- 172.21.15.12:0/3200923944 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a8f54e570 con 0x559a8f6a9410 2024-05-27T07:41:11.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.725+0000 7fdc53400700 10 client.17646 handle_command_reply: tid=0 2024-05-27T07:41:11.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.725+0000 7fdc82feab80 1 client.17646 shutdown 2024-05-27T07:41:11.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.725+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 msgr2=0x559a8f632620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.725+0000 7fdc82feab80 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f632620 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fdc58025ab0 tx=0x7fdc58025b90 comp rx=0 tx=0).stop 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.725+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 shutdown_connections 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x559a8f6a9410 0x559a8f6ab830 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559a8f62f2c0 0x559a8f6a7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559a8f4a5e40 0x559a8f6320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 1 --2- 172.21.15.12:0/3200923944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559a8f5919f0 0x559a8f632620 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc53400700 0 client.17646 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 >> 172.21.15.12:0/3200923944 conn(0x559a8f53f390 msgr2=0x559a8f62d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 shutdown_connections 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 1 -- 172.21.15.12:0/3200923944 wait complete. 2024-05-27T07:41:11.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.726+0000 7fdc82feab80 20 client.17646 trim_cache size 0 max 16384 2024-05-27T07:41:11.726 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:41:11.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc540b29f0 msgr2=0x7fdc540b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc540b29f0 0x7fdc540b4f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdc58005fd0 tx=0x7fdc58005dc0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 msgr2=0x7fdc6c19cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:11.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c19cd50 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fdc600057f0 tx=0x7fdc60009f90 comp rx=0 tx=0).stop 2024-05-27T07:41:11.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 shutdown_connections 2024-05-27T07:41:11.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fdc540b29f0 0x7fdc540b4f00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc6c1109a0 0x7fdc6c1a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c075d20 0x7fdc6c19cd50 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 --2- 172.21.15.12:0/1617246483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc6c076670 0x7fdc6c19d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 >> 172.21.15.12:0/1617246483 conn(0x7fdc6c079d00 msgr2=0x7fdc6c100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 shutdown_connections 2024-05-27T07:41:11.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:11.727+0000 7fdc72e00700 1 -- 172.21.15.12:0/1617246483 wait complete. 2024-05-27T07:41:12.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:12 smithi012 ceph-mon[31473]: pgmap v640: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 284 op/s 2024-05-27T07:41:12.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:12 smithi162 ceph-mon[35676]: pgmap v640: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 284 op/s 2024-05-27T07:41:12.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:12 smithi062 ceph-mon[35304]: pgmap v640: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 284 op/s 2024-05-27T07:41:13.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:13 smithi162 ceph-mon[35676]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:41:13.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:13 smithi162 ceph-mon[35676]: pgmap v642: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 176 op/s 2024-05-27T07:41:13.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:13 smithi062 ceph-mon[35304]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:41:13.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:13 smithi062 ceph-mon[35304]: pgmap v642: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 176 op/s 2024-05-27T07:41:14.098 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:13 smithi012 ceph-mon[31473]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:41:14.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:13 smithi012 ceph-mon[31473]: pgmap v642: 129 pgs: 129 active+clean; 495 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 176 op/s 2024-05-27T07:41:14.292 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:41:14.292 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:41:14.823 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:16.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:16 smithi012 ceph-mon[31473]: pgmap v643: 129 pgs: 129 active+clean; 475 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:41:16.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:16 smithi162 ceph-mon[35676]: pgmap v643: 129 pgs: 129 active+clean; 475 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:41:16.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:16 smithi062 ceph-mon[35304]: pgmap v643: 129 pgs: 129 active+clean; 475 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:41:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 -- 172.21.15.12:0/3487059130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 msgr2=0x7fed0c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 --2- 172.21.15.12:0/3487059130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c1038c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fed000099c0 tx=0x7fed00009cd0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 -- 172.21.15.12:0/3487059130 shutdown_connections 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 --2- 172.21.15.12:0/3487059130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c108730 0x7fed0c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 --2- 172.21.15.12:0/3487059130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c1038c0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 --2- 172.21.15.12:0/3487059130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c107de0 0x7fed0c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 -- 172.21.15.12:0/3487059130 >> 172.21.15.12:0/3487059130 conn(0x7fed0c0fedb0 msgr2=0x7fed0c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 -- 172.21.15.12:0/3487059130 shutdown_connections 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.013+0000 7fed14400700 1 -- 172.21.15.12:0/3487059130 wait complete. 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.015+0000 7fed14400700 1 Processor -- start 2024-05-27T07:41:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.015+0000 7fed14400700 1 -- start start 2024-05-27T07:41:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed14400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c0ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed14400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c107de0 0x7fed0c100200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed14400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c108730 0x7fed0c100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed14400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c113210 con 0x7fed0c1034b0 2024-05-27T07:41:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed14400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c113090 con 0x7fed0c107de0 2024-05-27T07:41:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed14400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c113390 con 0x7fed0c108730 2024-05-27T07:41:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed11a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c0ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed11a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c0ffcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41540/0 (socket says 172.21.15.12:41540) 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed11a00700 1 -- 172.21.15.12:0/4273010147 learned_addr learned my addr 172.21.15.12:0/4273010147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.016+0000 7fed11000700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c107de0 0x7fed0c100200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed12400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c108730 0x7fed0c100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed11a00700 1 -- 172.21.15.12:0/4273010147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c108730 msgr2=0x7fed0c100740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed11a00700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c108730 0x7fed0c100740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed11a00700 1 -- 172.21.15.12:0/4273010147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c107de0 msgr2=0x7fed0c100200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed11a00700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c107de0 0x7fed0c100200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed11a00700 1 -- 172.21.15.12:0/4273010147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0c1a90b0 con 0x7fed0c1034b0 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed12400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c108730 0x7fed0c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.017+0000 7fed11a00700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c0ffcc0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fed00006010 tx=0x7fed00005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.018+0000 7fed0a000700 1 -- 172.21.15.12:0/4273010147 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00026070 con 0x7fed0c1034b0 2024-05-27T07:41:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.018+0000 7fed0a000700 1 -- 172.21.15.12:0/4273010147 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed00003c50 con 0x7fed0c1034b0 2024-05-27T07:41:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.018+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0c1a9340 con 0x7fed0c1034b0 2024-05-27T07:41:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.018+0000 7fed0a000700 1 -- 172.21.15.12:0/4273010147 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0000fb10 con 0x7fed0c1034b0 2024-05-27T07:41:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.018+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0c1a97f0 con 0x7fed0c1034b0 2024-05-27T07:41:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.024+0000 7fed0a000700 1 -- 172.21.15.12:0/4273010147 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fed00029070 con 0x7fed0c1034b0 2024-05-27T07:41:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.025+0000 7fed22fd4b80 1 Processor -- start 2024-05-27T07:41:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.025+0000 7fed22fd4b80 1 -- start start 2024-05-27T07:41:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed0a000700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fecf40b2a70 0x7fecf40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 0x5615e447c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e447cdb0 0x5615e44768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 0x5615e4477240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5615e43a4ec0 con 0x5615e447cdb0 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5615e4394a60 con 0x5615e42f7ec0 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615e4398aa0 con 0x5615e4476e30 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed0a000700 1 -- 172.21.15.12:0/4273010147 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fed000e31e0 con 0x7fed0c1034b0 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.026+0000 7fed12400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fecf40b2a70 0x7fecf40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e447cdb0 0x5615e44768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e447cdb0 0x5615e44768f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41544/0 (socket says 172.21.15.12:41544) 2024-05-27T07:41:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 -- 172.21.15.12:0/4026543604 learned_addr learned my addr 172.21.15.12:0/4026543604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 -- 172.21.15.12:0/4026543604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 msgr2=0x5615e4477240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 --2- 172.21.15.12:0/4026543604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 0x5615e4477240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 -- 172.21.15.12:0/4026543604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 msgr2=0x5615e447c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 --2- 172.21.15.12:0/4026543604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 0x5615e447c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.027+0000 7fed11000700 1 -- 172.21.15.12:0/4026543604 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e435d4a0 con 0x5615e447cdb0 2024-05-27T07:41:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.028+0000 7fed11000700 1 --2- 172.21.15.12:0/4026543604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e447cdb0 0x5615e44768f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009cf0 tx=0x7fecf8011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf61a072e767090 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.028+0000 7fed12400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fecf40b2a70 0x7fecf40b4f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fed04009460 tx=0x7fed04009920 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.028+0000 7fecfea00700 1 -- 172.21.15.12:0/4026543604 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8011e30 con 0x5615e447cdb0 2024-05-27T07:41:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.029+0000 7fecfea00700 1 -- 172.21.15.12:0/4026543604 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fecf80044e0 con 0x5615e447cdb0 2024-05-27T07:41:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.029+0000 7fecfea00700 1 -- 172.21.15.12:0/4026543604 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8005280 con 0x5615e447cdb0 2024-05-27T07:41:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 -- 172.21.15.12:0/4026543604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e447cdb0 msgr2=0x5615e44768f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/4026543604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e447cdb0 0x5615e44768f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009cf0 tx=0x7fecf8011440 comp rx=0 tx=0).stop 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 -- 172.21.15.12:0/4026543604 shutdown_connections 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/4026543604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 0x5615e447c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/4026543604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 0x5615e4477240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/4026543604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e447cdb0 0x5615e44768f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 -- 172.21.15.12:0/4026543604 >> 172.21.15.12:0/4026543604 conn(0x5615e4358040 msgr2=0x5615e4471950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 -- 172.21.15.12:0/4026543604 shutdown_connections 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.030+0000 7fed22fd4b80 1 -- 172.21.15.12:0/4026543604 wait complete. 2024-05-27T07:41:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.031+0000 7fed22fd4b80 1 -- start start 2024-05-27T07:41:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.032+0000 7fed22fd4b80 1 Processor -- start 2024-05-27T07:41:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.032+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44805b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.032+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 0x5615e4480af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.032+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e447cdb0 0x5615e4481030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.032+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5615e429b020 con 0x5615e42f7ec0 2024-05-27T07:41:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.032+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5615e43357e0 con 0x5615e447cdb0 2024-05-27T07:41:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.032+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615e448c090 con 0x5615e4476e30 2024-05-27T07:41:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.033+0000 7fed22fd4b80 10 client.0 _ll_register_callbacks cb 0x5615e433ecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.033+0000 7fed11a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.033+0000 7fed11a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44805b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41556/0 (socket says 172.21.15.12:41556) 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.033+0000 7fed11a00700 1 -- 172.21.15.12:0/310777132 learned_addr learned my addr 172.21.15.12:0/310777132 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.033+0000 7fed12400700 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e447cdb0 0x5615e4481030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.033+0000 7fed11000700 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 0x5615e4480af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fed11a00700 1 -- 172.21.15.12:0/310777132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 msgr2=0x5615e4480af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fed11a00700 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 0x5615e4480af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fed11a00700 1 -- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e447cdb0 msgr2=0x5615e4481030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fed11a00700 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e447cdb0 0x5615e4481030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fed11a00700 1 -- 172.21.15.12:0/310777132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e42e4950 con 0x5615e42f7ec0 2024-05-27T07:41:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fed11a00700 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44805b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fed000e0580 tx=0x7fed00009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb09fe962e2d680d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fecf3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.034+0000 7fecf3400700 1 -- 172.21.15.12:0/310777132 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed000aa840 con 0x5615e42f7ec0 2024-05-27T07:41:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.035+0000 7fecf3400700 1 -- 172.21.15.12:0/310777132 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fed00003e00 con 0x5615e42f7ec0 2024-05-27T07:41:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.035+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf80044e0 con 0x5615e42f7ec0 2024-05-27T07:41:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.035+0000 7fecf3400700 1 -- 172.21.15.12:0/310777132 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fed000aa140 con 0x5615e42f7ec0 2024-05-27T07:41:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.035+0000 7fed22fd4b80 10 client.17658 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.035+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5615e43650e0 con 0x5615e42f7ec0 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fecf3400700 1 -- 172.21.15.12:0/310777132 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fecf80044e0 con 0x5615e42f7ec0 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fecf3400700 1 -- 172.21.15.12:0/310777132 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fed000b2a80 con 0x5615e42f7ec0 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fed22fd4b80 10 client.17658 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fed22fd4b80 10 client.17658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fed22fd4b80 20 client.17658 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4499050 0x5615e449b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fed22fd4b80 4 client.17658 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.036+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615e43a6e10 con 0x5615e4499050 2024-05-27T07:41:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.037+0000 7fed11000700 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4499050 0x5615e449b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.038+0000 7fed11000700 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4499050 0x5615e449b470 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b7ff5cdedc5d568 server_cookie=d9022aa4d67631c8 in_seq=0 out_seq=0 2024-05-27T07:41:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.038+0000 7fecf3400700 10 client.17658 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:18.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fecf3400700 1 -- 172.21.15.12:0/310777132 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5615e43a6e10 con 0x5615e4499050 2024-05-27T07:41:18.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fecf3400700 10 client.17658 handle_command_reply: tid=0 2024-05-27T07:41:18.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 client.17658 shutdown 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 msgr2=0x5615e44805b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44805b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fed000e0580 tx=0x7fed00009e60 comp rx=0 tx=0).stop 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 shutdown_connections 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4499050 0x5615e449b470 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e447cdb0 0x5615e4481030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44805b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/310777132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e4476e30 0x5615e4480af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 >> 172.21.15.12:0/310777132 conn(0x5615e4358040 msgr2=0x5615e4471600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.041+0000 7fecf3400700 0 client.17658 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:18.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.042+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 shutdown_connections 2024-05-27T07:41:18.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.042+0000 7fed22fd4b80 1 -- 172.21.15.12:0/310777132 wait complete. 2024-05-27T07:41:18.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.042+0000 7fed22fd4b80 20 client.17658 trim_cache size 0 max 16384 2024-05-27T07:41:18.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.068+0000 7fed22fd4b80 1 Processor -- start 2024-05-27T07:41:18.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.068+0000 7fed22fd4b80 1 -- start start 2024-05-27T07:41:18.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e4476e30 0x5615e447acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e4482a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1ad7a0 con 0x5615e42f7ec0 2024-05-27T07:41:18.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1aca00 con 0x5615e4476e30 2024-05-27T07:41:18.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1ad0d0 con 0x5615e447cdb0 2024-05-27T07:41:18.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed11a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed11a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44768f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41572/0 (socket says 172.21.15.12:41572) 2024-05-27T07:41:18.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed11a00700 1 -- 172.21.15.12:0/3452672100 learned_addr learned my addr 172.21.15.12:0/3452672100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:18.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.069+0000 7fed12400700 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e4482a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.070+0000 7fed11000700 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e4476e30 0x5615e447acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.070+0000 7fed11a00700 1 -- 172.21.15.12:0/3452672100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 msgr2=0x5615e4482a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.070+0000 7fed11a00700 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e4482a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.070+0000 7fed11a00700 1 -- 172.21.15.12:0/3452672100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e4476e30 msgr2=0x5615e447acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.070+0000 7fed11a00700 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e4476e30 0x5615e447acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.070+0000 7fed11a00700 1 -- 172.21.15.12:0/3452672100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e4477bb0 con 0x5615e42f7ec0 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.070+0000 7fed11a00700 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44768f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fed000a73b0 tx=0x7fed000e3170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2931ad8e542c8299 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.071+0000 7fecfe000700 1 -- 172.21.15.12:0/3452672100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed000ae4f0 con 0x5615e42f7ec0 2024-05-27T07:41:18.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.071+0000 7fecfe000700 1 -- 172.21.15.12:0/3452672100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed000b2f10 con 0x5615e42f7ec0 2024-05-27T07:41:18.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.071+0000 7fecfe000700 1 -- 172.21.15.12:0/3452672100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed000b2bf0 con 0x5615e42f7ec0 2024-05-27T07:41:18.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 -- 172.21.15.12:0/3452672100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 msgr2=0x5615e44768f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44768f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fed000a73b0 tx=0x7fed000e3170 comp rx=0 tx=0).stop 2024-05-27T07:41:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 -- 172.21.15.12:0/3452672100 shutdown_connections 2024-05-27T07:41:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e4482a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e42f7ec0 0x5615e44768f0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/3452672100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e4476e30 0x5615e447acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 -- 172.21.15.12:0/3452672100 >> 172.21.15.12:0/3452672100 conn(0x5615e4358040 msgr2=0x5615e4480390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.072+0000 7fed22fd4b80 1 -- 172.21.15.12:0/3452672100 shutdown_connections 2024-05-27T07:41:18.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.073+0000 7fed22fd4b80 1 -- 172.21.15.12:0/3452672100 wait complete. 2024-05-27T07:41:18.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.073+0000 7fed22fd4b80 1 -- start start 2024-05-27T07:41:18.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.074+0000 7fed22fd4b80 1 Processor -- start 2024-05-27T07:41:18.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.074+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 0x5615e44912e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e4476e30 0x5615e4491820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed22fd4b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e448e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5615e44727c0 con 0x5615e4476e30 2024-05-27T07:41:18.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5615e44816c0 con 0x5615e42f7ec0 2024-05-27T07:41:18.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed22fd4b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615e447bde0 con 0x5615e447cdb0 2024-05-27T07:41:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed22fd4b80 10 client.0 _ll_register_callbacks cb 0x5615e433ecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed11000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e4476e30 0x5615e4491820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed11000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e4476e30 0x5615e4491820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41580/0 (socket says 172.21.15.12:41580) 2024-05-27T07:41:18.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.075+0000 7fed11000700 1 -- 172.21.15.12:0/2620407126 learned_addr learned my addr 172.21.15.12:0/2620407126 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed12400700 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e448e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed11a00700 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 0x5615e44912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed11000700 1 -- 172.21.15.12:0/2620407126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 msgr2=0x5615e448e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed11000700 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e448e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed11000700 1 -- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 msgr2=0x5615e44912e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed11000700 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 0x5615e44912e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed11000700 1 -- 172.21.15.12:0/2620407126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e448e820 con 0x5615e4476e30 2024-05-27T07:41:18.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fed11000700 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e4476e30 0x5615e4491820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fecf80097c0 tx=0x7fecf8004240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92265914deade087 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:18.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.076+0000 7fecf3400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:18.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.077+0000 7fecf3400700 1 -- 172.21.15.12:0/2620407126 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8012e60 con 0x5615e4476e30 2024-05-27T07:41:18.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.077+0000 7fecf3400700 1 -- 172.21.15.12:0/2620407126 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fecf802ce00 con 0x5615e4476e30 2024-05-27T07:41:18.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.077+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed000b2f10 con 0x5615e4476e30 2024-05-27T07:41:18.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.077+0000 7fecf3400700 1 -- 172.21.15.12:0/2620407126 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fecf803b050 con 0x5615e4476e30 2024-05-27T07:41:18.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.077+0000 7fed22fd4b80 10 client.17664 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.077+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5615e45042d0 con 0x5615e4476e30 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fecf3400700 1 -- 172.21.15.12:0/2620407126 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed000b2f10 con 0x5615e4476e30 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fecf3400700 1 -- 172.21.15.12:0/2620407126 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fecf8018070 con 0x5615e4476e30 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fed22fd4b80 10 client.17664 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fed22fd4b80 10 client.17664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fed22fd4b80 20 client.17664 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4504a10 0x5615e4506e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fed22fd4b80 4 client.17664 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:41:18.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.078+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5615e43a05f0 con 0x5615e4504a10 2024-05-27T07:41:18.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.079+0000 7fed11a00700 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4504a10 0x5615e4506e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:18.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.080+0000 7fed11a00700 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4504a10 0x5615e4506e30 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c1191061a4ba46b server_cookie=caff6837e02c28e7 in_seq=0 out_seq=0 2024-05-27T07:41:18.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.080+0000 7fecf3400700 10 client.17664 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:18.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.085+0000 7fecf3400700 1 -- 172.21.15.12:0/2620407126 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5615e43a05f0 con 0x5615e4504a10 2024-05-27T07:41:18.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.085+0000 7fecf3400700 10 client.17664 handle_command_reply: tid=0 2024-05-27T07:41:18.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.085+0000 7fed22fd4b80 1 client.17664 shutdown 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e4476e30 msgr2=0x5615e4491820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e4476e30 0x5615e4491820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fecf80097c0 tx=0x7fecf8004240 comp rx=0 tx=0).stop 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 shutdown_connections 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5615e4504a10 0x5615e4506e30 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5615e42f7ec0 0x5615e44912e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e447cdb0 0x5615e448e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 --2- 172.21.15.12:0/2620407126 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5615e4476e30 0x5615e4491820 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 >> 172.21.15.12:0/2620407126 conn(0x5615e4358040 msgr2=0x5615e447aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.086+0000 7fecf3400700 0 client.17664 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.087+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 shutdown_connections 2024-05-27T07:41:18.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.087+0000 7fed22fd4b80 1 -- 172.21.15.12:0/2620407126 wait complete. 2024-05-27T07:41:18.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.087+0000 7fed22fd4b80 20 client.17664 trim_cache size 0 max 16384 2024-05-27T07:41:18.086 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:41:18.086 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:41:18.089 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "5b653bd5-b0ff-4df9-a093-80178b81a31b", 2024-05-27T07:41:18.089 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:41:18.089 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fecf40b2a70 msgr2=0x7fecf40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fecf40b2a70 0x7fecf40b4f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fed04009460 tx=0x7fed04009920 comp rx=0 tx=0).stop 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 msgr2=0x7fed0c0ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c0ffcc0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fed00006010 tx=0x7fed00005280 comp rx=0 tx=0).stop 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 shutdown_connections 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fecf40b2a70 0x7fecf40b4f80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c108730 0x7fed0c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed0c1034b0 0x7fed0c0ffcc0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 --2- 172.21.15.12:0/4273010147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c107de0 0x7fed0c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.088+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 >> 172.21.15.12:0/4273010147 conn(0x7fed0c0fedb0 msgr2=0x7fed0c101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:18.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.089+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 shutdown_connections 2024-05-27T07:41:18.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:18.089+0000 7fed14400700 1 -- 172.21.15.12:0/4273010147 wait complete. 2024-05-27T07:41:18.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:18 smithi162 ceph-mon[35676]: pgmap v644: 129 pgs: 129 active+clean; 475 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 258 op/s 2024-05-27T07:41:18.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:18 smithi062 ceph-mon[35304]: pgmap v644: 129 pgs: 129 active+clean; 475 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 258 op/s 2024-05-27T07:41:18.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:18 smithi012 ceph-mon[31473]: pgmap v644: 129 pgs: 129 active+clean; 475 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 258 op/s 2024-05-27T07:41:18.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:41:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:19 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:41:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:19 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:19 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:41:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:19 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:41:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:19 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:41:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:19 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:19 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:19 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:41:19.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:19 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:19 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:41:19.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:19 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:41:19.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:19 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:41:19.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:19 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:19 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:19 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:41:19.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:19 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:19 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:41:19.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:19 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:41:19.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:19 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:41:19.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:19 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:19.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:19 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:20.700 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:20 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:20.701 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:20 smithi012 ceph-mon[31473]: pgmap v645: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 223 op/s 2024-05-27T07:41:20.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:20 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:20.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:20 smithi162 ceph-mon[35676]: pgmap v645: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 223 op/s 2024-05-27T07:41:20.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:20 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:41:20.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:20 smithi062 ceph-mon[35304]: pgmap v645: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 223 op/s 2024-05-27T07:41:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.884+0000 7fad8ac00700 1 -- 172.21.15.12:0/2388335038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 msgr2=0x7fad7c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.884+0000 7fad8ac00700 1 --2- 172.21.15.12:0/2388335038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c0a5310 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fad84069810 tx=0x7fad8406a480 comp rx=0 tx=0).stop 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.885+0000 7fad8ac00700 1 -- 172.21.15.12:0/2388335038 shutdown_connections 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.885+0000 7fad8ac00700 1 --2- 172.21.15.12:0/2388335038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad7c095340 0x7fad7c095750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.885+0000 7fad8ac00700 1 --2- 172.21.15.12:0/2388335038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c0a5310 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.885+0000 7fad8ac00700 1 --2- 172.21.15.12:0/2388335038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad7c095d20 0x7fad7c0a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.885+0000 7fad8ac00700 1 -- 172.21.15.12:0/2388335038 >> 172.21.15.12:0/2388335038 conn(0x7fad7c00b050 msgr2=0x7fad7c00b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.885+0000 7fad8ac00700 1 -- 172.21.15.12:0/2388335038 shutdown_connections 2024-05-27T07:41:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.885+0000 7fad8ac00700 1 -- 172.21.15.12:0/2388335038 wait complete. 2024-05-27T07:41:21.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.887+0000 7fad8ac00700 1 Processor -- start 2024-05-27T07:41:21.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.887+0000 7fad8ac00700 1 -- start start 2024-05-27T07:41:21.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad7c095340 0x7fad7c12f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8ac00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad7c095d20 0x7fad7c12f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8ac00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c133a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8ac00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c0a8710 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c0a8590 con 0x7fad7c095340 2024-05-27T07:41:21.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8ac00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c0a8890 con 0x7fad7c095d20 2024-05-27T07:41:21.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c133a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8a200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c133a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38594/0 (socket says 172.21.15.12:38594) 2024-05-27T07:41:21.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad8a200700 1 -- 172.21.15.12:0/829930026 learned_addr learned my addr 172.21.15.12:0/829930026 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:21.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad89800700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad7c095340 0x7fad7c12f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.888+0000 7fad88e00700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad7c095d20 0x7fad7c12f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.889+0000 7fad8a200700 1 -- 172.21.15.12:0/829930026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad7c095d20 msgr2=0x7fad7c12f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.889+0000 7fad8a200700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad7c095d20 0x7fad7c12f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.889+0000 7fad8a200700 1 -- 172.21.15.12:0/829930026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad7c095340 msgr2=0x7fad7c12f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.889+0000 7fad8a200700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad7c095340 0x7fad7c12f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.889+0000 7fad8a200700 1 -- 172.21.15.12:0/829930026 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7c1340d0 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.889+0000 7fad8a200700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c133a30 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fad84051580 tx=0x7fad840806c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:21.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.890+0000 7fad82000700 1 -- 172.21.15.12:0/829930026 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8408e070 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.890+0000 7fad82000700 1 -- 172.21.15.12:0/829930026 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad84080d90 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.890+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7c134360 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.890+0000 7fad82000700 1 -- 172.21.15.12:0/829930026 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84085040 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.890+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7c1347c0 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.894+0000 7fad82000700 1 -- 172.21.15.12:0/829930026 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fad84069d80 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.896+0000 7fad9ad5eb80 1 Processor -- start 2024-05-27T07:41:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.896+0000 7fad9ad5eb80 1 -- start start 2024-05-27T07:41:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1bf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf0e0e40 con 0x55a2bf033e40 2024-05-27T07:41:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf0d09e0 con 0x55a2bf11f9f0 2024-05-27T07:41:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf0d4a20 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad82000700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad6c0b29f0 0x7fad6c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad89800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad89800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38598/0 (socket says 172.21.15.12:38598) 2024-05-27T07:41:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.897+0000 7fad89800700 1 -- 172.21.15.12:0/2136768589 learned_addr learned my addr 172.21.15.12:0/2136768589 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad82000700 1 -- 172.21.15.12:0/829930026 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fad84150fe0 con 0x7fad7c0a2ef0 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad88e00700 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1bf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad8a200700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad6c0b29f0 0x7fad6c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad88e00700 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad89800700 1 -- 172.21.15.12:0/2136768589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 msgr2=0x55a2bf1bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad89800700 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1bf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad89800700 1 -- 172.21.15.12:0/2136768589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 msgr2=0x55a2bf1b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad89800700 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.898+0000 7fad89800700 1 -- 172.21.15.12:0/2136768589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2bf099420 con 0x55a2bf033e40 2024-05-27T07:41:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.899+0000 7fad89800700 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1b3c90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fad78007cf0 tx=0x7fad780103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d76637bd929eb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.899+0000 7fad8a200700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad6c0b29f0 0x7fad6c0b4f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fad840525c0 tx=0x7fad84082040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.899+0000 7fad76a00700 1 -- 172.21.15.12:0/2136768589 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78010dc0 con 0x55a2bf033e40 2024-05-27T07:41:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.899+0000 7fad76a00700 1 -- 172.21.15.12:0/2136768589 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad780118d0 con 0x55a2bf033e40 2024-05-27T07:41:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.899+0000 7fad76a00700 1 -- 172.21.15.12:0/2136768589 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78019750 con 0x55a2bf033e40 2024-05-27T07:41:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/2136768589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 msgr2=0x55a2bf1b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1b3c90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fad78007cf0 tx=0x7fad780103d0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/2136768589 shutdown_connections 2024-05-27T07:41:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1bf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/2136768589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1b3c90 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/2136768589 >> 172.21.15.12:0/2136768589 conn(0x55a2bf0cd390 msgr2=0x55a2bf1ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/2136768589 shutdown_connections 2024-05-27T07:41:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.901+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/2136768589 wait complete. 2024-05-27T07:41:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.902+0000 7fad9ad5eb80 1 -- start start 2024-05-27T07:41:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.902+0000 7fad9ad5eb80 1 Processor -- start 2024-05-27T07:41:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.903+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1ba250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.903+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1ba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.903+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.903+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a2befd6fa0 con 0x55a2bf033e40 2024-05-27T07:41:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.903+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf071760 con 0x55a2bf11f9f0 2024-05-27T07:41:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.903+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf1bfe00 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.903+0000 7fad9ad5eb80 10 client.0 _ll_register_callbacks cb 0x55a2bf07ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.904+0000 7fad89800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.904+0000 7fad88e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1ba250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.904+0000 7fad88e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1ba250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38608/0 (socket says 172.21.15.12:38608) 2024-05-27T07:41:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.904+0000 7fad88e00700 1 -- 172.21.15.12:0/1222126556 learned_addr learned my addr 172.21.15.12:0/1222126556 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.905+0000 7fad88e00700 1 -- 172.21.15.12:0/1222126556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 msgr2=0x55a2bf1bacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.905+0000 7fad88e00700 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.905+0000 7fad88e00700 1 -- 172.21.15.12:0/1222126556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 msgr2=0x55a2bf1ba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.905+0000 7fad88e00700 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1ba790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.905+0000 7fad88e00700 1 -- 172.21.15.12:0/1222126556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2bf0208d0 con 0x55a2bf033e40 2024-05-27T07:41:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.906+0000 7fad88e00700 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1ba250 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fad7c008f00 tx=0x7fad700045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9e2fd6223507f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.906+0000 7fad6b400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.907+0000 7fad6b400700 1 -- 172.21.15.12:0/1222126556 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70018070 con 0x55a2bf033e40 2024-05-27T07:41:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.907+0000 7fad6b400700 1 -- 172.21.15.12:0/1222126556 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fad7001d460 con 0x55a2bf033e40 2024-05-27T07:41:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.907+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad780118d0 con 0x55a2bf033e40 2024-05-27T07:41:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.908+0000 7fad6b400700 1 -- 172.21.15.12:0/1222126556 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fad7001d660 con 0x55a2bf033e40 2024-05-27T07:41:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.908+0000 7fad9ad5eb80 10 client.17676 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.908+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a2bf0a1060 con 0x55a2bf033e40 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.911+0000 7fad6b400700 1 -- 172.21.15.12:0/1222126556 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad780118d0 con 0x55a2bf033e40 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.913+0000 7fad6b400700 1 -- 172.21.15.12:0/1222126556 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fad700128d0 con 0x55a2bf033e40 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.913+0000 7fad9ad5eb80 10 client.17676 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.913+0000 7fad9ad5eb80 10 client.17676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.913+0000 7fad9ad5eb80 20 client.17676 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.913+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf1cd0e0 0x55a2bf1cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.913+0000 7fad9ad5eb80 4 client.17676 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:21.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.913+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2bf0e2d90 con 0x55a2bf1cd0e0 2024-05-27T07:41:21.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.914+0000 7fad89800700 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf1cd0e0 0x55a2bf1cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.915+0000 7fad89800700 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf1cd0e0 0x55a2bf1cf590 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ec1d44d13ac6eb3 server_cookie=298631292bf2debe in_seq=0 out_seq=0 2024-05-27T07:41:21.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.915+0000 7fad6b400700 10 client.17676 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:21.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.918+0000 7fad6b400700 1 -- 172.21.15.12:0/1222126556 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a2bf0e2d90 con 0x55a2bf1cd0e0 2024-05-27T07:41:21.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.918+0000 7fad6b400700 10 client.17676 handle_command_reply: tid=0 2024-05-27T07:41:21.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.918+0000 7fad9ad5eb80 1 client.17676 shutdown 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.920+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 msgr2=0x55a2bf1ba250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.920+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1ba250 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fad7c008f00 tx=0x7fad700045a0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 shutdown_connections 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf033e40 0x55a2bf1ba250 secure :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fad7c008f00 tx=0x7fad700045a0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf1cd0e0 0x55a2bf1cf590 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/1222126556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1ba790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad6b400700 0 client.17676 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:21.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 >> 172.21.15.12:0/1222126556 conn(0x55a2bf0cd390 msgr2=0x55a2bf1ae300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:21.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.921+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 shutdown_connections 2024-05-27T07:41:21.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.922+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/1222126556 wait complete. 2024-05-27T07:41:21.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.922+0000 7fad9ad5eb80 20 client.17676 trim_cache size 0 max 16384 2024-05-27T07:41:21.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 Processor -- start 2024-05-27T07:41:21.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 -- start start 2024-05-27T07:41:21.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c144060 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c1432c0 con 0x55a2bf11f9f0 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.958+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c143990 con 0x55a2bf1ae740 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38620/0 (socket says 172.21.15.12:38620) 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 -- 172.21.15.12:0/3335972175 learned_addr learned my addr 172.21.15.12:0/3335972175 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad88e00700 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad88e00700 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 -- 172.21.15.12:0/3335972175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 msgr2=0x55a2bf1d4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1d4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 -- 172.21.15.12:0/3335972175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 msgr2=0x55a2bf1b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad89800700 1 -- 172.21.15.12:0/3335972175 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2bf1b87c0 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.959+0000 7fad88e00700 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:21.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.960+0000 7fad89800700 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c8f00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fad78005fd0 tx=0x7fad780077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d60a95214bb455 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.960+0000 7fad76000700 1 -- 172.21.15.12:0/3335972175 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78010e80 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.960+0000 7fad76000700 1 -- 172.21.15.12:0/3335972175 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad7802b030 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.960+0000 7fad76000700 1 -- 172.21.15.12:0/3335972175 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78004d10 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/3335972175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf1bd2c0 msgr2=0x55a2bf1c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c8f00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fad78005fd0 tx=0x7fad780077f0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/3335972175 shutdown_connections 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1d4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/3335972175 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c8f00 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/3335972175 >> 172.21.15.12:0/3335972175 conn(0x55a2bf0cd390 msgr2=0x55a2bf1b56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:21.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.961+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/3335972175 shutdown_connections 2024-05-27T07:41:21.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.962+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/3335972175 wait complete. 2024-05-27T07:41:21.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.962+0000 7fad9ad5eb80 1 -- start start 2024-05-27T07:41:21.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 1 Processor -- start 2024-05-27T07:41:21.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1c5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf1aaba0 con 0x55a2bf11f9f0 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf0e2d90 con 0x55a2bf1bd2c0 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf1cfb70 con 0x55a2bf1ae740 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.963+0000 7fad9ad5eb80 10 client.0 _ll_register_callbacks cb 0x55a2bf07ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1c5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38632/0 (socket says 172.21.15.12:38632) 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 -- 172.21.15.12:0/219299924 learned_addr learned my addr 172.21.15.12:0/219299924 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad89800700 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 -- 172.21.15.12:0/219299924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 msgr2=0x55a2bf1c1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1c1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 -- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf1bd2c0 msgr2=0x55a2bf1c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.964+0000 7fad88e00700 1 -- 172.21.15.12:0/219299924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2bf1c2650 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.965+0000 7fad88e00700 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1c5700 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fad70012cd0 tx=0x7fad700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a921dc9937c6e4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.965+0000 7fad6b400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.965+0000 7fad6b400700 1 -- 172.21.15.12:0/219299924 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70018030 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.965+0000 7fad6b400700 1 -- 172.21.15.12:0/219299924 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fad70012450 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.965+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad7802b030 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.965+0000 7fad6b400700 1 -- 172.21.15.12:0/219299924 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fad7003e050 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 10 client.17682 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x55a2bf2412a0 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad6b400700 1 -- 172.21.15.12:0/219299924 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad7802b030 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad6b400700 1 -- 172.21.15.12:0/219299924 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7fad70011a50 con 0x55a2bf11f9f0 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 10 client.17682 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 10 client.17682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 20 client.17682 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf241950 0x55a2bf243d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 4 client.17682 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:41:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.966+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a2bf0dc570 con 0x55a2bf241950 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.967+0000 7fad89800700 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf241950 0x55a2bf243d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad89800700 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf241950 0x55a2bf243d70 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a8e15a14224f69c server_cookie=a438a11772d04cef in_seq=0 out_seq=0 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad6b400700 10 client.17682 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad6b400700 1 -- 172.21.15.12:0/219299924 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a2bf0dc570 con 0x55a2bf241950 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad6b400700 10 client.17682 handle_command_reply: tid=0 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad9ad5eb80 1 client.17682 shutdown 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf11f9f0 msgr2=0x55a2bf1c5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1c5700 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fad70012cd0 tx=0x7fad700047c0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.968+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 shutdown_connections 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55a2bf241950 0x55a2bf243d70 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2bf1bd2c0 0x55a2bf1c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2bf11f9f0 0x55a2bf1c5700 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 1 --2- 172.21.15.12:0/219299924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf1ae740 0x55a2bf1c1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad6b400700 0 client.17682 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 >> 172.21.15.12:0/219299924 conn(0x55a2bf0cd390 msgr2=0x55a2bf1ad730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 shutdown_connections 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 1 -- 172.21.15.12:0/219299924 wait complete. 2024-05-27T07:41:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.969+0000 7fad9ad5eb80 20 client.17682 trim_cache size 0 max 16384 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: "5b653bd5-b0ff-4df9-a093-80178b81a31b": { 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "5b653bd5-b0ff-4df9-a093-80178b81a31b", 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:41:21.969 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:41:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad6c0b29f0 msgr2=0x7fad6c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad6c0b29f0 0x7fad6c0b4f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fad840525c0 tx=0x7fad84082040 comp rx=0 tx=0).stop 2024-05-27T07:41:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 msgr2=0x7fad7c133a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c133a30 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fad84051580 tx=0x7fad840806c0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 shutdown_connections 2024-05-27T07:41:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad6c0b29f0 0x7fad6c0b4f00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad7c095340 0x7fad7c12f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad7c0a2ef0 0x7fad7c133a30 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 --2- 172.21.15.12:0/829930026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad7c095d20 0x7fad7c12f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 >> 172.21.15.12:0/829930026 conn(0x7fad7c00b050 msgr2=0x7fad7c0a44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 shutdown_connections 2024-05-27T07:41:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:21.970+0000 7fad8ac00700 1 -- 172.21.15.12:0/829930026 wait complete. 2024-05-27T07:41:22.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:22 smithi162 ceph-mon[35676]: pgmap v646: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 223 op/s 2024-05-27T07:41:22.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:22 smithi062 ceph-mon[35304]: pgmap v646: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 223 op/s 2024-05-27T07:41:22.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:22 smithi012 ceph-mon[31473]: pgmap v646: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 223 op/s 2024-05-27T07:41:22.979 INFO:tasks.cephfs.filesystem:scrub status for tag:5b653bd5-b0ff-4df9-a093-80178b81a31b - {'path': '/', 'tag': '5b653bd5-b0ff-4df9-a093-80178b81a31b', 'options': 'recursive,force'} 2024-05-27T07:41:24.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:24 smithi162 ceph-mon[35676]: pgmap v647: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-05-27T07:41:24.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:24 smithi062 ceph-mon[35304]: pgmap v647: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-05-27T07:41:24.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:24 smithi012 ceph-mon[31473]: pgmap v647: 129 pgs: 129 active+clean; 473 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-05-27T07:41:26.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:26 smithi162 ceph-mon[35676]: pgmap v648: 129 pgs: 129 active+clean; 485 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 280 op/s 2024-05-27T07:41:26.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:26 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:26.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:26 smithi062 ceph-mon[35304]: pgmap v648: 129 pgs: 129 active+clean; 485 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 280 op/s 2024-05-27T07:41:26.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:26 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:26.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:26 smithi012 ceph-mon[31473]: pgmap v648: 129 pgs: 129 active+clean; 485 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 280 op/s 2024-05-27T07:41:26.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:26 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:27.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:27 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:27.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:27 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:27.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:27 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:41:27.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:27 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:27.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:27 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:27.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:27 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:41:27.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:27 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:27.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:27 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:27.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:27 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:41:28.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:28 smithi162 ceph-mon[35676]: pgmap v649: 129 pgs: 129 active+clean; 485 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-05-27T07:41:28.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:28 smithi062 ceph-mon[35304]: pgmap v649: 129 pgs: 129 active+clean; 485 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-05-27T07:41:28.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:28 smithi012 ceph-mon[31473]: pgmap v649: 129 pgs: 129 active+clean; 485 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-05-27T07:41:30.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:30 smithi062 ceph-mon[35304]: pgmap v650: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 295 op/s 2024-05-27T07:41:30.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:30 smithi012 ceph-mon[31473]: pgmap v650: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 295 op/s 2024-05-27T07:41:30.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:30 smithi162 ceph-mon[35676]: pgmap v650: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 295 op/s 2024-05-27T07:41:31.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:31 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:41:31.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:31 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:41:31.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:31 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:41:31.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:31 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:41:31.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:31 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:41:31.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:31 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:41:32.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:32 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:41:32.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:32 smithi012 ceph-mon[31473]: pgmap v651: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-27T07:41:32.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:32 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:41:32.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:32 smithi162 ceph-mon[35676]: pgmap v651: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-27T07:41:32.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:32 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:41:32.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:32 smithi062 ceph-mon[35304]: pgmap v651: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-27T07:41:34.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:34 smithi012 ceph-mon[31473]: pgmap v652: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-27T07:41:34.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:34 smithi162 ceph-mon[35676]: pgmap v652: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-27T07:41:34.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:34 smithi062 ceph-mon[35304]: pgmap v652: 129 pgs: 129 active+clean; 496 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-27T07:41:36.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:36 smithi012 ceph-mon[31473]: pgmap v653: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 300 op/s 2024-05-27T07:41:36.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:36 smithi162 ceph-mon[35676]: pgmap v653: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 300 op/s 2024-05-27T07:41:36.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:36 smithi062 ceph-mon[35304]: pgmap v653: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 300 op/s 2024-05-27T07:41:38.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:38 smithi012 ceph-mon[31473]: pgmap v654: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:38.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:38 smithi162 ceph-mon[35676]: pgmap v654: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:38.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:38 smithi062 ceph-mon[35304]: pgmap v654: 129 pgs: 129 active+clean; 481 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-05-27T07:41:40.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:40 smithi012 ceph-mon[31473]: pgmap v655: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:41:40.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:40 smithi162 ceph-mon[35676]: pgmap v655: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:41:40.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:40 smithi062 ceph-mon[35304]: pgmap v655: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 294 op/s 2024-05-27T07:41:42.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:42 smithi012 ceph-mon[31473]: pgmap v656: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:41:42.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:42 smithi162 ceph-mon[35676]: pgmap v656: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:41:42.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:42 smithi062 ceph-mon[35304]: pgmap v656: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:41:44.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:44 smithi012 ceph-mon[31473]: pgmap v657: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:41:44.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:44 smithi162 ceph-mon[35676]: pgmap v657: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:41:44.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:44 smithi062 ceph-mon[35304]: pgmap v657: 129 pgs: 129 active+clean; 490 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:41:46.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:46 smithi012 ceph-mon[31473]: pgmap v658: 129 pgs: 129 active+clean; 499 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-05-27T07:41:46.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:46 smithi162 ceph-mon[35676]: pgmap v658: 129 pgs: 129 active+clean; 499 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-05-27T07:41:46.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:46 smithi062 ceph-mon[35304]: pgmap v658: 129 pgs: 129 active+clean; 499 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-05-27T07:41:48.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:48 smithi012 ceph-mon[31473]: pgmap v659: 129 pgs: 129 active+clean; 499 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:41:48.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:48 smithi162 ceph-mon[35676]: pgmap v659: 129 pgs: 129 active+clean; 499 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:41:48.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:48 smithi062 ceph-mon[35304]: pgmap v659: 129 pgs: 129 active+clean; 499 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 188 op/s 2024-05-27T07:41:50.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:50 smithi012 ceph-mon[31473]: pgmap v660: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 270 op/s 2024-05-27T07:41:50.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:50 smithi162 ceph-mon[35676]: pgmap v660: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 270 op/s 2024-05-27T07:41:50.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:50 smithi062 ceph-mon[35304]: pgmap v660: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 270 op/s 2024-05-27T07:41:52.979 INFO:tasks.workunit.client.0.smithi012.stdout:All 50000 operations completed A-OK! 2024-05-27T07:41:52.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:41:52.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:52 smithi162 ceph-mon[35676]: pgmap v661: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 176 op/s 2024-05-27T07:41:52.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:52 smithi062 ceph-mon[35304]: pgmap v661: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 176 op/s 2024-05-27T07:41:53.037 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T07:41:53.038 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T07:41:53.050 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:52 smithi012 ceph-mon[31473]: pgmap v661: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 176 op/s 2024-05-27T07:41:53.431 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:54.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:54 smithi012 ceph-mon[31473]: pgmap v662: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 176 op/s 2024-05-27T07:41:54.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:54 smithi162 ceph-mon[35676]: pgmap v662: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 176 op/s 2024-05-27T07:41:54.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:54 smithi062 ceph-mon[35304]: pgmap v662: 129 pgs: 129 active+clean; 507 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 176 op/s 2024-05-27T07:41:55.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:55 smithi162 ceph-mon[35676]: pgmap v663: 129 pgs: 129 active+clean; 508 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 238 op/s 2024-05-27T07:41:55.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:55 smithi062 ceph-mon[35304]: pgmap v663: 129 pgs: 129 active+clean; 508 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 238 op/s 2024-05-27T07:41:56.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.020+0000 7f78d9200700 1 -- 172.21.15.12:0/2254244383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410af10 msgr2=0x7f78d410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.020+0000 7f78d9200700 1 --2- 172.21.15.12:0/2254244383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410af10 0x7f78d410b320 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0099c0 tx=0x7f78bc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.020+0000 7f78d9200700 1 -- 172.21.15.12:0/2254244383 shutdown_connections 2024-05-27T07:41:56.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.020+0000 7f78d9200700 1 --2- 172.21.15.12:0/2254244383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78d4106630 0x7f78d4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.020+0000 7f78d9200700 1 --2- 172.21.15.12:0/2254244383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78d410b860 0x7f78d41131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.020+0000 7f78d9200700 1 --2- 172.21.15.12:0/2254244383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410af10 0x7f78d410b320 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.020+0000 7f78d9200700 1 -- 172.21.15.12:0/2254244383 >> 172.21.15.12:0/2254244383 conn(0x7f78d4102020 msgr2=0x7f78d4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:56.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.021+0000 7f78d9200700 1 -- 172.21.15.12:0/2254244383 shutdown_connections 2024-05-27T07:41:56.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.021+0000 7f78d9200700 1 -- 172.21.15.12:0/2254244383 wait complete. 2024-05-27T07:41:56.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:55 smithi012 ceph-mon[31473]: pgmap v663: 129 pgs: 129 active+clean; 508 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 238 op/s 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.022+0000 7f78d9200700 1 Processor -- start 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.022+0000 7f78d9200700 1 -- start start 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78d4106630 0x7f78d41a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d9200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78d410af10 0x7f78d41a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d9200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410b860 0x7f78d419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d9200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78d4116320 con 0x7f78d410b860 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d9200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41161a0 con 0x7f78d4106630 2024-05-27T07:41:56.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d9200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41164a0 con 0x7f78d410af10 2024-05-27T07:41:56.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410b860 0x7f78d419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410b860 0x7f78d419cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56928/0 (socket says 172.21.15.12:56928) 2024-05-27T07:41:56.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d3e00700 1 -- 172.21.15.12:0/1695281480 learned_addr learned my addr 172.21.15.12:0/1695281480 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:56.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78cbe00700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78d410af10 0x7f78d41a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.023+0000 7f78d3400700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78d4106630 0x7f78d41a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.024+0000 7f78d3e00700 1 -- 172.21.15.12:0/1695281480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78d410af10 msgr2=0x7f78d41a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.024+0000 7f78d3e00700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78d410af10 0x7f78d41a2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.024+0000 7f78d3e00700 1 -- 172.21.15.12:0/1695281480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78d4106630 msgr2=0x7f78d41a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.024+0000 7f78d3e00700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78d4106630 0x7f78d41a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.024+0000 7f78d3e00700 1 -- 172.21.15.12:0/1695281480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d419d270 con 0x7f78d410b860 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.024+0000 7f78d3e00700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410b860 0x7f78d419cb40 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f78cc007ee0 tx=0x7f78cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.025+0000 7f78d0c00700 1 -- 172.21.15.12:0/1695281480 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc004bb0 con 0x7f78d410b860 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.025+0000 7f78d0c00700 1 -- 172.21.15.12:0/1695281480 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78cc004d50 con 0x7f78d410b860 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.025+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d419d560 con 0x7f78d410b860 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.025+0000 7f78d0c00700 1 -- 172.21.15.12:0/1695281480 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc0058b0 con 0x7f78d410b860 2024-05-27T07:41:56.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.025+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d41a9760 con 0x7f78d410b860 2024-05-27T07:41:56.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.029+0000 7f78d0c00700 1 -- 172.21.15.12:0/1695281480 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f78cc01a020 con 0x7f78d410b860 2024-05-27T07:41:56.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.031+0000 7f78e928eb80 1 Processor -- start 2024-05-27T07:41:56.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.031+0000 7f78e928eb80 1 -- start start 2024-05-27T07:41:56.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8546ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a8554020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78e928eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5563a8475e40 con 0x5563a8547820 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78e928eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5563a84659e0 con 0x5563a8547410 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78e928eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5563a8469a20 con 0x5563a83c8e40 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78d0c00700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f78b80b29f0 0x7f78b80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78d0c00700 1 -- 172.21.15.12:0/1695281480 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7f78cc0e2100 con 0x7f78d410b860 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a8554020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a8554020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56942/0 (socket says 172.21.15.12:56942) 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78cbe00700 1 -- 172.21.15.12:0/3122385692 learned_addr learned my addr 172.21.15.12:0/3122385692 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:56.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78d3e00700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f78b80b29f0 0x7f78b80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8546ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.032+0000 7f78d3400700 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.033+0000 7f78cbe00700 1 -- 172.21.15.12:0/3122385692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 msgr2=0x5563a8546ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.033+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8546ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.033+0000 7f78cbe00700 1 -- 172.21.15.12:0/3122385692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 msgr2=0x5563a8551ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.033+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.033+0000 7f78cbe00700 1 -- 172.21.15.12:0/3122385692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a842e420 con 0x5563a8547820 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.033+0000 7f78d3e00700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f78b80b29f0 0x7f78b80b4f00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0a4ff0 tx=0x7f78cc007f10 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.033+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a8554020 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009fd0 tx=0x7f78bc005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980230110f3cff61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.034+0000 7f78c8c00700 1 -- 172.21.15.12:0/3122385692 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0050f0 con 0x5563a8547820 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.034+0000 7f78c8c00700 1 -- 172.21.15.12:0/3122385692 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78bc00b790 con 0x5563a8547820 2024-05-27T07:41:56.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.034+0000 7f78c8c00700 1 -- 172.21.15.12:0/3122385692 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc004840 con 0x5563a8547820 2024-05-27T07:41:56.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 -- 172.21.15.12:0/3122385692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 msgr2=0x5563a8554020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a8554020 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009fd0 tx=0x7f78bc005a90 comp rx=0 tx=0).stop 2024-05-27T07:41:56.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 -- 172.21.15.12:0/3122385692 shutdown_connections 2024-05-27T07:41:56.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a8554020 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8546ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3122385692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 -- 172.21.15.12:0/3122385692 >> 172.21.15.12:0/3122385692 conn(0x5563a8428fc0 msgr2=0x5563a8542d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:56.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 -- 172.21.15.12:0/3122385692 shutdown_connections 2024-05-27T07:41:56.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.035+0000 7f78e928eb80 1 -- 172.21.15.12:0/3122385692 wait complete. 2024-05-27T07:41:56.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.036+0000 7f78e928eb80 1 -- start start 2024-05-27T07:41:56.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.036+0000 7f78e928eb80 1 Processor -- start 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8551590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8552010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78e928eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5563a836bfa0 con 0x5563a8547410 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78e928eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5563a8406760 con 0x5563a8547820 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78e928eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5563a8554930 con 0x5563a83c8e40 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78e928eb80 10 client.0 _ll_register_callbacks cb 0x5563a840fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:56.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78d3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56950/0 (socket says 172.21.15.12:56950) 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78d3400700 1 -- 172.21.15.12:0/3289052737 learned_addr learned my addr 172.21.15.12:0/3289052737 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8551590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.037+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8552010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78d3400700 1 -- 172.21.15.12:0/3289052737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 msgr2=0x5563a8551590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78d3400700 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8551590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78d3400700 1 -- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 msgr2=0x5563a8552010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78d3400700 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8552010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78d3400700 1 -- 172.21.15.12:0/3289052737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a83b58d0 con 0x5563a8547410 2024-05-27T07:41:56.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78d3400700 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547410 0x5563a8551ad0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f78c400bb70 tx=0x7f78c400bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fba9695a73cfd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:56.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78c1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:56.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.038+0000 7f78c1600700 1 -- 172.21.15.12:0/3289052737 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4003d60 con 0x5563a8547410 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78c1600700 1 -- 172.21.15.12:0/3289052737 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7f78c4004830 con 0x5563a8547410 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78bc00b790 con 0x5563a8547410 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78c1600700 1 -- 172.21.15.12:0/3289052737 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f78c4024050 con 0x5563a8547410 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78e928eb80 10 client.17694 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5563a8436060 con 0x5563a8547410 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78c1600700 1 -- 172.21.15.12:0/3289052737 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78bc00b790 con 0x5563a8547410 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78c1600700 1 -- 172.21.15.12:0/3289052737 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f78c4011440 con 0x5563a8547410 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78e928eb80 10 client.17694 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.039+0000 7f78e928eb80 10 client.17694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.040+0000 7f78e928eb80 20 client.17694 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.040+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a8561d50 0x5563a85641a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.040+0000 7f78e928eb80 4 client.17694 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:41:56.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.040+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563a8477d90 con 0x5563a8561d50 2024-05-27T07:41:56.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.040+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a8561d50 0x5563a85641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.041+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a8561d50 0x5563a85641a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93abcbd198f78213 server_cookie=f6fda9d6333faafe in_seq=0 out_seq=0 2024-05-27T07:41:56.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.041+0000 7f78c1600700 10 client.17694 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.043+0000 7f78c1600700 1 -- 172.21.15.12:0/3289052737 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5563a8477d90 con 0x5563a8561d50 2024-05-27T07:41:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.043+0000 7f78c1600700 10 client.17694 handle_command_reply: tid=0 2024-05-27T07:41:56.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.043+0000 7f78e928eb80 1 client.17694 shutdown 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547410 msgr2=0x5563a8551ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547410 0x5563a8551ad0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f78c400bb70 tx=0x7f78c400bf30 comp rx=0 tx=0).stop 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 shutdown_connections 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a8561d50 0x5563a85641a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8552010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a8551590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3289052737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547410 0x5563a8551ad0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 >> 172.21.15.12:0/3289052737 conn(0x5563a8428fc0 msgr2=0x5563a8541ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:56.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78c1600700 0 client.17694 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:56.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 shutdown_connections 2024-05-27T07:41:56.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 1 -- 172.21.15.12:0/3289052737 wait complete. 2024-05-27T07:41:56.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.044+0000 7f78e928eb80 20 client.17694 trim_cache size 0 max 16384 2024-05-27T07:41:56.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 Processor -- start 2024-05-27T07:41:56.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 -- start start 2024-05-27T07:41:56.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a854b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8553990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a855da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41b1b10 con 0x5563a8547820 2024-05-27T07:41:56.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41b0d70 con 0x5563a8547410 2024-05-27T07:41:56.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.068+0000 7f78e928eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41b1440 con 0x5563a83c8e40 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.069+0000 7f78d3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8553990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.069+0000 7f78d3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8553990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:44672/0 (socket says 172.21.15.12:44672) 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.069+0000 7f78d3400700 1 -- 172.21.15.12:0/2126174207 learned_addr learned my addr 172.21.15.12:0/2126174207 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.069+0000 7f78cbe00700 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a855da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.069+0000 7f78cbe00700 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a854b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.070+0000 7f78d3400700 1 -- 172.21.15.12:0/2126174207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 msgr2=0x5563a854b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.070+0000 7f78d3400700 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a854b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.070+0000 7f78d3400700 1 -- 172.21.15.12:0/2126174207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 msgr2=0x5563a855da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.070+0000 7f78d3400700 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a855da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.070+0000 7f78d3400700 1 -- 172.21.15.12:0/2126174207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a8544350 con 0x5563a8547410 2024-05-27T07:41:56.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.070+0000 7f78d3400700 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8553990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78c4000c00 tx=0x7f78c400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ffbc243b352052 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:56.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.071+0000 7f78c3e00700 1 -- 172.21.15.12:0/2126174207 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4003e20 con 0x5563a8547410 2024-05-27T07:41:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.071+0000 7f78c3e00700 1 -- 172.21.15.12:0/2126174207 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78c402c030 con 0x5563a8547410 2024-05-27T07:41:56.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.071+0000 7f78c3e00700 1 -- 172.21.15.12:0/2126174207 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4019980 con 0x5563a8547410 2024-05-27T07:41:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 -- 172.21.15.12:0/2126174207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 msgr2=0x5563a8553990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8553990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78c4000c00 tx=0x7f78c400b480 comp rx=0 tx=0).stop 2024-05-27T07:41:56.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 -- 172.21.15.12:0/2126174207 shutdown_connections 2024-05-27T07:41:56.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a8547820 0x5563a855da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a83c8e40 0x5563a854b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 --2- 172.21.15.12:0/2126174207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547410 0x5563a8553990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 -- 172.21.15.12:0/2126174207 >> 172.21.15.12:0/2126174207 conn(0x5563a8428fc0 msgr2=0x5563a8551d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.072+0000 7f78e928eb80 1 -- 172.21.15.12:0/2126174207 shutdown_connections 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.073+0000 7f78e928eb80 1 -- 172.21.15.12:0/2126174207 wait complete. 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.073+0000 7f78e928eb80 1 -- start start 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.073+0000 7f78e928eb80 1 Processor -- start 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a83c8e40 0x5563a8555990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a8547410 0x5563a8555ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78e928eb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8556410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78e928eb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5563a855df40 con 0x5563a83c8e40 2024-05-27T07:41:56.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78e928eb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5563a8543fe0 con 0x5563a8547820 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78e928eb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5563a855e0c0 con 0x5563a8547410 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78e928eb80 10 client.0 _ll_register_callbacks cb 0x5563a840fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.074+0000 7f78cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a83c8e40 0x5563a8555990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a83c8e40 0x5563a8555990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56970/0 (socket says 172.21.15.12:56970) 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 -- 172.21.15.12:0/3127140586 learned_addr learned my addr 172.21.15.12:0/3127140586 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78d3400700 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a8547410 0x5563a8555ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8556410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 -- 172.21.15.12:0/3127140586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a8547410 msgr2=0x5563a8555ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a8547410 0x5563a8555ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 -- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 msgr2=0x5563a8556410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:41:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8556410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 -- 172.21.15.12:0/3127140586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a85cb830 con 0x5563a83c8e40 2024-05-27T07:41:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78cbe00700 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a83c8e40 0x5563a8555990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009d10 tx=0x7f78bc019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb3f768d9a413ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.075+0000 7f78c1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:41:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78c1600700 1 -- 172.21.15.12:0/3127140586 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc030040 con 0x5563a83c8e40 2024-05-27T07:41:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78c1600700 1 -- 172.21.15.12:0/3127140586 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7f78bc00be40 con 0x5563a83c8e40 2024-05-27T07:41:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c402c030 con 0x5563a83c8e40 2024-05-27T07:41:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78c1600700 1 -- 172.21.15.12:0/3127140586 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f78bc049050 con 0x5563a83c8e40 2024-05-27T07:41:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78e928eb80 10 client.17700 fetch_fsmap learned FSMap version 119 2024-05-27T07:41:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5563a855df40 con 0x5563a83c8e40 2024-05-27T07:41:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78c1600700 1 -- 172.21.15.12:0/3127140586 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f78c402c030 con 0x5563a83c8e40 2024-05-27T07:41:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78c1600700 1 -- 172.21.15.12:0/3127140586 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 119) ==== 2937+0+0 (secure 0 0 0) 0x7f78bc036040 con 0x5563a83c8e40 2024-05-27T07:41:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78e928eb80 10 client.17700 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:41:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.076+0000 7f78e928eb80 10 client.17700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.077+0000 7f78e928eb80 20 client.17700 populate_metadata read hostname 'smithi012' 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.077+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a85ccbc0 0x5563a85cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.077+0000 7f78e928eb80 4 client.17700 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.077+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5563a8471570 con 0x5563a85ccbc0 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.077+0000 7f78d3400700 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a85ccbc0 0x5563a85cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.078+0000 7f78d3400700 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a85ccbc0 0x5563a85cefe0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50694f1c21ed8b2c server_cookie=b209770d4eab1637 in_seq=0 out_seq=0 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.078+0000 7f78c1600700 10 client.17700 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78c1600700 1 -- 172.21.15.12:0/3127140586 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5563a8471570 con 0x5563a85ccbc0 2024-05-27T07:41:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78c1600700 10 client.17700 handle_command_reply: tid=0 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 client.17700 shutdown 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a83c8e40 msgr2=0x5563a8555990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a83c8e40 0x5563a8555990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009d10 tx=0x7f78bc019040 comp rx=0 tx=0).stop 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 shutdown_connections 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5563a85ccbc0 0x5563a85cefe0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5563a8547820 0x5563a8556410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563a83c8e40 0x5563a8555990 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 --2- 172.21.15.12:0/3127140586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563a8547410 0x5563a8555ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 >> 172.21.15.12:0/3127140586 conn(0x5563a8428fc0 msgr2=0x5563a8551d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:56.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.079+0000 7f78c1600700 0 client.17700 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:41:56.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.080+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 shutdown_connections 2024-05-27T07:41:56.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.080+0000 7f78e928eb80 1 -- 172.21.15.12:0/3127140586 wait complete. 2024-05-27T07:41:56.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.080+0000 7f78e928eb80 20 client.17700 trim_cache size 0 max 16384 2024-05-27T07:41:56.083 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:41:56.084 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:41:56.084 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:41:56.084 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.080+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f78b80b29f0 msgr2=0x7f78b80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.080+0000 7f78d9200700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f78b80b29f0 0x7f78b80b4f00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0a4ff0 tx=0x7f78cc007f10 comp rx=0 tx=0).stop 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.080+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410b860 msgr2=0x7f78d419cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.080+0000 7f78d9200700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410b860 0x7f78d419cb40 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f78cc007ee0 tx=0x7f78cc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 shutdown_connections 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f78b80b29f0 0x7f78b80b4f00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78d4106630 0x7f78d41a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78d410b860 0x7f78d419cb40 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 --2- 172.21.15.12:0/1695281480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78d410af10 0x7f78d41a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 >> 172.21.15.12:0/1695281480 conn(0x7f78d4102020 msgr2=0x7f78d410a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 shutdown_connections 2024-05-27T07:41:56.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:41:56.081+0000 7f78d9200700 1 -- 172.21.15.12:0/1695281480 wait complete. 2024-05-27T07:41:57.028 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:41:57.028 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:41:57.641 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:58.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:58 smithi012 ceph-mon[31473]: pgmap v664: 129 pgs: 129 active+clean; 508 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 143 op/s 2024-05-27T07:41:58.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:58 smithi162 ceph-mon[35676]: pgmap v664: 129 pgs: 129 active+clean; 508 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 143 op/s 2024-05-27T07:41:58.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:58 smithi062 ceph-mon[35304]: pgmap v664: 129 pgs: 129 active+clean; 508 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 143 op/s 2024-05-27T07:41:59.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:41:59 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:41:59.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:41:59 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:41:59.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:41:59 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:00.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:00 smithi012 ceph-mon[31473]: pgmap v665: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 -- 172.21.15.12:0/789768585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 msgr2=0x7fb3d4109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 --2- 172.21.15.12:0/789768585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 0x7fb3d4109320 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00099c0 tx=0x7fb3c0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 -- 172.21.15.12:0/789768585 shutdown_connections 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 --2- 172.21.15.12:0/789768585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d4109860 0x7fb3d41111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 --2- 172.21.15.12:0/789768585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d41045e0 0x7fb3d41049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 --2- 172.21.15.12:0/789768585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 0x7fb3d4109320 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 -- 172.21.15.12:0/789768585 >> 172.21.15.12:0/789768585 conn(0x7fb3d4100030 msgr2=0x7fb3d4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.430+0000 7fb3d9600700 1 -- 172.21.15.12:0/789768585 shutdown_connections 2024-05-27T07:42:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.431+0000 7fb3d9600700 1 -- 172.21.15.12:0/789768585 wait complete. 2024-05-27T07:42:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.432+0000 7fb3d9600700 1 Processor -- start 2024-05-27T07:42:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.432+0000 7fb3d9600700 1 -- start start 2024-05-27T07:42:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.432+0000 7fb3d9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d41045e0 0x7fb3d4075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.432+0000 7fb3d9600700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 0x7fb3d4075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.432+0000 7fb3d9600700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d4109860 0x7fb3d4076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d9600700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d4114320 con 0x7fb3d4108f10 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d9600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41141a0 con 0x7fb3d41045e0 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d9600700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41144a0 con 0x7fb3d4109860 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d41045e0 0x7fb3d4075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d41045e0 0x7fb3d4075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:44692/0 (socket says 172.21.15.12:44692) 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 -- 172.21.15.12:0/3997808219 learned_addr learned my addr 172.21.15.12:0/3997808219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2000700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 0x7fb3d4075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d3400700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d4109860 0x7fb3d4076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 -- 172.21.15.12:0/3997808219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d4109860 msgr2=0x7fb3d4076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d4109860 0x7fb3d4076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 -- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 msgr2=0x7fb3d4075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 0x7fb3d4075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2a00700 1 -- 172.21.15.12:0/3997808219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d41a9110 con 0x7fb3d41045e0 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d2000700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 0x7fb3d4075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.433+0000 7fb3d3400700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d4109860 0x7fb3d4076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.434+0000 7fb3d2a00700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d41045e0 0x7fb3d4075ab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400d950 tx=0x7fb3c400dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.434+0000 7fb3cb400700 1 -- 172.21.15.12:0/3997808219 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4016070 con 0x7fb3d41045e0 2024-05-27T07:42:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.434+0000 7fb3cb400700 1 -- 172.21.15.12:0/3997808219 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3c4010460 con 0x7fb3d41045e0 2024-05-27T07:42:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.434+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d41a9400 con 0x7fb3d41045e0 2024-05-27T07:42:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.434+0000 7fb3cb400700 1 -- 172.21.15.12:0/3997808219 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c400f620 con 0x7fb3d41045e0 2024-05-27T07:42:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.435+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d41a9910 con 0x7fb3d41045e0 2024-05-27T07:42:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.436+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d40512b0 con 0x7fb3d41045e0 2024-05-27T07:42:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.438+0000 7fb3cb400700 1 -- 172.21.15.12:0/3997808219 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb3c4019070 con 0x7fb3d41045e0 2024-05-27T07:42:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.440+0000 7fb3cb400700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb3b80b29f0 0x7fb3b80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.440+0000 7fb3cb400700 1 -- 172.21.15.12:0/3997808219 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fb3c40e26d0 con 0x7fb3d41045e0 2024-05-27T07:42:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.440+0000 7fb3d2000700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb3b80b29f0 0x7fb3b80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.441+0000 7fb3d2000700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb3b80b29f0 0x7fb3b80b4f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0000c00 tx=0x7fb3c001a040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.445+0000 7fb3cb400700 1 -- 172.21.15.12:0/3997808219 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb3c40a9c40 con 0x7fb3d41045e0 2024-05-27T07:42:00.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:00 smithi162 ceph-mon[35676]: pgmap v665: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-05-27T07:42:00.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:00 smithi062 ceph-mon[35304]: pgmap v665: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-05-27T07:42:00.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.779+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3d402e5e0 con 0x7fb3d41045e0 2024-05-27T07:42:00.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.781+0000 7fb3cb400700 1 -- 172.21.15.12:0/3997808219 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+4411 (secure 0 0 0) 0x7fb3c40a9540 con 0x7fb3d41045e0 2024-05-27T07:42:00.780 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:42:00.780 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":120,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:41:57.658907+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34415,34439,34475]},"id":1}]} 2024-05-27T07:42:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb3b80b29f0 msgr2=0x7fb3b80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb3b80b29f0 0x7fb3b80b4f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0000c00 tx=0x7fb3c001a040 comp rx=0 tx=0).stop 2024-05-27T07:42:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d41045e0 msgr2=0x7fb3d4075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:00.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d41045e0 0x7fb3d4075ab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400d950 tx=0x7fb3c400dd10 comp rx=0 tx=0).stop 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 shutdown_connections 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fb3b80b29f0 0x7fb3b80b4f00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d41045e0 0x7fb3d4075ab0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d4109860 0x7fb3d4076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 --2- 172.21.15.12:0/3997808219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3d4108f10 0x7fb3d4075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 >> 172.21.15.12:0/3997808219 conn(0x7fb3d4100030 msgr2=0x7fb3d4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.784+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 shutdown_connections 2024-05-27T07:42:00.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:00.785+0000 7fb3d9600700 1 -- 172.21.15.12:0/3997808219 wait complete. 2024-05-27T07:42:00.785 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 120 2024-05-27T07:42:01.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3997808219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:01.519 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:01 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3997808219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:01.597 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:42:01.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:01 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3997808219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:42:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:02.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:02 smithi062 ceph-mon[35304]: pgmap v666: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 669 KiB/s wr, 94 op/s 2024-05-27T07:42:02.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:02 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:02.503 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:02 smithi012 ceph-mon[31473]: pgmap v666: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 669 KiB/s wr, 94 op/s 2024-05-27T07:42:02.503 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:02 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:02.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:02 smithi162 ceph-mon[35676]: pgmap v666: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 669 KiB/s wr, 94 op/s 2024-05-27T07:42:02.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:02 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:04.344 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:04 smithi012 ceph-mon[31473]: pgmap v667: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 669 KiB/s wr, 94 op/s 2024-05-27T07:42:04.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:04 smithi062 ceph-mon[35304]: pgmap v667: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 669 KiB/s wr, 94 op/s 2024-05-27T07:42:04.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 -- 172.21.15.12:0/1867965534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2941045c0 msgr2=0x7fe2941049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 --2- 172.21.15.12:0/1867965534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2941045c0 0x7fe2941049d0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe2840099c0 tx=0x7fe284009cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 -- 172.21.15.12:0/1867965534 shutdown_connections 2024-05-27T07:42:04.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 --2- 172.21.15.12:0/1867965534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe294109840 0x7fe2941111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 --2- 172.21.15.12:0/1867965534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2941045c0 0x7fe2941049d0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 --2- 172.21.15.12:0/1867965534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe294108ef0 0x7fe294109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 -- 172.21.15.12:0/1867965534 >> 172.21.15.12:0/1867965534 conn(0x7fe2940ffe90 msgr2=0x7fe2941022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:04.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 -- 172.21.15.12:0/1867965534 shutdown_connections 2024-05-27T07:42:04.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.603+0000 7fe29aa00700 1 -- 172.21.15.12:0/1867965534 wait complete. 2024-05-27T07:42:04.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.604+0000 7fe29aa00700 1 Processor -- start 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe29aa00700 1 -- start start 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe29aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2941045c0 0x7fe29419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe29aa00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe294108ef0 0x7fe29419d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe29aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe294109840 0x7fe2941a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe29aa00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe294114370 con 0x7fe294109840 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe29aa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941141f0 con 0x7fe2941045c0 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe29aa00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941144f0 con 0x7fe294108ef0 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe298a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe294109840 0x7fe2941a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.605+0000 7fe298a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe294109840 0x7fe2941a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43410/0 (socket says 172.21.15.12:43410) 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe298a00700 1 -- 172.21.15.12:0/3031563956 learned_addr learned my addr 172.21.15.12:0/3031563956 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe28be00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe294108ef0 0x7fe29419d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe293e00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2941045c0 0x7fe29419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe298a00700 1 -- 172.21.15.12:0/3031563956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe294108ef0 msgr2=0x7fe29419d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe298a00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe294108ef0 0x7fe29419d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe298a00700 1 -- 172.21.15.12:0/3031563956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2941045c0 msgr2=0x7fe29419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe298a00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2941045c0 0x7fe29419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.606+0000 7fe298a00700 1 -- 172.21.15.12:0/3031563956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2941a1e10 con 0x7fe294109840 2024-05-27T07:42:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.607+0000 7fe298a00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe294109840 0x7fe2941a1770 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe28c007ee0 tx=0x7fe28c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.607+0000 7fe291600700 1 -- 172.21.15.12:0/3031563956 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c00dbd0 con 0x7fe294109840 2024-05-27T07:42:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.607+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2941a2100 con 0x7fe294109840 2024-05-27T07:42:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.607+0000 7fe291600700 1 -- 172.21.15.12:0/3031563956 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe28c00dd70 con 0x7fe294109840 2024-05-27T07:42:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.607+0000 7fe291600700 1 -- 172.21.15.12:0/3031563956 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c0058d0 con 0x7fe294109840 2024-05-27T07:42:04.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.607+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2941a9a20 con 0x7fe294109840 2024-05-27T07:42:04.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.611+0000 7fe291600700 1 -- 172.21.15.12:0/3031563956 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe28c01a020 con 0x7fe294109840 2024-05-27T07:42:04.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.612+0000 7fe2a9626b80 1 Processor -- start 2024-05-27T07:42:04.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.612+0000 7fe2a9626b80 1 -- start start 2024-05-27T07:42:04.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6be699f0 0x556d6befc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6bd7de40 0x556d6befc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 0x556d6bf09110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556d6be2ae40 con 0x556d6be699f0 2024-05-27T07:42:04.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556d6be1a9e0 con 0x556d6bd7de40 2024-05-27T07:42:04.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d6be1ea20 con 0x556d6befce70 2024-05-27T07:42:04.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe291600700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2780b29f0 0x7fe2780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6be699f0 0x556d6befc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6be699f0 0x556d6befc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43418/0 (socket says 172.21.15.12:43418) 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.613+0000 7fe28be00700 1 -- 172.21.15.12:0/2836582421 learned_addr learned my addr 172.21.15.12:0/2836582421 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe293e00700 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6bd7de40 0x556d6befc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe291600700 1 -- 172.21.15.12:0/3031563956 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fe28c0e22b0 con 0x7fe294109840 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe298a00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2780b29f0 0x7fe2780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe28be00700 1 -- 172.21.15.12:0/2836582421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 msgr2=0x556d6bf09110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe28be00700 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 0x556d6bf09110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe28be00700 1 -- 172.21.15.12:0/2836582421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6bd7de40 msgr2=0x556d6befc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe28be00700 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6bd7de40 0x556d6befc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe28be00700 1 -- 172.21.15.12:0/2836582421 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6bde3420 con 0x556d6be699f0 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.614+0000 7fe28be00700 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6be699f0 0x556d6befc330 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00bd90 tx=0x7fe27c011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93384b515cd2dc9d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.615+0000 7fe298a00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2780b29f0 0x7fe2780b4f00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0a51a0 tx=0x7fe28c00b040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.615+0000 7fe289600700 1 -- 172.21.15.12:0/2836582421 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01a400 con 0x556d6be699f0 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.615+0000 7fe289600700 1 -- 172.21.15.12:0/2836582421 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe27c01aa30 con 0x556d6be699f0 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.615+0000 7fe289600700 1 -- 172.21.15.12:0/2836582421 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c012750 con 0x556d6be699f0 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2836582421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6be699f0 msgr2=0x556d6befc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6be699f0 0x556d6befc330 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00bd90 tx=0x7fe27c011660 comp rx=0 tx=0).stop 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2836582421 shutdown_connections 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 0x556d6bf09110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6be699f0 0x556d6befc330 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2836582421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6bd7de40 0x556d6befc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2836582421 >> 172.21.15.12:0/2836582421 conn(0x556d6be17390 msgr2=0x556d6bef84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.616+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2836582421 shutdown_connections 2024-05-27T07:42:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.617+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2836582421 wait complete. 2024-05-27T07:42:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.617+0000 7fe2a9626b80 1 -- start start 2024-05-27T07:42:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.617+0000 7fe2a9626b80 1 Processor -- start 2024-05-27T07:42:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf06700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6be699f0 0x556d6bf06c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 0x556d6bf07180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556d6bd20fa0 con 0x556d6bd7de40 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556d6bdbb760 con 0x556d6be699f0 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d6bf09b80 con 0x556d6befce70 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe2a9626b80 10 client.0 _ll_register_callbacks cb 0x556d6bdc4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.618+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf06700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf06700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43426/0 (socket says 172.21.15.12:43426) 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe28be00700 1 -- 172.21.15.12:0/1757484634 learned_addr learned my addr 172.21.15.12:0/1757484634 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe293e00700 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6be699f0 0x556d6bf06c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe28be00700 1 -- 172.21.15.12:0/1757484634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 msgr2=0x556d6bf07180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe28be00700 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 0x556d6bf07180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe28be00700 1 -- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6be699f0 msgr2=0x556d6bf06c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe28be00700 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6be699f0 0x556d6bf06c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.619+0000 7fe28be00700 1 -- 172.21.15.12:0/1757484634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d6bd6a8d0 con 0x556d6bd7de40 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe28be00700 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf06700 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00bfe0 tx=0x7fe27c005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17afc219c95bf9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe282000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe282000700 1 -- 172.21.15.12:0/1757484634 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01ae50 con 0x556d6bd7de40 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe282000700 1 -- 172.21.15.12:0/1757484634 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fe27c01a610 con 0x556d6bd7de40 2024-05-27T07:42:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27c01aa30 con 0x556d6bd7de40 2024-05-27T07:42:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe282000700 1 -- 172.21.15.12:0/1757484634 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7fe27c032050 con 0x556d6bd7de40 2024-05-27T07:42:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe2a9626b80 10 client.17718 fetch_fsmap learned FSMap version 121 2024-05-27T07:42:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.620+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x556d6bdeb060 con 0x556d6bd7de40 2024-05-27T07:42:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe282000700 1 -- 172.21.15.12:0/1757484634 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe27c01aa30 con 0x556d6bd7de40 2024-05-27T07:42:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe282000700 1 -- 172.21.15.12:0/1757484634 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 121) ==== 2945+0+0 (secure 0 0 0) 0x7fe27c02d030 con 0x556d6bd7de40 2024-05-27T07:42:04.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe2a9626b80 10 client.17718 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe2a9626b80 10 client.17718 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe2a9626b80 20 client.17718 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf16d10 0x556d6bf191a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe2a9626b80 4 client.17718 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.621+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d6be2cd90 con 0x556d6bf16d10 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.622+0000 7fe293e00700 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf16d10 0x556d6bf191a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.623+0000 7fe293e00700 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf16d10 0x556d6bf191a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=270f144801bd602d server_cookie=626016bb9b6a31da in_seq=0 out_seq=0 2024-05-27T07:42:04.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.623+0000 7fe282000700 10 client.17718 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:42:04.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe282000700 1 -- 172.21.15.12:0/1757484634 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556d6be2cd90 con 0x556d6bf16d10 2024-05-27T07:42:04.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe282000700 10 client.17718 handle_command_reply: tid=0 2024-05-27T07:42:04.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 client.17718 shutdown 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 msgr2=0x556d6bf06700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf06700 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00bfe0 tx=0x7fe27c005cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 shutdown_connections 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6befce70 0x556d6bf07180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf06700 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf16d10 0x556d6bf191a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1757484634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6be699f0 0x556d6bf06c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 >> 172.21.15.12:0/1757484634 conn(0x556d6be17390 msgr2=0x556d6befc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:04.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe282000700 0 client.17718 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:42:04.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 shutdown_connections 2024-05-27T07:42:04.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.626+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1757484634 wait complete. 2024-05-27T07:42:04.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.627+0000 7fe2a9626b80 20 client.17718 trim_cache size 0 max 16384 2024-05-27T07:42:04.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.644+0000 7fe2a9626b80 1 Processor -- start 2024-05-27T07:42:04.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.644+0000 7fe2a9626b80 1 -- start start 2024-05-27T07:42:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf05660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf01ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 0x556d6bf12c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b1e30 con 0x556d6bd7de40 2024-05-27T07:42:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b1060 con 0x556d6befce70 2024-05-27T07:42:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b1760 con 0x556d6be699f0 2024-05-27T07:42:04.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf05660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf05660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43440/0 (socket says 172.21.15.12:43440) 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe28be00700 1 -- 172.21.15.12:0/1224789016 learned_addr learned my addr 172.21.15.12:0/1224789016 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.645+0000 7fe293e00700 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf01ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe28be00700 1 -- 172.21.15.12:0/1224789016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 msgr2=0x556d6bf01ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe28be00700 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf01ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe28be00700 1 -- 172.21.15.12:0/1224789016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 msgr2=0x556d6bf12c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe28be00700 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 0x556d6bf12c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe28be00700 1 -- 172.21.15.12:0/1224789016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6bf074f0 con 0x556d6bd7de40 2024-05-27T07:42:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe28be00700 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf05660 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe27c011ae0 tx=0x7fe27c00b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd8a6bc13f3bbc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe288c00700 1 -- 172.21.15.12:0/1224789016 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c03a070 con 0x556d6bd7de40 2024-05-27T07:42:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe288c00700 1 -- 172.21.15.12:0/1224789016 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe27c011e80 con 0x556d6bd7de40 2024-05-27T07:42:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.646+0000 7fe288c00700 1 -- 172.21.15.12:0/1224789016 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c024e60 con 0x556d6bd7de40 2024-05-27T07:42:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.647+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1224789016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 msgr2=0x556d6bf05660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.647+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf05660 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe27c011ae0 tx=0x7fe27c00b700 comp rx=0 tx=0).stop 2024-05-27T07:42:04.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.647+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1224789016 shutdown_connections 2024-05-27T07:42:04.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.647+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 0x556d6bf12c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.647+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf05660 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.647+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/1224789016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf01ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.647+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1224789016 >> 172.21.15.12:0/1224789016 conn(0x556d6be17390 msgr2=0x556d6beffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:04.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.648+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1224789016 shutdown_connections 2024-05-27T07:42:04.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.648+0000 7fe2a9626b80 1 -- 172.21.15.12:0/1224789016 wait complete. 2024-05-27T07:42:04.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.648+0000 7fe2a9626b80 1 -- start start 2024-05-27T07:42:04.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.648+0000 7fe2a9626b80 1 Processor -- start 2024-05-27T07:42:04.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf0ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf0b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe2a9626b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 0x556d6bf0b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x556d6bf04c70 con 0x556d6bd7de40 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556d6bf19780 con 0x556d6befce70 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe2a9626b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d6befc1b0 con 0x556d6be699f0 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe2a9626b80 10 client.0 _ll_register_callbacks cb 0x556d6bdc4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf0ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe28be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf0ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43450/0 (socket says 172.21.15.12:43450) 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.649+0000 7fe28be00700 1 -- 172.21.15.12:0/2695456636 learned_addr learned my addr 172.21.15.12:0/2695456636 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe293e00700 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf0b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe28be00700 1 -- 172.21.15.12:0/2695456636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 msgr2=0x556d6bf0b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe28be00700 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf0b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe28be00700 1 -- 172.21.15.12:0/2695456636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 msgr2=0x556d6bf0b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe28be00700 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 0x556d6bf0b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe28be00700 1 -- 172.21.15.12:0/2695456636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d6bf80a30 con 0x556d6bd7de40 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe28be00700 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf0ad90 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe27c005950 tx=0x7fe27c00b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb51024e799c99b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe282000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.650+0000 7fe282000700 1 -- 172.21.15.12:0/2695456636 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c042030 con 0x556d6bd7de40 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe282000700 1 -- 172.21.15.12:0/2695456636 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9554+0+0 (secure 0 0 0) 0x7fe27c01a9e0 con 0x556d6bd7de40 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556d6be26570 con 0x556d6bd7de40 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe282000700 1 -- 172.21.15.12:0/2695456636 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7fe27c04c050 con 0x556d6bd7de40 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 10 client.17724 fetch_fsmap learned FSMap version 121 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x556d6bf81c90 con 0x556d6bd7de40 2024-05-27T07:42:04.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe282000700 1 -- 172.21.15.12:0/2695456636 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x556d6be26570 con 0x556d6bd7de40 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe282000700 1 -- 172.21.15.12:0/2695456636 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 121) ==== 2945+0+0 (secure 0 0 0) 0x7fe27c025a40 con 0x556d6bd7de40 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 10 client.17724 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 10 client.17724 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 20 client.17724 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf81ff0 0x556d6bf84410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 4 client.17724 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.651+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d6bc9d250 con 0x556d6bf81ff0 2024-05-27T07:42:04.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.652+0000 7fe293e00700 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf81ff0 0x556d6bf84410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.653+0000 7fe293e00700 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf81ff0 0x556d6bf84410 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=77f0aa9a52e0236f server_cookie=7ee22a51e909fc19 in_seq=0 out_seq=0 2024-05-27T07:42:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.653+0000 7fe282000700 10 client.17724 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:42:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe282000700 1 -- 172.21.15.12:0/2695456636 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d6bc9d250 con 0x556d6bf81ff0 2024-05-27T07:42:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe282000700 10 client.17724 handle_command_reply: tid=0 2024-05-27T07:42:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 client.17724 shutdown 2024-05-27T07:42:04.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 msgr2=0x556d6bf0ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf0ad90 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe27c005950 tx=0x7fe27c00b510 comp rx=0 tx=0).stop 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 shutdown_connections 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x556d6bf81ff0 0x556d6bf84410 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d6befce70 0x556d6bf0b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x556d6bd7de40 0x556d6bf0ad90 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 --2- 172.21.15.12:0/2695456636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d6be699f0 0x556d6bf0b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 >> 172.21.15.12:0/2695456636 conn(0x556d6be17390 msgr2=0x556d6bf07a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe282000700 0 client.17724 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 shutdown_connections 2024-05-27T07:42:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.654+0000 7fe2a9626b80 1 -- 172.21.15.12:0/2695456636 wait complete. 2024-05-27T07:42:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.655+0000 7fe2a9626b80 20 client.17724 trim_cache size 0 max 16384 2024-05-27T07:42:04.655 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.655+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2780b29f0 msgr2=0x7fe2780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.655+0000 7fe29aa00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2780b29f0 0x7fe2780b4f00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0a51a0 tx=0x7fe28c00b040 comp rx=0 tx=0).stop 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.655+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe294109840 msgr2=0x7fe2941a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.655+0000 7fe29aa00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe294109840 0x7fe2941a1770 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe28c007ee0 tx=0x7fe28c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 shutdown_connections 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2780b29f0 0x7fe2780b4f00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2941045c0 0x7fe29419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe294109840 0x7fe2941a1770 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 --2- 172.21.15.12:0/3031563956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe294108ef0 0x7fe29419d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 >> 172.21.15.12:0/3031563956 conn(0x7fe2940ffe90 msgr2=0x7fe29410a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 shutdown_connections 2024-05-27T07:42:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:04.656+0000 7fe29aa00700 1 -- 172.21.15.12:0/3031563956 wait complete. 2024-05-27T07:42:04.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:04 smithi162 ceph-mon[35676]: pgmap v667: 129 pgs: 129 active+clean; 510 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 669 KiB/s wr, 94 op/s 2024-05-27T07:42:05.610 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:42:06.041 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:06.537 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:06 smithi012 ceph-mon[31473]: pgmap v668: 129 pgs: 129 active+clean; 514 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-27T07:42:06.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:06 smithi162 ceph-mon[35676]: pgmap v668: 129 pgs: 129 active+clean; 514 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-27T07:42:06.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:06 smithi062 ceph-mon[35304]: pgmap v668: 129 pgs: 129 active+clean; 514 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-27T07:42:07.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:07 smithi012 ceph-mon[31473]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:42:07.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:07 smithi162 ceph-mon[35676]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:42:07.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:07 smithi062 ceph-mon[35304]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:42:08.335 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:08 smithi012 ceph-mon[31473]: pgmap v670: 129 pgs: 129 active+clean; 514 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 654 KiB/s wr, 90 op/s 2024-05-27T07:42:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 -- 172.21.15.12:0/2057825479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 msgr2=0x7fe1c4114c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/2057825479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c4114c30 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009990 tx=0x7fe1ac009ca0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 -- 172.21.15.12:0/2057825479 shutdown_connections 2024-05-27T07:42:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/2057825479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1c4075b50 0x7fe1c4075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/2057825479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4115170 0x7fe1c4117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/2057825479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c4114c30 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 -- 172.21.15.12:0/2057825479 >> 172.21.15.12:0/2057825479 conn(0x7fe1c4079d00 msgr2=0x7fe1c407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 -- 172.21.15.12:0/2057825479 shutdown_connections 2024-05-27T07:42:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.718+0000 7fe1c8a00700 1 -- 172.21.15.12:0/2057825479 wait complete. 2024-05-27T07:42:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.719+0000 7fe1c8a00700 1 Processor -- start 2024-05-27T07:42:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.720+0000 7fe1c8a00700 1 -- start start 2024-05-27T07:42:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.720+0000 7fe1c8a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4075b50 0x7fe1c41a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.720+0000 7fe1c8a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c41a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.720+0000 7fe1c8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1c4115170 0x7fe1c41a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.720+0000 7fe1c8a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c411a730 con 0x7fe1c40764a0 2024-05-27T07:42:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.720+0000 7fe1c8a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c411a5b0 con 0x7fe1c4115170 2024-05-27T07:42:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.720+0000 7fe1c8a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c411a8b0 con 0x7fe1c4075b50 2024-05-27T07:42:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c41a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1c3400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4075b50 0x7fe1c41a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1c3400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4075b50 0x7fe1c41a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:53726/0 (socket says 172.21.15.12:53726) 2024-05-27T07:42:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1c3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1c4115170 0x7fe1c41a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1c3400700 1 -- 172.21.15.12:0/1700176404 learned_addr learned my addr 172.21.15.12:0/1700176404 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c41a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43470/0 (socket says 172.21.15.12:43470) 2024-05-27T07:42:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1bbe00700 1 -- 172.21.15.12:0/1700176404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4075b50 msgr2=0x7fe1c41a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4075b50 0x7fe1c41a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1bbe00700 1 -- 172.21.15.12:0/1700176404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1c4115170 msgr2=0x7fe1c41a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1c4115170 0x7fe1c41a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.721+0000 7fe1bbe00700 1 -- 172.21.15.12:0/1700176404 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c41a6030 con 0x7fe1c40764a0 2024-05-27T07:42:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.722+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c41a1700 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0059d0 tx=0x7fe1ac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.722+0000 7fe1c0c00700 1 -- 172.21.15.12:0/1700176404 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac027070 con 0x7fe1c40764a0 2024-05-27T07:42:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.722+0000 7fe1c0c00700 1 -- 172.21.15.12:0/1700176404 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1ac00fb00 con 0x7fe1c40764a0 2024-05-27T07:42:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.722+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c41a62c0 con 0x7fe1c40764a0 2024-05-27T07:42:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.722+0000 7fe1c0c00700 1 -- 172.21.15.12:0/1700176404 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac022ac0 con 0x7fe1c40764a0 2024-05-27T07:42:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.722+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c41a67c0 con 0x7fe1c40764a0 2024-05-27T07:42:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.728+0000 7fe1c0c00700 1 -- 172.21.15.12:0/1700176404 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe1ac02a070 con 0x7fe1c40764a0 2024-05-27T07:42:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.729+0000 7fe1d8b05b80 1 Processor -- start 2024-05-27T07:42:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.729+0000 7fe1d8b05b80 1 -- start start 2024-05-27T07:42:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.729+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e975f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e95dfe40 0x5635e975fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e976b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5635e968ce40 con 0x5635e95dfe40 2024-05-27T07:42:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5635e967c9e0 con 0x5635e96cb9f0 2024-05-27T07:42:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5635e9680a20 con 0x5635e97692c0 2024-05-27T07:42:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1c0c00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe1a80b2a70 0x7fe1a80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe1a80b2a70 0x7fe1a80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1c0c00700 1 -- 172.21.15.12:0/1700176404 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 9642+0+0 (secure 0 0 0) 0x7fe1ac0e3c40 con 0x7fe1c40764a0 2024-05-27T07:42:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1c3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e95dfe40 0x5635e975fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1c3400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e95dfe40 0x5635e975fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43480/0 (socket says 172.21.15.12:43480) 2024-05-27T07:42:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1c3400700 1 -- 172.21.15.12:0/3873774656 learned_addr learned my addr 172.21.15.12:0/3873774656 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e975f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.730+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e976b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3400700 1 -- 172.21.15.12:0/3873774656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 msgr2=0x5635e976b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3400700 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e976b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3400700 1 -- 172.21.15.12:0/3873774656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 msgr2=0x5635e975f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3400700 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e975f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3400700 1 -- 172.21.15.12:0/3873774656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e9645420 con 0x5635e95dfe40 2024-05-27T07:42:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e975f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:42:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe1a80b2a70 0x7fe1a80b4f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009550 tx=0x7fe1bc009c10 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.731+0000 7fe1c3400700 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e95dfe40 0x5635e975fc90 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400bea0 tx=0x7fe1b40106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0940c56ed1f2a87 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.732+0000 7fe1b8c00700 1 -- 172.21.15.12:0/3873774656 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4011440 con 0x5635e95dfe40 2024-05-27T07:42:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.732+0000 7fe1b8c00700 1 -- 172.21.15.12:0/3873774656 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1b4011a70 con 0x5635e95dfe40 2024-05-27T07:42:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.732+0000 7fe1b8c00700 1 -- 172.21.15.12:0/3873774656 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40197f0 con 0x5635e95dfe40 2024-05-27T07:42:08.736 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:08 smithi162 ceph-mon[35676]: pgmap v670: 129 pgs: 129 active+clean; 514 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 654 KiB/s wr, 90 op/s 2024-05-27T07:42:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3873774656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e95dfe40 msgr2=0x5635e975fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e95dfe40 0x5635e975fc90 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400bea0 tx=0x7fe1b40106b0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3873774656 shutdown_connections 2024-05-27T07:42:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e975f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e976b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3873774656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e95dfe40 0x5635e975fc90 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3873774656 >> 172.21.15.12:0/3873774656 conn(0x5635e9679390 msgr2=0x5635e975a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3873774656 shutdown_connections 2024-05-27T07:42:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.733+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3873774656 wait complete. 2024-05-27T07:42:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.734+0000 7fe1d8b05b80 1 -- start start 2024-05-27T07:42:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.734+0000 7fe1d8b05b80 1 Processor -- start 2024-05-27T07:42:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9766230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e96cb9f0 0x5635e9766770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e9766cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5635e9582fa0 con 0x5635e96cb9f0 2024-05-27T07:42:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5635e961d760 con 0x5635e95dfe40 2024-05-27T07:42:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5635e976be00 con 0x5635e97692c0 2024-05-27T07:42:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1d8b05b80 10 client.0 _ll_register_callbacks cb 0x5635e9626c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e96cb9f0 0x5635e9766770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.735+0000 7fe1c3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9766230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1c3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9766230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:53710/0 (socket says 172.21.15.12:53710) 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e96cb9f0 0x5635e9766770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43496/0 (socket says 172.21.15.12:43496) 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1c3400700 1 -- 172.21.15.12:0/3798032608 learned_addr learned my addr 172.21.15.12:0/3798032608 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e9766cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1bbe00700 1 -- 172.21.15.12:0/3798032608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 msgr2=0x5635e9766cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e9766cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1bbe00700 1 -- 172.21.15.12:0/3798032608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 msgr2=0x5635e9766230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9766230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1bbe00700 1 -- 172.21.15.12:0/3798032608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e95cc8d0 con 0x5635e96cb9f0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e96cb9f0 0x5635e9766770 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0e0ed0 tx=0x7fe1ac0ab030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d720c78df5bf305 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.736+0000 7fe1b1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1b1600700 1 -- 172.21.15.12:0/3798032608 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0a6280 con 0x5635e96cb9f0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1b1600700 1 -- 172.21.15.12:0/3798032608 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9642+0+0 (secure 0 0 0) 0x7fe1ac00f880 con 0x5635e96cb9f0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b4011a70 con 0x5635e96cb9f0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1b1600700 1 -- 172.21.15.12:0/3798032608 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7fe1ac0b3200 con 0x5635e96cb9f0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1d8b05b80 10 client.17736 fetch_fsmap learned FSMap version 121 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x5635e964d060 con 0x5635e96cb9f0 2024-05-27T07:42:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1b1600700 1 -- 172.21.15.12:0/3798032608 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1b4011a70 con 0x5635e96cb9f0 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.737+0000 7fe1b1600700 1 -- 172.21.15.12:0/3798032608 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 121) ==== 2945+0+0 (secure 0 0 0) 0x7fe1ac0710d0 con 0x5635e96cb9f0 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.738+0000 7fe1d8b05b80 10 client.17736 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.738+0000 7fe1d8b05b80 10 client.17736 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.738+0000 7fe1d8b05b80 20 client.17736 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.738+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e9778de0 0x5635e977b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.738+0000 7fe1d8b05b80 4 client.17736 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.738+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635e968ed90 con 0x5635e9778de0 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.738+0000 7fe1c3400700 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e9778de0 0x5635e977b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.739+0000 7fe1c3400700 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e9778de0 0x5635e977b290 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=14e90a7d4bbb97e1 server_cookie=f72458d789fbc46f in_seq=0 out_seq=0 2024-05-27T07:42:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.739+0000 7fe1b1600700 10 client.17736 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:42:08.746 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:08 smithi062 ceph-mon[35304]: pgmap v670: 129 pgs: 129 active+clean; 514 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 654 KiB/s wr, 90 op/s 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.751+0000 7fe1b1600700 1 -- 172.21.15.12:0/3798032608 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5635e968ed90 con 0x5635e9778de0 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.751+0000 7fe1b1600700 10 client.17736 handle_command_reply: tid=0 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 client.17736 shutdown 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e96cb9f0 msgr2=0x5635e9766770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e96cb9f0 0x5635e9766770 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0e0ed0 tx=0x7fe1ac0ab030 comp rx=0 tx=0).stop 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 shutdown_connections 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e9778de0 0x5635e977b290 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9766230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e97692c0 0x5635e9766cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/3798032608 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e96cb9f0 0x5635e9766770 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 >> 172.21.15.12:0/3798032608 conn(0x5635e9679390 msgr2=0x5635e975a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1b1600700 0 client.17736 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:42:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 shutdown_connections 2024-05-27T07:42:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.752+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/3798032608 wait complete. 2024-05-27T07:42:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.753+0000 7fe1d8b05b80 20 client.17736 trim_cache size 0 max 16384 2024-05-27T07:42:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.770+0000 7fe1d8b05b80 1 Processor -- start 2024-05-27T07:42:08.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.771+0000 7fe1d8b05b80 1 -- start start 2024-05-27T07:42:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.771+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e95dfe40 0x5635e975a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.771+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e9765190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.771+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e9774ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.771+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c41b6060 con 0x5635e97692c0 2024-05-27T07:42:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.771+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c41b5290 con 0x5635e96cb9f0 2024-05-27T07:42:08.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.771+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c41b5990 con 0x5635e95dfe40 2024-05-27T07:42:08.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e9774ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e9774ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43512/0 (socket says 172.21.15.12:43512) 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e95dfe40 0x5635e975a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 -- 172.21.15.12:0/350949919 learned_addr learned my addr 172.21.15.12:0/350949919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e9765190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 -- 172.21.15.12:0/350949919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e95dfe40 msgr2=0x5635e975a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e95dfe40 0x5635e975a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 -- 172.21.15.12:0/350949919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 msgr2=0x5635e9765190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e9765190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.772+0000 7fe1c3e00700 1 -- 172.21.15.12:0/350949919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e97647a0 con 0x5635e97692c0 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.773+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e9774ea0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0199f0 tx=0x7fe1bc00d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b00baca8dcdf829 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.773+0000 7fe1b3e00700 1 -- 172.21.15.12:0/350949919 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc004bb0 con 0x5635e97692c0 2024-05-27T07:42:08.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.773+0000 7fe1b3e00700 1 -- 172.21.15.12:0/350949919 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1bc004d50 con 0x5635e97692c0 2024-05-27T07:42:08.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.773+0000 7fe1b3e00700 1 -- 172.21.15.12:0/350949919 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc027760 con 0x5635e97692c0 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/350949919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 msgr2=0x5635e9774ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e9774ea0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0199f0 tx=0x7fe1bc00d000 comp rx=0 tx=0).stop 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/350949919 shutdown_connections 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e9774ea0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e95dfe40 0x5635e975a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/350949919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e96cb9f0 0x5635e9765190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/350949919 >> 172.21.15.12:0/350949919 conn(0x5635e9679390 msgr2=0x5635e975e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/350949919 shutdown_connections 2024-05-27T07:42:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.774+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/350949919 wait complete. 2024-05-27T07:42:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.775+0000 7fe1d8b05b80 1 -- start start 2024-05-27T07:42:08.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.775+0000 7fe1d8b05b80 1 Processor -- start 2024-05-27T07:42:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9771900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e96cb9f0 0x5635e976ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1d8b05b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e976d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5635e968ed90 con 0x5635e97692c0 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5635e975b1b0 con 0x5635e95dfe40 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1d8b05b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5635e97676c0 con 0x5635e96cb9f0 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1d8b05b80 10 client.0 _ll_register_callbacks cb 0x5635e9626c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e976d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1c3e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e976d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43514/0 (socket says 172.21.15.12:43514) 2024-05-27T07:42:08.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1c3e00700 1 -- 172.21.15.12:0/186152221 learned_addr learned my addr 172.21.15.12:0/186152221 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1c3400700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9771900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.776+0000 7fe1bbe00700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e96cb9f0 0x5635e976ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1c3e00700 1 -- 172.21.15.12:0/186152221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e96cb9f0 msgr2=0x5635e976ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e96cb9f0 0x5635e976ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1c3e00700 1 -- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 msgr2=0x5635e9771900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9771900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1c3e00700 1 -- 172.21.15.12:0/186152221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e976dad0 con 0x5635e97692c0 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1c3400700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9771900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1c3e00700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e976d300 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009140 tx=0x7fe1bc01bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee8f766e5f409ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.777+0000 7fe1b1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.778+0000 7fe1b1600700 1 -- 172.21.15.12:0/186152221 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc02b040 con 0x5635e97692c0 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.778+0000 7fe1b1600700 1 -- 172.21.15.12:0/186152221 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 9642+0+0 (secure 0 0 0) 0x7fe1bc027470 con 0x5635e97692c0 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.778+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1bc004d50 con 0x5635e97692c0 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.778+0000 7fe1b1600700 1 -- 172.21.15.12:0/186152221 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7fe1bc03e050 con 0x5635e97692c0 2024-05-27T07:42:08.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.778+0000 7fe1d8b05b80 10 client.17742 fetch_fsmap learned FSMap version 121 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.778+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x5635e97e3990 con 0x5635e97692c0 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.778+0000 7fe1b1600700 1 -- 172.21.15.12:0/186152221 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe1bc004d50 con 0x5635e97692c0 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1b1600700 1 -- 172.21.15.12:0/186152221 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 121) ==== 2945+0+0 (secure 0 0 0) 0x7fe1bc015070 con 0x5635e97692c0 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1d8b05b80 10 client.17742 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1d8b05b80 10 client.17742 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1d8b05b80 20 client.17742 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e97e3eb0 0x5635e97e62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1d8b05b80 4 client.17742 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:42:08.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5635e9688570 con 0x5635e97e3eb0 2024-05-27T07:42:08.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.779+0000 7fe1c3400700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e97e3eb0 0x5635e97e62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:08.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.780+0000 7fe1c3400700 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e97e3eb0 0x5635e97e62d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=757ed49bdd7709dc server_cookie=8a3cb5aa86851c20 in_seq=0 out_seq=0 2024-05-27T07:42:08.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.780+0000 7fe1b1600700 10 client.17742 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:42:08.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.792+0000 7fe1b1600700 1 -- 172.21.15.12:0/186152221 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5635e9688570 con 0x5635e97e3eb0 2024-05-27T07:42:08.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.792+0000 7fe1b1600700 10 client.17742 handle_command_reply: tid=0 2024-05-27T07:42:08.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.792+0000 7fe1d8b05b80 1 client.17742 shutdown 2024-05-27T07:42:08.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.792+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 msgr2=0x5635e976d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.792+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e976d300 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009140 tx=0x7fe1bc01bca0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 shutdown_connections 2024-05-27T07:42:08.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5635e97e3eb0 0x5635e97e62d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635e95dfe40 0x5635e9771900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5635e97692c0 0x5635e976d300 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 --2- 172.21.15.12:0/186152221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635e96cb9f0 0x5635e976ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1b1600700 0 client.17742 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:42:08.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 >> 172.21.15.12:0/186152221 conn(0x5635e9679390 msgr2=0x5635e97618e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 shutdown_connections 2024-05-27T07:42:08.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 1 -- 172.21.15.12:0/186152221 wait complete. 2024-05-27T07:42:08.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.793+0000 7fe1d8b05b80 20 client.17742 trim_cache size 0 max 16384 2024-05-27T07:42:08.794 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:42:08.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe1a80b2a70 msgr2=0x7fe1a80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe1a80b2a70 0x7fe1a80b4f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009550 tx=0x7fe1bc009c10 comp rx=0 tx=0).stop 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 msgr2=0x7fe1c41a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c41a1700 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0059d0 tx=0x7fe1ac0047b0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 shutdown_connections 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe1a80b2a70 0x7fe1a80b4f80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1c4115170 0x7fe1c41a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4075b50 0x7fe1c41a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 --2- 172.21.15.12:0/1700176404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe1c40764a0 0x7fe1c41a1700 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.794+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 >> 172.21.15.12:0/1700176404 conn(0x7fe1c4079d00 msgr2=0x7fe1c4100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.795+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 shutdown_connections 2024-05-27T07:42:08.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:08.795+0000 7fe1c8a00700 1 -- 172.21.15.12:0/1700176404 wait complete. 2024-05-27T07:42:09.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:42:10.109 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:10.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:10 smithi012 ceph-mon[31473]: pgmap v671: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 KiB/s wr, 98 op/s 2024-05-27T07:42:10.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:10 smithi162 ceph-mon[35676]: pgmap v671: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 KiB/s wr, 98 op/s 2024-05-27T07:42:10.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:10 smithi062 ceph-mon[35304]: pgmap v671: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 KiB/s wr, 98 op/s 2024-05-27T07:42:12.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:12 smithi012 ceph-mon[31473]: pgmap v672: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 KiB/s wr, 98 op/s 2024-05-27T07:42:12.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:12 smithi162 ceph-mon[35676]: pgmap v672: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 KiB/s wr, 98 op/s 2024-05-27T07:42:12.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:12 smithi062 ceph-mon[35304]: pgmap v672: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 KiB/s wr, 98 op/s 2024-05-27T07:42:13.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 -- 172.21.15.12:0/707655103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 msgr2=0x7f7164108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 --2- 172.21.15.12:0/707655103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f7164108020 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f71500099c0 tx=0x7f7150009cd0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 -- 172.21.15.12:0/707655103 shutdown_connections 2024-05-27T07:42:13.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 --2- 172.21.15.12:0/707655103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7164108560 0x7f716410fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 --2- 172.21.15.12:0/707655103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71641032e0 0x7f71641036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 --2- 172.21.15.12:0/707655103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f7164108020 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 -- 172.21.15.12:0/707655103 >> 172.21.15.12:0/707655103 conn(0x7f7164078af0 msgr2=0x7f7164078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:13.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 -- 172.21.15.12:0/707655103 shutdown_connections 2024-05-27T07:42:13.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.162+0000 7f716ae00700 1 -- 172.21.15.12:0/707655103 wait complete. 2024-05-27T07:42:13.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.164+0000 7f716ae00700 1 Processor -- start 2024-05-27T07:42:13.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.164+0000 7f716ae00700 1 -- start start 2024-05-27T07:42:13.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.165+0000 7f716ae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71641032e0 0x7f71641a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.165+0000 7f716ae00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f71641a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.165+0000 7f716ae00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7164108560 0x7f716419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.165+0000 7f716ae00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7164113070 con 0x7f7164107c10 2024-05-27T07:42:13.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.165+0000 7f716ae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7164112ef0 con 0x7f71641032e0 2024-05-27T07:42:13.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.165+0000 7f716ae00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71641131f0 con 0x7f7164108560 2024-05-27T07:42:13.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.165+0000 7f7169000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f71641a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f71641a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45410/0 (socket says 172.21.15.12:45410) 2024-05-27T07:42:13.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169000700 1 -- 172.21.15.12:0/654609983 learned_addr learned my addr 172.21.15.12:0/654609983 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:13.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169a00700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71641032e0 0x7f71641a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169000700 1 -- 172.21.15.12:0/654609983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7164108560 msgr2=0x7f716419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169000700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7164108560 0x7f716419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169000700 1 -- 172.21.15.12:0/654609983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71641032e0 msgr2=0x7f71641a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169000700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71641032e0 0x7f71641a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.166+0000 7f7169000700 1 -- 172.21.15.12:0/654609983 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716419b0d0 con 0x7f7164107c10 2024-05-27T07:42:13.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.167+0000 7f7169000700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f71641a0d40 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f71500098e0 tx=0x7f71500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:13.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.167+0000 7f7162000700 1 -- 172.21.15.12:0/654609983 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150027070 con 0x7f7164107c10 2024-05-27T07:42:13.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.167+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716419b360 con 0x7f7164107c10 2024-05-27T07:42:13.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.167+0000 7f7162000700 1 -- 172.21.15.12:0/654609983 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f715000fd30 con 0x7f7164107c10 2024-05-27T07:42:13.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.167+0000 7f7162000700 1 -- 172.21.15.12:0/654609983 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150022e30 con 0x7f7164107c10 2024-05-27T07:42:13.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.167+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71641a7690 con 0x7f7164107c10 2024-05-27T07:42:13.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.171+0000 7f7162000700 1 -- 172.21.15.12:0/654609983 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7150022600 con 0x7f7164107c10 2024-05-27T07:42:13.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.173+0000 7f717b029b80 1 Processor -- start 2024-05-27T07:42:13.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.173+0000 7f7162000700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f714c0b2a70 0x7f714c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.173+0000 7f717b029b80 1 -- start start 2024-05-27T07:42:13.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.173+0000 7f7162000700 1 -- 172.21.15.12:0/654609983 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f71500e3890 con 0x7f7164107c10 2024-05-27T07:42:13.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f716a400700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f714c0b2a70 0x7f714c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f717b029b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f717b029b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563cc100 0x5607563cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f717b029b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563daee0 0x5607563dd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f717b029b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5607562fae40 con 0x5607563cc100 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f717b029b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5607562ea9e0 con 0x5607563daee0 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f717b029b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5607562eea20 con 0x56075624de40 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f7169000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563cc100 0x5607563cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f7169000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563cc100 0x5607563cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45424/0 (socket says 172.21.15.12:45424) 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f7169000700 1 -- 172.21.15.12:0/2886701377 learned_addr learned my addr 172.21.15.12:0/2886701377 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.174+0000 7f7169a00700 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f716a400700 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563daee0 0x5607563dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f7169000700 1 -- 172.21.15.12:0/2886701377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 msgr2=0x5607563cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f7169000700 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f716a400700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f714c0b2a70 0x7f714c0b4f80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f715c009710 tx=0x7f715c009ef0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f7169000700 1 -- 172.21.15.12:0/2886701377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563daee0 msgr2=0x5607563dd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f7169000700 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563daee0 0x5607563dd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f7169000700 1 -- 172.21.15.12:0/2886701377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607562b3420 con 0x5607563cc100 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.175+0000 7f7169000700 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563cc100 0x5607563cc510 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7150024000 tx=0x7f71500aa160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1bf64f39214796 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.176+0000 7f7156a00700 1 -- 172.21.15.12:0/2886701377 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500aa2a0 con 0x5607563cc100 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.176+0000 7f7156a00700 1 -- 172.21.15.12:0/2886701377 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71500aae90 con 0x5607563cc100 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.176+0000 7f7156a00700 1 -- 172.21.15.12:0/2886701377 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500922d0 con 0x5607563cc100 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 -- 172.21.15.12:0/2886701377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563cc100 msgr2=0x5607563cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563cc100 0x5607563cc510 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7150024000 tx=0x7f71500aa160 comp rx=0 tx=0).stop 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 -- 172.21.15.12:0/2886701377 shutdown_connections 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563daee0 0x5607563dd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 --2- 172.21.15.12:0/2886701377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563cc100 0x5607563cc510 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 -- 172.21.15.12:0/2886701377 >> 172.21.15.12:0/2886701377 conn(0x5607562adfc0 msgr2=0x5607563c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:13.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 -- 172.21.15.12:0/2886701377 shutdown_connections 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.177+0000 7f717b029b80 1 -- 172.21.15.12:0/2886701377 wait complete. 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.178+0000 7f717b029b80 1 -- start start 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.178+0000 7f717b029b80 1 Processor -- start 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f717b029b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563d7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f717b029b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f717b029b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563daee0 0x5607563d8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f717b029b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5607561f0fa0 con 0x5607563daee0 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f717b029b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56075628b760 con 0x5607563cc100 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f717b029b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5607563ddb10 con 0x56075624de40 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f717b029b80 10 client.0 _ll_register_callbacks cb 0x560756294c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f716a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563daee0 0x5607563d8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f716a400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563daee0 0x5607563d8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45440/0 (socket says 172.21.15.12:45440) 2024-05-27T07:42:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f716a400700 1 -- 172.21.15.12:0/2059964947 learned_addr learned my addr 172.21.15.12:0/2059964947 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f7169000700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.179+0000 7f7169a00700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563d7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f716a400700 1 -- 172.21.15.12:0/2059964947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 msgr2=0x5607563d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f716a400700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563d7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f716a400700 1 -- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 msgr2=0x5607563d8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f716a400700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f716a400700 1 -- 172.21.15.12:0/2059964947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56075623a8d0 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f7169000700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f716a400700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563daee0 0x5607563d8a50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f715c0189b0 tx=0x7f715c006040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b02c07b174ceeb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f714b400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.180+0000 7f714b400700 1 -- 172.21.15.12:0/2059964947 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c018e20 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.181+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71500aae90 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.181+0000 7f714b400700 1 -- 172.21.15.12:0/2059964947 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f715c012070 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.181+0000 7f714b400700 1 -- 172.21.15.12:0/2059964947 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7f715c02f050 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.181+0000 7f717b029b80 10 client.17754 fetch_fsmap learned FSMap version 121 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.181+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x5607562bb060 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.181+0000 7f714b400700 1 -- 172.21.15.12:0/2059964947 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71500aae90 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.181+0000 7f714b400700 1 -- 172.21.15.12:0/2059964947 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 121) ==== 2945+0+0 (secure 0 0 0) 0x7f715c021b80 con 0x5607563daee0 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.182+0000 7f717b029b80 10 client.17754 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.182+0000 7f717b029b80 10 client.17754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.182+0000 7f717b029b80 20 client.17754 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:13.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.182+0000 7f717b029b80 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563eae90 0x5607563ed340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.182+0000 7f717b029b80 4 client.17754 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:13.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.182+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607562fcd90 con 0x5607563eae90 2024-05-27T07:42:13.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.182+0000 7f7169a00700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563eae90 0x5607563ed340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.183+0000 7f7169a00700 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563eae90 0x5607563ed340 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd110809fe0cd3ba server_cookie=bc1fa77d13de70f0 in_seq=0 out_seq=0 2024-05-27T07:42:13.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.183+0000 7f714b400700 10 client.17754 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f714b400700 1 -- 172.21.15.12:0/2059964947 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5607562fcd90 con 0x5607563eae90 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f714b400700 10 client.17754 handle_command_reply: tid=0 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 client.17754 shutdown 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563daee0 msgr2=0x5607563d8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563daee0 0x5607563d8a50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f715c0189b0 tx=0x7f715c006040 comp rx=0 tx=0).stop 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 shutdown_connections 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5607563daee0 0x5607563d8a50 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56075624de40 0x5607563d7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563eae90 0x5607563ed340 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 --2- 172.21.15.12:0/2059964947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 >> 172.21.15.12:0/2059964947 conn(0x5607562adfc0 msgr2=0x5607563c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f714b400700 0 client.17754 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.185+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 shutdown_connections 2024-05-27T07:42:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.186+0000 7f717b029b80 1 -- 172.21.15.12:0/2059964947 wait complete. 2024-05-27T07:42:13.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.186+0000 7f717b029b80 20 client.17754 trim_cache size 0 max 16384 2024-05-27T07:42:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.211+0000 7f717b029b80 1 Processor -- start 2024-05-27T07:42:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.211+0000 7f717b029b80 1 -- start start 2024-05-27T07:42:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.211+0000 7f717b029b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563da470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.211+0000 7f717b029b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563d3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.211+0000 7f717b029b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563e6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f717b029b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f71641ad780 con 0x56075624de40 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f717b029b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f71641ac9b0 con 0x5607563cc100 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f717b029b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71641ad0b0 con 0x5607562adfc0 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f7169000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563d3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f7169000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563d3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45450/0 (socket says 172.21.15.12:45450) 2024-05-27T07:42:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f7169000700 1 -- 172.21.15.12:0/379569895 learned_addr learned my addr 172.21.15.12:0/379569895 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f716a400700 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563e6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.212+0000 7f7169a00700 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563da470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.213+0000 7f7169000700 1 -- 172.21.15.12:0/379569895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 msgr2=0x5607563da470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.213+0000 7f7169000700 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563da470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.213+0000 7f7169000700 1 -- 172.21.15.12:0/379569895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 msgr2=0x5607563e6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.213+0000 7f7169000700 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563e6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.213+0000 7f7169000700 1 -- 172.21.15.12:0/379569895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607563d72f0 con 0x56075624de40 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.213+0000 7f716a400700 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563e6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.213+0000 7f7169000700 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563d3590 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f71500aa720 tx=0x7f71500cd0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e5f101d6cc0497 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.214+0000 7f7156000700 1 -- 172.21.15.12:0/379569895 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500c3020 con 0x56075624de40 2024-05-27T07:42:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.214+0000 7f7156000700 1 -- 172.21.15.12:0/379569895 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71500aae90 con 0x56075624de40 2024-05-27T07:42:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.214+0000 7f7156000700 1 -- 172.21.15.12:0/379569895 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500920d0 con 0x56075624de40 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 -- 172.21.15.12:0/379569895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 msgr2=0x5607563d3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563d3590 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f71500aa720 tx=0x7f71500cd0f0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 -- 172.21.15.12:0/379569895 shutdown_connections 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563e6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563da470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 --2- 172.21.15.12:0/379569895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563d3590 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 -- 172.21.15.12:0/379569895 >> 172.21.15.12:0/379569895 conn(0x5607563c0f60 msgr2=0x5607563d2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 -- 172.21.15.12:0/379569895 shutdown_connections 2024-05-27T07:42:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.215+0000 7f717b029b80 1 -- 172.21.15.12:0/379569895 wait complete. 2024-05-27T07:42:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.216+0000 7f717b029b80 1 -- start start 2024-05-27T07:42:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.216+0000 7f717b029b80 1 Processor -- start 2024-05-27T07:42:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f717b029b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563e2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f717b029b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563e3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f717b029b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f717b029b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5607563d3ad0 con 0x56075624de40 2024-05-27T07:42:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f717b029b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5607563d6540 con 0x5607563cc100 2024-05-27T07:42:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f717b029b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5607563c8000 con 0x5607562adfc0 2024-05-27T07:42:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f717b029b80 10 client.0 _ll_register_callbacks cb 0x560756294c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f7169a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563e2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f7169a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563e2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45464/0 (socket says 172.21.15.12:45464) 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f7169a00700 1 -- 172.21.15.12:0/2686409079 learned_addr learned my addr 172.21.15.12:0/2686409079 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.217+0000 7f7169000700 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563e3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f716a400700 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f7169a00700 1 -- 172.21.15.12:0/2686409079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 msgr2=0x5607563e3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f7169a00700 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563e3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f7169a00700 1 -- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 msgr2=0x5607563d5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f7169a00700 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f7169a00700 1 -- 172.21.15.12:0/2686409079 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607563d5ac0 con 0x56075624de40 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f7169a00700 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563e2dd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7158000c00 tx=0x7f715800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376bc6c7abe089f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.218+0000 7f714b400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.219+0000 7f714b400700 1 -- 172.21.15.12:0/2686409079 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158003d90 con 0x56075624de40 2024-05-27T07:42:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.219+0000 7f714b400700 1 -- 172.21.15.12:0/2686409079 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f7158003f30 con 0x56075624de40 2024-05-27T07:42:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.219+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71500aae90 con 0x56075624de40 2024-05-27T07:42:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.219+0000 7f714b400700 1 -- 172.21.15.12:0/2686409079 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7f7158030050 con 0x56075624de40 2024-05-27T07:42:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.219+0000 7f717b029b80 10 client.17760 fetch_fsmap learned FSMap version 121 2024-05-27T07:42:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.219+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x5607563dc410 con 0x56075624de40 2024-05-27T07:42:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.220+0000 7f714b400700 1 -- 172.21.15.12:0/2686409079 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71500aae90 con 0x56075624de40 2024-05-27T07:42:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.220+0000 7f714b400700 1 -- 172.21.15.12:0/2686409079 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 121) ==== 2945+0+0 (secure 0 0 0) 0x7f7158027420 con 0x56075624de40 2024-05-27T07:42:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.221+0000 7f717b029b80 10 client.17760 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:42:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.221+0000 7f717b029b80 10 client.17760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.221+0000 7f717b029b80 20 client.17760 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.221+0000 7f717b029b80 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563daee0 0x560756457c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.221+0000 7f717b029b80 4 client.17760 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:42:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.221+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5607563dd770 con 0x5607563daee0 2024-05-27T07:42:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.221+0000 7f7169000700 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563daee0 0x560756457c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.222+0000 7f7169000700 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563daee0 0x560756457c80 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3adaad09e9c2e28f server_cookie=e100514433b10296 in_seq=0 out_seq=0 2024-05-27T07:42:13.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.222+0000 7f714b400700 10 client.17760 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:13.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f714b400700 1 -- 172.21.15.12:0/2686409079 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5607563dd770 con 0x5607563daee0 2024-05-27T07:42:13.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f714b400700 10 client.17760 handle_command_reply: tid=0 2024-05-27T07:42:13.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 client.17760 shutdown 2024-05-27T07:42:13.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 msgr2=0x5607563e2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563e2dd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7158000c00 tx=0x7f715800bd50 comp rx=0 tx=0).stop 2024-05-27T07:42:13.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 shutdown_connections 2024-05-27T07:42:13.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x5607563daee0 0x560756457c80 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607563cc100 0x5607563d5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56075624de40 0x5607563e2dd0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 --2- 172.21.15.12:0/2686409079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607562adfc0 0x5607563e3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 >> 172.21.15.12:0/2686409079 conn(0x5607563c0f60 msgr2=0x5607563cd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f714b400700 0 client.17760 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.223+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 shutdown_connections 2024-05-27T07:42:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.224+0000 7f717b029b80 1 -- 172.21.15.12:0/2686409079 wait complete. 2024-05-27T07:42:13.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.224+0000 7f717b029b80 20 client.17760 trim_cache size 0 max 16384 2024-05-27T07:42:13.223 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f714c0b2a70 msgr2=0x7f714c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f714c0b2a70 0x7f714c0b4f80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f715c009710 tx=0x7f715c009ef0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 msgr2=0x7f71641a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f71641a0d40 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f71500098e0 tx=0x7f71500047b0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 shutdown_connections 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f714c0b2a70 0x7f714c0b4f80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71641032e0 0x7f71641a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7164108560 0x7f716419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 --2- 172.21.15.12:0/654609983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164107c10 0x7f71641a0d40 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 >> 172.21.15.12:0/654609983 conn(0x7f7164078af0 msgr2=0x7f716406c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 shutdown_connections 2024-05-27T07:42:13.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:13.225+0000 7f716ae00700 1 -- 172.21.15.12:0/654609983 wait complete. 2024-05-27T07:42:13.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:13 smithi062 ceph-mon[35304]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:42:13.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:13 smithi062 ceph-mon[35304]: pgmap v674: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 585 KiB/s wr, 58 op/s 2024-05-27T07:42:14.011 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:13 smithi012 ceph-mon[31473]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:42:14.011 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:13 smithi012 ceph-mon[31473]: pgmap v674: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 585 KiB/s wr, 58 op/s 2024-05-27T07:42:14.231 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:13 smithi162 ceph-mon[35676]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:42:14.232 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:13 smithi162 ceph-mon[35676]: pgmap v674: 129 pgs: 129 active+clean; 517 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 585 KiB/s wr, 58 op/s 2024-05-27T07:42:15.554 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:42:15.554 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:42:16.031 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:16.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:16 smithi162 ceph-mon[35676]: pgmap v675: 129 pgs: 129 active+clean; 521 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1005 KiB/s wr, 126 op/s 2024-05-27T07:42:16.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:16 smithi062 ceph-mon[35304]: pgmap v675: 129 pgs: 129 active+clean; 521 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1005 KiB/s wr, 126 op/s 2024-05-27T07:42:16.519 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:16 smithi012 ceph-mon[31473]: pgmap v675: 129 pgs: 129 active+clean; 521 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1005 KiB/s wr, 126 op/s 2024-05-27T07:42:18.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:18 smithi162 ceph-mon[35676]: pgmap v676: 129 pgs: 129 active+clean; 521 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 893 KiB/s wr, 112 op/s 2024-05-27T07:42:18.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:18 smithi062 ceph-mon[35304]: pgmap v676: 129 pgs: 129 active+clean; 521 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 893 KiB/s wr, 112 op/s 2024-05-27T07:42:18.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:18 smithi012 ceph-mon[31473]: pgmap v676: 129 pgs: 129 active+clean; 521 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 893 KiB/s wr, 112 op/s 2024-05-27T07:42:19.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.199+0000 7fa3c2a00700 1 -- 172.21.15.12:0/651408177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 msgr2=0x7fa3bc10e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.199+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/651408177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 0x7fa3bc10e3b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009990 tx=0x7fa3a8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.200+0000 7fa3c2a00700 1 -- 172.21.15.12:0/651408177 shutdown_connections 2024-05-27T07:42:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.200+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/651408177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3bc10e8f0 0x7fa3bc110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.200+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/651408177 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 0x7fa3bc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.200+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/651408177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 0x7fa3bc10e3b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.200+0000 7fa3c2a00700 1 -- 172.21.15.12:0/651408177 >> 172.21.15.12:0/651408177 conn(0x7fa3bc0fedb0 msgr2=0x7fa3bc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.201+0000 7fa3c2a00700 1 -- 172.21.15.12:0/651408177 shutdown_connections 2024-05-27T07:42:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.201+0000 7fa3c2a00700 1 -- 172.21.15.12:0/651408177 wait complete. 2024-05-27T07:42:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.203+0000 7fa3c2a00700 1 Processor -- start 2024-05-27T07:42:19.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.203+0000 7fa3c2a00700 1 -- start start 2024-05-27T07:42:19.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.204+0000 7fa3c2a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 0x7fa3bc10b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.204+0000 7fa3c2a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 0x7fa3bc10bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.204+0000 7fa3c2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3bc10e8f0 0x7fa3bc10c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.204+0000 7fa3c2a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc104b40 con 0x7fa3bc105510 2024-05-27T07:42:19.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.204+0000 7fa3c2a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc1049c0 con 0x7fa3bc10e8f0 2024-05-27T07:42:19.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.204+0000 7fa3c2a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc104cc0 con 0x7fa3bc105e60 2024-05-27T07:42:19.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.205+0000 7fa3bb400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 0x7fa3bc10bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.205+0000 7fa3bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 0x7fa3bc10b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.205+0000 7fa3bbe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 0x7fa3bc10b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45482/0 (socket says 172.21.15.12:45482) 2024-05-27T07:42:19.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.205+0000 7fa3bbe00700 1 -- 172.21.15.12:0/1994866937 learned_addr learned my addr 172.21.15.12:0/1994866937 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:19.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.205+0000 7fa3c0a00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3bc10e8f0 0x7fa3bc10c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.206+0000 7fa3bbe00700 1 -- 172.21.15.12:0/1994866937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 msgr2=0x7fa3bc10bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.206+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 0x7fa3bc10bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.206+0000 7fa3bbe00700 1 -- 172.21.15.12:0/1994866937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3bc10e8f0 msgr2=0x7fa3bc10c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.206+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3bc10e8f0 0x7fa3bc10c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.206+0000 7fa3bbe00700 1 -- 172.21.15.12:0/1994866937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3bc1ad470 con 0x7fa3bc105510 2024-05-27T07:42:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.207+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 0x7fa3bc10b7e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00e9a0 tx=0x7fa3ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.208+0000 7fa3b8c00700 1 -- 172.21.15.12:0/1994866937 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac00cbd0 con 0x7fa3bc105510 2024-05-27T07:42:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.208+0000 7fa3b8c00700 1 -- 172.21.15.12:0/1994866937 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa3ac00cd70 con 0x7fa3bc105510 2024-05-27T07:42:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.208+0000 7fa3b8c00700 1 -- 172.21.15.12:0/1994866937 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac018680 con 0x7fa3bc105510 2024-05-27T07:42:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.208+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3bc1ad760 con 0x7fa3bc105510 2024-05-27T07:42:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.209+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3bc1adc50 con 0x7fa3bc105510 2024-05-27T07:42:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.217+0000 7fa3b8c00700 1 -- 172.21.15.12:0/1994866937 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa3ac010430 con 0x7fa3bc105510 2024-05-27T07:42:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.220+0000 7fa3b8c00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa3a00b2a70 0x7fa3a00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.220+0000 7fa3d151fb80 1 Processor -- start 2024-05-27T07:42:19.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.220+0000 7fa3b8c00700 1 -- 172.21.15.12:0/1994866937 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fa3ac0e2db0 con 0x7fa3bc105510 2024-05-27T07:42:19.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.220+0000 7fa3d151fb80 1 -- start start 2024-05-27T07:42:19.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.221+0000 7fa3bb400700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa3a00b2a70 0x7fa3a00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.221+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e15870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.221+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e0f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.221+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e10240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.221+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ba97d3dec0 con 0x55ba97e0fe30 2024-05-27T07:42:19.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.221+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ba97d2da60 con 0x55ba97e15db0 2024-05-27T07:42:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.221+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ba97d31aa0 con 0x55ba97c90ec0 2024-05-27T07:42:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.222+0000 7fa3bbe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e15870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.222+0000 7fa3bbe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e15870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35820/0 (socket says 172.21.15.12:35820) 2024-05-27T07:42:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.222+0000 7fa3bbe00700 1 -- 172.21.15.12:0/328851324 learned_addr learned my addr 172.21.15.12:0/328851324 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.222+0000 7fa3bb400700 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e0f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.223+0000 7fa3bbe00700 1 -- 172.21.15.12:0/328851324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 msgr2=0x55ba97e0f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:19.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.223+0000 7fa3c0a00700 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e10240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.224+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e0f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.224+0000 7fa3bbe00700 1 -- 172.21.15.12:0/328851324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 msgr2=0x55ba97e10240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:19.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.224+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e10240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.225+0000 7fa3bbe00700 1 -- 172.21.15.12:0/328851324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba97cf64a0 con 0x55ba97c90ec0 2024-05-27T07:42:19.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.225+0000 7fa3c0a00700 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e10240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:42:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.227+0000 7fa3bb400700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa3a00b2a70 0x7fa3a00b4f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc101970 tx=0x7fa3a80096c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.227+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e15870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0a9c10 tx=0x7fa3ac0a93f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdcdc03e26dd640b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.227+0000 7fa3b1600700 1 -- 172.21.15.12:0/328851324 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0aa380 con 0x55ba97c90ec0 2024-05-27T07:42:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.228+0000 7fa3b1600700 1 -- 172.21.15.12:0/328851324 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa3ac0a9c80 con 0x55ba97c90ec0 2024-05-27T07:42:19.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.228+0000 7fa3b1600700 1 -- 172.21.15.12:0/328851324 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac010720 con 0x55ba97c90ec0 2024-05-27T07:42:19.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.229+0000 7fa3d151fb80 1 -- 172.21.15.12:0/328851324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 msgr2=0x55ba97e15870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.229+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e15870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0a9c10 tx=0x7fa3ac0a93f0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.230+0000 7fa3d151fb80 1 -- 172.21.15.12:0/328851324 shutdown_connections 2024-05-27T07:42:19.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.230+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e10240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.230+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e15870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.230+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/328851324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e0f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.230+0000 7fa3d151fb80 1 -- 172.21.15.12:0/328851324 >> 172.21.15.12:0/328851324 conn(0x55ba97cf1040 msgr2=0x55ba97e0a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:19.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.230+0000 7fa3d151fb80 1 -- 172.21.15.12:0/328851324 shutdown_connections 2024-05-27T07:42:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.230+0000 7fa3d151fb80 1 -- 172.21.15.12:0/328851324 wait complete. 2024-05-27T07:42:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.231+0000 7fa3d151fb80 1 -- start start 2024-05-27T07:42:19.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.232+0000 7fa3d151fb80 1 Processor -- start 2024-05-27T07:42:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.233+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97c90ec0 0x55ba97e196b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.233+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97e0fe30 0x55ba97e19bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.233+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.233+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ba97c34020 con 0x55ba97c90ec0 2024-05-27T07:42:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.233+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ba97cce7e0 con 0x55ba97e15db0 2024-05-27T07:42:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.233+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ba97e1c880 con 0x55ba97e0fe30 2024-05-27T07:42:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.233+0000 7fa3d151fb80 10 client.0 _ll_register_callbacks cb 0x55ba97cd7cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.234+0000 7fa3c0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.234+0000 7fa3c0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:43294/0 (socket says 172.21.15.12:43294) 2024-05-27T07:42:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.234+0000 7fa3c0a00700 1 -- 172.21.15.12:0/2392452740 learned_addr learned my addr 172.21.15.12:0/2392452740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:19.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.235+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97c90ec0 0x55ba97e196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.235+0000 7fa3bb400700 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97e0fe30 0x55ba97e19bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.236+0000 7fa3c0a00700 1 -- 172.21.15.12:0/2392452740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97e0fe30 msgr2=0x55ba97e19bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:19.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.236+0000 7fa3c0a00700 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97e0fe30 0x55ba97e19bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.236+0000 7fa3c0a00700 1 -- 172.21.15.12:0/2392452740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97c90ec0 msgr2=0x55ba97e196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:19.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.236+0000 7fa3c0a00700 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97c90ec0 0x55ba97e196b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.236+0000 7fa3c0a00700 1 -- 172.21.15.12:0/2392452740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba97c7d950 con 0x55ba97e15db0 2024-05-27T07:42:19.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.237+0000 7fa3c0a00700 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1a130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009fd0 tx=0x7fa3b4007d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45c9c6ea6d1ecd78 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:19.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.237+0000 7fa3a6000700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:42:19.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.241+0000 7fa3a6000700 1 -- 172.21.15.12:0/2392452740 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4003c00 con 0x55ba97e15db0 2024-05-27T07:42:19.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.241+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3ac0a9c80 con 0x55ba97e15db0 2024-05-27T07:42:19.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.241+0000 7fa3a6000700 1 -- 172.21.15.12:0/2392452740 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fa3b4003da0 con 0x55ba97e15db0 2024-05-27T07:42:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.249+0000 7fa3a6000700 1 -- 172.21.15.12:0/2392452740 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fa3b40106e0 con 0x55ba97e15db0 2024-05-27T07:42:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.249+0000 7fa3d151fb80 10 client.37094 fetch_fsmap learned FSMap version 122 2024-05-27T07:42:19.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.249+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=4+}) -- 0x55ba97cfe0e0 con 0x55ba97e15db0 2024-05-27T07:42:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.251+0000 7fa3a6000700 1 -- 172.21.15.12:0/2392452740 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa3ac0a9c80 con 0x55ba97e15db0 2024-05-27T07:42:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.251+0000 7fa3a6000700 1 -- 172.21.15.12:0/2392452740 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 122) ==== 2949+0+0 (secure 0 0 0) 0x7fa3b4004520 con 0x55ba97e15db0 2024-05-27T07:42:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.251+0000 7fa3d151fb80 10 client.37094 fetch_fsmap finished waiting for FSMap version 122 2024-05-27T07:42:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.251+0000 7fa3d151fb80 10 client.37094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.251+0000 7fa3d151fb80 20 client.37094 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.251+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97e2be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.252+0000 7fa3d151fb80 4 client.37094 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.252+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba97d3fe10 con 0x55ba97e299a0 2024-05-27T07:42:19.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.253+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97e2be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.254+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97e2be10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0f82cb566a133ce server_cookie=56fe48fd44dc735 in_seq=0 out_seq=0 2024-05-27T07:42:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.254+0000 7fa3a6000700 10 client.37094 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:19.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.258+0000 7fa3a6000700 1 -- 172.21.15.12:0/2392452740 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ba97d3fe10 con 0x55ba97e299a0 2024-05-27T07:42:19.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.258+0000 7fa3a6000700 10 client.37094 handle_command_reply: tid=0 2024-05-27T07:42:19.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.258+0000 7fa3d151fb80 1 client.37094 shutdown 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 msgr2=0x55ba97e1a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1a130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009fd0 tx=0x7fa3b4007d60 comp rx=0 tx=0).stop 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 shutdown_connections 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97e2be10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1a130 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3a6000700 0 client.37094 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97c90ec0 0x55ba97e196b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/2392452740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97e0fe30 0x55ba97e19bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.259+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 >> 172.21.15.12:0/2392452740 conn(0x55ba97cf1040 msgr2=0x55ba97e0a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:19.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.260+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 shutdown_connections 2024-05-27T07:42:19.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.261+0000 7fa3d151fb80 1 -- 172.21.15.12:0/2392452740 wait complete. 2024-05-27T07:42:19.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.261+0000 7fa3d151fb80 20 client.37094 trim_cache size 0 max 16384 2024-05-27T07:42:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.309+0000 7fa3d151fb80 1 Processor -- start 2024-05-27T07:42:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.309+0000 7fa3d151fb80 1 -- start start 2024-05-27T07:42:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e0f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e258e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc1b6340 con 0x55ba97e0fe30 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc1b5540 con 0x55ba97e15db0 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc1b5c70 con 0x55ba97c90ec0 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3bb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e258e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3bbe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e0f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3bbe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e0f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:35840/0 (socket says 172.21.15.12:35840) 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.310+0000 7fa3bbe00700 1 -- 172.21.15.12:0/3438770701 learned_addr learned my addr 172.21.15.12:0/3438770701 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.311+0000 7fa3bbe00700 1 -- 172.21.15.12:0/3438770701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 msgr2=0x55ba97e1ab10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:42:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.311+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1ab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.311+0000 7fa3bbe00700 1 -- 172.21.15.12:0/3438770701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 msgr2=0x55ba97e258e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.311+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e258e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.311+0000 7fa3bbe00700 1 -- 172.21.15.12:0/3438770701 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba97e18690 con 0x55ba97c90ec0 2024-05-27T07:42:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.312+0000 7fa3bb400700 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e258e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.313+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e0f8f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0e01a0 tx=0x7fa3ac023040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=781a3d48a4f3d58b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.313+0000 7fa3b0c00700 1 -- 172.21.15.12:0/3438770701 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0c9070 con 0x55ba97c90ec0 2024-05-27T07:42:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.314+0000 7fa3b0c00700 1 -- 172.21.15.12:0/3438770701 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa3ac0ae030 con 0x55ba97c90ec0 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.314+0000 7fa3b0c00700 1 -- 172.21.15.12:0/3438770701 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac08c050 con 0x55ba97c90ec0 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.315+0000 7fa3d151fb80 1 -- 172.21.15.12:0/3438770701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 msgr2=0x55ba97e0f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.315+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e0f8f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0e01a0 tx=0x7fa3ac023040 comp rx=0 tx=0).stop 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.315+0000 7fa3d151fb80 1 -- 172.21.15.12:0/3438770701 shutdown_connections 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.315+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e1ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.315+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e0f8f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.315+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/3438770701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e258e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.315+0000 7fa3d151fb80 1 -- 172.21.15.12:0/3438770701 >> 172.21.15.12:0/3438770701 conn(0x55ba97cf1040 msgr2=0x55ba97e13bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.318+0000 7fa3d151fb80 1 -- 172.21.15.12:0/3438770701 shutdown_connections 2024-05-27T07:42:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.319+0000 7fa3d151fb80 1 -- 172.21.15.12:0/3438770701 wait complete. 2024-05-27T07:42:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.319+0000 7fa3d151fb80 1 -- start start 2024-05-27T07:42:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.320+0000 7fa3d151fb80 1 Processor -- start 2024-05-27T07:42:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.321+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e20ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.321+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e21010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.321+0000 7fa3d151fb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e21660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.321+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ba97e17c40 con 0x55ba97e0fe30 2024-05-27T07:42:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.321+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ba97e1a4a0 con 0x55ba97e15db0 2024-05-27T07:42:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.321+0000 7fa3d151fb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ba97e150d0 con 0x55ba97c90ec0 2024-05-27T07:42:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.321+0000 7fa3d151fb80 10 client.0 _ll_register_callbacks cb 0x55ba97cd7cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.322+0000 7fa3bb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e21010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.322+0000 7fa3bb400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e21010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45534/0 (socket says 172.21.15.12:45534) 2024-05-27T07:42:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.322+0000 7fa3bb400700 1 -- 172.21.15.12:0/369982930 learned_addr learned my addr 172.21.15.12:0/369982930 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.323+0000 7fa3c0a00700 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e21660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.323+0000 7fa3bb400700 1 -- 172.21.15.12:0/369982930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 msgr2=0x55ba97e20ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.323+0000 7fa3bb400700 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e20ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.323+0000 7fa3bb400700 1 -- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 msgr2=0x55ba97e21660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.323+0000 7fa3bb400700 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e21660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.323+0000 7fa3bb400700 1 -- 172.21.15.12:0/369982930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba97e21e00 con 0x55ba97e0fe30 2024-05-27T07:42:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.324+0000 7fa3bb400700 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e21010 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8000c00 tx=0x7fa3a8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0cfa9ad3fdd4526 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.324+0000 7fa3a6000700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.325+0000 7fa3a6000700 1 -- 172.21.15.12:0/369982930 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8021040 con 0x55ba97e0fe30 2024-05-27T07:42:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.327+0000 7fa3a6000700 1 -- 172.21.15.12:0/369982930 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fa3a800f870 con 0x55ba97e0fe30 2024-05-27T07:42:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.327+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ba97d395f0 con 0x55ba97e0fe30 2024-05-27T07:42:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.328+0000 7fa3a6000700 1 -- 172.21.15.12:0/369982930 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fa3a800fa70 con 0x55ba97e0fe30 2024-05-27T07:42:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.328+0000 7fa3d151fb80 10 client.17775 fetch_fsmap learned FSMap version 122 2024-05-27T07:42:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.328+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=4+}) -- 0x55ba97ea5d00 con 0x55ba97e0fe30 2024-05-27T07:42:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3a6000700 1 -- 172.21.15.12:0/369982930 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55ba97d395f0 con 0x55ba97e0fe30 2024-05-27T07:42:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3a6000700 1 -- 172.21.15.12:0/369982930 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 122) ==== 2949+0+0 (secure 0 0 0) 0x7fa3a8031070 con 0x55ba97e0fe30 2024-05-27T07:42:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3d151fb80 10 client.17775 fetch_fsmap finished waiting for FSMap version 122 2024-05-27T07:42:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3d151fb80 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3d151fb80 20 client.17775 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97ea84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3d151fb80 4 client.17775 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:42:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.329+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ba97bb02d0 con 0x55ba97e299a0 2024-05-27T07:42:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.331+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97ea84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.332+0000 7fa3bbe00700 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97ea84a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b2534aa02b4e6b2 server_cookie=307536cafcec6caf in_seq=0 out_seq=0 2024-05-27T07:42:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.332+0000 7fa3a6000700 10 client.17775 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.334+0000 7fa3a6000700 1 -- 172.21.15.12:0/369982930 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ba97bb02d0 con 0x55ba97e299a0 2024-05-27T07:42:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.334+0000 7fa3a6000700 10 client.17775 handle_command_reply: tid=0 2024-05-27T07:42:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.334+0000 7fa3d151fb80 1 client.17775 shutdown 2024-05-27T07:42:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.335+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 msgr2=0x55ba97e21010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.335+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e21010 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8000c00 tx=0x7fa3a8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.336+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 shutdown_connections 2024-05-27T07:42:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.336+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55ba97e299a0 0x55ba97ea84a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.336+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba97e15db0 0x55ba97e21660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.336+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba97c90ec0 0x55ba97e20ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.336+0000 7fa3d151fb80 1 --2- 172.21.15.12:0/369982930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ba97e0fe30 0x55ba97e21010 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.336+0000 7fa3a6000700 0 client.17775 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.336+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 >> 172.21.15.12:0/369982930 conn(0x55ba97cf1040 msgr2=0x55ba97e13bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.337+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 shutdown_connections 2024-05-27T07:42:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.337+0000 7fa3d151fb80 1 -- 172.21.15.12:0/369982930 wait complete. 2024-05-27T07:42:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.337+0000 7fa3d151fb80 20 client.17775 trim_cache size 0 max 16384 2024-05-27T07:42:19.337 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:42:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "return_code": 0, 2024-05-27T07:42:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "scrub_tag": "1fbb49a7-7866-4b2e-9d0f-8e20b9886ecd", 2024-05-27T07:42:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": "asynchronous" 2024-05-27T07:42:19.337 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:42:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.339+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa3a00b2a70 msgr2=0x7fa3a00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.339+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa3a00b2a70 0x7fa3a00b4f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc101970 tx=0x7fa3a80096c0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.339+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 msgr2=0x7fa3bc10b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.339+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 0x7fa3bc10b7e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00e9a0 tx=0x7fa3ac00ecb0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.340+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 shutdown_connections 2024-05-27T07:42:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.340+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fa3a00b2a70 0x7fa3a00b4f80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.340+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3bc10e8f0 0x7fa3bc10c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.340+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa3bc105510 0x7fa3bc10b7e0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.340+0000 7fa3c2a00700 1 --2- 172.21.15.12:0/1994866937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3bc105e60 0x7fa3bc10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.340+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 >> 172.21.15.12:0/1994866937 conn(0x7fa3bc0fedb0 msgr2=0x7fa3bc0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.341+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 shutdown_connections 2024-05-27T07:42:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:19.341+0000 7fa3c2a00700 1 -- 172.21.15.12:0/1994866937 wait complete. 2024-05-27T07:42:19.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:19 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:19.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:19 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:19.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:19 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:20.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:42:20.601 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: pgmap v677: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 863 KiB/s wr, 130 op/s 2024-05-27T07:42:20.601 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub queued for path: / 2024-05-27T07:42:20.601 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:42:20.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:42:20.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:42:20.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: active paths [/] 2024-05-27T07:42:20.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.602 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:20 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: pgmap v677: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 863 KiB/s wr, 130 op/s 2024-05-27T07:42:20.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub queued for path: / 2024-05-27T07:42:20.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:42:20.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:42:20.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:42:20.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: active paths [/] 2024-05-27T07:42:20.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:20 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: pgmap v677: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 863 KiB/s wr, 130 op/s 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub queued for path: / 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: active paths [/] 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:20 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:20.802 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:21.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:21 smithi012 ceph-mon[31473]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:21.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:21 smithi162 ceph-mon[35676]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:21.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:21 smithi062 ceph-mon[35304]: scrub summary: idle+waiting paths [/] 2024-05-27T07:42:22.525 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:22 smithi012 ceph-mon[31473]: pgmap v678: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 863 KiB/s wr, 130 op/s 2024-05-27T07:42:22.525 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:22 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:22.645 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:22 smithi062 ceph-mon[35304]: pgmap v678: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 863 KiB/s wr, 130 op/s 2024-05-27T07:42:22.646 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:22 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:22.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:22 smithi162 ceph-mon[35676]: pgmap v678: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 863 KiB/s wr, 130 op/s 2024-05-27T07:42:22.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:22 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:23.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 -- 172.21.15.12:0/2842103031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078076670 msgr2=0x7fe078110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 --2- 172.21.15.12:0/2842103031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078076670 0x7fe078110650 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe064009990 tx=0x7fe064009ca0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 -- 172.21.15.12:0/2842103031 shutdown_connections 2024-05-27T07:42:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 --2- 172.21.15.12:0/2842103031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe078110b90 0x7fe078112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 --2- 172.21.15.12:0/2842103031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe078075d20 0x7fe078076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 --2- 172.21.15.12:0/2842103031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078076670 0x7fe078110650 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 -- 172.21.15.12:0/2842103031 >> 172.21.15.12:0/2842103031 conn(0x7fe078100030 msgr2=0x7fe078102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 -- 172.21.15.12:0/2842103031 shutdown_connections 2024-05-27T07:42:23.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.152+0000 7fe080400700 1 -- 172.21.15.12:0/2842103031 wait complete. 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.153+0000 7fe080400700 1 Processor -- start 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.154+0000 7fe080400700 1 -- start start 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.154+0000 7fe080400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe078075d20 0x7fe07819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.154+0000 7fe080400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe078076670 0x7fe07819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.154+0000 7fe080400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078110b90 0x7fe0781a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.154+0000 7fe080400700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe078116540 con 0x7fe078110b90 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.154+0000 7fe080400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe0781163c0 con 0x7fe078076670 2024-05-27T07:42:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.154+0000 7fe080400700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0781166c0 con 0x7fe078075d20 2024-05-27T07:42:23.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078110b90 0x7fe0781a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078110b90 0x7fe0781a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42874/0 (socket says 172.21.15.12:42874) 2024-05-27T07:42:23.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 -- 172.21.15.12:0/393890476 learned_addr learned my addr 172.21.15.12:0/393890476 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:23.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07da00700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe078075d20 0x7fe07819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07d000700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe078076670 0x7fe07819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 -- 172.21.15.12:0/393890476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe078075d20 msgr2=0x7fe07819cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe078075d20 0x7fe07819cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 -- 172.21.15.12:0/393890476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe078076670 msgr2=0x7fe07819d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe078076670 0x7fe07819d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07e400700 1 -- 172.21.15.12:0/393890476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0781a1d90 con 0x7fe078110b90 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.155+0000 7fe07d000700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe078076670 0x7fe07819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:23.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.156+0000 7fe07e400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078110b90 0x7fe0781a17a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fe0700094f0 tx=0x7fe07000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:23.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.156+0000 7fe076000700 1 -- 172.21.15.12:0/393890476 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07000ddd0 con 0x7fe078110b90 2024-05-27T07:42:23.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.156+0000 7fe076000700 1 -- 172.21.15.12:0/393890476 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe070009e20 con 0x7fe078110b90 2024-05-27T07:42:23.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.156+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0781a2080 con 0x7fe078110b90 2024-05-27T07:42:23.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.156+0000 7fe076000700 1 -- 172.21.15.12:0/393890476 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070015400 con 0x7fe078110b90 2024-05-27T07:42:23.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.156+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0781a2570 con 0x7fe078110b90 2024-05-27T07:42:23.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.161+0000 7fe076000700 1 -- 172.21.15.12:0/393890476 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe070005020 con 0x7fe078110b90 2024-05-27T07:42:23.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.162+0000 7fe08f025b80 1 Processor -- start 2024-05-27T07:42:23.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe08f025b80 1 -- start start 2024-05-27T07:42:23.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc76511330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76511870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe08f025b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dc7643fe40 con 0x55dc76392e40 2024-05-27T07:42:23.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe08f025b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dc7642f9e0 con 0x55dc76511e70 2024-05-27T07:42:23.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe08f025b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dc76433a20 con 0x55dc7647e9f0 2024-05-27T07:42:23.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe076000700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0600b29f0 0x7fe0600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.163+0000 7fe07da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76511870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe076000700 1 -- 172.21.15.12:0/393890476 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fe0700e24c0 con 0x7fe078110b90 2024-05-27T07:42:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07e400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0600b29f0 0x7fe0600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76511870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42886/0 (socket says 172.21.15.12:42886) 2024-05-27T07:42:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07da00700 1 -- 172.21.15.12:0/1256585520 learned_addr learned my addr 172.21.15.12:0/1256585520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07d000700 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc76511330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07d000700 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07da00700 1 -- 172.21.15.12:0/1256585520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 msgr2=0x55dc76511330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07da00700 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc76511330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07da00700 1 -- 172.21.15.12:0/1256585520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 msgr2=0x55dc7651e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07da00700 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.164+0000 7fe07da00700 1 -- 172.21.15.12:0/1256585520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc763f8420 con 0x55dc76392e40 2024-05-27T07:42:23.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.165+0000 7fe07d000700 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.165+0000 7fe07da00700 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76511870 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00ad70 tx=0x7fe06c00f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637f2d15d6d6b704 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.165+0000 7fe07e400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0600b29f0 0x7fe0600b4f00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe0700a9030 tx=0x7fe0700096d0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.165+0000 7fe06aa00700 1 -- 172.21.15.12:0/1256585520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c004020 con 0x55dc76392e40 2024-05-27T07:42:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.165+0000 7fe06aa00700 1 -- 172.21.15.12:0/1256585520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe06c003680 con 0x55dc76392e40 2024-05-27T07:42:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.165+0000 7fe06aa00700 1 -- 172.21.15.12:0/1256585520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c0186a0 con 0x55dc76392e40 2024-05-27T07:42:23.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.166+0000 7fe08f025b80 1 -- 172.21.15.12:0/1256585520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 msgr2=0x55dc76511870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.166+0000 7fe08f025b80 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76511870 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00ad70 tx=0x7fe06c00f6a0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 -- 172.21.15.12:0/1256585520 shutdown_connections 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc76511330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 --2- 172.21.15.12:0/1256585520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76511870 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 -- 172.21.15.12:0/1256585520 >> 172.21.15.12:0/1256585520 conn(0x55dc7642c390 msgr2=0x55dc7650d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 -- 172.21.15.12:0/1256585520 shutdown_connections 2024-05-27T07:42:23.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 -- 172.21.15.12:0/1256585520 wait complete. 2024-05-27T07:42:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.167+0000 7fe08f025b80 1 -- start start 2024-05-27T07:42:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 1 Processor -- start 2024-05-27T07:42:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc7651b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc7651bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dc76335fa0 con 0x55dc76392e40 2024-05-27T07:42:23.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dc763d0760 con 0x55dc76511e70 2024-05-27T07:42:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dc7651eb80 con 0x55dc7647e9f0 2024-05-27T07:42:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.168+0000 7fe08f025b80 10 client.0 _ll_register_callbacks cb 0x55dc763d9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc7651b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc7651b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42898/0 (socket says 172.21.15.12:42898) 2024-05-27T07:42:23.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 -- 172.21.15.12:0/845152561 learned_addr learned my addr 172.21.15.12:0/845152561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:23.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07da00700 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc7651bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 -- 172.21.15.12:0/845152561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 msgr2=0x55dc7651bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc7651bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 -- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 msgr2=0x55dc7651c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.169+0000 7fe07d000700 1 -- 172.21.15.12:0/845152561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc7637f8d0 con 0x55dc76392e40 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe07d000700 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc7651b670 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe064005b40 tx=0x7fe064005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fe791e9f8a4689 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe05f400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe05f400700 1 -- 172.21.15.12:0/845152561 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06402b060 con 0x55dc76392e40 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe06c003680 con 0x55dc76392e40 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe05f400700 1 -- 172.21.15.12:0/845152561 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fe064003b50 con 0x55dc76392e40 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe05f400700 1 -- 172.21.15.12:0/845152561 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7fe064032050 con 0x55dc76392e40 2024-05-27T07:42:23.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe08f025b80 10 client.17787 fetch_fsmap learned FSMap version 123 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.170+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x55dc76400060 con 0x55dc76392e40 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe05f400700 1 -- 172.21.15.12:0/845152561 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe06c003680 con 0x55dc76392e40 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe05f400700 1 -- 172.21.15.12:0/845152561 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 123) ==== 2957+0+0 (secure 0 0 0) 0x7fe06400f580 con 0x55dc76392e40 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe08f025b80 10 client.17787 fetch_fsmap finished waiting for FSMap version 123 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe08f025b80 10 client.17787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe08f025b80 20 client.17787 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe08f025b80 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc7652be20 0x55dc7652e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe08f025b80 4 client.17787 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:23.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.171+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc76441d90 con 0x55dc7652be20 2024-05-27T07:42:23.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.172+0000 7fe07da00700 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc7652be20 0x55dc7652e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.173+0000 7fe07da00700 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc7652be20 0x55dc7652e2f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=266b4377fe889365 server_cookie=249df5086a0c952b in_seq=0 out_seq=0 2024-05-27T07:42:23.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.173+0000 7fe05f400700 10 client.17787 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe05f400700 1 -- 172.21.15.12:0/845152561 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dc76441d90 con 0x55dc7652be20 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe05f400700 10 client.17787 handle_command_reply: tid=0 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 client.17787 shutdown 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 msgr2=0x55dc7651b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc7651b670 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe064005b40 tx=0x7fe064005fb0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 shutdown_connections 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc7652be20 0x55dc7652e2f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76511e70 0x55dc7651c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc7651b670 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 --2- 172.21.15.12:0/845152561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc7647e9f0 0x55dc7651bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 >> 172.21.15.12:0/845152561 conn(0x55dc7642c390 msgr2=0x55dc76511310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:23.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe05f400700 0 client.17787 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.175+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 shutdown_connections 2024-05-27T07:42:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.176+0000 7fe08f025b80 1 -- 172.21.15.12:0/845152561 wait complete. 2024-05-27T07:42:23.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.176+0000 7fe08f025b80 20 client.17787 trim_cache size 0 max 16384 2024-05-27T07:42:23.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 Processor -- start 2024-05-27T07:42:23.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 -- start start 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76527c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc7647e9f0 0x55dc76533ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 0x55dc76533ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dc7651a7f0 con 0x55dc76392e40 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dc7650be50 con 0x55dc7647e9f0 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.200+0000 7fe08f025b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0781b1e90 con 0x55dc76511e70 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76527c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76527c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42910/0 (socket says 172.21.15.12:42910) 2024-05-27T07:42:23.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 -- 172.21.15.12:0/3440932930 learned_addr learned my addr 172.21.15.12:0/3440932930 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07da00700 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc7647e9f0 0x55dc76533ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 -- 172.21.15.12:0/3440932930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 msgr2=0x55dc76533ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 0x55dc76533ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 -- 172.21.15.12:0/3440932930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc7647e9f0 msgr2=0x55dc76533ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc7647e9f0 0x55dc76533ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.201+0000 7fe07d000700 1 -- 172.21.15.12:0/3440932930 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc76509240 con 0x55dc76392e40 2024-05-27T07:42:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.202+0000 7fe07d000700 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76527c50 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe0640098e0 tx=0x7fe064003840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62920b8b3a212488 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.202+0000 7fe06a000700 1 -- 172.21.15.12:0/3440932930 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06403b070 con 0x55dc76392e40 2024-05-27T07:42:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.202+0000 7fe06a000700 1 -- 172.21.15.12:0/3440932930 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe06400fb70 con 0x55dc76392e40 2024-05-27T07:42:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.202+0000 7fe06a000700 1 -- 172.21.15.12:0/3440932930 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06402b040 con 0x55dc76392e40 2024-05-27T07:42:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 -- 172.21.15.12:0/3440932930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 msgr2=0x55dc76527c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76527c50 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe0640098e0 tx=0x7fe064003840 comp rx=0 tx=0).stop 2024-05-27T07:42:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 -- 172.21.15.12:0/3440932930 shutdown_connections 2024-05-27T07:42:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 0x55dc76533ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc76392e40 0x55dc76527c50 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 --2- 172.21.15.12:0/3440932930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc7647e9f0 0x55dc76533ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 -- 172.21.15.12:0/3440932930 >> 172.21.15.12:0/3440932930 conn(0x55dc7642c390 msgr2=0x55dc76511720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.203+0000 7fe08f025b80 1 -- 172.21.15.12:0/3440932930 shutdown_connections 2024-05-27T07:42:23.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.204+0000 7fe08f025b80 1 -- 172.21.15.12:0/3440932930 wait complete. 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.204+0000 7fe08f025b80 1 -- start start 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.204+0000 7fe08f025b80 1 Processor -- start 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76392e40 0x55dc7651fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc7647e9f0 0x55dc76520150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe08f025b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 0x55dc76520690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe08f025b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0781b1790 con 0x55dc7647e9f0 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe08f025b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe0781b1060 con 0x55dc76392e40 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe08f025b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dc76595980 con 0x55dc76511e70 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe08f025b80 10 client.0 _ll_register_callbacks cb 0x55dc763d9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe07da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc7647e9f0 0x55dc76520150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe07da00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc7647e9f0 0x55dc76520150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42924/0 (socket says 172.21.15.12:42924) 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.205+0000 7fe07da00700 1 -- 172.21.15.12:0/4058894176 learned_addr learned my addr 172.21.15.12:0/4058894176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07d000700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76392e40 0x55dc7651fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07d000700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 0x55dc76520690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07da00700 1 -- 172.21.15.12:0/4058894176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 msgr2=0x55dc76520690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07da00700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 0x55dc76520690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07da00700 1 -- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76392e40 msgr2=0x55dc7651fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07da00700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76392e40 0x55dc7651fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07da00700 1 -- 172.21.15.12:0/4058894176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc76595b30 con 0x55dc7647e9f0 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07d000700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76392e40 0x55dc7651fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe07da00700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc7647e9f0 0x55dc76520150 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00a630 tx=0x7fe06c00ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1e03461a5bd084 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:23.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.206+0000 7fe05f400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:23.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.207+0000 7fe05f400700 1 -- 172.21.15.12:0/4058894176 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c005040 con 0x55dc7647e9f0 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.207+0000 7fe05f400700 1 -- 172.21.15.12:0/4058894176 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fe06c0051e0 con 0x55dc7647e9f0 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.207+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe06400fb70 con 0x55dc7647e9f0 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.207+0000 7fe05f400700 1 -- 172.21.15.12:0/4058894176 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7fe06c03a050 con 0x55dc7647e9f0 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.207+0000 7fe08f025b80 10 client.17793 fetch_fsmap learned FSMap version 123 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.207+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x55dc76596de0 con 0x55dc7647e9f0 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.207+0000 7fe05f400700 1 -- 172.21.15.12:0/4058894176 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe06400fb70 con 0x55dc7647e9f0 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe05f400700 1 -- 172.21.15.12:0/4058894176 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 123) ==== 2957+0+0 (secure 0 0 0) 0x7fe06c003680 con 0x55dc7647e9f0 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe08f025b80 10 client.17793 fetch_fsmap finished waiting for FSMap version 123 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe08f025b80 10 client.17793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe08f025b80 20 client.17793 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe08f025b80 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc76597500 0x55dc76599920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe08f025b80 4 client.17793 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:42:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dc7643b570 con 0x55dc76597500 2024-05-27T07:42:23.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.208+0000 7fe07d000700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc76597500 0x55dc76599920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.209+0000 7fe07d000700 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc76597500 0x55dc76599920 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca30577daec1e2a9 server_cookie=cfa0c3a7c89aec07 in_seq=0 out_seq=0 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.209+0000 7fe05f400700 10 client.17793 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe05f400700 1 -- 172.21.15.12:0/4058894176 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dc7643b570 con 0x55dc76597500 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe05f400700 10 client.17793 handle_command_reply: tid=0 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 client.17793 shutdown 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc7647e9f0 msgr2=0x55dc76520150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc7647e9f0 0x55dc76520150 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00a630 tx=0x7fe06c00ffa0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 shutdown_connections 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55dc76597500 0x55dc76599920 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dc76392e40 0x55dc7651fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc76511e70 0x55dc76520690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 --2- 172.21.15.12:0/4058894176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dc7647e9f0 0x55dc76520150 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 >> 172.21.15.12:0/4058894176 conn(0x55dc7642c390 msgr2=0x55dc76511720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:23.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.210+0000 7fe05f400700 0 client.17793 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.211+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 shutdown_connections 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.211+0000 7fe08f025b80 1 -- 172.21.15.12:0/4058894176 wait complete. 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.211+0000 7fe08f025b80 20 client.17793 trim_cache size 0 max 16384 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": { 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stdout: "1fbb49a7-7866-4b2e-9d0f-8e20b9886ecd": { 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stdout: "path": "/", 2024-05-27T07:42:23.210 INFO:teuthology.orchestra.run.smithi012.stdout: "tag": "1fbb49a7-7866-4b2e-9d0f-8e20b9886ecd", 2024-05-27T07:42:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: "options": "recursive,force" 2024-05-27T07:42:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:42:23.211 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:42:23.211 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:42:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0600b29f0 msgr2=0x7fe0600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0600b29f0 0x7fe0600b4f00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe0700a9030 tx=0x7fe0700096d0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078110b90 msgr2=0x7fe0781a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078110b90 0x7fe0781a17a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fe0700094f0 tx=0x7fe07000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 shutdown_connections 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0600b29f0 0x7fe0600b4f00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe078110b90 0x7fe0781a17a0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe078075d20 0x7fe07819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 --2- 172.21.15.12:0/393890476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe078076670 0x7fe07819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 >> 172.21.15.12:0/393890476 conn(0x7fe078100030 msgr2=0x7fe0781006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 shutdown_connections 2024-05-27T07:42:23.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:23.212+0000 7fe080400700 1 -- 172.21.15.12:0/393890476 wait complete. 2024-05-27T07:42:24.083 INFO:tasks.cephfs.filesystem:scrub status for tag:1fbb49a7-7866-4b2e-9d0f-8e20b9886ecd - {'path': '/', 'tag': '1fbb49a7-7866-4b2e-9d0f-8e20b9886ecd', 'options': 'recursive,force'} 2024-05-27T07:42:24.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:24 smithi012 ceph-mon[31473]: pgmap v679: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 828 KiB/s wr, 125 op/s 2024-05-27T07:42:24.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:24 smithi162 ceph-mon[35676]: pgmap v679: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 828 KiB/s wr, 125 op/s 2024-05-27T07:42:24.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:24 smithi062 ceph-mon[35304]: pgmap v679: 129 pgs: 129 active+clean; 525 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 828 KiB/s wr, 125 op/s 2024-05-27T07:42:26.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:26 smithi012 ceph-mon[31473]: pgmap v680: 129 pgs: 129 active+clean; 530 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-05-27T07:42:26.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:26 smithi162 ceph-mon[35676]: pgmap v680: 129 pgs: 129 active+clean; 530 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-05-27T07:42:26.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:26 smithi062 ceph-mon[35304]: pgmap v680: 129 pgs: 129 active+clean; 530 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-05-27T07:42:27.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:27 smithi012 ceph-mon[31473]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:27.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:27 smithi162 ceph-mon[35676]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:27.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:27 smithi062 ceph-mon[35304]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2024-05-27T07:42:28.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:28 smithi012 ceph-mon[31473]: pgmap v681: 129 pgs: 129 active+clean; 530 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 747 KiB/s wr, 114 op/s 2024-05-27T07:42:28.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:28 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:42:28.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:28 smithi162 ceph-mon[35676]: pgmap v681: 129 pgs: 129 active+clean; 530 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 747 KiB/s wr, 114 op/s 2024-05-27T07:42:28.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:28 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:42:28.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:28 smithi062 ceph-mon[35304]: pgmap v681: 129 pgs: 129 active+clean; 530 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 747 KiB/s wr, 114 op/s 2024-05-27T07:42:28.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:28 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:42:29.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:29 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:42:29.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:29 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:42:29.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:29 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:42:29.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:29 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:42:29.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:29 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:42:29.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:29 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:42:29.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:29 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:42:29.743 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:29 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:42:29.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:29 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:42:30.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:30 smithi012 ceph-mon[31473]: pgmap v682: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:42:30.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:30 smithi012 ceph-mon[31473]: pgmap v683: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:30.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:30 smithi162 ceph-mon[35676]: pgmap v682: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:42:30.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:30 smithi162 ceph-mon[35676]: pgmap v683: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:30.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:30 smithi062 ceph-mon[35304]: pgmap v682: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.1 MiB/s wr, 186 op/s 2024-05-27T07:42:30.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:30 smithi062 ceph-mon[35304]: pgmap v683: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:31.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:31 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:42:31.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:31 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:42:31.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:31 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:42:31.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:31 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:42:31.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:31 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:42:31.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:31 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:42:32.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:32 smithi012 ceph-mon[31473]: scrub summary: idle 2024-05-27T07:42:32.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:32 smithi012 ceph-mon[31473]: pgmap v684: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:32.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:32 smithi162 ceph-mon[35676]: scrub summary: idle 2024-05-27T07:42:32.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:32 smithi162 ceph-mon[35676]: pgmap v684: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:32.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:32 smithi062 ceph-mon[35304]: scrub summary: idle 2024-05-27T07:42:32.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:32 smithi062 ceph-mon[35304]: pgmap v684: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:34.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:34 smithi012 ceph-mon[31473]: pgmap v685: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:34.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:34 smithi162 ceph-mon[35676]: pgmap v685: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:34.995 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:34 smithi062 ceph-mon[35304]: pgmap v685: 129 pgs: 129 active+clean; 534 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 882 KiB/s wr, 157 op/s 2024-05-27T07:42:36.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:36 smithi012 ceph-mon[31473]: pgmap v686: 129 pgs: 129 active+clean; 537 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 737 KiB/s wr, 99 op/s 2024-05-27T07:42:36.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:36 smithi162 ceph-mon[35676]: pgmap v686: 129 pgs: 129 active+clean; 537 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 737 KiB/s wr, 99 op/s 2024-05-27T07:42:36.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:36 smithi062 ceph-mon[35304]: pgmap v686: 129 pgs: 129 active+clean; 537 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 737 KiB/s wr, 99 op/s 2024-05-27T07:42:38.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:38 smithi012 ceph-mon[31473]: pgmap v687: 129 pgs: 129 active+clean; 537 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 737 KiB/s wr, 99 op/s 2024-05-27T07:42:38.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:38 smithi162 ceph-mon[35676]: pgmap v687: 129 pgs: 129 active+clean; 537 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 737 KiB/s wr, 99 op/s 2024-05-27T07:42:38.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:38 smithi062 ceph-mon[35304]: pgmap v687: 129 pgs: 129 active+clean; 537 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 737 KiB/s wr, 99 op/s 2024-05-27T07:42:40.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:40 smithi012 ceph-mon[31473]: pgmap v688: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 750 KiB/s wr, 106 op/s 2024-05-27T07:42:40.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:40 smithi162 ceph-mon[35676]: pgmap v688: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 750 KiB/s wr, 106 op/s 2024-05-27T07:42:40.993 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:40 smithi062 ceph-mon[35304]: pgmap v688: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 750 KiB/s wr, 106 op/s 2024-05-27T07:42:41.670 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-05-27T07:42:41.670 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-27T07:42:42.837 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:42 smithi012 ceph-mon[31473]: pgmap v689: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 625 KiB/s wr, 88 op/s 2024-05-27T07:42:42.840 DEBUG:teuthology.parallel:result is None 2024-05-27T07:42:42.841 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-27T07:42:42.927 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-27T07:42:42.927 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-27T07:42:42.937 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-27T07:42:42.937 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:42:42.937 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:42:42.937 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-27T07:42:42.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:42 smithi162 ceph-mon[35676]: pgmap v689: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 625 KiB/s wr, 88 op/s 2024-05-27T07:42:42.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:42 smithi062 ceph-mon[35304]: pgmap v689: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 625 KiB/s wr, 88 op/s 2024-05-27T07:42:43.000 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-27T07:42:43.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T07:42:43.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.355+0000 7fe09a200640 1 -- 172.21.15.12:0/3273885039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 msgr2=0x7fe0941107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.355+0000 7fe09a200640 1 --2- 172.21.15.12:0/3273885039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe0941107a0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009ca0 tx=0x7fe08c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.356+0000 7fe09a200640 1 -- 172.21.15.12:0/3273885039 shutdown_connections 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.356+0000 7fe09a200640 1 --2- 172.21.15.12:0/3273885039 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe0941107a0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.356+0000 7fe09a200640 1 --2- 172.21.15.12:0/3273885039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09410a120 0x7fe094103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.356+0000 7fe09a200640 1 --2- 172.21.15.12:0/3273885039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe094109c80 0x7fe094103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.356+0000 7fe09a200640 1 -- 172.21.15.12:0/3273885039 >> 172.21.15.12:0/3273885039 conn(0x7fe094079290 msgr2=0x7fe0940796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.356+0000 7fe09a200640 1 -- 172.21.15.12:0/3273885039 shutdown_connections 2024-05-27T07:42:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.356+0000 7fe09a200640 1 -- 172.21.15.12:0/3273885039 wait complete. 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 Processor -- start 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 -- start start 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe09419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe094109c80 0x7fe09419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09410a120 0x7fe09419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe094113d90 con 0x7fe0941043d0 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe094113c10 con 0x7fe09410a120 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.357+0000 7fe09a200640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe094113f10 con 0x7fe094109c80 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe09419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe09419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33752/0 (socket says 172.21.15.12:33752) 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 -- 172.21.15.12:0/979849458 learned_addr learned my addr 172.21.15.12:0/979849458 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093e00640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09410a120 0x7fe09419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 -- 172.21.15.12:0/979849458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe094109c80 msgr2=0x7fe09419acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe094109c80 0x7fe09419acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 -- 172.21.15.12:0/979849458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09410a120 msgr2=0x7fe09419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09410a120 0x7fe09419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 -- 172.21.15.12:0/979849458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09419f570 con 0x7fe0941043d0 2024-05-27T07:42:43.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe093400640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe09419a790 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe08400c9a0 tx=0x7fe08400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:43.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.358+0000 7fe08be00640 1 -- 172.21.15.12:0/979849458 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe084007bb0 con 0x7fe0941043d0 2024-05-27T07:42:43.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.359+0000 7fe08be00640 1 -- 172.21.15.12:0/979849458 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe084007d50 con 0x7fe0941043d0 2024-05-27T07:42:43.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.359+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09419f860 con 0x7fe0941043d0 2024-05-27T07:42:43.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.359+0000 7fe08be00640 1 -- 172.21.15.12:0/979849458 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe084018660 con 0x7fe0941043d0 2024-05-27T07:42:43.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.359+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe09419fd60 con 0x7fe0941043d0 2024-05-27T07:42:43.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.360+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe094050860 con 0x7fe0941043d0 2024-05-27T07:42:43.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.362+0000 7fe08be00640 1 -- 172.21.15.12:0/979849458 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe08400f970 con 0x7fe0941043d0 2024-05-27T07:42:43.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.364+0000 7fe08be00640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0780b2950 0x7fe0780b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.364+0000 7fe08be00640 1 -- 172.21.15.12:0/979849458 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fe0840e2fa0 con 0x7fe0941043d0 2024-05-27T07:42:43.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.364+0000 7fe092a00640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0780b2950 0x7fe0780b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:43.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.365+0000 7fe092a00640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0780b2950 0x7fe0780b4e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe080005940 tx=0x7fe080005c20 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:43.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.366+0000 7fe08be00640 1 -- 172.21.15.12:0/979849458 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe084014030 con 0x7fe0941043d0 2024-05-27T07:42:43.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.627+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fe094063100 con 0x7fe0780b2950 2024-05-27T07:42:43.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.644+0000 7fe08be00640 1 -- 172.21.15.12:0/979849458 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fe094063100 con 0x7fe0780b2950 2024-05-27T07:42:43.642 INFO:teuthology.orchestra.run.smithi012.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:22:40", "first": "2024-05-27T07:23:00", "last": "2024-05-27T07:42:00", "last_pruned": "2024-05-27T07:42:01", "created_count": 20, "pruned_count": 14, "active": true} 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0780b2950 msgr2=0x7fe0780b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0780b2950 0x7fe0780b4e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe080005940 tx=0x7fe080005c20 comp rx=0 tx=0).stop 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 msgr2=0x7fe09419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe09419a790 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe08400c9a0 tx=0x7fe08400ce60 comp rx=0 tx=0).stop 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 shutdown_connections 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe0780b2950 0x7fe0780b4e60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09410a120 0x7fe09419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe094109c80 0x7fe09419acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 --2- 172.21.15.12:0/979849458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0941043d0 0x7fe09419a790 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 >> 172.21.15.12:0/979849458 conn(0x7fe094079290 msgr2=0x7fe094100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:43.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 shutdown_connections 2024-05-27T07:42:43.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.647+0000 7fe09a200640 1 -- 172.21.15.12:0/979849458 wait complete. 2024-05-27T07:42:43.675 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 -- 172.21.15.12:0/1034935652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef01121b0 msgr2=0x7fbef01145d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 --2- 172.21.15.12:0/1034935652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef01121b0 0x7fbef01145d0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009900 tx=0x7fbed802a3f0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 -- 172.21.15.12:0/1034935652 shutdown_connections 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 --2- 172.21.15.12:0/1034935652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef0114b10 0x7fbef0116f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 --2- 172.21.15.12:0/1034935652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef01121b0 0x7fbef01145d0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 --2- 172.21.15.12:0/1034935652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbef0078720 0x7fbef0078b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 -- 172.21.15.12:0/1034935652 >> 172.21.15.12:0/1034935652 conn(0x7fbef01016b0 msgr2=0x7fbef0103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:43.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 -- 172.21.15.12:0/1034935652 shutdown_connections 2024-05-27T07:42:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.973+0000 7fbef6e00640 1 -- 172.21.15.12:0/1034935652 wait complete. 2024-05-27T07:42:43.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.974+0000 7fbef6e00640 1 Processor -- start 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.974+0000 7fbef6e00640 1 -- start start 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbef6e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef0078720 0x7fbef01a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbef6e00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbef01121b0 0x7fbef01a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbef6e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef0114b10 0x7fbef01a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbef6e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbef011a380 con 0x7fbef0078720 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbef6e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbef011a200 con 0x7fbef0114b10 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbef6e00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbef011a500 con 0x7fbef01121b0 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbeefe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef0078720 0x7fbef01a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbeefe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef0078720 0x7fbef01a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33772/0 (socket says 172.21.15.12:33772) 2024-05-27T07:42:43.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbeefe00640 1 -- 172.21.15.12:0/2725964517 learned_addr learned my addr 172.21.15.12:0/2725964517 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbeef400640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbef01121b0 0x7fbef01a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.975+0000 7fbef4e00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef0114b10 0x7fbef01a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeefe00640 1 -- 172.21.15.12:0/2725964517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbef01121b0 msgr2=0x7fbef01a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeefe00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbef01121b0 0x7fbef01a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeefe00640 1 -- 172.21.15.12:0/2725964517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef0114b10 msgr2=0x7fbef01a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeefe00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef0114b10 0x7fbef01a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeefe00640 1 -- 172.21.15.12:0/2725964517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef01a5be0 con 0x7fbef0078720 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeefe00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef0078720 0x7fbef01a0e00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fbee000eab0 tx=0x7fbee000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeecc00640 1 -- 172.21.15.12:0/2725964517 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee000ce40 con 0x7fbef0078720 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeecc00640 1 -- 172.21.15.12:0/2725964517 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbee0005020 con 0x7fbef0078720 2024-05-27T07:42:43.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef01a5ed0 con 0x7fbef0078720 2024-05-27T07:42:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.976+0000 7fbeecc00640 1 -- 172.21.15.12:0/2725964517 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee00116d0 con 0x7fbef0078720 2024-05-27T07:42:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.977+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef01a63a0 con 0x7fbef0078720 2024-05-27T07:42:43.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.978+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef01a2150 con 0x7fbef0078720 2024-05-27T07:42:43.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.981+0000 7fbeecc00640 1 -- 172.21.15.12:0/2725964517 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbee0004a40 con 0x7fbef0078720 2024-05-27T07:42:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.982+0000 7fbeecc00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbed40b2950 0x7fbed40b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.983+0000 7fbeecc00640 1 -- 172.21.15.12:0/2725964517 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fbee00e3100 con 0x7fbef0078720 2024-05-27T07:42:43.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.983+0000 7fbeef400640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbed40b2950 0x7fbed40b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:43.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.983+0000 7fbeef400640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbed40b2950 0x7fbed40b4e60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbed80057c0 tx=0x7fbed8000bc0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:43.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:43.985+0000 7fbeecc00640 1 -- 172.21.15.12:0/2725964517 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbee0016030 con 0x7fbef0078720 2024-05-27T07:42:44.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.225+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 --> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fbef0063100 con 0x7fbed40b2950 2024-05-27T07:42:44.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.244+0000 7fbeecc00640 1 -- 172.21.15.12:0/2725964517 <== mgr.24991 v2:172.21.15.12:6800/3926981258 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fbef0063100 con 0x7fbed40b2950 2024-05-27T07:42:44.242 INFO:teuthology.orchestra.run.smithi012.stdout:/ 1m 6m3h 2024-05-27T07:42:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbed40b2950 msgr2=0x7fbed40b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbed40b2950 0x7fbed40b4e60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbed80057c0 tx=0x7fbed8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef0078720 msgr2=0x7fbef01a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef0078720 0x7fbef01a0e00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fbee000eab0 tx=0x7fbee000ef70 comp rx=0 tx=0).stop 2024-05-27T07:42:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 shutdown_connections 2024-05-27T07:42:44.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbed40b2950 0x7fbed40b4e60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef0114b10 0x7fbef01a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbef01121b0 0x7fbef01a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 --2- 172.21.15.12:0/2725964517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbef0078720 0x7fbef01a0e00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 >> 172.21.15.12:0/2725964517 conn(0x7fbef01016b0 msgr2=0x7fbef007b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 shutdown_connections 2024-05-27T07:42:44.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:44.247+0000 7fbef6e00640 1 -- 172.21.15.12:0/2725964517 wait complete. 2024-05-27T07:42:44.272 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-27T07:42:44.282 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-27T07:42:44.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:44 smithi012 ceph-mon[31473]: pgmap v690: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 625 KiB/s wr, 88 op/s 2024-05-27T07:42:44.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:44 smithi162 ceph-mon[35676]: pgmap v690: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 625 KiB/s wr, 88 op/s 2024-05-27T07:42:44.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:44 smithi062 ceph-mon[35304]: pgmap v690: 129 pgs: 129 active+clean; 541 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 625 KiB/s wr, 88 op/s 2024-05-27T07:42:45.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:45 smithi012 ceph-mon[31473]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:42:45.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:45 smithi012 ceph-mon[31473]: from='client.17805 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:42:45.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:45 smithi162 ceph-mon[35676]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:42:45.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:45 smithi162 ceph-mon[35676]: from='client.17805 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:42:45.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:45 smithi062 ceph-mon[35304]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:42:45.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:45 smithi062 ceph-mon[35304]: from='client.17805 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:42:46.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:46 smithi012 ceph-mon[31473]: pgmap v691: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 770 KiB/s wr, 120 op/s 2024-05-27T07:42:46.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:46 smithi162 ceph-mon[35676]: pgmap v691: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 770 KiB/s wr, 120 op/s 2024-05-27T07:42:46.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:46 smithi062 ceph-mon[35304]: pgmap v691: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 770 KiB/s wr, 120 op/s 2024-05-27T07:42:48.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:48 smithi012 ceph-mon[31473]: pgmap v692: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 506 KiB/s wr, 109 op/s 2024-05-27T07:42:48.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:48 smithi162 ceph-mon[35676]: pgmap v692: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 506 KiB/s wr, 109 op/s 2024-05-27T07:42:48.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:48 smithi062 ceph-mon[35304]: pgmap v692: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 506 KiB/s wr, 109 op/s 2024-05-27T07:42:50.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:50 smithi012 ceph-mon[31473]: pgmap v693: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 520 KiB/s wr, 115 op/s 2024-05-27T07:42:50.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:50 smithi162 ceph-mon[35676]: pgmap v693: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 520 KiB/s wr, 115 op/s 2024-05-27T07:42:50.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:50 smithi062 ceph-mon[35304]: pgmap v693: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 520 KiB/s wr, 115 op/s 2024-05-27T07:42:52.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:52 smithi012 ceph-mon[31473]: pgmap v694: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 159 KiB/s wr, 38 op/s 2024-05-27T07:42:52.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:52 smithi162 ceph-mon[35676]: pgmap v694: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 159 KiB/s wr, 38 op/s 2024-05-27T07:42:52.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:52 smithi062 ceph-mon[35304]: pgmap v694: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 159 KiB/s wr, 38 op/s 2024-05-27T07:42:54.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:42:54.553 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:54.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:54 smithi162 ceph-mon[35676]: pgmap v695: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 159 KiB/s wr, 38 op/s 2024-05-27T07:42:54.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:54 smithi062 ceph-mon[35304]: pgmap v695: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 159 KiB/s wr, 38 op/s 2024-05-27T07:42:55.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:54 smithi012 ceph-mon[31473]: pgmap v695: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 159 KiB/s wr, 38 op/s 2024-05-27T07:42:56.868 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:56 smithi012 ceph-mon[31473]: pgmap v696: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 165 KiB/s wr, 39 op/s 2024-05-27T07:42:56.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:56 smithi162 ceph-mon[35676]: pgmap v696: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 165 KiB/s wr, 39 op/s 2024-05-27T07:42:56.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:56 smithi062 ceph-mon[35304]: pgmap v696: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 165 KiB/s wr, 39 op/s 2024-05-27T07:42:57.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.277+0000 7f99b7e00700 1 -- 172.21.15.12:0/3876729298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 msgr2=0x7f99b81131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.277+0000 7f99b7e00700 1 --2- 172.21.15.12:0/3876729298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81131f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f99b000b0d0 tx=0x7f99b000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.277+0000 7f99b7e00700 1 -- 172.21.15.12:0/3876729298 shutdown_connections 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.277+0000 7f99b7e00700 1 --2- 172.21.15.12:0/3876729298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8106610 0x7f99b8106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.277+0000 7f99b7e00700 1 --2- 172.21.15.12:0/3876729298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81131f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.277+0000 7f99b7e00700 1 --2- 172.21.15.12:0/3876729298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b810af40 0x7f99b810b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.277+0000 7f99b7e00700 1 -- 172.21.15.12:0/3876729298 >> 172.21.15.12:0/3876729298 conn(0x7f99b8102020 msgr2=0x7f99b8104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.278+0000 7f99b7e00700 1 -- 172.21.15.12:0/3876729298 shutdown_connections 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.278+0000 7f99b7e00700 1 -- 172.21.15.12:0/3876729298 wait complete. 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 Processor -- start 2024-05-27T07:42:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 -- start start 2024-05-27T07:42:57.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8106610 0x7f99b819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b810af40 0x7f99b819d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81163c0 con 0x7f99b810b890 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8116240 con 0x7f99b8106610 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.279+0000 7f99b7e00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8116540 con 0x7f99b810af40 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39648/0 (socket says 172.21.15.12:39648) 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 -- 172.21.15.12:0/923071180 learned_addr learned my addr 172.21.15.12:0/923071180 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b6a00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8106610 0x7f99b819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b6000700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b810af40 0x7f99b819d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 -- 172.21.15.12:0/923071180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b810af40 msgr2=0x7f99b819d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b810af40 0x7f99b819d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 -- 172.21.15.12:0/923071180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8106610 msgr2=0x7f99b819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8106610 0x7f99b819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.280+0000 7f99b7400700 1 -- 172.21.15.12:0/923071180 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b81a1c00 con 0x7f99b810b890 2024-05-27T07:42:57.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.281+0000 7f99b7400700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81a1560 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f99b00079a0 tx=0x7f99b0007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:57.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.281+0000 7f99bca00700 1 -- 172.21.15.12:0/923071180 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b001f070 con 0x7f99b810b890 2024-05-27T07:42:57.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.281+0000 7f99bca00700 1 -- 172.21.15.12:0/923071180 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99b0007d90 con 0x7f99b810b890 2024-05-27T07:42:57.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.281+0000 7f99bca00700 1 -- 172.21.15.12:0/923071180 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b001b690 con 0x7f99b810b890 2024-05-27T07:42:57.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.281+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b81a1e90 con 0x7f99b810b890 2024-05-27T07:42:57.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.281+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b81a9810 con 0x7f99b810b890 2024-05-27T07:42:57.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.285+0000 7f99bca00700 1 -- 172.21.15.12:0/923071180 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f99b0022070 con 0x7f99b810b890 2024-05-27T07:42:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.287+0000 7f99bca00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f999c0b2a70 0x7f999c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.287+0000 7f99bca00700 1 -- 172.21.15.12:0/923071180 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f99b000f020 con 0x7f99b810b890 2024-05-27T07:42:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.287+0000 7f99cc8edb80 1 Processor -- start 2024-05-27T07:42:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.287+0000 7f99b6a00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f999c0b2a70 0x7f999c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.287+0000 7f99cc8edb80 1 -- start start 2024-05-27T07:42:57.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977dfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f9976f8e40 con 0x55f99764be40 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f9976e89e0 con 0x55f9977379f0 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f9976eca20 con 0x55f9977cbf90 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99b6a00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f999c0b2a70 0x7f999c0b4f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99a8005fd0 tx=0x7f99a8005f60 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99b6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99b6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39662/0 (socket says 172.21.15.12:39662) 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.288+0000 7f99b6000700 1 -- 172.21.15.12:0/1615002210 learned_addr learned my addr 172.21.15.12:0/1615002210 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b6a00700 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b7400700 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977dfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b6000700 1 -- 172.21.15.12:0/1615002210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 msgr2=0x55f9977dfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b6000700 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977dfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b6000700 1 -- 172.21.15.12:0/1615002210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 msgr2=0x55f9977cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b6000700 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b6000700 1 -- 172.21.15.12:0/1615002210 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9976b1420 con 0x55f99764be40 2024-05-27T07:42:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.289+0000 7f99b6000700 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977cb9b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009c60 tx=0x7f99a4009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc50a6e1b4b9b847 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.290+0000 7f99acc00700 1 -- 172.21.15.12:0/1615002210 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a4010c90 con 0x55f99764be40 2024-05-27T07:42:57.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.290+0000 7f99acc00700 1 -- 172.21.15.12:0/1615002210 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99a4010e30 con 0x55f99764be40 2024-05-27T07:42:57.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.290+0000 7f99acc00700 1 -- 172.21.15.12:0/1615002210 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a4011440 con 0x55f99764be40 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1615002210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 msgr2=0x55f9977cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977cb9b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009c60 tx=0x7f99a4009f70 comp rx=0 tx=0).stop 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1615002210 shutdown_connections 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977dfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1615002210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977cb9b0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1615002210 >> 172.21.15.12:0/1615002210 conn(0x55f9976e5390 msgr2=0x55f9977c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1615002210 shutdown_connections 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.291+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1615002210 wait complete. 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.292+0000 7f99cc8edb80 1 -- start start 2024-05-27T07:42:57.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.292+0000 7f99cc8edb80 1 Processor -- start 2024-05-27T07:42:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977dd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977ddb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f9975eefa0 con 0x55f99764be40 2024-05-27T07:42:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f997689760 con 0x55f9977379f0 2024-05-27T07:42:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f9977e0530 con 0x55f9977cbf90 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99cc8edb80 10 client.0 _ll_register_callbacks cb 0x55f997692c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977dd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977dd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39666/0 (socket says 172.21.15.12:39666) 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99b6a00700 1 -- 172.21.15.12:0/976491099 learned_addr learned my addr 172.21.15.12:0/976491099 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99b7400700 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977ddb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.293+0000 7f99b6000700 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99b6a00700 1 -- 172.21.15.12:0/976491099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 msgr2=0x55f9977ddb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99b6a00700 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977ddb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99b6a00700 1 -- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 msgr2=0x55f9977dd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99b6a00700 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99b6a00700 1 -- 172.21.15.12:0/976491099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9976388d0 con 0x55f99764be40 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99b6a00700 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977dd090 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ff40 tx=0x7f99a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e420e350c5d7a2e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:57.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99a1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99a1600700 1 -- 172.21.15.12:0/976491099 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a801f630 con 0x55f99764be40 2024-05-27T07:42:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99a1600700 1 -- 172.21.15.12:0/976491099 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f99a801f7d0 con 0x55f99764be40 2024-05-27T07:42:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.294+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99a4010e30 con 0x55f99764be40 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99a1600700 1 -- 172.21.15.12:0/976491099 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f99a8030050 con 0x55f99764be40 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 10 client.17817 fetch_fsmap learned FSMap version 124 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x55f9976b9060 con 0x55f99764be40 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99a1600700 1 -- 172.21.15.12:0/976491099 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99a4010e30 con 0x55f99764be40 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99a1600700 1 -- 172.21.15.12:0/976491099 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7f99a8010910 con 0x55f99764be40 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 10 client.17817 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 10 client.17817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 20 client.17817 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f9977ed8e0 0x55f9977efd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 4 client.17817 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:42:57.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.295+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9976fad90 con 0x55f9977ed8e0 2024-05-27T07:42:57.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.296+0000 7f99b6000700 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f9977ed8e0 0x55f9977efd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.297+0000 7f99b6000700 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f9977ed8e0 0x55f9977efd50 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eea41169f05a0d90 server_cookie=a04b9c9a870105c6 in_seq=0 out_seq=0 2024-05-27T07:42:57.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.297+0000 7f99a1600700 10 client.17817 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:57.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.299+0000 7f99a1600700 1 -- 172.21.15.12:0/976491099 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f9976fad90 con 0x55f9977ed8e0 2024-05-27T07:42:57.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.299+0000 7f99a1600700 10 client.17817 handle_command_reply: tid=0 2024-05-27T07:42:57.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 client.17817 shutdown 2024-05-27T07:42:57.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 msgr2=0x55f9977dd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977dd090 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ff40 tx=0x7f99a8004750 comp rx=0 tx=0).stop 2024-05-27T07:42:57.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 shutdown_connections 2024-05-27T07:42:57.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977cbf90 0x55f9977ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977dd090 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f9977ed8e0 0x55f9977efd50 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/976491099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 >> 172.21.15.12:0/976491099 conn(0x55f9976e5390 msgr2=0x55f9977cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99a1600700 0 client.17817 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 shutdown_connections 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.300+0000 7f99cc8edb80 1 -- 172.21.15.12:0/976491099 wait complete. 2024-05-27T07:42:57.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.301+0000 7f99cc8edb80 20 client.17817 trim_cache size 0 max 16384 2024-05-27T07:42:57.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.323+0000 7f99cc8edb80 1 Processor -- start 2024-05-27T07:42:57.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99cc8edb80 1 -- start start 2024-05-27T07:42:57.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f99764be40 0x55f9977e9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9977cbf90 0x55f9977d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81ad7c0 con 0x55f9977cbf90 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81ac960 con 0x55f9977379f0 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81ad090 con 0x55f99764be40 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9977cbf90 0x55f9977d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99b7400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9977cbf90 0x55f9977d6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39676/0 (socket says 172.21.15.12:39676) 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.324+0000 7f99b7400700 1 -- 172.21.15.12:0/1372318943 learned_addr learned my addr 172.21.15.12:0/1372318943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b6000700 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b6a00700 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f99764be40 0x55f9977e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b7400700 1 -- 172.21.15.12:0/1372318943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f99764be40 msgr2=0x55f9977e9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b7400700 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f99764be40 0x55f9977e9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b7400700 1 -- 172.21.15.12:0/1372318943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 msgr2=0x55f9977dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b7400700 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b7400700 1 -- 172.21.15.12:0/1372318943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9977df930 con 0x55f9977cbf90 2024-05-27T07:42:57.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99b7400700 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9977cbf90 0x55f9977d6a80 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009fd0 tx=0x7f99b00aaa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913aefe923cff4dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:57.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.325+0000 7f99a3e00700 1 -- 172.21.15.12:0/1372318943 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00c4070 con 0x55f9977cbf90 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.326+0000 7f99a3e00700 1 -- 172.21.15.12:0/1372318943 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99b00b1040 con 0x55f9977cbf90 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.326+0000 7f99a3e00700 1 -- 172.21.15.12:0/1372318943 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00cbe60 con 0x55f9977cbf90 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.326+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1372318943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9977cbf90 msgr2=0x55f9977d6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.326+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9977cbf90 0x55f9977d6a80 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009fd0 tx=0x7f99b00aaa90 comp rx=0 tx=0).stop 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1372318943 shutdown_connections 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f9977cbf90 0x55f9977d6a80 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f99764be40 0x55f9977e9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1372318943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977379f0 0x55f9977dee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1372318943 >> 172.21.15.12:0/1372318943 conn(0x55f9976e5390 msgr2=0x55f9977de740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1372318943 shutdown_connections 2024-05-27T07:42:57.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1372318943 wait complete. 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 -- start start 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.327+0000 7f99cc8edb80 1 Processor -- start 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977e5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977379f0 0x55f9977d7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99cc8edb80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977cbf90 0x55f9977d7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f9977c2ba0 con 0x55f99764be40 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f9977de3e0 con 0x55f9977cbf90 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99cc8edb80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f9977e0070 con 0x55f9977379f0 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99cc8edb80 10 client.0 _ll_register_callbacks cb 0x55f997692c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977e5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977e5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39690/0 (socket says 172.21.15.12:39690) 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.328+0000 7f99b6a00700 1 -- 172.21.15.12:0/1649705311 learned_addr learned my addr 172.21.15.12:0/1649705311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b7400700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977cbf90 0x55f9977d7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6000700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977379f0 0x55f9977d7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6a00700 1 -- 172.21.15.12:0/1649705311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977379f0 msgr2=0x55f9977d7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6a00700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977379f0 0x55f9977d7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6a00700 1 -- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977cbf90 msgr2=0x55f9977d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6a00700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977cbf90 0x55f9977d7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6a00700 1 -- 172.21.15.12:0/1649705311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9977d8390 con 0x55f99764be40 2024-05-27T07:42:57.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6000700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977379f0 0x55f9977d7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99b6a00700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977e5fe0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f99a801cf00 tx=0x7f99a8010ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c49e0653b3574f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.329+0000 7f99a1600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.330+0000 7f99a1600700 1 -- 172.21.15.12:0/1649705311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a803a070 con 0x55f99764be40 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.330+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b00b1040 con 0x55f99764be40 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.330+0000 7f99a1600700 1 -- 172.21.15.12:0/1649705311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7f99a8005220 con 0x55f99764be40 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.330+0000 7f99a1600700 1 -- 172.21.15.12:0/1649705311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f99a8045050 con 0x55f99764be40 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.330+0000 7f99cc8edb80 10 client.17823 fetch_fsmap learned FSMap version 124 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.330+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x55f997858420 con 0x55f99764be40 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.330+0000 7f99a1600700 1 -- 172.21.15.12:0/1649705311 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99b00b1040 con 0x55f99764be40 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99a1600700 1 -- 172.21.15.12:0/1649705311 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7f99a8033040 con 0x55f99764be40 2024-05-27T07:42:57.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99cc8edb80 10 client.17823 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:42:57.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99cc8edb80 10 client.17823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:42:57.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99cc8edb80 20 client.17823 populate_metadata read hostname 'smithi012' 2024-05-27T07:42:57.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f997858b00 0x55f99785af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:42:57.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99cc8edb80 4 client.17823 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:42:57.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f9976f4570 con 0x55f997858b00 2024-05-27T07:42:57.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.331+0000 7f99b6000700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f997858b00 0x55f99785af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.332+0000 7f99b6000700 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f997858b00 0x55f99785af20 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab42e94dbebcd5c3 server_cookie=cbbe7a5addaf50c9 in_seq=0 out_seq=0 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.332+0000 7f99a1600700 10 client.17823 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.333+0000 7f99a1600700 1 -- 172.21.15.12:0/1649705311 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f9976f4570 con 0x55f997858b00 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.333+0000 7f99a1600700 10 client.17823 handle_command_reply: tid=0 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.333+0000 7f99cc8edb80 1 client.17823 shutdown 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.333+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 msgr2=0x55f9977e5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.333+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977e5fe0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f99a801cf00 tx=0x7f99a8010ab0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 shutdown_connections 2024-05-27T07:42:57.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x55f997858b00 0x55f99785af20 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f9977cbf90 0x55f9977d7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f99764be40 0x55f9977e5fe0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 --2- 172.21.15.12:0/1649705311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f9977379f0 0x55f9977d7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 >> 172.21.15.12:0/1649705311 conn(0x55f9976e5390 msgr2=0x55f9977dce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99a1600700 0 client.17823 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 shutdown_connections 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 1 -- 172.21.15.12:0/1649705311 wait complete. 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.334+0000 7f99cc8edb80 20 client.17823 trim_cache size 0 max 16384 2024-05-27T07:42:57.334 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:42:57.335 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "no active scrubs running", 2024-05-27T07:42:57.335 INFO:teuthology.orchestra.run.smithi012.stdout: "scrubs": {} 2024-05-27T07:42:57.335 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:42:57.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f999c0b2a70 msgr2=0x7f999c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f999c0b2a70 0x7f999c0b4f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99a8005fd0 tx=0x7f99a8005f60 comp rx=0 tx=0).stop 2024-05-27T07:42:57.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 msgr2=0x7f99b81a1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:42:57.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81a1560 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f99b00079a0 tx=0x7f99b0007a80 comp rx=0 tx=0).stop 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 shutdown_connections 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f999c0b2a70 0x7f999c0b4f80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8106610 0x7f99b819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99b810b890 0x7f99b81a1560 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 --2- 172.21.15.12:0/923071180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b810af40 0x7f99b819d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 >> 172.21.15.12:0/923071180 conn(0x7f99b8102020 msgr2=0x7f99b810a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 shutdown_connections 2024-05-27T07:42:57.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:42:57.335+0000 7f99b7e00700 1 -- 172.21.15.12:0/923071180 wait complete. 2024-05-27T07:42:58.433 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:42:58.434 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:42:58.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:42:58 smithi012 ceph-mon[31473]: pgmap v697: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-05-27T07:42:58.854 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:42:58.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:42:58 smithi162 ceph-mon[35676]: pgmap v697: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-05-27T07:42:58.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:42:58 smithi062 ceph-mon[35304]: pgmap v697: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-05-27T07:43:00.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:00 smithi012 ceph-mon[31473]: pgmap v698: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 21 KiB/s wr, 7 op/s 2024-05-27T07:43:00.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:00 smithi162 ceph-mon[35676]: pgmap v698: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 21 KiB/s wr, 7 op/s 2024-05-27T07:43:00.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:00 smithi062 ceph-mon[35304]: pgmap v698: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 21 KiB/s wr, 7 op/s 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.308+0000 7fe2b7a00700 1 -- 172.21.15.12:0/3358394160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0105450 msgr2=0x7fe2b0105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.308+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/3358394160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0105450 0x7fe2b0105860 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009990 tx=0x7fe2a4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.308+0000 7fe2b7a00700 1 -- 172.21.15.12:0/3358394160 shutdown_connections 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.308+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/3358394160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b010e8c0 0x7fe2b0110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.308+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/3358394160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b0105450 0x7fe2b0105860 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.308+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/3358394160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0105da0 0x7fe2b010e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.308+0000 7fe2b7a00700 1 -- 172.21.15.12:0/3358394160 >> 172.21.15.12:0/3358394160 conn(0x7fe2b00fedb0 msgr2=0x7fe2b01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.309+0000 7fe2b7a00700 1 -- 172.21.15.12:0/3358394160 shutdown_connections 2024-05-27T07:43:01.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.309+0000 7fe2b7a00700 1 -- 172.21.15.12:0/3358394160 wait complete. 2024-05-27T07:43:01.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.310+0000 7fe2b7a00700 1 Processor -- start 2024-05-27T07:43:01.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.310+0000 7fe2b7a00700 1 -- start start 2024-05-27T07:43:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.311+0000 7fe2b7a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0105450 0x7fe2b019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.311+0000 7fe2b7a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0105da0 0x7fe2b019d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.311+0000 7fe2b7a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b010e8c0 0x7fe2b01a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.311+0000 7fe2b7a00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0114310 con 0x7fe2b010e8c0 2024-05-27T07:43:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.311+0000 7fe2b7a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0114190 con 0x7fe2b0105450 2024-05-27T07:43:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.311+0000 7fe2b7a00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0114490 con 0x7fe2b0105da0 2024-05-27T07:43:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b010e8c0 0x7fe2b01a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b010e8c0 0x7fe2b01a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39708/0 (socket says 172.21.15.12:39708) 2024-05-27T07:43:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 -- 172.21.15.12:0/2700362139 learned_addr learned my addr 172.21.15.12:0/2700362139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5000700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0105450 0x7fe2b019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2afe00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0105da0 0x7fe2b019d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 -- 172.21.15.12:0/2700362139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0105da0 msgr2=0x7fe2b019d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0105da0 0x7fe2b019d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 -- 172.21.15.12:0/2700362139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0105450 msgr2=0x7fe2b019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0105450 0x7fe2b019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.312+0000 7fe2b5a00700 1 -- 172.21.15.12:0/2700362139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b01a1da0 con 0x7fe2b010e8c0 2024-05-27T07:43:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.313+0000 7fe2b5a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b010e8c0 0x7fe2b01a17b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007ee0 tx=0x7fe2a800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.313+0000 7fe2ad600700 1 -- 172.21.15.12:0/2700362139 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a800dbd0 con 0x7fe2b010e8c0 2024-05-27T07:43:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.313+0000 7fe2ad600700 1 -- 172.21.15.12:0/2700362139 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2a800dd70 con 0x7fe2b010e8c0 2024-05-27T07:43:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.313+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b01a2090 con 0x7fe2b010e8c0 2024-05-27T07:43:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.313+0000 7fe2ad600700 1 -- 172.21.15.12:0/2700362139 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a80058d0 con 0x7fe2b010e8c0 2024-05-27T07:43:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.313+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b01a25d0 con 0x7fe2b010e8c0 2024-05-27T07:43:01.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.314+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b00512b0 con 0x7fe2b010e8c0 2024-05-27T07:43:01.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.317+0000 7fe2ad600700 1 -- 172.21.15.12:0/2700362139 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe2a801a020 con 0x7fe2b010e8c0 2024-05-27T07:43:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.319+0000 7fe2ad600700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2980b2a70 0x7fe2980b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.319+0000 7fe2ad600700 1 -- 172.21.15.12:0/2700362139 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fe2a80e2860 con 0x7fe2b010e8c0 2024-05-27T07:43:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.319+0000 7fe2b5000700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2980b2a70 0x7fe2980b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:01.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.320+0000 7fe2b5000700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2980b2a70 0x7fe2980b4f80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400b5c0 tx=0x7fe2a4005fb0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:01.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.322+0000 7fe2ad600700 1 -- 172.21.15.12:0/2700362139 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe2a8010070 con 0x7fe2b010e8c0 2024-05-27T07:43:01.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.612+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2b002e5e0 con 0x7fe2b010e8c0 2024-05-27T07:43:01.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.614+0000 7fe2ad600700 1 -- 172.21.15.12:0/2700362139 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+4419 (secure 0 0 0) 0x7fe2a80a9dd0 con 0x7fe2b010e8c0 2024-05-27T07:43:01.613 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:43:01.613 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":124,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:17:09.825542+0000","modified":"2024-05-27T07:42:25.613728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":34475,"mds_1":34439,"mds_2":34415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34415":{"gid":34415,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":35,"addr":"172.21.15.62:6835/989260451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":989260451},{"type":"v1","addr":"172.21.15.62:6835","nonce":989260451}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34439":{"gid":34439,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":32,"addr":"172.21.15.62:6837/1993227402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":1993227402},{"type":"v1","addr":"172.21.15.62:6837","nonce":1993227402}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34475":{"gid":34475,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.62:6839/994488749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":994488749},{"type":"v1","addr":"172.21.15.62:6839","nonce":994488749}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34475,"qdb_cluster":[34475,34415,34439]},"id":1}]} 2024-05-27T07:43:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2980b2a70 msgr2=0x7fe2980b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2980b2a70 0x7fe2980b4f80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400b5c0 tx=0x7fe2a4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b010e8c0 msgr2=0x7fe2b01a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b010e8c0 0x7fe2b01a17b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007ee0 tx=0x7fe2a800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 shutdown_connections 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe2980b2a70 0x7fe2980b4f80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0105450 0x7fe2b019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2b010e8c0 0x7fe2b01a17b0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 --2- 172.21.15.12:0/2700362139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2b0105da0 0x7fe2b019d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 >> 172.21.15.12:0/2700362139 conn(0x7fe2b00fedb0 msgr2=0x7fe2b01031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 shutdown_connections 2024-05-27T07:43:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:01.617+0000 7fe2b7a00700 1 -- 172.21.15.12:0/2700362139 wait complete. 2024-05-27T07:43:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 124 2024-05-27T07:43:01.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:01 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/2700362139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:01.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:01 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/2700362139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:02.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:01 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/2700362139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:43:02.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:43:02.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:02 smithi012 ceph-mon[31473]: pgmap v699: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s wr, 0 op/s 2024-05-27T07:43:02.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:02 smithi162 ceph-mon[35676]: pgmap v699: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s wr, 0 op/s 2024-05-27T07:43:02.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:02 smithi062 ceph-mon[35304]: pgmap v699: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s wr, 0 op/s 2024-05-27T07:43:03.006 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:03.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:03 smithi012 ceph-mon[31473]: pgmap v700: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s wr, 0 op/s 2024-05-27T07:43:03.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:03 smithi162 ceph-mon[35676]: pgmap v700: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s wr, 0 op/s 2024-05-27T07:43:03.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:03 smithi062 ceph-mon[35304]: pgmap v700: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s wr, 0 op/s 2024-05-27T07:43:05.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 -- 172.21.15.12:0/131829965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 msgr2=0x7fad9c106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 --2- 172.21.15.12:0/131829965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c106980 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fad88009990 tx=0x7fad88009ca0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 -- 172.21.15.12:0/131829965 shutdown_connections 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 --2- 172.21.15.12:0/131829965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad9c10c5b0 0x7fad9c10c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 --2- 172.21.15.12:0/131829965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad9c106ec0 0x7fad9c113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 --2- 172.21.15.12:0/131829965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c106980 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 -- 172.21.15.12:0/131829965 >> 172.21.15.12:0/131829965 conn(0x7fad9c102070 msgr2=0x7fad9c1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 -- 172.21.15.12:0/131829965 shutdown_connections 2024-05-27T07:43:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.347+0000 7fada4200700 1 -- 172.21.15.12:0/131829965 wait complete. 2024-05-27T07:43:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.348+0000 7fada4200700 1 Processor -- start 2024-05-27T07:43:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada4200700 1 -- start start 2024-05-27T07:43:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada4200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c19d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada4200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad9c106ec0 0x7fad9c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada4200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad9c10c5b0 0x7fad9c1a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada4200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c116590 con 0x7fad9c106570 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada4200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c116410 con 0x7fad9c10c5b0 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada4200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c116710 con 0x7fad9c106ec0 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c19d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c19d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58010/0 (socket says 172.21.15.12:58010) 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.349+0000 7fada1800700 1 -- 172.21.15.12:0/3484789872 learned_addr learned my addr 172.21.15.12:0/3484789872 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada0e00700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad9c106ec0 0x7fad9c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada2200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad9c10c5b0 0x7fad9c1a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada1800700 1 -- 172.21.15.12:0/3484789872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad9c106ec0 msgr2=0x7fad9c19d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada1800700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad9c106ec0 0x7fad9c19d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada1800700 1 -- 172.21.15.12:0/3484789872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad9c10c5b0 msgr2=0x7fad9c1a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada1800700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad9c10c5b0 0x7fad9c1a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada1800700 1 -- 172.21.15.12:0/3484789872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9c1a1f10 con 0x7fad9c106570 2024-05-27T07:43:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.350+0000 7fada1800700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c19d040 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fad9000d980 tx=0x7fad9000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.351+0000 7fad9a000700 1 -- 172.21.15.12:0/3484789872 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad90017070 con 0x7fad9c106570 2024-05-27T07:43:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.351+0000 7fad9a000700 1 -- 172.21.15.12:0/3484789872 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad90010460 con 0x7fad9c106570 2024-05-27T07:43:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.351+0000 7fad9a000700 1 -- 172.21.15.12:0/3484789872 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9000de60 con 0x7fad9c106570 2024-05-27T07:43:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.351+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad9c1a2200 con 0x7fad9c106570 2024-05-27T07:43:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.351+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad9c1a9b20 con 0x7fad9c106570 2024-05-27T07:43:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.355+0000 7fad9a000700 1 -- 172.21.15.12:0/3484789872 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fad90010610 con 0x7fad9c106570 2024-05-27T07:43:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.356+0000 7fadb2dfab80 1 Processor -- start 2024-05-27T07:43:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.356+0000 7fadb2dfab80 1 -- start start 2024-05-27T07:43:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.356+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4ac77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.356+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac7d30 0x5618e4ac1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.356+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4ac21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5618e49efe40 con 0x5618e4942e40 2024-05-27T07:43:05.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5618e49df9e0 con 0x5618e4ac1db0 2024-05-27T07:43:05.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5618e49e3a20 con 0x5618e4ac7d30 2024-05-27T07:43:05.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fada2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4ac77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fada2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4ac77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58022/0 (socket says 172.21.15.12:58022) 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fada2200700 1 -- 172.21.15.12:0/3543700422 learned_addr learned my addr 172.21.15.12:0/3543700422 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fad9a000700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad840b2a70 0x7fad840b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fada1800700 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac7d30 0x5618e4ac1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fad9a000700 1 -- 172.21.15.12:0/3484789872 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fad900e24e0 con 0x7fad9c106570 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.357+0000 7fada0e00700 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4ac21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.358+0000 7fada2200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad840b2a70 0x7fad840b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.358+0000 7fada2200700 1 -- 172.21.15.12:0/3543700422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac7d30 msgr2=0x5618e4ac1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.358+0000 7fada2200700 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac7d30 0x5618e4ac1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.358+0000 7fada2200700 1 -- 172.21.15.12:0/3543700422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 msgr2=0x5618e4ac21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.358+0000 7fada2200700 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4ac21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.358+0000 7fada2200700 1 -- 172.21.15.12:0/3543700422 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618e49a8420 con 0x5618e4942e40 2024-05-27T07:43:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.358+0000 7fada2200700 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4ac77f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fad94007a20 tx=0x7fad94007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea0cf2cc48268ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.359+0000 7fad8ea00700 1 -- 172.21.15.12:0/3543700422 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9401d040 con 0x5618e4942e40 2024-05-27T07:43:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.359+0000 7fada2200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad840b2a70 0x7fad840b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fad9401a000 tx=0x7fad9401c000 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.359+0000 7fad8ea00700 1 -- 172.21.15.12:0/3543700422 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad94016de0 con 0x5618e4942e40 2024-05-27T07:43:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.359+0000 7fad8ea00700 1 -- 172.21.15.12:0/3543700422 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94012040 con 0x5618e4942e40 2024-05-27T07:43:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3543700422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 msgr2=0x5618e4ac77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4ac77f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fad94007a20 tx=0x7fad94007d30 comp rx=0 tx=0).stop 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3543700422 shutdown_connections 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4ac21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4ac77f0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3543700422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac7d30 0x5618e4ac1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3543700422 >> 172.21.15.12:0/3543700422 conn(0x5618e49a2fc0 msgr2=0x5618e4abc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3543700422 shutdown_connections 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.360+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3543700422 wait complete. 2024-05-27T07:43:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.361+0000 7fadb2dfab80 1 -- start start 2024-05-27T07:43:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.361+0000 7fadb2dfab80 1 Processor -- start 2024-05-27T07:43:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4acb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4acbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac7d30 0x5618e4acc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5618e48e5fa0 con 0x5618e4942e40 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5618e4980760 con 0x5618e4ac7d30 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5618e4ace7a0 con 0x5618e4ac1db0 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fadb2dfab80 10 client.0 _ll_register_callbacks cb 0x5618e4989c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4acb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4acb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58032/0 (socket says 172.21.15.12:58032) 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 -- 172.21.15.12:0/3829264762 learned_addr learned my addr 172.21.15.12:0/3829264762 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada0e00700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4acbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada2200700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac7d30 0x5618e4acc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 -- 172.21.15.12:0/3829264762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 msgr2=0x5618e4acbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4acbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 -- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac7d30 msgr2=0x5618e4acc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac7d30 0x5618e4acc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.362+0000 7fada1800700 1 -- 172.21.15.12:0/3829264762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618e492f8d0 con 0x5618e4942e40 2024-05-27T07:43:05.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fada0e00700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4acbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:05.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fada2200700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac7d30 0x5618e4acc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:05.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fada1800700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4acb5a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fad900a53d0 tx=0x7fad9006ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b93b87c901cf16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:05.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fad83400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fad83400700 1 -- 172.21.15.12:0/3829264762 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad90020040 con 0x5618e4942e40 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fad83400700 1 -- 172.21.15.12:0/3829264762 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fad9006eac0 con 0x5618e4942e40 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad94016de0 con 0x5618e4942e40 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fad83400700 1 -- 172.21.15.12:0/3829264762 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7fad900a9a50 con 0x5618e4942e40 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fadb2dfab80 10 client.17841 fetch_fsmap learned FSMap version 124 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.363+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x5618e49b0060 con 0x5618e4942e40 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fad83400700 1 -- 172.21.15.12:0/3829264762 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad94016de0 con 0x5618e4942e40 2024-05-27T07:43:05.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fad83400700 1 -- 172.21.15.12:0/3829264762 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7fad900e2120 con 0x5618e4942e40 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fadb2dfab80 10 client.17841 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fadb2dfab80 10 client.17841 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fadb2dfab80 20 client.17841 populate_metadata read hostname 'smithi012' 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4adbab0 0x5618e4addf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fadb2dfab80 4 client.17841 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.364+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618e49f1d90 con 0x5618e4adbab0 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.365+0000 7fada0e00700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4adbab0 0x5618e4addf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.366+0000 7fada0e00700 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4adbab0 0x5618e4addf40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54bc436c54ed6061 server_cookie=d46dda099e574da2 in_seq=0 out_seq=0 2024-05-27T07:43:05.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.366+0000 7fad83400700 10 client.17841 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:43:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fad83400700 1 -- 172.21.15.12:0/3829264762 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5618e49f1d90 con 0x5618e4adbab0 2024-05-27T07:43:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fad83400700 10 client.17841 handle_command_reply: tid=0 2024-05-27T07:43:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 client.17841 shutdown 2024-05-27T07:43:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 msgr2=0x5618e4acb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4acb5a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fad900a53d0 tx=0x7fad9006ece0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 shutdown_connections 2024-05-27T07:43:05.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4adbab0 0x5618e4addf40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac7d30 0x5618e4acc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4942e40 0x5618e4acb5a0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3829264762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4acbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 >> 172.21.15.12:0/3829264762 conn(0x5618e49a2fc0 msgr2=0x5618e4ac21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.368+0000 7fad83400700 0 client.17841 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.369+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 shutdown_connections 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.369+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3829264762 wait complete. 2024-05-27T07:43:05.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.369+0000 7fadb2dfab80 20 client.17841 trim_cache size 0 max 16384 2024-05-27T07:43:05.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 Processor -- start 2024-05-27T07:43:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 -- start start 2024-05-27T07:43:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4942e40 0x5618e4ac1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4acda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ac5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5618e4ac2800 con 0x5618e4ac7d30 2024-05-27T07:43:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c1b1fb0 con 0x5618e4ac1db0 2024-05-27T07:43:05.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.386+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c1b1150 con 0x5618e4942e40 2024-05-27T07:43:05.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ac5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ac5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58046/0 (socket says 172.21.15.12:58046) 2024-05-27T07:43:05.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 -- 172.21.15.12:0/464403336 learned_addr learned my addr 172.21.15.12:0/464403336 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada0e00700 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4acda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada1800700 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4942e40 0x5618e4ac1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 -- 172.21.15.12:0/464403336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4942e40 msgr2=0x5618e4ac1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4942e40 0x5618e4ac1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 -- 172.21.15.12:0/464403336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 msgr2=0x5618e4acda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4acda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.387+0000 7fada2200700 1 -- 172.21.15.12:0/464403336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618e4acc390 con 0x5618e4ac7d30 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.388+0000 7fada2200700 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ac5920 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fad94009410 tx=0x7fad94007950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d70b239cdc7e4ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.388+0000 7fad8e000700 1 -- 172.21.15.12:0/464403336 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94031070 con 0x5618e4ac7d30 2024-05-27T07:43:05.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.388+0000 7fad8e000700 1 -- 172.21.15.12:0/464403336 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad94019400 con 0x5618e4ac7d30 2024-05-27T07:43:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.388+0000 7fad8e000700 1 -- 172.21.15.12:0/464403336 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9401d040 con 0x5618e4ac7d30 2024-05-27T07:43:05.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 -- 172.21.15.12:0/464403336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 msgr2=0x5618e4ac5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ac5920 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fad94009410 tx=0x7fad94007950 comp rx=0 tx=0).stop 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 -- 172.21.15.12:0/464403336 shutdown_connections 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ac5920 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4942e40 0x5618e4ac1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/464403336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4ac1db0 0x5618e4acda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 -- 172.21.15.12:0/464403336 >> 172.21.15.12:0/464403336 conn(0x5618e49a2fc0 msgr2=0x5618e4ac5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 -- 172.21.15.12:0/464403336 shutdown_connections 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.389+0000 7fadb2dfab80 1 -- 172.21.15.12:0/464403336 wait complete. 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.390+0000 7fadb2dfab80 1 -- start start 2024-05-27T07:43:05.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.390+0000 7fadb2dfab80 1 Processor -- start 2024-05-27T07:43:05.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.390+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4942e40 0x5618e4ad3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.390+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4ad4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.390+0000 7fadb2dfab80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ae2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.390+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c1b1880 con 0x5618e4ac7d30 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.390+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5618e4ae3200 con 0x5618e4942e40 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fadb2dfab80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5618e4ae3400 con 0x5618e4ac1db0 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fadb2dfab80 10 client.0 _ll_register_callbacks cb 0x5618e4989c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ae2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ae2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58058/0 (socket says 172.21.15.12:58058) 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 -- 172.21.15.12:0/3802495725 learned_addr learned my addr 172.21.15.12:0/3802495725 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada0e00700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4ad4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada1800700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4942e40 0x5618e4ad3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 -- 172.21.15.12:0/3802495725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 msgr2=0x5618e4ad4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4ad4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 -- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4942e40 msgr2=0x5618e4ad3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:05.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4942e40 0x5618e4ad3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada2200700 1 -- 172.21.15.12:0/3802495725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618e4ae3600 con 0x5618e4ac7d30 2024-05-27T07:43:05.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.391+0000 7fada0e00700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4ad4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:05.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fada2200700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ae2b50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fad94009010 tx=0x7fad94019dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2326920765e07260 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:05.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fad83400700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:43:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fad83400700 1 -- 172.21.15.12:0/3802495725 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9403c060 con 0x5618e4ac7d30 2024-05-27T07:43:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fad83400700 1 -- 172.21.15.12:0/3802495725 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9554+0+0 (secure 0 0 0) 0x7fad94016de0 con 0x5618e4ac7d30 2024-05-27T07:43:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad94019400 con 0x5618e4ac7d30 2024-05-27T07:43:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fad83400700 1 -- 172.21.15.12:0/3802495725 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7fad94031030 con 0x5618e4ac7d30 2024-05-27T07:43:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fadb2dfab80 10 client.17847 fetch_fsmap learned FSMap version 124 2024-05-27T07:43:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.392+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x5618e4b462f0 con 0x5618e4ac7d30 2024-05-27T07:43:05.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fad83400700 1 -- 172.21.15.12:0/3802495725 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad94019400 con 0x5618e4ac7d30 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fad83400700 1 -- 172.21.15.12:0/3802495725 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7fad94018040 con 0x5618e4ac7d30 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fadb2dfab80 10 client.17847 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fadb2dfab80 10 client.17847 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fadb2dfab80 20 client.17847 populate_metadata read hostname 'smithi012' 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4b46720 0x5618e4b48b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fadb2dfab80 4 client.17847 mds_command: new command op to 34415 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.393+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 --> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5618e49eb570 con 0x5618e4b46720 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.394+0000 7fada1800700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4b46720 0x5618e4b48b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fada1800700 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4b46720 0x5618e4b48b40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9883c0f5820167b server_cookie=d0804bb96bd9367b in_seq=0 out_seq=0 2024-05-27T07:43:05.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fad83400700 10 client.17847 ms_handle_connect on v2:172.21.15.62:6834/989260451 2024-05-27T07:43:05.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fad83400700 1 -- 172.21.15.12:0/3802495725 <== mds.2 v2:172.21.15.62:6834/989260451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5618e49eb570 con 0x5618e4b46720 2024-05-27T07:43:05.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fad83400700 10 client.17847 handle_command_reply: tid=0 2024-05-27T07:43:05.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 client.17847 shutdown 2024-05-27T07:43:05.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 msgr2=0x5618e4ae2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ae2b50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fad94009010 tx=0x7fad94019dd0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 shutdown_connections 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:6834/989260451,v1:172.21.15.62:6835/989260451] conn(0x5618e4b46720 0x5618e4b48b40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618e4942e40 0x5618e4ad3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5618e4ac7d30 0x5618e4ae2b50 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 --2- 172.21.15.12:0/3802495725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5618e4ac1db0 0x5618e4ad4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.395+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 >> 172.21.15.12:0/3802495725 conn(0x5618e49a2fc0 msgr2=0x5618e4ac5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.396+0000 7fad83400700 0 client.17847 ms_handle_reset on v2:172.21.15.62:6834/989260451 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.396+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 shutdown_connections 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.396+0000 7fadb2dfab80 1 -- 172.21.15.12:0/3802495725 wait complete. 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.396+0000 7fadb2dfab80 20 client.17847 trim_cache size 0 max 16384 2024-05-27T07:43:05.396 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:43:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.396+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad840b2a70 msgr2=0x7fad840b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad840b2a70 0x7fad840b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fad9401a000 tx=0x7fad9401c000 comp rx=0 tx=0).stop 2024-05-27T07:43:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 msgr2=0x7fad9c19d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c19d040 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fad9000d980 tx=0x7fad9000dd40 comp rx=0 tx=0).stop 2024-05-27T07:43:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 shutdown_connections 2024-05-27T07:43:05.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fad840b2a70 0x7fad840b4f80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad9c10c5b0 0x7fad9c1a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad9c106570 0x7fad9c19d040 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 --2- 172.21.15.12:0/3484789872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad9c106ec0 0x7fad9c19d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 >> 172.21.15.12:0/3484789872 conn(0x7fad9c102070 msgr2=0x7fad9c1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 shutdown_connections 2024-05-27T07:43:05.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:05.397+0000 7fada4200700 1 -- 172.21.15.12:0/3484789872 wait complete. 2024-05-27T07:43:06.101 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:06 smithi012 ceph-mon[31473]: pgmap v701: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 52 KiB/s wr, 17 op/s 2024-05-27T07:43:06.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:43:06.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:06 smithi162 ceph-mon[35676]: pgmap v701: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 52 KiB/s wr, 17 op/s 2024-05-27T07:43:06.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:06 smithi062 ceph-mon[35304]: pgmap v701: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 52 KiB/s wr, 17 op/s 2024-05-27T07:43:06.940 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:07.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:07 smithi012 ceph-mon[31473]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:43:07.481 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:07 smithi162 ceph-mon[35676]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:43:07.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:07 smithi062 ceph-mon[35304]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:43:08.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:08 smithi012 ceph-mon[31473]: pgmap v703: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:08.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:08 smithi162 ceph-mon[35676]: pgmap v703: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:08.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:08 smithi062 ceph-mon[35304]: pgmap v703: 129 pgs: 129 active+clean; 543 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:09.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.849+0000 7fd096200700 1 -- 172.21.15.12:0/3399122834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090108f10 msgr2=0x7fd090109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.849+0000 7fd096200700 1 --2- 172.21.15.12:0/3399122834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090108f10 0x7fd090109320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd0780099c0 tx=0x7fd078009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.849+0000 7fd096200700 1 -- 172.21.15.12:0/3399122834 shutdown_connections 2024-05-27T07:43:09.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.849+0000 7fd096200700 1 --2- 172.21.15.12:0/3399122834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0901045e0 0x7fd0901049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.849+0000 7fd096200700 1 --2- 172.21.15.12:0/3399122834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd090109860 0x7fd0901111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.849+0000 7fd096200700 1 --2- 172.21.15.12:0/3399122834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090108f10 0x7fd090109320 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.850+0000 7fd096200700 1 -- 172.21.15.12:0/3399122834 >> 172.21.15.12:0/3399122834 conn(0x7fd090100030 msgr2=0x7fd090102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.851+0000 7fd096200700 1 -- 172.21.15.12:0/3399122834 shutdown_connections 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.852+0000 7fd096200700 1 -- 172.21.15.12:0/3399122834 wait complete. 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.854+0000 7fd096200700 1 Processor -- start 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.854+0000 7fd096200700 1 -- start start 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.856+0000 7fd096200700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0901045e0 0x7fd0901108d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.856+0000 7fd096200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd090108f10 0x7fd090110e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.856+0000 7fd096200700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090109860 0x7fd09010dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.856+0000 7fd096200700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd090076920 con 0x7fd0901045e0 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.856+0000 7fd096200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900767a0 con 0x7fd090108f10 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.856+0000 7fd096200700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd090076aa0 con 0x7fd090109860 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.857+0000 7fd08fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090109860 0x7fd09010dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.857+0000 7fd08fe00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090109860 0x7fd09010dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:60152/0 (socket says 172.21.15.12:60152) 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.857+0000 7fd08fe00700 1 -- 172.21.15.12:0/1851375974 learned_addr learned my addr 172.21.15.12:0/1851375974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.857+0000 7fd086a00700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd090108f10 0x7fd090110e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.857+0000 7fd08f400700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0901045e0 0x7fd0901108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.859+0000 7fd08fe00700 1 -- 172.21.15.12:0/1851375974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd090108f10 msgr2=0x7fd090110e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.859+0000 7fd08fe00700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd090108f10 0x7fd090110e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.859+0000 7fd08fe00700 1 -- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0901045e0 msgr2=0x7fd0901108d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.859+0000 7fd08fe00700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0901045e0 0x7fd0901108d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.859+0000 7fd08fe00700 1 -- 172.21.15.12:0/1851375974 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09010e2c0 con 0x7fd090109860 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.861+0000 7fd08fe00700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090109860 0x7fd09010dbc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c970 tx=0x7fd08800cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.862+0000 7fd08cc00700 1 -- 172.21.15.12:0/1851375974 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088004a30 con 0x7fd090109860 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.862+0000 7fd08cc00700 1 -- 172.21.15.12:0/1851375974 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0880054b0 con 0x7fd090109860 2024-05-27T07:43:09.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.862+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09010e550 con 0x7fd090109860 2024-05-27T07:43:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.862+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0901a36c0 con 0x7fd090109860 2024-05-27T07:43:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.862+0000 7fd08cc00700 1 -- 172.21.15.12:0/1851375974 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08800eac0 con 0x7fd090109860 2024-05-27T07:43:09.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.868+0000 7fd08cc00700 1 -- 172.21.15.12:0/1851375974 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd08800ec60 con 0x7fd090109860 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.870+0000 7fd08cc00700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd0740b2b40 0x7fd0740b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.870+0000 7fd08cc00700 1 -- 172.21.15.12:0/1851375974 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7fd0880e32e0 con 0x7fd090109860 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.871+0000 7fd08f400700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd0740b2b40 0x7fd0740b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.872+0000 7fd08f400700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd0740b2b40 0x7fd0740b5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd09010ed00 tx=0x7fd08000b040 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.872+0000 7fd0a4d3db80 1 Processor -- start 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.872+0000 7fd0a4d3db80 1 -- start start 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.873+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x56537551e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.874+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375518870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.874+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x5653755191c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.874+0000 7fd08f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x56537551e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.874+0000 7fd08f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x56537551e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58078/0 (socket says 172.21.15.12:58078) 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.874+0000 7fd08f400700 1 -- 172.21.15.12:0/1881837652 learned_addr learned my addr 172.21.15.12:0/1881837652 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.875+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x565375446e40 con 0x565375399e40 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.875+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5653754369e0 con 0x56537551ed30 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.875+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56537543aa20 con 0x565375518db0 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.875+0000 7fd08fe00700 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x5653755191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.875+0000 7fd086a00700 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375518870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.876+0000 7fd08f400700 1 -- 172.21.15.12:0/1881837652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 msgr2=0x5653755191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.876+0000 7fd08f400700 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x5653755191c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.876+0000 7fd08f400700 1 -- 172.21.15.12:0/1881837652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 msgr2=0x565375518870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.876+0000 7fd08f400700 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375518870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.876+0000 7fd08f400700 1 -- 172.21.15.12:0/1881837652 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653753ff420 con 0x565375399e40 2024-05-27T07:43:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.876+0000 7fd086a00700 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375518870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:09.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.877+0000 7fd08f400700 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x56537551e7f0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd0800176e0 tx=0x7fd080014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcecdb6050bc9a60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:09.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.877+0000 7fd084c00700 1 -- 172.21.15.12:0/1881837652 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080004db0 con 0x565375399e40 2024-05-27T07:43:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.878+0000 7fd084c00700 1 -- 172.21.15.12:0/1881837652 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd08001b070 con 0x565375399e40 2024-05-27T07:43:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.878+0000 7fd084c00700 1 -- 172.21.15.12:0/1881837652 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0800053e0 con 0x565375399e40 2024-05-27T07:43:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.879+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 msgr2=0x56537551e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.879+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x56537551e7f0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd0800176e0 tx=0x7fd080014000 comp rx=0 tx=0).stop 2024-05-27T07:43:09.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.880+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 shutdown_connections 2024-05-27T07:43:09.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.880+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x5653755191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.880+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x56537551e7f0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.880+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1881837652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375518870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.880+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 >> 172.21.15.12:0/1881837652 conn(0x5653753f9fc0 msgr2=0x5653755138d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:09.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.880+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 shutdown_connections 2024-05-27T07:43:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.881+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1881837652 wait complete. 2024-05-27T07:43:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.882+0000 7fd0a4d3db80 1 -- start start 2024-05-27T07:43:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.882+0000 7fd0a4d3db80 1 Processor -- start 2024-05-27T07:43:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.883+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x5653755224b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.883+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375518db0 0x5653755229f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.883+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375522f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.884+0000 7fd086a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375518db0 0x5653755229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.885+0000 7fd086a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375518db0 0x5653755229f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58080/0 (socket says 172.21.15.12:58080) 2024-05-27T07:43:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.885+0000 7fd08f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x5653755224b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.885+0000 7fd08f400700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x5653755224b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.12:60170/0 (socket says 172.21.15.12:60170) 2024-05-27T07:43:09.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.885+0000 7fd08fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375522f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.885+0000 7fd086a00700 1 -- 172.21.15.12:0/51808013 learned_addr learned my addr 172.21.15.12:0/51808013 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.885+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56537533cfa0 con 0x565375518db0 2024-05-27T07:43:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.886+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5653753d7760 con 0x56537551ed30 2024-05-27T07:43:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.886+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565375525800 con 0x565375399e40 2024-05-27T07:43:09.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.887+0000 7fd0a4d3db80 10 client.0 _ll_register_callbacks cb 0x5653753e0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:09.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.887+0000 7fd08f400700 1 -- 172.21.15.12:0/51808013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 msgr2=0x565375522f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.887+0000 7fd08f400700 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375522f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.887+0000 7fd08f400700 1 -- 172.21.15.12:0/51808013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375518db0 msgr2=0x5653755229f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.887+0000 7fd08f400700 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375518db0 0x5653755229f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.887+0000 7fd08f400700 1 -- 172.21.15.12:0/51808013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653753868d0 con 0x565375399e40 2024-05-27T07:43:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.888+0000 7fd08f400700 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x5653755224b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd080015f40 tx=0x7fd0800158d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5797e9848e69ae3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:09.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.888+0000 7fd07d600700 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-05-27T07:43:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.889+0000 7fd07d600700 1 -- 172.21.15.12:0/51808013 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080015bb0 con 0x565375399e40 2024-05-27T07:43:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.889+0000 7fd07d600700 1 -- 172.21.15.12:0/51808013 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7fd080005450 con 0x565375399e40 2024-05-27T07:43:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.889+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd08001b070 con 0x565375399e40 2024-05-27T07:43:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.891+0000 7fd07d600700 1 -- 172.21.15.12:0/51808013 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7fd08003b050 con 0x565375399e40 2024-05-27T07:43:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.891+0000 7fd0a4d3db80 10 client.27172 fetch_fsmap learned FSMap version 124 2024-05-27T07:43:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.891+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x565375407060 con 0x565375399e40 2024-05-27T07:43:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.891+0000 7fd07d600700 1 -- 172.21.15.12:0/51808013 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd08001b070 con 0x565375399e40 2024-05-27T07:43:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.892+0000 7fd07d600700 1 -- 172.21.15.12:0/51808013 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7fd080020070 con 0x565375399e40 2024-05-27T07:43:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.892+0000 7fd0a4d3db80 10 client.27172 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:43:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.892+0000 7fd0a4d3db80 10 client.27172 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:43:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.892+0000 7fd0a4d3db80 20 client.27172 populate_metadata read hostname 'smithi012' 2024-05-27T07:43:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.892+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5653755329f0 0x565375534ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.892+0000 7fd0a4d3db80 4 client.27172 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.892+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565375448d90 con 0x5653755329f0 2024-05-27T07:43:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.895+0000 7fd086a00700 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5653755329f0 0x565375534ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.896+0000 7fd086a00700 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5653755329f0 0x565375534ee0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a26e1fbf4e1a352 server_cookie=a0a79648b25b3c70 in_seq=0 out_seq=0 2024-05-27T07:43:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.896+0000 7fd07d600700 10 client.27172 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:43:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.897+0000 7fd07d600700 1 -- 172.21.15.12:0/51808013 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x565375448d90 con 0x5653755329f0 2024-05-27T07:43:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.898+0000 7fd07d600700 10 client.27172 handle_command_reply: tid=0 2024-05-27T07:43:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.898+0000 7fd0a4d3db80 1 client.27172 shutdown 2024-05-27T07:43:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.898+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 msgr2=0x5653755224b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.898+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x5653755224b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd080015f40 tx=0x7fd0800158d0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 shutdown_connections 2024-05-27T07:43:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x5653755329f0 0x565375534ee0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375522f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd07d600700 0 client.27172 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:43:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x5653755224b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/51808013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375518db0 0x5653755229f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 >> 172.21.15.12:0/51808013 conn(0x5653753f9fc0 msgr2=0x565375513580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 shutdown_connections 2024-05-27T07:43:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/51808013 wait complete. 2024-05-27T07:43:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.899+0000 7fd0a4d3db80 20 client.27172 trim_cache size 0 max 16384 2024-05-27T07:43:09.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 Processor -- start 2024-05-27T07:43:09.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 -- start start 2024-05-27T07:43:09.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x565375518870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x56537551cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x56537552e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c00e5f0 con 0x565375399e40 2024-05-27T07:43:09.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c00d790 con 0x56537551ed30 2024-05-27T07:43:09.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.931+0000 7fd0a4d3db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c00dec0 con 0x565375518db0 2024-05-27T07:43:09.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.932+0000 7fd08f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x565375518870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.932+0000 7fd08f400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x565375518870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58096/0 (socket says 172.21.15.12:58096) 2024-05-27T07:43:09.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd08fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x56537552e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd08fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x56537552e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:56350/0 (socket says 172.21.15.12:56350) 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd086a00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x56537551cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd08f400700 1 -- 172.21.15.12:0/1716046139 learned_addr learned my addr 172.21.15.12:0/1716046139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd086a00700 1 -- 172.21.15.12:0/1716046139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 msgr2=0x56537552e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd086a00700 1 --2- 172.21.15.12:0/1716046139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x56537552e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd086a00700 1 -- 172.21.15.12:0/1716046139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 msgr2=0x565375518870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd086a00700 1 --2- 172.21.15.12:0/1716046139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x565375518870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.933+0000 7fd086a00700 1 -- 172.21.15.12:0/1716046139 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653755232a0 con 0x565375518db0 2024-05-27T07:43:09.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.934+0000 7fd086a00700 1 --2- 172.21.15.12:0/1716046139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x56537551cb70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x565375519760 tx=0x7fd078005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ed93b87346e5a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.934+0000 7fd07fe00700 1 -- 172.21.15.12:0/1716046139 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07800b790 con 0x565375518db0 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.934+0000 7fd07fe00700 1 -- 172.21.15.12:0/1716046139 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd07800bdc0 con 0x565375518db0 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1716046139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 msgr2=0x56537551cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd07fe00700 1 -- 172.21.15.12:0/1716046139 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07800f460 con 0x565375518db0 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1716046139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x56537551cb70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x565375519760 tx=0x7fd078005040 comp rx=0 tx=0).stop 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1716046139 shutdown_connections 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1716046139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x56537552e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1716046139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x565375399e40 0x565375518870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/1716046139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375518db0 0x56537551cb70 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x565375519760 tx=0x7fd078005040 comp rx=0 tx=0).stop 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.936+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1716046139 >> 172.21.15.12:0/1716046139 conn(0x5653753f9fc0 msgr2=0x565375521410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:09.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.937+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1716046139 shutdown_connections 2024-05-27T07:43:09.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.937+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/1716046139 wait complete. 2024-05-27T07:43:09.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.938+0000 7fd0a4d3db80 1 -- start start 2024-05-27T07:43:09.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.938+0000 7fd0a4d3db80 1 Processor -- start 2024-05-27T07:43:09.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.940+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x56537552b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.940+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375539fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.940+0000 7fd0a4d3db80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56537553a500 0x56537553a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.940+0000 7fd0a4d3db80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x565375521b20 con 0x56537553a500 2024-05-27T07:43:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.940+0000 7fd0a4d3db80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5653755235c0 con 0x56537551ed30 2024-05-27T07:43:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.940+0000 7fd0a4d3db80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565375514700 con 0x565375399e40 2024-05-27T07:43:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.940+0000 7fd0a4d3db80 10 client.0 _ll_register_callbacks cb 0x5653753e0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:09.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.941+0000 7fd086a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375539fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.941+0000 7fd086a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375539fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.12:56360/0 (socket says 172.21.15.12:56360) 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.941+0000 7fd086a00700 1 -- 172.21.15.12:0/2538676291 learned_addr learned my addr 172.21.15.12:0/2538676291 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.941+0000 7fd08fe00700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56537553a500 0x56537553a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.942+0000 7fd08f400700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x56537552b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.942+0000 7fd086a00700 1 -- 172.21.15.12:0/2538676291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 msgr2=0x56537552b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.942+0000 7fd086a00700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x56537552b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.942+0000 7fd086a00700 1 -- 172.21.15.12:0/2538676291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56537553a500 msgr2=0x56537553a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.942+0000 7fd086a00700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56537553a500 0x56537553a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.942+0000 7fd086a00700 1 -- 172.21.15.12:0/2538676291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56537559c6a0 con 0x56537551ed30 2024-05-27T07:43:09.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.942+0000 7fd08f400700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x56537552b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:09.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.944+0000 7fd086a00700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375539fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd078005ff0 tx=0x7fd078009710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b01a796c73d246a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:09.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.944+0000 7fd07d600700 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-05-27T07:43:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.944+0000 7fd07d600700 1 -- 172.21.15.12:0/2538676291 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07800b730 con 0x56537551ed30 2024-05-27T07:43:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.945+0000 7fd07d600700 1 -- 172.21.15.12:0/2538676291 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7fd07800b8d0 con 0x56537551ed30 2024-05-27T07:43:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.946+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd07800bdc0 con 0x56537551ed30 2024-05-27T07:43:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.946+0000 7fd07d600700 1 -- 172.21.15.12:0/2538676291 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7fd07800bad0 con 0x56537551ed30 2024-05-27T07:43:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.946+0000 7fd0a4d3db80 10 client.37163 fetch_fsmap learned FSMap version 124 2024-05-27T07:43:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.946+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x56537559d900 con 0x56537551ed30 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd07d600700 1 -- 172.21.15.12:0/2538676291 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd07800bdc0 con 0x56537551ed30 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd07d600700 1 -- 172.21.15.12:0/2538676291 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7fd07800f460 con 0x56537551ed30 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd0a4d3db80 10 client.37163 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd0a4d3db80 10 client.37163 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd0a4d3db80 20 client.37163 populate_metadata read hostname 'smithi012' 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56537559dc40 0x5653755a0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd0a4d3db80 4 client.37163 mds_command: new command op to 34439 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:43:09.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.948+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 --> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565375442570 con 0x56537559dc40 2024-05-27T07:43:09.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.949+0000 7fd08f400700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56537559dc40 0x5653755a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:09.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.950+0000 7fd08f400700 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56537559dc40 0x5653755a0060 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3200719f2e45eb8c server_cookie=4dc6f0aa48a9e584 in_seq=0 out_seq=0 2024-05-27T07:43:09.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.950+0000 7fd07d600700 10 client.37163 ms_handle_connect on v2:172.21.15.62:6836/1993227402 2024-05-27T07:43:09.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.952+0000 7fd07d600700 1 -- 172.21.15.12:0/2538676291 <== mds.1 v2:172.21.15.62:6836/1993227402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565375442570 con 0x56537559dc40 2024-05-27T07:43:09.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.952+0000 7fd07d600700 10 client.37163 handle_command_reply: tid=0 2024-05-27T07:43:09.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.952+0000 7fd0a4d3db80 1 client.37163 shutdown 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 msgr2=0x565375539fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375539fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd078005ff0 tx=0x7fd078009710 comp rx=0 tx=0).stop 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 shutdown_connections 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:6836/1993227402,v1:172.21.15.62:6837/1993227402] conn(0x56537559dc40 0x5653755a0060 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56537553a500 0x56537553a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565375399e40 0x56537552b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 --2- 172.21.15.12:0/2538676291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56537551ed30 0x565375539fc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 >> 172.21.15.12:0/2538676291 conn(0x5653753f9fc0 msgr2=0x565375521410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.954+0000 7fd07d600700 0 client.37163 ms_handle_reset on v2:172.21.15.62:6836/1993227402 2024-05-27T07:43:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.955+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 shutdown_connections 2024-05-27T07:43:09.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.956+0000 7fd0a4d3db80 1 -- 172.21.15.12:0/2538676291 wait complete. 2024-05-27T07:43:09.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.956+0000 7fd0a4d3db80 20 client.37163 trim_cache size 0 max 16384 2024-05-27T07:43:09.955 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:43:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.957+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd0740b2b40 msgr2=0x7fd0740b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.957+0000 7fd096200700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd0740b2b40 0x7fd0740b5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd09010ed00 tx=0x7fd08000b040 comp rx=0 tx=0).stop 2024-05-27T07:43:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.957+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090109860 msgr2=0x7fd09010dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:09.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.957+0000 7fd096200700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090109860 0x7fd09010dbc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c970 tx=0x7fd08800cc80 comp rx=0 tx=0).stop 2024-05-27T07:43:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 shutdown_connections 2024-05-27T07:43:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fd0740b2b40 0x7fd0740b5050 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd090109860 0x7fd09010dbc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0901045e0 0x7fd0901108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 --2- 172.21.15.12:0/1851375974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd090108f10 0x7fd090110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 >> 172.21.15.12:0/1851375974 conn(0x7fd090100030 msgr2=0x7fd0901063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:09.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 shutdown_connections 2024-05-27T07:43:09.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:09.958+0000 7fd096200700 1 -- 172.21.15.12:0/1851375974 wait complete. 2024-05-27T07:43:10.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:10 smithi012 ceph-mon[31473]: pgmap v704: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:10.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:10 smithi162 ceph-mon[35676]: pgmap v704: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:10.493 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:10 smithi062 ceph-mon[35304]: pgmap v704: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:11.687 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:43:12.152 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:12.483 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:12 smithi162 ceph-mon[35676]: pgmap v705: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:12.487 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:12 smithi012 ceph-mon[31473]: pgmap v705: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:12.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:12 smithi062 ceph-mon[35304]: pgmap v705: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:14.349 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:14 smithi012 ceph-mon[31473]: pgmap v706: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:14.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:14 smithi162 ceph-mon[35676]: pgmap v706: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:14.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:14 smithi062 ceph-mon[35304]: pgmap v706: 129 pgs: 13 active+clean+snaptrim, 84 active+clean, 32 active+clean+snaptrim_wait; 539 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 20 op/s 2024-05-27T07:43:15.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 -- 172.21.15.12:0/1486415580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 msgr2=0x7f09c0110dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 --2- 172.21.15.12:0/1486415580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0110dd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f09b800b0a0 tx=0x7f09b800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 -- 172.21.15.12:0/1486415580 shutdown_connections 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 --2- 172.21.15.12:0/1486415580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0110dd0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 --2- 172.21.15.12:0/1486415580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c0105450 0x7f09c0105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 --2- 172.21.15.12:0/1486415580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0105da0 0x7f09c010e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 -- 172.21.15.12:0/1486415580 >> 172.21.15.12:0/1486415580 conn(0x7f09c00fede0 msgr2=0x7f09c0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.006+0000 7f09c6000700 1 -- 172.21.15.12:0/1486415580 shutdown_connections 2024-05-27T07:43:15.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.007+0000 7f09c6000700 1 -- 172.21.15.12:0/1486415580 wait complete. 2024-05-27T07:43:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.008+0000 7f09c6000700 1 Processor -- start 2024-05-27T07:43:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.008+0000 7f09c6000700 1 -- start start 2024-05-27T07:43:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.008+0000 7f09c6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0105450 0x7f09c0103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.008+0000 7f09c6000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c0105da0 0x7f09c0103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.008+0000 7f09c6000700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09c6000700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0114370 con 0x7f09c010e8c0 2024-05-27T07:43:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09c6000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01141f0 con 0x7f09c0105450 2024-05-27T07:43:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09c6000700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01144f0 con 0x7f09c0105da0 2024-05-27T07:43:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09bfe00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0103bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36704/0 (socket says 172.21.15.12:36704) 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09bfe00700 1 -- 172.21.15.12:0/1520386770 learned_addr learned my addr 172.21.15.12:0/1520386770 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09bf400700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0105450 0x7f09c0103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.009+0000 7f09b6a00700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c0105da0 0x7f09c0103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bfe00700 1 -- 172.21.15.12:0/1520386770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c0105da0 msgr2=0x7f09c0103680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bfe00700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c0105da0 0x7f09c0103680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bfe00700 1 -- 172.21.15.12:0/1520386770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0105450 msgr2=0x7f09c0103140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bfe00700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0105450 0x7f09c0103140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bfe00700 1 -- 172.21.15.12:0/1520386770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c01a9020 con 0x7f09c010e8c0 2024-05-27T07:43:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bf400700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0105450 0x7f09c0103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:43:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bfe00700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0103bc0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f09b800b070 tx=0x7f09b8004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.010+0000 7f09bcc00700 1 -- 172.21.15.12:0/1520386770 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8021070 con 0x7f09c010e8c0 2024-05-27T07:43:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.011+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c01a92b0 con 0x7f09c010e8c0 2024-05-27T07:43:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.011+0000 7f09bcc00700 1 -- 172.21.15.12:0/1520386770 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09b8007c00 con 0x7f09c010e8c0 2024-05-27T07:43:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.011+0000 7f09bcc00700 1 -- 172.21.15.12:0/1520386770 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8004850 con 0x7f09c010e8c0 2024-05-27T07:43:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.011+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c01a9700 con 0x7f09c010e8c0 2024-05-27T07:43:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.015+0000 7f09bcc00700 1 -- 172.21.15.12:0/1520386770 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f09b8003fb0 con 0x7f09c010e8c0 2024-05-27T07:43:15.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.016+0000 7f09d4ab0b80 1 Processor -- start 2024-05-27T07:43:15.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.016+0000 7f09d4ab0b80 1 -- start start 2024-05-27T07:43:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e96330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e96870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574ea3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564574dc4e40 con 0x564574d17e40 2024-05-27T07:43:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564574db49e0 con 0x564574e039f0 2024-05-27T07:43:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564574db8a20 con 0x564574e96e70 2024-05-27T07:43:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09bcc00700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b29f0 0x7f09a40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e96870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.017+0000 7f09bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e96870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36708/0 (socket says 172.21.15.12:36708) 2024-05-27T07:43:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09bf400700 1 -- 172.21.15.12:0/3311602248 learned_addr learned my addr 172.21.15.12:0/3311602248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e96330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574ea3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09bcc00700 1 -- 172.21.15.12:0/1520386770 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7f09b800f020 con 0x7f09c010e8c0 2024-05-27T07:43:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09bfe00700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b29f0 0x7f09a40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09b6a00700 1 -- 172.21.15.12:0/3311602248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 msgr2=0x564574ea3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574ea3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09b6a00700 1 -- 172.21.15.12:0/3311602248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 msgr2=0x564574e96870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e96870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.018+0000 7f09b6a00700 1 -- 172.21.15.12:0/3311602248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564574d7d420 con 0x564574e039f0 2024-05-27T07:43:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.019+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e96330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f09c0076b30 tx=0x7f09a8009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf51d641ce4f1a3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.019+0000 7f09bfe00700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b29f0 0x7f09a40b4f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09b80e1310 tx=0x7f09b80075c0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.019+0000 7f09b4c00700 1 -- 172.21.15.12:0/3311602248 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8012cb0 con 0x564574e039f0 2024-05-27T07:43:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.019+0000 7f09b4c00700 1 -- 172.21.15.12:0/3311602248 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09a8012e50 con 0x564574e039f0 2024-05-27T07:43:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.019+0000 7f09b4c00700 1 -- 172.21.15.12:0/3311602248 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a801e740 con 0x564574e039f0 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.020+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3311602248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 msgr2=0x564574e96330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.020+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e96330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f09c0076b30 tx=0x7f09a8009670 comp rx=0 tx=0).stop 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3311602248 shutdown_connections 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e96330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574ea3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3311602248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e96870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3311602248 >> 172.21.15.12:0/3311602248 conn(0x564574db1390 msgr2=0x564574e924c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3311602248 shutdown_connections 2024-05-27T07:43:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3311602248 wait complete. 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.021+0000 7f09d4ab0b80 1 -- start start 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 1 Processor -- start 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574d17e40 0x564574ea0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574ea0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e96e70 0x564574ea1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564574cbafa0 con 0x564574e039f0 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564574d55760 con 0x564574e96e70 2024-05-27T07:43:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564574ea3b80 con 0x564574d17e40 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.022+0000 7f09d4ab0b80 10 client.0 _ll_register_callbacks cb 0x564574d5ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574ea0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574ea0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36722/0 (socket says 172.21.15.12:36722) 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 -- 172.21.15.12:0/3252519600 learned_addr learned my addr 172.21.15.12:0/3252519600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e96e70 0x564574ea1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574d17e40 0x564574ea0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 -- 172.21.15.12:0/3252519600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574d17e40 msgr2=0x564574ea0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574d17e40 0x564574ea0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 -- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e96e70 msgr2=0x564574ea1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e96e70 0x564574ea1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09bf400700 1 -- 172.21.15.12:0/3252519600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564574d048d0 con 0x564574e039f0 2024-05-27T07:43:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.023+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574d17e40 0x564574ea0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09bf400700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574ea0d40 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f09b00124d0 tx=0x7f09b0012890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff114675923ae7e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09ad600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09ad600700 1 -- 172.21.15.12:0/3252519600 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b00107e0 con 0x564574e039f0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a8012e50 con 0x564574e039f0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09ad600700 1 -- 172.21.15.12:0/3252519600 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7f09b0010980 con 0x564574e039f0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09ad600700 1 -- 172.21.15.12:0/3252519600 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f09b0027050 con 0x564574e039f0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09d4ab0b80 10 client.17874 fetch_fsmap learned FSMap version 124 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.024+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x564574d85060 con 0x564574e039f0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09ad600700 1 -- 172.21.15.12:0/3252519600 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09a8012e50 con 0x564574e039f0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09ad600700 1 -- 172.21.15.12:0/3252519600 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7f09b001cc50 con 0x564574e039f0 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09d4ab0b80 10 client.17874 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09d4ab0b80 10 client.17874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09d4ab0b80 20 client.17874 populate_metadata read hostname 'smithi012' 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574eb0fd0 0x564574eb3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09d4ab0b80 4 client.17874 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:43:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.025+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564574dc6d90 con 0x564574eb0fd0 2024-05-27T07:43:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.026+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574eb0fd0 0x564574eb3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.026+0000 7f09b6a00700 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574eb0fd0 0x564574eb3420 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1be35d6168ac4f5 server_cookie=834174996f985167 in_seq=0 out_seq=0 2024-05-27T07:43:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.026+0000 7f09ad600700 10 client.17874 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:43:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.028+0000 7f09ad600700 1 -- 172.21.15.12:0/3252519600 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564574dc6d90 con 0x564574eb0fd0 2024-05-27T07:43:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.028+0000 7f09ad600700 10 client.17874 handle_command_reply: tid=0 2024-05-27T07:43:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.028+0000 7f09d4ab0b80 1 client.17874 shutdown 2024-05-27T07:43:15.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 msgr2=0x564574ea0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574ea0d40 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f09b00124d0 tx=0x7f09b0012890 comp rx=0 tx=0).stop 2024-05-27T07:43:15.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 shutdown_connections 2024-05-27T07:43:15.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574eb0fd0 0x564574eb3420 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e96e70 0x564574ea1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574d17e40 0x564574ea0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/3252519600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574ea0d40 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 >> 172.21.15.12:0/3252519600 conn(0x564574db1390 msgr2=0x564574e96410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09ad600700 0 client.17874 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:43:15.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 shutdown_connections 2024-05-27T07:43:15.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/3252519600 wait complete. 2024-05-27T07:43:15.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.029+0000 7f09d4ab0b80 20 client.17874 trim_cache size 0 max 16384 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.047+0000 7f09d4ab0b80 1 Processor -- start 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09d4ab0b80 1 -- start start 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e9f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e968a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564574e90e50 con 0x564574d17e40 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01b1eb0 con 0x564574e039f0 2024-05-27T07:43:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01b1050 con 0x564574e96e70 2024-05-27T07:43:15.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.048+0000 7f09b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e9f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e9f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36738/0 (socket says 172.21.15.12:36738) 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 -- 172.21.15.12:0/2840330512 learned_addr learned my addr 172.21.15.12:0/2840330512 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09bf400700 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 -- 172.21.15.12:0/2840330512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 msgr2=0x564574e9bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 -- 172.21.15.12:0/2840330512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 msgr2=0x564574e968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 -- 172.21.15.12:0/2840330512 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564574eb39f0 con 0x564574d17e40 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.049+0000 7f09b6a00700 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e9f760 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f09c0076b30 tx=0x7f09a8014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fbfb708284aafa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.050+0000 7f09afe00700 1 -- 172.21.15.12:0/2840330512 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a802c070 con 0x564574d17e40 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.050+0000 7f09afe00700 1 -- 172.21.15.12:0/2840330512 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09a8012e50 con 0x564574d17e40 2024-05-27T07:43:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.050+0000 7f09afe00700 1 -- 172.21.15.12:0/2840330512 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a800f7a0 con 0x564574d17e40 2024-05-27T07:43:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/2840330512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 msgr2=0x564574e9f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e9f760 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f09c0076b30 tx=0x7f09a8014040 comp rx=0 tx=0).stop 2024-05-27T07:43:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/2840330512 shutdown_connections 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574d17e40 0x564574e9f760 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/2840330512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574e039f0 0x564574e968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/2840330512 >> 172.21.15.12:0/2840330512 conn(0x564574db1390 msgr2=0x564574e96190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/2840330512 shutdown_connections 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/2840330512 wait complete. 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.051+0000 7f09d4ab0b80 1 -- start start 2024-05-27T07:43:15.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 1 Processor -- start 2024-05-27T07:43:15.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574d17e40 0x564574e9bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574e9a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564574e90e50 con 0x564574e039f0 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01b1780 con 0x564574d17e40 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564574e8e240 con 0x564574e96e70 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.052+0000 7f09d4ab0b80 10 client.0 _ll_register_callbacks cb 0x564574d5ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574e9a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574e9a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36740/0 (socket says 172.21.15.12:36740) 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 -- 172.21.15.12:0/700216807 learned_addr learned my addr 172.21.15.12:0/700216807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09b6a00700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574d17e40 0x564574e9bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09b6a00700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 -- 172.21.15.12:0/700216807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 msgr2=0x564574e9a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 -- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574d17e40 msgr2=0x564574e9bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574d17e40 0x564574e9bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 -- 172.21.15.12:0/700216807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564574e9af90 con 0x564574e039f0 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09b6a00700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09bf400700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574e9a270 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f09b000e530 tx=0x7f09b000e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e60f827c7069127 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.053+0000 7f09ad600700 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.054+0000 7f09ad600700 1 -- 172.21.15.12:0/700216807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0030030 con 0x564574e039f0 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.054+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a8012e50 con 0x564574e039f0 2024-05-27T07:43:15.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.054+0000 7f09ad600700 1 -- 172.21.15.12:0/700216807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7f09b000e9d0 con 0x564574e039f0 2024-05-27T07:43:15.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.054+0000 7f09ad600700 1 -- 172.21.15.12:0/700216807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f09b000ebd0 con 0x564574e039f0 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.054+0000 7f09d4ab0b80 10 client.17880 fetch_fsmap learned FSMap version 124 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.054+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=4+}) -- 0x564574ea8d80 con 0x564574e039f0 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09ad600700 1 -- 172.21.15.12:0/700216807 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09a8012e50 con 0x564574e039f0 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09ad600700 1 -- 172.21.15.12:0/700216807 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 124) ==== 2961+0+0 (secure 0 0 0) 0x7f09b000f580 con 0x564574e039f0 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09d4ab0b80 10 client.17880 fetch_fsmap finished waiting for FSMap version 124 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09d4ab0b80 10 client.17880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09d4ab0b80 20 client.17880 populate_metadata read hostname 'smithi012' 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574ea9540 0x564574f1db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09d4ab0b80 4 client.17880 mds_command: new command op to 34475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 --> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564574dc0570 con 0x564574ea9540 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.055+0000 7f09b6a00700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574ea9540 0x564574f1db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.056+0000 7f09b6a00700 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574ea9540 0x564574f1db00 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=666209a4c347a850 server_cookie=3b411987f0e2b964 in_seq=0 out_seq=0 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.056+0000 7f09ad600700 10 client.17880 ms_handle_connect on v2:172.21.15.62:6838/994488749 2024-05-27T07:43:15.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09ad600700 1 -- 172.21.15.12:0/700216807 <== mds.0 v2:172.21.15.62:6838/994488749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564574dc0570 con 0x564574ea9540 2024-05-27T07:43:15.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09ad600700 10 client.17880 handle_command_reply: tid=0 2024-05-27T07:43:15.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 client.17880 shutdown 2024-05-27T07:43:15.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 msgr2=0x564574e9a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574e9a270 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f09b000e530 tx=0x7f09b000e350 comp rx=0 tx=0).stop 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 shutdown_connections 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:6838/994488749,v1:172.21.15.62:6839/994488749] conn(0x564574ea9540 0x564574f1db00 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564574d17e40 0x564574e9bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564574e96e70 0x564574e9a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 --2- 172.21.15.12:0/700216807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564574e039f0 0x564574e9a270 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 >> 172.21.15.12:0/700216807 conn(0x564574db1390 msgr2=0x564574e97ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09ad600700 0 client.17880 ms_handle_reset on v2:172.21.15.62:6838/994488749 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 shutdown_connections 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 1 -- 172.21.15.12:0/700216807 wait complete. 2024-05-27T07:43:15.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.057+0000 7f09d4ab0b80 20 client.17880 trim_cache size 0 max 16384 2024-05-27T07:43:15.059 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-05-27T07:43:15.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b29f0 msgr2=0x7f09a40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b29f0 0x7f09a40b4f00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09b80e1310 tx=0x7f09b80075c0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 msgr2=0x7f09c0103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:15.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0103bc0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f09b800b070 tx=0x7f09b8004d30 comp rx=0 tx=0).stop 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 shutdown_connections 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f09a40b29f0 0x7f09a40b4f00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09c0105450 0x7f09c0103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c010e8c0 0x7f09c0103bc0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 --2- 172.21.15.12:0/1520386770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09c0105da0 0x7f09c0103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 >> 172.21.15.12:0/1520386770 conn(0x7f09c00fede0 msgr2=0x7f09c0100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.058+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 shutdown_connections 2024-05-27T07:43:15.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:15.059+0000 7f09c6000700 1 -- 172.21.15.12:0/1520386770 wait complete. 2024-05-27T07:43:16.279 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-05-27T07:43:16.280 INFO:tasks.fwd_scrub:done joining 2024-05-27T07:43:16.280 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-27T07:43:16.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T07:43:16.376 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:16 smithi012 ceph-mon[31473]: pgmap v707: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s wr, 0 op/s 2024-05-27T07:43:16.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:16 smithi162 ceph-mon[35676]: pgmap v707: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s wr, 0 op/s 2024-05-27T07:43:16.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:16 smithi062 ceph-mon[35304]: pgmap v707: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s wr, 0 op/s 2024-05-27T07:43:16.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 -- 172.21.15.12:0/3084239191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 msgr2=0x7f85ac105830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:16.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 --2- 172.21.15.12:0/3084239191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac105830 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f859c009900 tx=0x7f859c02a410 comp rx=0 tx=0).stop 2024-05-27T07:43:16.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 -- 172.21.15.12:0/3084239191 shutdown_connections 2024-05-27T07:43:16.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 --2- 172.21.15.12:0/3084239191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac1063e0 0x7f85ac1127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 --2- 172.21.15.12:0/3084239191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85ac10c130 0x7f85ac105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 --2- 172.21.15.12:0/3084239191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac105830 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 -- 172.21.15.12:0/3084239191 >> 172.21.15.12:0/3084239191 conn(0x7f85ac1013d0 msgr2=0x7f85ac1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:16.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.594+0000 7f85b0a00640 1 -- 172.21.15.12:0/3084239191 shutdown_connections 2024-05-27T07:43:16.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.595+0000 7f85b0a00640 1 -- 172.21.15.12:0/3084239191 wait complete. 2024-05-27T07:43:16.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.596+0000 7f85b0a00640 1 Processor -- start 2024-05-27T07:43:16.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.596+0000 7f85b0a00640 1 -- start start 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.596+0000 7f85b0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac1063e0 0x7f85ac19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85b0a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85b0a00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85ac10c130 0x7f85ac1a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85b0a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac115d80 con 0x7f85ac10bc90 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85b0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac115c00 con 0x7f85ac1063e0 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85b0a00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac115f00 con 0x7f85ac10c130 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85a2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85a2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36754/0 (socket says 172.21.15.12:36754) 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85a2a00640 1 -- 172.21.15.12:0/709625410 learned_addr learned my addr 172.21.15.12:0/709625410 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:16.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85ab400640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac1063e0 0x7f85ac19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85abe00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85ac10c130 0x7f85ac1a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85a2a00640 1 -- 172.21.15.12:0/709625410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85ac10c130 msgr2=0x7f85ac1a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:16.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85a2a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85ac10c130 0x7f85ac1a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85a2a00640 1 -- 172.21.15.12:0/709625410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac1063e0 msgr2=0x7f85ac19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.597+0000 7f85a2a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac1063e0 0x7f85ac19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.598+0000 7f85a2a00640 1 -- 172.21.15.12:0/709625410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac1a1640 con 0x7f85ac10bc90 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.598+0000 7f85a2a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac19cd40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8594005d00 tx=0x7f859400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.598+0000 7f85a8c00640 1 -- 172.21.15.12:0/709625410 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594017070 con 0x7f85ac10bc90 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.598+0000 7f85a8c00640 1 -- 172.21.15.12:0/709625410 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8594009c30 con 0x7f85ac10bc90 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.598+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac1a1930 con 0x7f85ac10bc90 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.598+0000 7f85a8c00640 1 -- 172.21.15.12:0/709625410 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859400f620 con 0x7f85ac10bc90 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.598+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac1a1e30 con 0x7f85ac10bc90 2024-05-27T07:43:16.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.600+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ac06a150 con 0x7f85ac10bc90 2024-05-27T07:43:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.604+0000 7f85a8c00640 1 -- 172.21.15.12:0/709625410 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f859401a070 con 0x7f85ac10bc90 2024-05-27T07:43:16.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.606+0000 7f85a8c00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f85900b2950 0x7f85900b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:16.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.607+0000 7f85a8c00640 1 -- 172.21.15.12:0/709625410 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7f85940e2550 con 0x7f85ac10bc90 2024-05-27T07:43:16.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.607+0000 7f85ab400640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f85900b2950 0x7f85900b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:16.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.608+0000 7f85ab400640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f85900b2950 0x7f85900b4e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f859c0057c0 tx=0x7f859c000bc0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.609+0000 7f85a8c00640 1 -- 172.21.15.12:0/709625410 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f85ac1a1e30 con 0x7f85ac10bc90 2024-05-27T07:43:16.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.823+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85ac1a9400 con 0x7f85ac10bc90 2024-05-27T07:43:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.828+0000 7f85a8c00640 1 -- 172.21.15.12:0/709625410 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) ==== 74+0+25507 (secure 0 0 0) 0x7f8594009de0 con 0x7f85ac10bc90 2024-05-27T07:43:16.827 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:43:16.827 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":158,"fsid":"55c251ba-1bf7-11ef-bc9b-c7b262605968","created":"2024-05-27T07:05:24.014715+0000","modified":"2024-05-27T07:43:06.053886+0000","last_up_change":"2024-05-27T07:15:45.450720+0000","last_in_change":"2024-05-27T07:15:18.901141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:10:07.718700+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:17:01.212613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:17:05.645973+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":37,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:17:19.864490+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":37,"snap_epoch":158,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e5c0c75c-b9be-4c85-8ad2-8c4555dfce22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6803","nonce":667573268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6805","nonce":667573268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6809","nonce":667573268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":667573268},{"type":"v1","addr":"172.21.15.12:6807","nonce":667573268}]},"public_addr":"172.21.15.12:6803/667573268","cluster_addr":"172.21.15.12:6805/667573268","heartbeat_back_addr":"172.21.15.12:6809/667573268","heartbeat_front_addr":"172.21.15.12:6807/667573268","state":["exists","up"]},{"osd":1,"uuid":"afd3e256-9590-46d2-b846-c3b3539d8de5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6801","nonce":1268848674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6803","nonce":1268848674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6807","nonce":1268848674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1268848674},{"type":"v1","addr":"172.21.15.62:6805","nonce":1268848674}]},"public_addr":"172.21.15.62:6801/1268848674","cluster_addr":"172.21.15.62:6803/1268848674","heartbeat_back_addr":"172.21.15.62:6807/1268848674","heartbeat_front_addr":"172.21.15.62:6805/1268848674","state":["exists","up"]},{"osd":2,"uuid":"3814d7d7-2bd7-4e53-958f-3fa857d074dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6801","nonce":2540739935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6803","nonce":2540739935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6807","nonce":2540739935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2540739935},{"type":"v1","addr":"172.21.15.162:6805","nonce":2540739935}]},"public_addr":"172.21.15.162:6801/2540739935","cluster_addr":"172.21.15.162:6803/2540739935","heartbeat_back_addr":"172.21.15.162:6807/2540739935","heartbeat_front_addr":"172.21.15.162:6805/2540739935","state":["exists","up"]},{"osd":3,"uuid":"e6c89349-9353-4f0c-9a8d-07e7305be298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6811","nonce":3716699952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6813","nonce":3716699952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6817","nonce":3716699952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3716699952},{"type":"v1","addr":"172.21.15.12:6815","nonce":3716699952}]},"public_addr":"172.21.15.12:6811/3716699952","cluster_addr":"172.21.15.12:6813/3716699952","heartbeat_back_addr":"172.21.15.12:6817/3716699952","heartbeat_front_addr":"172.21.15.12:6815/3716699952","state":["exists","up"]},{"osd":4,"uuid":"50344032-dab9-44e9-a549-f57d219d7959","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6809","nonce":2898003339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6811","nonce":2898003339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6815","nonce":2898003339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2898003339},{"type":"v1","addr":"172.21.15.62:6813","nonce":2898003339}]},"public_addr":"172.21.15.62:6809/2898003339","cluster_addr":"172.21.15.62:6811/2898003339","heartbeat_back_addr":"172.21.15.62:6815/2898003339","heartbeat_front_addr":"172.21.15.62:6813/2898003339","state":["exists","up"]},{"osd":5,"uuid":"c7e9391c-de97-406f-aa88-e12252972a6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6809","nonce":1419812101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6811","nonce":1419812101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6815","nonce":1419812101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1419812101},{"type":"v1","addr":"172.21.15.162:6813","nonce":1419812101}]},"public_addr":"172.21.15.162:6809/1419812101","cluster_addr":"172.21.15.162:6811/1419812101","heartbeat_back_addr":"172.21.15.162:6815/1419812101","heartbeat_front_addr":"172.21.15.162:6813/1419812101","state":["exists","up"]},{"osd":6,"uuid":"b9511154-fa9d-432a-b0be-c67dacf0d0f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6819","nonce":3556220791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6821","nonce":3556220791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6825","nonce":3556220791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":3556220791},{"type":"v1","addr":"172.21.15.12:6823","nonce":3556220791}]},"public_addr":"172.21.15.12:6819/3556220791","cluster_addr":"172.21.15.12:6821/3556220791","heartbeat_back_addr":"172.21.15.12:6825/3556220791","heartbeat_front_addr":"172.21.15.12:6823/3556220791","state":["exists","up"]},{"osd":7,"uuid":"ff91153a-4b0f-4275-96a2-6bde3d6f192c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6817","nonce":3439452424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6819","nonce":3439452424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6823","nonce":3439452424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3439452424},{"type":"v1","addr":"172.21.15.62:6821","nonce":3439452424}]},"public_addr":"172.21.15.62:6817/3439452424","cluster_addr":"172.21.15.62:6819/3439452424","heartbeat_back_addr":"172.21.15.62:6823/3439452424","heartbeat_front_addr":"172.21.15.62:6821/3439452424","state":["exists","up"]},{"osd":8,"uuid":"0775d1a0-a6b0-4a31-9f58-7b310ad4eeba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6817","nonce":1474732705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6819","nonce":1474732705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6823","nonce":1474732705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1474732705},{"type":"v1","addr":"172.21.15.162:6821","nonce":1474732705}]},"public_addr":"172.21.15.162:6817/1474732705","cluster_addr":"172.21.15.162:6819/1474732705","heartbeat_back_addr":"172.21.15.162:6823/1474732705","heartbeat_front_addr":"172.21.15.162:6821/1474732705","state":["exists","up"]},{"osd":9,"uuid":"c7623211-c464-44f4-b7bd-9bd337ae5c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6827","nonce":667220154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6829","nonce":667220154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6833","nonce":667220154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":667220154},{"type":"v1","addr":"172.21.15.12:6831","nonce":667220154}]},"public_addr":"172.21.15.12:6827/667220154","cluster_addr":"172.21.15.12:6829/667220154","heartbeat_back_addr":"172.21.15.12:6833/667220154","heartbeat_front_addr":"172.21.15.12:6831/667220154","state":["exists","up"]},{"osd":10,"uuid":"9449f46e-d354-4305-9f57-ff89f196c55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6825","nonce":3686530646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6827","nonce":3686530646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6831","nonce":3686530646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3686530646},{"type":"v1","addr":"172.21.15.62:6829","nonce":3686530646}]},"public_addr":"172.21.15.62:6825/3686530646","cluster_addr":"172.21.15.62:6827/3686530646","heartbeat_back_addr":"172.21.15.62:6831/3686530646","heartbeat_front_addr":"172.21.15.62:6829/3686530646","state":["exists","up"]},{"osd":11,"uuid":"ae11ac4c-de1a-4d93-9b88-a11e2f1a71b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6825","nonce":1037138310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6827","nonce":1037138310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6831","nonce":1037138310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1037138310},{"type":"v1","addr":"172.21.15.162:6829","nonce":1037138310}]},"public_addr":"172.21.15.162:6825/1037138310","cluster_addr":"172.21.15.162:6827/1037138310","heartbeat_back_addr":"172.21.15.162:6831/1037138310","heartbeat_front_addr":"172.21.15.162:6829/1037138310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:08:54.408795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:09:30.446852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:04.911951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:10:42.259999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:20.994004+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:11:56.855402+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:12:34.744601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:13.363750+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:13:49.944820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:14:27.335380+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:05.952923+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:15:42.911679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/435630590":"2024-05-28T07:22:22.282926+0000","172.21.15.12:6801/2856894494":"2024-05-28T07:22:22.282926+0000","172.21.15.12:6800/2856894494":"2024-05-28T07:22:22.282926+0000","172.21.15.12:0/4121651169":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6801/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/2115905110":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/725627344":"2024-05-28T07:22:22.282926+0000","172.21.15.12:0/1619685219":"2024-05-28T07:22:22.282926+0000","172.21.15.12:0/1199837030":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/949043977":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1631546627":"2024-05-28T07:06:45.507849+0000","172.21.15.12:0/3259206461":"2024-05-28T07:22:22.282926+0000","172.21.15.12:6801/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/1064527569":"2024-05-28T07:06:45.507849+0000","172.21.15.12:6800/579288494":"2024-05-28T07:06:09.346476+0000","172.21.15.12:0/3780513737":"2024-05-28T07:06:09.346476+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":24,"length":1},{"begin":37,"length":1}]},{"pool":4,"snaps":[{"begin":24,"length":1},{"begin":37,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":24,"length":1},{"begin":37,"length":1}]},{"pool":4,"snaps":[{"begin":24,"length":1},{"begin":37,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:43:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f85900b2950 msgr2=0x7f85900b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f85900b2950 0x7f85900b4e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f859c0057c0 tx=0x7f859c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 msgr2=0x7f85ac19cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac19cd40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8594005d00 tx=0x7f859400dde0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 shutdown_connections 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85ac10c130 0x7f85ac1a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f85ac10bc90 0x7f85ac19cd40 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f85900b2950 0x7f85900b4e60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 --2- 172.21.15.12:0/709625410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac1063e0 0x7f85ac19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 >> 172.21.15.12:0/709625410 conn(0x7f85ac1013d0 msgr2=0x7f85ac1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 shutdown_connections 2024-05-27T07:43:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:16.831+0000 7f85b0a00640 1 -- 172.21.15.12:0/709625410 wait complete. 2024-05-27T07:43:16.860 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:10:07.718700+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:17:01.212613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:17:05.645973+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 37, 'snap_epoch': 158, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:17:19.864490+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 37, 'snap_epoch': 158, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T07:43:16.860 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T07:43:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 -- 172.21.15.12:0/3815055840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 msgr2=0x7f91e41104e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3815055840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e41104e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009ca0 tx=0x7f91dc02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 -- 172.21.15.12:0/3815055840 shutdown_connections 2024-05-27T07:43:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3815055840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e41104e0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3815055840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4109ef0 0x7f91e4103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3815055840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4109a50 0x7f91e4103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 -- 172.21.15.12:0/3815055840 >> 172.21.15.12:0/3815055840 conn(0x7f91e40ff0e0 msgr2=0x7f91e4101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 -- 172.21.15.12:0/3815055840 shutdown_connections 2024-05-27T07:43:17.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.149+0000 7f91e3e00640 1 -- 172.21.15.12:0/3815055840 wait complete. 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.150+0000 7f91e3e00640 1 Processor -- start 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.150+0000 7f91e3e00640 1 -- start start 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4109a50 0x7f91e419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e3e00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4109ef0 0x7f91e41a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4113b90 con 0x7f91e41041b0 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4113a10 con 0x7f91e4109a50 2024-05-27T07:43:17.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e3e00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4113d10 con 0x7f91e4109ef0 2024-05-27T07:43:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e2a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36774/0 (socket says 172.21.15.12:36774) 2024-05-27T07:43:17.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.151+0000 7f91e2a00640 1 -- 172.21.15.12:0/3953868092 learned_addr learned my addr 172.21.15.12:0/3953868092 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2000640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4109a50 0x7f91e419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e3400640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4109ef0 0x7f91e41a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2a00640 1 -- 172.21.15.12:0/3953868092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4109ef0 msgr2=0x7f91e41a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2a00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4109ef0 0x7f91e41a0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2a00640 1 -- 172.21.15.12:0/3953868092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4109a50 msgr2=0x7f91e419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2a00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4109a50 0x7f91e419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2a00640 1 -- 172.21.15.12:0/3953868092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e41a1570 con 0x7f91e41041b0 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2000640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4109a50 0x7f91e419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e2a00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e419c710 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f91d400e9f0 tx=0x7f91d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91db400640 1 -- 172.21.15.12:0/3953868092 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d400ccc0 con 0x7f91e41041b0 2024-05-27T07:43:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91db400640 1 -- 172.21.15.12:0/3953868092 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91d40108c0 con 0x7f91e41041b0 2024-05-27T07:43:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91db400640 1 -- 172.21.15.12:0/3953868092 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4018750 con 0x7f91e41041b0 2024-05-27T07:43:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e41a1860 con 0x7f91e41041b0 2024-05-27T07:43:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.152+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e41a91e0 con 0x7f91e41041b0 2024-05-27T07:43:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.154+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e4069d00 con 0x7f91e41041b0 2024-05-27T07:43:17.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.158+0000 7f91db400640 1 -- 172.21.15.12:0/3953868092 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f91d4010430 con 0x7f91e41041b0 2024-05-27T07:43:17.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.160+0000 7f91db400640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91c80b2950 0x7f91c80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.160+0000 7f91db400640 1 -- 172.21.15.12:0/3953868092 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7f91d40e3440 con 0x7f91e41041b0 2024-05-27T07:43:17.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.160+0000 7f91e2000640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91c80b2950 0x7f91c80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.161+0000 7f91e2000640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91c80b2950 0x7f91c80b4e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f91e4075670 tx=0x7f91d0005e70 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:17.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.163+0000 7f91db400640 1 -- 172.21.15.12:0/3953868092 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f91d40aa960 con 0x7f91e41041b0 2024-05-27T07:43:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.375+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f91e41a93b0 con 0x7f91e41041b0 2024-05-27T07:43:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.376+0000 7f91db400640 1 -- 172.21.15.12:0/3953868092 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) ==== 93+0+10 (secure 0 0 0) 0x7f91d40aa260 con 0x7f91e41041b0 2024-05-27T07:43:17.375 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-05-27T07:43:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91c80b2950 msgr2=0x7f91c80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91c80b2950 0x7f91c80b4e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f91e4075670 tx=0x7f91d0005e70 comp rx=0 tx=0).stop 2024-05-27T07:43:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 msgr2=0x7f91e419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e419c710 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f91d400e9f0 tx=0x7f91d400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 shutdown_connections 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4109ef0 0x7f91e41a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f91c80b2950 0x7f91c80b4e60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4109a50 0x7f91e419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 --2- 172.21.15.12:0/3953868092 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91e41041b0 0x7f91e419c710 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 >> 172.21.15.12:0/3953868092 conn(0x7f91e40ff0e0 msgr2=0x7f91e4100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 shutdown_connections 2024-05-27T07:43:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.379+0000 7f91e3e00640 1 -- 172.21.15.12:0/3953868092 wait complete. 2024-05-27T07:43:17.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:17 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/709625410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:43:17.404 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T07:43:17.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:17 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/709625410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:43:17.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:17 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/709625410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:43:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 -- 172.21.15.12:0/3233653988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410f360 msgr2=0x7fbdb4116b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/3233653988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410f360 0x7fbdb4116b00 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009ca0 tx=0x7fbdac02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 -- 172.21.15.12:0/3233653988 shutdown_connections 2024-05-27T07:43:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/3233653988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410f360 0x7fbdb4116b00 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/3233653988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdb410e8a0 0x7fbdb410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/3233653988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb410a360 0x7fbdb410a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 -- 172.21.15.12:0/3233653988 >> 172.21.15.12:0/3233653988 conn(0x7fbdb4076000 msgr2=0x7fbdb4076410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:17.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 -- 172.21.15.12:0/3233653988 shutdown_connections 2024-05-27T07:43:17.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.696+0000 7fbdb3e00640 1 -- 172.21.15.12:0/3233653988 wait complete. 2024-05-27T07:43:17.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.697+0000 7fbdb3e00640 1 Processor -- start 2024-05-27T07:43:17.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.697+0000 7fbdb3e00640 1 -- start start 2024-05-27T07:43:17.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb410a360 0x7fbdb41a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb3e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410e8a0 0x7fbdb41a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb3e00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdb410f360 0x7fbdb41a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb3e00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb411a150 con 0x7fbdb410e8a0 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb4119fd0 con 0x7fbdb410a360 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb3e00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb411a2d0 con 0x7fbdb410f360 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb2000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410e8a0 0x7fbdb41a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb2000640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410e8a0 0x7fbdb41a6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36802/0 (socket says 172.21.15.12:36802) 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb2000640 1 -- 172.21.15.12:0/1019759405 learned_addr learned my addr 172.21.15.12:0/1019759405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb2a00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb410a360 0x7fbdb41a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.698+0000 7fbdb3400640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdb410f360 0x7fbdb41a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb2000640 1 -- 172.21.15.12:0/1019759405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdb410f360 msgr2=0x7fbdb41a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb2000640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdb410f360 0x7fbdb41a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb2000640 1 -- 172.21.15.12:0/1019759405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb410a360 msgr2=0x7fbdb41a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb2000640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb410a360 0x7fbdb41a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb2000640 1 -- 172.21.15.12:0/1019759405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb41a10b0 con 0x7fbdb410e8a0 2024-05-27T07:43:17.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb2000640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410e8a0 0x7fbdb41a6d40 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005fd0 tx=0x7fbda000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:17.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb8a00640 1 -- 172.21.15.12:0/1019759405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0017070 con 0x7fbdb410e8a0 2024-05-27T07:43:17.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb41a13a0 con 0x7fbdb410e8a0 2024-05-27T07:43:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb8a00640 1 -- 172.21.15.12:0/1019759405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbda0009c30 con 0x7fbdb410e8a0 2024-05-27T07:43:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.699+0000 7fbdb8a00640 1 -- 172.21.15.12:0/1019759405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0003c70 con 0x7fbdb410e8a0 2024-05-27T07:43:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.700+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb41ad680 con 0x7fbdb410e8a0 2024-05-27T07:43:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.701+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdb40692d0 con 0x7fbdb410e8a0 2024-05-27T07:43:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.707+0000 7fbdb8a00640 1 -- 172.21.15.12:0/1019759405 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbda0003e10 con 0x7fbdb410e8a0 2024-05-27T07:43:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.708+0000 7fbdb8a00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbd980b2950 0x7fbd980b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.708+0000 7fbdb8a00640 1 -- 172.21.15.12:0/1019759405 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9642+0+0 (secure 0 0 0) 0x7fbda00e2f80 con 0x7fbdb410e8a0 2024-05-27T07:43:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.709+0000 7fbdb2a00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbd980b2950 0x7fbd980b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.710+0000 7fbdb2a00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbd980b2950 0x7fbd980b4e60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbda4005fd0 tx=0x7fbda4005850 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.710+0000 7fbdb8a00640 1 -- 172.21.15.12:0/1019759405 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbda0020030 con 0x7fbdb410e8a0 2024-05-27T07:43:17.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.942+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbdb4112180 con 0x7fbdb410e8a0 2024-05-27T07:43:17.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.943+0000 7fbdb8a00640 1 -- 172.21.15.12:0/1019759405 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v159) ==== 104+0+11 (secure 0 0 0) 0x7fbda00aa4a0 con 0x7fbdb410e8a0 2024-05-27T07:43:17.942 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbd980b2950 msgr2=0x7fbd980b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbd980b2950 0x7fbd980b4e60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbda4005fd0 tx=0x7fbda4005850 comp rx=0 tx=0).stop 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410e8a0 msgr2=0x7fbdb41a6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410e8a0 0x7fbdb41a6d40 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005fd0 tx=0x7fbda000dde0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 shutdown_connections 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdb410f360 0x7fbdb41a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdb410e8a0 0x7fbdb41a6d40 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fbd980b2950 0x7fbd980b4e60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 --2- 172.21.15.12:0/1019759405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb410a360 0x7fbdb41a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 >> 172.21.15.12:0/1019759405 conn(0x7fbdb4076000 msgr2=0x7fbdb4107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:17.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 shutdown_connections 2024-05-27T07:43:17.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:17.946+0000 7fbdb3e00640 1 -- 172.21.15.12:0/1019759405 wait complete. 2024-05-27T07:43:17.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T07:43:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 -- 172.21.15.12:0/1000424749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010e870 msgr2=0x7f352010eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 --2- 172.21.15.12:0/1000424749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010e870 0x7f352010eca0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f3510009ca0 tx=0x7f351002a8b0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 -- 172.21.15.12:0/1000424749 shutdown_connections 2024-05-27T07:43:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 --2- 172.21.15.12:0/1000424749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f352010f330 0x7f3520116ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 --2- 172.21.15.12:0/1000424749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010e870 0x7f352010eca0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 --2- 172.21.15.12:0/1000424749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352010a2e0 0x7f352010a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 -- 172.21.15.12:0/1000424749 >> 172.21.15.12:0/1000424749 conn(0x7f3520076000 msgr2=0x7f3520076410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 -- 172.21.15.12:0/1000424749 shutdown_connections 2024-05-27T07:43:18.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.263+0000 7f3525800640 1 -- 172.21.15.12:0/1000424749 wait complete. 2024-05-27T07:43:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.264+0000 7f3525800640 1 Processor -- start 2024-05-27T07:43:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.264+0000 7f3525800640 1 -- start start 2024-05-27T07:43:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f3525800640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010a2e0 0x7f35201a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f3525800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352010e870 0x7f35201a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f3525800640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f352010f330 0x7f35201a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f3525800640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f352011a0c0 con 0x7f352010a2e0 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f3525800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3520119f40 con 0x7f352010e870 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f3525800640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f352011a240 con 0x7f352010f330 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f351fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010a2e0 0x7f35201a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f351fe00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010a2e0 0x7f35201a67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36822/0 (socket says 172.21.15.12:36822) 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f351fe00640 1 -- 172.21.15.12:0/1498854096 learned_addr learned my addr 172.21.15.12:0/1498854096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f3524e00640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f352010f330 0x7f35201a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f351f400640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352010e870 0x7f35201a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f351fe00640 1 -- 172.21.15.12:0/1498854096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f352010f330 msgr2=0x7f35201a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f351fe00640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f352010f330 0x7f35201a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.265+0000 7f351fe00640 1 -- 172.21.15.12:0/1498854096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352010e870 msgr2=0x7f35201a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f351fe00640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352010e870 0x7f35201a6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f351fe00640 1 -- 172.21.15.12:0/1498854096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35201a0ff0 con 0x7f352010a2e0 2024-05-27T07:43:18.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f351fe00640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010a2e0 0x7f35201a67f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f350800d950 tx=0x7f350800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:18.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f351cc00640 1 -- 172.21.15.12:0/1498854096 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3508017070 con 0x7f352010a2e0 2024-05-27T07:43:18.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35201a12e0 con 0x7f352010a2e0 2024-05-27T07:43:18.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f351cc00640 1 -- 172.21.15.12:0/1498854096 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3508009c30 con 0x7f352010a2e0 2024-05-27T07:43:18.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f351cc00640 1 -- 172.21.15.12:0/1498854096 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350800f620 con 0x7f352010a2e0 2024-05-27T07:43:18.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.266+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35201a1800 con 0x7f352010a2e0 2024-05-27T07:43:18.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.267+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3520069d00 con 0x7f352010a2e0 2024-05-27T07:43:18.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.270+0000 7f351cc00640 1 -- 172.21.15.12:0/1498854096 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3508005030 con 0x7f352010a2e0 2024-05-27T07:43:18.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.272+0000 7f351cc00640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35040b2950 0x7f35040b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.272+0000 7f351cc00640 1 -- 172.21.15.12:0/1498854096 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9554+0+0 (secure 0 0 0) 0x7f35080e2730 con 0x7f352010a2e0 2024-05-27T07:43:18.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.272+0000 7f351f400640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35040b2950 0x7f35040b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.273+0000 7f351f400640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35040b2950 0x7f35040b4e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3520109230 tx=0x7f3510030f00 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:18.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.275+0000 7f351cc00640 1 -- 172.21.15.12:0/1498854096 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f35080a9ca0 con 0x7f352010a2e0 2024-05-27T07:43:18.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:18 smithi012 ceph-mon[31473]: pgmap v708: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 749 B/s wr, 0 op/s 2024-05-27T07:43:18.348 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:18 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3953868092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:43:18.349 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:18 smithi012 ceph-mon[31473]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:43:18.349 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:18 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1019759405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:43:18.384 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:18 smithi162 ceph-mon[35676]: pgmap v708: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 749 B/s wr, 0 op/s 2024-05-27T07:43:18.384 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:18 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3953868092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:43:18.384 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:18 smithi162 ceph-mon[35676]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:43:18.384 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:18 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1019759405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:43:18.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.488+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f35201ad6d0 con 0x7f352010a2e0 2024-05-27T07:43:18.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.489+0000 7f351cc00640 1 -- 172.21.15.12:0/1498854096 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v159) ==== 100+0+11 (secure 0 0 0) 0x7f3508009de0 con 0x7f352010a2e0 2024-05-27T07:43:18.488 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35040b2950 msgr2=0x7f35040b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35040b2950 0x7f35040b4e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3520109230 tx=0x7f3510030f00 comp rx=0 tx=0).stop 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010a2e0 msgr2=0x7f35201a67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010a2e0 0x7f35201a67f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f350800d950 tx=0x7f350800de10 comp rx=0 tx=0).stop 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 shutdown_connections 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f352010f330 0x7f35201a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35040b2950 0x7f35040b4e60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352010e870 0x7f35201a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 --2- 172.21.15.12:0/1498854096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f352010a2e0 0x7f35201a67f0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 >> 172.21.15.12:0/1498854096 conn(0x7f3520076000 msgr2=0x7f3520107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:18.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 shutdown_connections 2024-05-27T07:43:18.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.493+0000 7f3525800640 1 -- 172.21.15.12:0/1498854096 wait complete. 2024-05-27T07:43:18.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:18 smithi062 ceph-mon[35304]: pgmap v708: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 749 B/s wr, 0 op/s 2024-05-27T07:43:18.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:18 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3953868092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:43:18.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:18 smithi062 ceph-mon[35304]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:43:18.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:18 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1019759405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:43:18.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T07:43:18.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 -- 172.21.15.12:0/3287051122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 msgr2=0x7f641810a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 --2- 172.21.15.12:0/3287051122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f641810a6f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f6408009900 tx=0x7f640802a410 comp rx=0 tx=0).stop 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 -- 172.21.15.12:0/3287051122 shutdown_connections 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 --2- 172.21.15.12:0/3287051122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f641810f330 0x7f6418116ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 --2- 172.21.15.12:0/3287051122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641810e870 0x7f641810eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 --2- 172.21.15.12:0/3287051122 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f641810a6f0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 -- 172.21.15.12:0/3287051122 >> 172.21.15.12:0/3287051122 conn(0x7f6418076c80 msgr2=0x7f64180770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.810+0000 7f6417200640 1 -- 172.21.15.12:0/3287051122 shutdown_connections 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.811+0000 7f6417200640 1 -- 172.21.15.12:0/3287051122 wait complete. 2024-05-27T07:43:18.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.812+0000 7f6417200640 1 Processor -- start 2024-05-27T07:43:18.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.812+0000 7f6417200640 1 -- start start 2024-05-27T07:43:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6417200640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f64181a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6417200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f641810e870 0x7f64181a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6417200640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641810f330 0x7f64181a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6417200640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f641811a160 con 0x7f641810a2e0 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6417200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6418119fe0 con 0x7f641810e870 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6417200640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f641811a2e0 con 0x7f641810f330 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f64181a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f64181a6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36832/0 (socket says 172.21.15.12:36832) 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415e00640 1 -- 172.21.15.12:0/1093074877 learned_addr learned my addr 172.21.15.12:0/1093074877 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415400640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f641810e870 0x7f64181a6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6416800640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641810f330 0x7f64181a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415e00640 1 -- 172.21.15.12:0/1093074877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641810f330 msgr2=0x7f64181a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415e00640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641810f330 0x7f64181a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415e00640 1 -- 172.21.15.12:0/1093074877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f641810e870 msgr2=0x7f64181a6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.813+0000 7f6415e00640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f641810e870 0x7f64181a6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:18.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f6415e00640 1 -- 172.21.15.12:0/1093074877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64181a10c0 con 0x7f641810a2e0 2024-05-27T07:43:18.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f6415400640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f641810e870 0x7f64181a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:18.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f6416800640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641810f330 0x7f64181a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:43:18.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f6415e00640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f64181a6810 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f640802a930 tx=0x7f6408003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:18.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f640ea00640 1 -- 172.21.15.12:0/1093074877 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640803f070 con 0x7f641810a2e0 2024-05-27T07:43:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f640ea00640 1 -- 172.21.15.12:0/1093074877 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6408003e40 con 0x7f641810a2e0 2024-05-27T07:43:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f640ea00640 1 -- 172.21.15.12:0/1093074877 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408035030 con 0x7f641810a2e0 2024-05-27T07:43:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64181a1350 con 0x7f641810a2e0 2024-05-27T07:43:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.814+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64181a17f0 con 0x7f641810a2e0 2024-05-27T07:43:18.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.816+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641806a150 con 0x7f641810a2e0 2024-05-27T07:43:18.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.818+0000 7f640ea00640 1 -- 172.21.15.12:0/1093074877 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6408042070 con 0x7f641810a2e0 2024-05-27T07:43:18.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.820+0000 7f640ea00640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f63fc0b2950 0x7f63fc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:18.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.820+0000 7f640ea00640 1 -- 172.21.15.12:0/1093074877 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9554+0+0 (secure 0 0 0) 0x7f6408103780 con 0x7f641810a2e0 2024-05-27T07:43:18.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.821+0000 7f6415400640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f63fc0b2950 0x7f63fc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:18.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.821+0000 7f6415400640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f63fc0b2950 0x7f63fc0b4e60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6418109940 tx=0x7f6404005e50 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:18.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:18.823+0000 7f640ea00640 1 -- 172.21.15.12:0/1093074877 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f64080cacf0 con 0x7f641810a2e0 2024-05-27T07:43:19.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.036+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f64181ad7b0 con 0x7f641810a2e0 2024-05-27T07:43:19.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.037+0000 7f640ea00640 1 -- 172.21.15.12:0/1093074877 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v159) ==== 103+0+11 (secure 0 0 0) 0x7f64080ca5f0 con 0x7f641810a2e0 2024-05-27T07:43:19.036 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-05-27T07:43:19.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f63fc0b2950 msgr2=0x7f63fc0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f63fc0b2950 0x7f63fc0b4e60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6418109940 tx=0x7f6404005e50 comp rx=0 tx=0).stop 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 msgr2=0x7f64181a6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f64181a6810 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f640802a930 tx=0x7f6408003c20 comp rx=0 tx=0).stop 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 shutdown_connections 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641810f330 0x7f64181a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f63fc0b2950 0x7f63fc0b4e60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f641810e870 0x7f64181a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 --2- 172.21.15.12:0/1093074877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f641810a2e0 0x7f64181a6810 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 >> 172.21.15.12:0/1093074877 conn(0x7f6418076c80 msgr2=0x7f6418107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 shutdown_connections 2024-05-27T07:43:19.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.040+0000 7f6417200640 1 -- 172.21.15.12:0/1093074877 wait complete. 2024-05-27T07:43:19.065 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-27T07:43:19.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.346+0000 7f1655a00640 1 -- 172.21.15.12:0/3991329213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16501063c0 msgr2=0x7f1650112790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:19.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.346+0000 7f1655a00640 1 --2- 172.21.15.12:0/3991329213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16501063c0 0x7f1650112790 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f1648009ca0 tx=0x7f164802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.346+0000 7f1655a00640 1 -- 172.21.15.12:0/3991329213 shutdown_connections 2024-05-27T07:43:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.346+0000 7f1655a00640 1 --2- 172.21.15.12:0/3991329213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16501063c0 0x7f1650112790 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.346+0000 7f1655a00640 1 --2- 172.21.15.12:0/3991329213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f165010c110 0x7f1650105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.346+0000 7f1655a00640 1 --2- 172.21.15.12:0/3991329213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f165010bc70 0x7f1650105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.346+0000 7f1655a00640 1 -- 172.21.15.12:0/3991329213 >> 172.21.15.12:0/3991329213 conn(0x7f165007d590 msgr2=0x7f165007d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.347+0000 7f1655a00640 1 -- 172.21.15.12:0/3991329213 shutdown_connections 2024-05-27T07:43:19.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.347+0000 7f1655a00640 1 -- 172.21.15.12:0/3991329213 wait complete. 2024-05-27T07:43:19.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.348+0000 7f1655a00640 1 Processor -- start 2024-05-27T07:43:19.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.348+0000 7f1655a00640 1 -- start start 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1655a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16501063c0 0x7f165019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1655a00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f165010bc70 0x7f165019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1655a00640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f165010c110 0x7f16501a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1655a00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1650115d60 con 0x7f165010bc70 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1655a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1650115be0 con 0x7f16501063c0 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1655a00640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1650115ee0 con 0x7f165010c110 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1647e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f165010bc70 0x7f165019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1647e00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f165010bc70 0x7f165019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36844/0 (socket says 172.21.15.12:36844) 2024-05-27T07:43:19.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f1647e00640 1 -- 172.21.15.12:0/1907441658 learned_addr learned my addr 172.21.15.12:0/1907441658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:19.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:19 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1498854096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:43:19.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:19 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1093074877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:43:19.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.349+0000 7f164fe00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16501063c0 0x7f165019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.350+0000 7f1655000640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f165010c110 0x7f16501a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.350+0000 7f1647e00640 1 -- 172.21.15.12:0/1907441658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f165010c110 msgr2=0x7f16501a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.350+0000 7f1647e00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f165010c110 0x7f16501a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.350+0000 7f1647e00640 1 -- 172.21.15.12:0/1907441658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16501063c0 msgr2=0x7f165019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.350+0000 7f1647e00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16501063c0 0x7f165019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.350+0000 7f1647e00640 1 -- 172.21.15.12:0/1907441658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16501a15a0 con 0x7f165010bc70 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.350+0000 7f1647e00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f165010bc70 0x7f165019cca0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f1638005fd0 tx=0x7f163800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.351+0000 7f164d600640 1 -- 172.21.15.12:0/1907441658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638017070 con 0x7f165010bc70 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.351+0000 7f164d600640 1 -- 172.21.15.12:0/1907441658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1638009c30 con 0x7f165010bc70 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.351+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16501a1890 con 0x7f165010bc70 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.351+0000 7f164d600640 1 -- 172.21.15.12:0/1907441658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638003c90 con 0x7f165010bc70 2024-05-27T07:43:19.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.351+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16501a9210 con 0x7f165010bc70 2024-05-27T07:43:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.352+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1650106f50 con 0x7f165010bc70 2024-05-27T07:43:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.356+0000 7f164d600640 1 -- 172.21.15.12:0/1907441658 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1638003e30 con 0x7f165010bc70 2024-05-27T07:43:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.358+0000 7f164d600640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f16340b2950 0x7f16340b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.358+0000 7f164d600640 1 -- 172.21.15.12:0/1907441658 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9554+0+0 (secure 0 0 0) 0x7f16380e2e50 con 0x7f165010bc70 2024-05-27T07:43:19.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.358+0000 7f164fe00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f16340b2950 0x7f16340b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:19.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.359+0000 7f164fe00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f16340b2950 0x7f16340b4e60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1650106950 tx=0x7f164000b390 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:19.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.360+0000 7f164d600640 1 -- 172.21.15.12:0/1907441658 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1638020030 con 0x7f165010bc70 2024-05-27T07:43:19.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:19 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1498854096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:43:19.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:19 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1093074877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:43:19.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:19 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1498854096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:43:19.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:19 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1093074877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:43:19.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.625+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f165010dd80 con 0x7f165010bc70 2024-05-27T07:43:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.661+0000 7f164d600640 1 -- 172.21.15.12:0/1907441658 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+816175 (secure 0 0 0) 0x7f16380aa3c0 con 0x7f165010bc70 2024-05-27T07:43:19.673 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 26, 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "active_gid": 24991, 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "active_name": "x", 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addrs": { 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6800", 2024-05-27T07:43:19.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3926981258 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6801", 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3926981258 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addr": "172.21.15.12:6801/3926981258", 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "active_change": "2024-05-27T07:22:22.283324+0000", 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "active_mgr_features": 4540701547738038271, 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "available": true, 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: "standbys": [ 2024-05-27T07:43:19.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 24988, 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "z", 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:19.676 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.677 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:43:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-05-27T07:43:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-05-27T07:43:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-05-27T07:43:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.692 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.693 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.694 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.695 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-05-27T07:43:19.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.697 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.698 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.699 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.700 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-05-27T07:43:19.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:43:19.703 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.704 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.705 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.706 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.707 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.708 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.709 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.710 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-05-27T07:43:19.711 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-05-27T07:43:19.712 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.713 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:19.714 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.717 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:19.718 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-05-27T07:43:19.719 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.720 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-05-27T07:43:19.721 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.722 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.723 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.724 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:19.725 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:19.726 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.727 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-05-27T07:43:19.728 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.729 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-05-27T07:43:19.730 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.731 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.733 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:43:19.734 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.735 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.736 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.737 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.738 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.739 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:43:19.740 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.741 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.742 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-05-27T07:43:19.743 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.744 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.745 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.746 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-05-27T07:43:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_samba": { 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_samba", 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-05-27T07:43:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.753 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.754 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.755 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-05-27T07:43:19.756 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.757 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:43:19.758 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.759 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "grafana_dashboards_path": { 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:43:19.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.761 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-05-27T07:43:19.762 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-05-27T07:43:19.763 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.764 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.765 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-05-27T07:43:19.766 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.768 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.769 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.771 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.772 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.773 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-05-27T07:43:19.774 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.775 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.776 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.777 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-05-27T07:43:19.778 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.779 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-05-27T07:43:19.780 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:43:19.781 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.782 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.783 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.784 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.785 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-05-27T07:43:19.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:43:19.788 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-05-27T07:43:19.789 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.790 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-05-27T07:43:19.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.793 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.794 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.795 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.796 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:43:19.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.799 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.800 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.801 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.802 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.803 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.804 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.805 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.806 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.807 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.809 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.811 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:43:19.812 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:43:19.813 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.814 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.815 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.816 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.817 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.818 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.819 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.820 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.821 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.822 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.823 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.824 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.825 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.826 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.827 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:43:19.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.829 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.830 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.831 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.832 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.833 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.834 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.835 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "[]", 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:43:19.836 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:43:19.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.838 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.840 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.841 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:43:19.842 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.845 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.846 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.847 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:43:19.848 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.849 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.850 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.851 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.852 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:43:19.853 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.854 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.855 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.856 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.857 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.858 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.860 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.861 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:43:19.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.863 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.864 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-05-27T07:43:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:43:19.868 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-05-27T07:43:19.869 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-05-27T07:43:19.870 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.872 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.873 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.874 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.875 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.876 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-05-27T07:43:19.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.878 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.879 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.880 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.881 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.883 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.885 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.886 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-05-27T07:43:19.887 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.888 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.889 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-05-27T07:43:19.890 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.892 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.893 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.894 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.895 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.896 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.897 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.898 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.899 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.900 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.901 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.902 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.903 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:43:19.904 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.905 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.906 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.907 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.908 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.909 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.910 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.911 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.912 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.913 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.914 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.915 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:19.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.918 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-05-27T07:43:19.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.920 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.921 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:43:19.922 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.923 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.924 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.925 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.926 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.927 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.928 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.929 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-05-27T07:43:19.931 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.932 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.933 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.934 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:19.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.936 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.937 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.939 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.940 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.941 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.942 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.944 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.945 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.946 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.947 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.948 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.950 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.951 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.952 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.953 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:43:19.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.955 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.957 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.958 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.959 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.960 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.961 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.962 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.964 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-05-27T07:43:19.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.967 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.968 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.969 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.970 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.972 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.973 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.974 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.975 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.976 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:43:19.977 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.978 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.979 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.981 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.982 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.983 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.984 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.985 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.986 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.987 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-05-27T07:43:19.988 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.990 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.993 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.995 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:19.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:19.998 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:19.999 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.000 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.002 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.003 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.004 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.005 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.006 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.007 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.008 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.009 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.010 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-05-27T07:43:20.011 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.014 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.015 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.016 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.017 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.018 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.019 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.020 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.021 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.022 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.023 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.024 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.025 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.027 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.029 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.030 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.031 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.032 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.033 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.034 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.035 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.036 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.037 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.038 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.038 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.038 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.038 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.038 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.038 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.041 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.042 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.044 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.046 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.047 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.048 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.049 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.051 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.052 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.053 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.054 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.056 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.057 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.058 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.059 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.060 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.062 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.063 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.064 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.065 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-05-27T07:43:20.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.067 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.068 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.069 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.070 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:20.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.072 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.073 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.074 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-05-27T07:43:20.075 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.077 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.078 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.079 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.081 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.082 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.083 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.084 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:43:20.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.086 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.087 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.088 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-05-27T07:43:20.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.090 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-05-27T07:43:20.091 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.093 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.094 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.095 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.097 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.098 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.099 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.100 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.101 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.102 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.104 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.106 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.107 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.108 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.109 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.110 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.111 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.113 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.114 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.115 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.116 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.117 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.118 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.119 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.120 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.121 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.122 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.124 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.125 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.126 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-05-27T07:43:20.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.130 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.131 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.132 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.133 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.134 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-05-27T07:43:20.135 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.136 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-05-27T07:43:20.137 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.138 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.139 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.140 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.141 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.142 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.143 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.144 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smb", 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.145 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.146 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.147 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.148 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.149 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.150 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.151 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.152 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.153 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "update_orchestration": { 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "update_orchestration", 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.154 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.155 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:43:20.156 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.157 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.158 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.159 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.160 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.161 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.162 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.163 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.166 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.166 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.166 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.166 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.166 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.166 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.167 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.168 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.169 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.170 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.171 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.172 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.173 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.174 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.175 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-05-27T07:43:20.176 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.177 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.179 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.180 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.181 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.183 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.184 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.190 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.191 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.192 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.193 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.194 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.196 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.197 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.198 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-05-27T07:43:20.199 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-05-27T07:43:20.200 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.201 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.203 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.204 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.205 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-05-27T07:43:20.206 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.207 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-05-27T07:43:20.208 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.210 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.211 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.211 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.211 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.212 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.214 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.214 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.214 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.214 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.214 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.214 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.214 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.215 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.216 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.216 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.216 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.216 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.216 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.218 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.218 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.218 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.218 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.218 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.218 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.218 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-05-27T07:43:20.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.220 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.221 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.222 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.223 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.224 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.226 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.227 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.228 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.229 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.230 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.231 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.232 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.234 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.235 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.236 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.237 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-05-27T07:43:20.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.240 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.241 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.242 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.245 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-05-27T07:43:20.246 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.247 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.249 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.250 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.251 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.252 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.253 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.254 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.255 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.256 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.257 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.260 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 34964, 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "y", 2024-05-27T07:43:20.261 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.263 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.264 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.265 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.268 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.269 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.270 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-05-27T07:43:20.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-05-27T07:43:20.273 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.274 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.275 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-05-27T07:43:20.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.277 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.278 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.279 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-05-27T07:43:20.280 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.281 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.281 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.281 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-05-27T07:43:20.281 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-05-27T07:43:20.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.282 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:43:20.283 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-05-27T07:43:20.284 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.285 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.286 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-05-27T07:43:20.287 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:43:20.289 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-05-27T07:43:20.290 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.291 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.293 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.294 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.295 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.296 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.297 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.298 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.299 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.300 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.301 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.302 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.303 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-05-27T07:43:20.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.306 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.307 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-05-27T07:43:20.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:43:20.310 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.311 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.312 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-05-27T07:43:20.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.314 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.315 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.316 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-05-27T07:43:20.317 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.318 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.319 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.320 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.321 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-05-27T07:43:20.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.323 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.324 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.325 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:43:20.326 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.327 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.328 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-05-27T07:43:20.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.331 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.332 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.333 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-05-27T07:43:20.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_samba": { 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_samba", 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.336 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.337 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.338 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-05-27T07:43:20.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.340 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.341 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.343 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.344 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.345 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.346 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "grafana_dashboards_path": { 2024-05-27T07:43:20.347 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.348 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.349 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-05-27T07:43:20.350 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.351 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.352 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.353 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.354 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-05-27T07:43:20.355 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.357 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.358 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.359 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.361 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.362 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.363 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.364 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.365 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.366 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.366 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.366 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:43:20.366 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.366 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.370 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.371 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.372 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.373 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.374 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.375 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-05-27T07:43:20.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.377 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.378 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.379 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.380 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:43:20.382 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.384 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-05-27T07:43:20.385 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.387 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.388 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.389 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.391 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.392 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.393 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.394 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.395 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.396 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.397 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.398 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.399 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:43:20.401 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.402 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.403 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.404 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:43:20.405 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.406 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.407 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.408 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:43:20.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.410 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.411 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.412 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:43:20.413 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.414 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.415 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.416 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.417 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.418 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.419 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.420 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.421 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:43:20.422 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.423 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.424 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.425 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.426 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.427 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.428 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.429 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.430 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.431 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:43:20.432 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "[]", 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.434 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.435 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.436 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:43:20.437 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.438 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.439 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.440 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.441 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:43:20.442 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.443 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.444 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.445 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:43:20.446 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.448 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.449 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.450 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:43:20.451 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.453 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.457 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.458 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.459 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.460 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.462 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.464 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-05-27T07:43:20.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:43:20.467 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.468 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.469 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.471 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.472 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.473 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.474 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.475 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.476 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.477 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.479 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.480 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.481 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.482 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.483 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.484 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-05-27T07:43:20.485 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-05-27T07:43:20.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.487 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-05-27T07:43:20.488 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.489 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.490 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-05-27T07:43:20.491 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.493 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.495 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.496 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.497 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.498 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.499 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.500 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.501 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.502 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.503 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-05-27T07:43:20.504 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.505 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:43:20.506 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.507 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-05-27T07:43:20.508 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.510 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.511 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.512 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.513 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.517 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-05-27T07:43:20.518 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.519 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.520 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.521 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-05-27T07:43:20.522 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.523 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:43:20.524 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.525 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.526 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.527 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.528 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.529 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.530 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.531 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.532 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.533 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.534 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.535 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.536 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-05-27T07:43:20.537 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:43:20.539 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.540 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.541 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.542 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.543 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.544 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.545 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.546 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.547 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.548 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.549 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.550 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.551 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.552 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.553 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.554 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.555 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.556 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.557 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-05-27T07:43:20.558 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.559 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.573 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.574 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.575 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.576 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.577 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:43:20.578 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-05-27T07:43:20.579 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.580 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.581 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.582 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:43:20.583 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.584 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.585 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.586 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.602 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.603 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.604 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.605 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.606 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.608 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.609 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.610 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.611 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.612 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.613 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.614 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.615 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.616 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.617 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.618 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.619 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.620 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.622 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.623 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.624 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.625 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.626 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.629 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.630 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.632 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.633 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.634 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.635 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.636 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.637 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.638 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.639 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.640 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.641 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.642 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.643 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.645 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-05-27T07:43:20.646 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.648 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.649 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.650 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.651 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.652 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.653 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.654 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:20.655 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.656 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.657 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.658 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.659 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.660 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.661 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.662 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.663 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.664 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.665 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-05-27T07:43:20.666 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.667 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.668 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.669 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-05-27T07:43:20.670 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.671 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.673 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.674 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.675 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-05-27T07:43:20.676 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.677 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.678 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.679 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.680 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.681 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.682 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.683 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.684 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:43:20.685 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.686 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.687 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.688 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.690 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.691 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.692 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.693 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.694 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.695 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.696 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.697 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.698 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.699 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.700 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.701 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.703 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.704 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.705 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.706 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.707 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.708 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.722 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.723 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.724 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.725 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.726 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.727 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.728 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.729 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.730 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-05-27T07:43:20.731 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.732 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.733 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.735 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-05-27T07:43:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-05-27T07:43:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smb", 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.749 INFO:teuthology.orchestra.run.smithi012.stdout: "update_orchestration": { 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "update_orchestration", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.750 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.751 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.752 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.753 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.754 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.755 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.756 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.757 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.758 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.759 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.761 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.762 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.763 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.764 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.765 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.766 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.767 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.768 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.769 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.770 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.771 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.772 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.773 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.774 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.775 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.775 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.775 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.778 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.778 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.778 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.779 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.780 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.781 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.782 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.783 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.784 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.786 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.788 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.791 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.792 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:43:20.793 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-05-27T07:43:20.796 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.797 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.798 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.799 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-05-27T07:43:20.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.802 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.803 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.804 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.805 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.806 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.807 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.808 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.809 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.810 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.811 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.812 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.813 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-05-27T07:43:20.816 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.818 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.819 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.820 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.821 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.822 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.823 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.824 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.825 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.827 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.828 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.829 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.830 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.831 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.832 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.833 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.834 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.835 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:43:20.836 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.837 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.838 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:43:20.839 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.840 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.841 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.842 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.843 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.844 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.845 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.846 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.847 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.848 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.849 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.850 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.851 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.852 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-05-27T07:43:20.853 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.854 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.855 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.856 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "modules": [ 2024-05-27T07:43:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard", 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "iostat", 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "nfs", 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "restful", 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "snap_schedule" 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.858 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:43:20.859 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.860 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.861 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.862 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.865 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.866 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.867 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-05-27T07:43:20.868 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-05-27T07:43:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-05-27T07:43:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-05-27T07:43:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-05-27T07:43:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:43:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:43:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-05-27T07:43:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.901 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.902 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.903 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-05-27T07:43:20.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.906 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.907 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.908 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:20.909 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-05-27T07:43:20.911 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.912 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-05-27T07:43:20.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.914 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:43:20.915 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.916 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.917 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-05-27T07:43:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.920 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.921 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:43:20.922 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.923 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.924 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:43:20.925 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.926 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-05-27T07:43:20.927 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.928 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.930 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.931 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_samba": { 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_samba", 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.932 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.933 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-05-27T07:43:20.934 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.935 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.937 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:43:20.938 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.939 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.940 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.941 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "grafana_dashboards_path": { 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.942 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.944 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.945 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.946 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.948 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-05-27T07:43:20.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.950 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.951 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.952 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.953 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.954 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.955 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.956 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.957 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.959 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.960 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.961 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-05-27T07:43:20.962 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.964 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.965 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.966 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.967 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.968 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:43:20.973 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.974 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.975 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-05-27T07:43:20.976 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.978 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.979 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.980 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.981 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.982 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.983 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:20.985 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.986 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.987 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.988 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:20.990 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.992 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.993 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.994 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:43:20.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.997 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:20.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:20.999 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.000 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:43:21.001 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.002 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.003 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.004 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.005 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.006 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:43:21.007 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.008 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.009 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.010 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.011 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:43:21.012 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.014 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.015 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.016 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.017 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:43:21.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "[]", 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.019 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.020 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.021 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.022 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.023 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.024 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.025 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.026 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.027 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.028 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.029 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.030 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.031 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:43:21.032 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.034 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.035 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.036 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.037 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.040 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.041 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.042 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.043 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:43:21.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:43:21.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.046 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.047 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.048 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.049 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.050 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.051 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-05-27T07:43:21.052 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.053 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.054 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.055 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.057 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.059 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.060 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.061 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.062 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:21.063 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.064 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.065 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.066 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.067 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.069 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-05-27T07:43:21.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.071 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.072 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.073 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.074 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.075 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.076 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.077 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.078 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.079 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.080 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.081 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.082 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.083 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.084 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:21.085 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.086 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-05-27T07:43:21.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.088 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.089 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.090 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.091 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.092 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.093 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.094 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.095 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-05-27T07:43:21.096 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-05-27T07:43:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:43:21.101 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.102 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.103 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.105 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.106 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.107 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.108 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.109 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.110 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.111 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-05-27T07:43:21.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.114 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.115 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:21.116 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.117 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-05-27T07:43:21.118 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.121 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.122 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.123 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.124 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.125 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.126 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.127 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.128 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.129 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.130 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.131 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.132 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.133 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-05-27T07:43:21.135 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.136 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.137 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.138 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.139 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.140 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.141 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.142 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.145 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.146 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.147 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.149 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.150 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.151 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.152 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:43:21.153 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.154 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.155 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-05-27T07:43:21.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.157 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.158 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:43:21.159 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.160 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.161 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.163 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.164 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.165 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.166 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.167 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.168 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.169 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.170 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.171 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.172 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.173 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.175 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.176 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.178 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.179 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.180 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.181 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.182 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.183 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.184 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.187 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.188 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.190 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.191 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.193 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.194 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.195 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.196 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.198 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.201 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.202 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.203 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.204 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.205 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.206 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.207 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.208 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.209 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.211 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.212 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.213 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:21.214 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.218 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:43:21.219 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.220 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.221 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.222 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.223 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.224 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.225 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.226 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.227 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.228 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.232 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.233 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.234 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.235 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.236 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.238 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.239 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.240 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.241 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:21.242 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:21.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.244 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.246 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.247 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-05-27T07:43:21.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.250 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.253 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:43:21.255 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.256 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.257 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.258 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.259 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.260 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.261 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.262 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.263 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.264 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.265 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.266 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.267 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.269 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.270 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.271 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.273 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.274 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.275 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.276 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.277 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.278 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.280 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.281 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.283 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.284 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.285 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.286 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.287 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.288 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-05-27T07:43:21.289 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.290 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.293 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.294 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.295 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.296 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-05-27T07:43:21.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.298 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.299 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-05-27T07:43:21.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.302 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.303 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.304 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.305 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.306 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.309 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.310 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smb", 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.312 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.313 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.314 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.315 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.316 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.317 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "update_orchestration": { 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "update_orchestration", 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.319 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.320 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.321 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.322 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.324 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.325 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.327 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.328 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.329 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.331 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.332 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.333 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.334 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.335 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.336 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.337 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.338 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.339 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.342 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.345 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-05-27T07:43:21.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:21.347 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.349 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.350 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.352 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.353 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.354 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-05-27T07:43:21.356 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:43:21.358 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.359 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.360 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-05-27T07:43:21.361 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.362 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.363 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.364 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.365 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-05-27T07:43:21.366 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.367 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.369 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.370 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.373 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.376 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.377 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.378 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.379 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.380 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.381 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.382 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.383 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.384 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.385 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.390 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.391 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.392 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.393 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.394 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.395 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.396 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.397 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.398 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.399 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.400 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.401 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.402 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.403 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.404 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.405 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:43:21.406 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.407 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.408 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-05-27T07:43:21.409 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-05-27T07:43:21.410 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.411 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.412 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.413 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.414 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.415 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.416 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.417 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.418 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-05-27T07:43:21.419 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.420 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-05-27T07:43:21.421 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.422 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-05-27T07:43:21.423 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: "services": { 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard": "https://172.21.15.12:8443/" 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: "always_on_modules": { 2024-05-27T07:43:21.424 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus": [ 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.425 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific": [ 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.426 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy": [ 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.427 INFO:teuthology.orchestra.run.smithi012.stdout: "reef": [ 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-05-27T07:43:21.428 INFO:teuthology.orchestra.run.smithi012.stdout: "squid": [ 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: "last_failure_osd_epoch": 90, 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: "active_clients": [ 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1715017267 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.430 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "libcephsqlite", 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2988981755 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 4052494821 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.432 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2835907532 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-05-27T07:43:21.433 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2882915565 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-05-27T07:43:21.434 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2054603597 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-05-27T07:43:21.435 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-05-27T07:43:21.436 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:20 smithi062 ceph-mon[35304]: pgmap v710: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:21.436 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:20 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1907441658' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:43:21.437 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:20 smithi162 ceph-mon[35676]: pgmap v710: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:21.437 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:20 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1907441658' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f16340b2950 msgr2=0x7f16340b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f16340b2950 0x7f16340b4e60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1650106950 tx=0x7f164000b390 comp rx=0 tx=0).stop 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f165010bc70 msgr2=0x7f165019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f165010bc70 0x7f165019cca0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f1638005fd0 tx=0x7f163800dde0 comp rx=0 tx=0).stop 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 shutdown_connections 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f165010c110 0x7f16501a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f165010bc70 0x7f165019cca0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f16340b2950 0x7f16340b4e60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 --2- 172.21.15.12:0/1907441658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16501063c0 0x7f165019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.675+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 >> 172.21.15.12:0/1907441658 conn(0x7f165007d590 msgr2=0x7f1650103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.676+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 shutdown_connections 2024-05-27T07:43:21.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:19.676+0000 7f1655a00640 1 -- 172.21.15.12:0/1907441658 wait complete. 2024-05-27T07:43:21.495 DEBUG:tasks.check_counter:Getting stats from a 2024-05-27T07:43:21.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-27T07:43:21.496 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:20 smithi012 ceph-mon[31473]: pgmap v710: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:21.496 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:20 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1907441658' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:43:21.952 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:22.434 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:22 smithi012 ceph-mon[31473]: pgmap v711: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:22.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:22 smithi162 ceph-mon[35676]: pgmap v711: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:22.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:22 smithi062 ceph-mon[35304]: pgmap v711: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:24.444 INFO:teuthology.orchestra.run.smithi012.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-05-27T07:43:24.444 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:24 smithi012 ceph-mon[31473]: pgmap v712: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:24.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:24 smithi162 ceph-mon[35676]: pgmap v712: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:24.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:24 smithi062 ceph-mon[35304]: pgmap v712: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-05-27T07:43:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-27T07:43:25.294 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T07:43:25.295 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-27T07:43:25.306 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-27T07:43:25.306 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:43:25.306 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:43:25.369 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:43:25.369 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:43:25.427 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:43:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:26.349 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:26 smithi012 ceph-mon[31473]: pgmap v713: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:26.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:26 smithi162 ceph-mon[35676]: pgmap v713: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:26.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:26 smithi062 ceph-mon[35304]: pgmap v713: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 -- 172.21.15.12:0/1453702138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c10b8b0 msgr2=0x7f356c113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 --2- 172.21.15.12:0/1453702138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c10b8b0 0x7f356c113210 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f356400b0d0 tx=0x7f356400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 -- 172.21.15.12:0/1453702138 shutdown_connections 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 --2- 172.21.15.12:0/1453702138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c10b8b0 0x7f356c113210 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 --2- 172.21.15.12:0/1453702138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f356c106630 0x7f356c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 --2- 172.21.15.12:0/1453702138 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f356c10af60 0x7f356c10b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 -- 172.21.15.12:0/1453702138 >> 172.21.15.12:0/1453702138 conn(0x7f356c101ec0 msgr2=0x7f356c104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 -- 172.21.15.12:0/1453702138 shutdown_connections 2024-05-27T07:43:28.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.228+0000 7f3572800700 1 -- 172.21.15.12:0/1453702138 wait complete. 2024-05-27T07:43:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 Processor -- start 2024-05-27T07:43:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 -- start start 2024-05-27T07:43:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c106630 0x7f356c1a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f356c10af60 0x7f356c1a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f356c10b8b0 0x7f356c19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f356c116390 con 0x7f356c106630 2024-05-27T07:43:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f356c116210 con 0x7f356c10b8b0 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.230+0000 7f3572800700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f356c116510 con 0x7f356c10af60 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c106630 0x7f356c1a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c106630 0x7f356c1a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43732/0 (socket says 172.21.15.12:43732) 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 -- 172.21.15.12:0/3823861247 learned_addr learned my addr 172.21.15.12:0/3823861247 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571e00700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f356c10b8b0 0x7f356c19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3570a00700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f356c10af60 0x7f356c1a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 -- 172.21.15.12:0/3823861247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f356c10af60 msgr2=0x7f356c1a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f356c10af60 0x7f356c1a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 -- 172.21.15.12:0/3823861247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f356c10b8b0 msgr2=0x7f356c19cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f356c10b8b0 0x7f356c19cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.231+0000 7f3571400700 1 -- 172.21.15.12:0/3823861247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356c19d260 con 0x7f356c106630 2024-05-27T07:43:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.232+0000 7f3571400700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c106630 0x7f356c1a29a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f356000d950 tx=0x7f356000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.232+0000 7f356a000700 1 -- 172.21.15.12:0/3823861247 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560016070 con 0x7f356c106630 2024-05-27T07:43:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.232+0000 7f356a000700 1 -- 172.21.15.12:0/3823861247 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3560010460 con 0x7f356c106630 2024-05-27T07:43:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.232+0000 7f356a000700 1 -- 172.21.15.12:0/3823861247 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356000f620 con 0x7f356c106630 2024-05-27T07:43:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.232+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356c19d550 con 0x7f356c106630 2024-05-27T07:43:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.232+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f356c1a9750 con 0x7f356c106630 2024-05-27T07:43:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.233+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f356c107ae0 con 0x7f356c106630 2024-05-27T07:43:28.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.238+0000 7f356a000700 1 -- 172.21.15.12:0/3823861247 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3560019070 con 0x7f356c106630 2024-05-27T07:43:28.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.239+0000 7f356a000700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35540b2b40 0x7f35540b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:28.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.240+0000 7f356a000700 1 -- 172.21.15.12:0/3823861247 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9554+0+0 (secure 0 0 0) 0x7f35600e2580 con 0x7f356c106630 2024-05-27T07:43:28.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.240+0000 7f3570a00700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35540b2b40 0x7f35540b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:28.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.241+0000 7f3570a00700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35540b2b40 0x7f35540b5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f356c19e260 tx=0x7f355800b5b0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:28.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.243+0000 7f356a000700 1 -- 172.21.15.12:0/3823861247 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f35600a9b70 con 0x7f356c106630 2024-05-27T07:43:28.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:28 smithi162 ceph-mon[35676]: pgmap v714: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:28.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:28 smithi062 ceph-mon[35304]: pgmap v714: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:28.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.502+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f356c02e5e0 con 0x7f356c106630 2024-05-27T07:43:28.501 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:28 smithi012 ceph-mon[31473]: pgmap v714: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:28.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.503+0000 7f356a000700 1 -- 172.21.15.12:0/3823861247 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v159) ==== 81+0+941 (secure 0 0 0) 0x7f35600a9470 con 0x7f356c106630 2024-05-27T07:43:28.502 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/435630590 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2856894494 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2856894494 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/4121651169 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/725627344 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1619685219 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1199837030 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/949043977 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1631546627 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3259206461 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1064527569 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:28.503 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:28.504 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3780513737 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:28.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35540b2b40 msgr2=0x7f35540b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35540b2b40 0x7f35540b5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f356c19e260 tx=0x7f355800b5b0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c106630 msgr2=0x7f356c1a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c106630 0x7f356c1a29a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f356000d950 tx=0x7f356000dd10 comp rx=0 tx=0).stop 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 shutdown_connections 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7f35540b2b40 0x7f35540b5050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f356c10b8b0 0x7f356c19cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f356c106630 0x7f356c1a29a0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 --2- 172.21.15.12:0/3823861247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f356c10af60 0x7f356c1a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 >> 172.21.15.12:0/3823861247 conn(0x7f356c101ec0 msgr2=0x7f356c10a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 shutdown_connections 2024-05-27T07:43:28.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:28.507+0000 7f3572800700 1 -- 172.21.15.12:0/3823861247 wait complete. 2024-05-27T07:43:28.508 INFO:teuthology.orchestra.run.smithi012.stderr:listed 16 entries 2024-05-27T07:43:29.254 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-05-27T07:43:29.255 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:43:29.316 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:43:29.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:29 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/3823861247' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:43:29.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:29 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/3823861247' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:43:29.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:29 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/3823861247' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:43:29.788 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:43:30.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:30 smithi162 ceph-mon[35676]: pgmap v715: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:30.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:30 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:43:30.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:30 smithi062 ceph-mon[35304]: pgmap v715: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:30.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:30 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:43:30.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:30 smithi012 ceph-mon[31473]: pgmap v715: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:30.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:30 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:43:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 -- 172.21.15.12:0/1251284698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c108d20 msgr2=0x7fe74c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 --2- 172.21.15.12:0/1251284698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c108d20 0x7fe74c109130 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fe7340099c0 tx=0x7fe734009cd0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 -- 172.21.15.12:0/1251284698 shutdown_connections 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 --2- 172.21.15.12:0/1251284698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe74c1043f0 0x7fe74c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 --2- 172.21.15.12:0/1251284698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe74c109670 0x7fe74c110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 --2- 172.21.15.12:0/1251284698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c108d20 0x7fe74c109130 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 -- 172.21.15.12:0/1251284698 >> 172.21.15.12:0/1251284698 conn(0x7fe74c079d00 msgr2=0x7fe74c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 -- 172.21.15.12:0/1251284698 shutdown_connections 2024-05-27T07:43:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.046+0000 7fe74be00700 1 -- 172.21.15.12:0/1251284698 wait complete. 2024-05-27T07:43:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.047+0000 7fe74be00700 1 Processor -- start 2024-05-27T07:43:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.047+0000 7fe74be00700 1 -- start start 2024-05-27T07:43:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.048+0000 7fe74be00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c1043f0 0x7fe74c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.048+0000 7fe74be00700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe74c108d20 0x7fe74c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.048+0000 7fe74be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe74c109670 0x7fe74c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.048+0000 7fe74be00700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c114180 con 0x7fe74c1043f0 2024-05-27T07:43:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.048+0000 7fe74be00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c114000 con 0x7fe74c109670 2024-05-27T07:43:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.048+0000 7fe74be00700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c114300 con 0x7fe74c108d20 2024-05-27T07:43:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.049+0000 7fe74aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c1043f0 0x7fe74c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.049+0000 7fe74aa00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c1043f0 0x7fe74c1a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38568/0 (socket says 172.21.15.12:38568) 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.049+0000 7fe74aa00700 1 -- 172.21.15.12:0/1314213264 learned_addr learned my addr 172.21.15.12:0/1314213264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.049+0000 7fe74a000700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe74c108d20 0x7fe74c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.049+0000 7fe74b400700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe74c109670 0x7fe74c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.049+0000 7fe74aa00700 1 -- 172.21.15.12:0/1314213264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe74c108d20 msgr2=0x7fe74c1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.049+0000 7fe74aa00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe74c108d20 0x7fe74c1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe74aa00700 1 -- 172.21.15.12:0/1314213264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe74c109670 msgr2=0x7fe74c19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe74aa00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe74c109670 0x7fe74c19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe74aa00700 1 -- 172.21.15.12:0/1314213264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c19b090 con 0x7fe74c1043f0 2024-05-27T07:43:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe74aa00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c1043f0 0x7fe74c1a07c0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00d950 tx=0x7fe73c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe743400700 1 -- 172.21.15.12:0/1314213264 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c016070 con 0x7fe74c1043f0 2024-05-27T07:43:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe743400700 1 -- 172.21.15.12:0/1314213264 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe73c009dd0 con 0x7fe74c1043f0 2024-05-27T07:43:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c19b380 con 0x7fe74c1043f0 2024-05-27T07:43:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe743400700 1 -- 172.21.15.12:0/1314213264 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c00f3c0 con 0x7fe74c1043f0 2024-05-27T07:43:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.050+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c1a7650 con 0x7fe74c1043f0 2024-05-27T07:43:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.052+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c0512b0 con 0x7fe74c1043f0 2024-05-27T07:43:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.054+0000 7fe743400700 1 -- 172.21.15.12:0/1314213264 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe73c019070 con 0x7fe74c1043f0 2024-05-27T07:43:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.056+0000 7fe743400700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7300b2a70 0x7fe7300b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:43:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.056+0000 7fe743400700 1 -- 172.21.15.12:0/1314213264 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9554+0+0 (secure 0 0 0) 0x7fe73c0e25f0 con 0x7fe74c1043f0 2024-05-27T07:43:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.056+0000 7fe74a000700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7300b2a70 0x7fe7300b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:43:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.057+0000 7fe74a000700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7300b2a70 0x7fe7300b4f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe734009990 tx=0x7fe734005fb0 comp rx=0 tx=0).ready entity=mgr.24991 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:43:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.058+0000 7fe743400700 1 -- 172.21.15.12:0/1314213264 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe73c0a9b60 con 0x7fe74c1043f0 2024-05-27T07:43:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.320+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe74c02e5e0 con 0x7fe74c1043f0 2024-05-27T07:43:32.320 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:32 smithi012 ceph-mon[31473]: pgmap v716: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.321+0000 7fe743400700 1 -- 172.21.15.12:0/1314213264 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v159) ==== 81+0+941 (secure 0 0 0) 0x7fe73c0a9460 con 0x7fe74c1043f0 2024-05-27T07:43:32.321 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/435630590 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:32.321 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2856894494 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:32.321 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2856894494 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:32.321 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/4121651169 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:32.321 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:32.321 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2115905110 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/725627344 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1619685219 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1199837030 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/949043977 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1631546627 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3259206461 2024-05-28T07:22:22.282926+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1064527569 2024-05-28T07:06:45.507849+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/579288494 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:32.322 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3780513737 2024-05-28T07:06:09.346476+0000 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7300b2a70 msgr2=0x7fe7300b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7300b2a70 0x7fe7300b4f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe734009990 tx=0x7fe734005fb0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c1043f0 msgr2=0x7fe74c1a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c1043f0 0x7fe74c1a07c0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00d950 tx=0x7fe73c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 shutdown_connections 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:6800/3926981258,v1:172.21.15.12:6801/3926981258] conn(0x7fe7300b2a70 0x7fe7300b4f80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe74c109670 0x7fe74c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c1043f0 0x7fe74c1a07c0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 --2- 172.21.15.12:0/1314213264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe74c108d20 0x7fe74c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:43:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.324+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 >> 172.21.15.12:0/1314213264 conn(0x7fe74c079d00 msgr2=0x7fe74c1018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:43:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.325+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 shutdown_connections 2024-05-27T07:43:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-05-27T07:43:32.325+0000 7fe74be00700 1 -- 172.21.15.12:0/1314213264 wait complete. 2024-05-27T07:43:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:listed 16 entries 2024-05-27T07:43:32.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:32 smithi162 ceph-mon[35676]: pgmap v716: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:32.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:32 smithi062 ceph-mon[35304]: pgmap v716: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:33.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:33 smithi012 ceph-mon[31473]: from='client.? 172.21.15.12:0/1314213264' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:43:33.356 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:33 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:33.356 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:33 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:33.415 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi012.front.sepia.ceph.com... 2024-05-27T07:43:33.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:43:33.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-27T07:43:33.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:33 smithi162 ceph-mon[35676]: from='client.? 172.21.15.12:0/1314213264' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:43:33.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:33 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:33.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:33 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:33.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi012.stderr:ceph-fuse[90645]: fuse finished with error 0 and tester_r 0 2024-05-27T07:43:33.498 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:33 smithi062 ceph-mon[35304]: from='client.? 172.21.15.12:0/1314213264' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:43:33.498 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:33 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:33.498 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:33 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:33.518 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:43:34.483 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:34 smithi162 ceph-mon[35676]: pgmap v717: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:34.483 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:34.483 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:34 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:34.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:34 smithi062 ceph-mon[35304]: pgmap v717: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:34.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:34.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:34 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:34.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:34 smithi012 ceph-mon[31473]: pgmap v717: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:34.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:34.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:34 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:35.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:35 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:35 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:35 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:35 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:35 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:43:35.496 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:35 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:43:35.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:35 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.598 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:35 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:35 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:35 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:35 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:43:35.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:35 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:43:35.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:35 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:35 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:35 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:35 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:35.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:35 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi062 to 176.4M 2024-05-27T07:43:35.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:35 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi062 to 185071308: error parsing value: Value '185071308' is below minimum 939524096 2024-05-27T07:43:36.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: pgmap v718: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:36.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:43:36.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:43:36.496 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:36.496 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:36 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:36.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: pgmap v718: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:36.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:43:36.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:43:36.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:36.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:36 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:36.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: pgmap v718: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:36.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:36.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi162 to 176.4M 2024-05-27T07:43:36.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi162 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-05-27T07:43:36.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:36.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:36 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:38.494 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: pgmap v719: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: pgmap v720: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: pgmap v721: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.495 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:38 smithi062 ceph-mon[35304]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: pgmap v719: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:43:38.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:43:38.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:43:38.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: pgmap v720: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: pgmap v721: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:38 smithi012 ceph-mon[31473]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:38.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: pgmap v719: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.732 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: Adjusting osd_memory_target on smithi012 to 177.5M 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: Unable to set osd_memory_target on smithi012 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: pgmap v720: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: pgmap v721: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:43:38.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:38 smithi162 ceph-mon[35676]: from='mgr.24991 172.21.15.12:0/4252222981' entity='mgr.x' 2024-05-27T07:43:39.620 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:43:39.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:43:39.620 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-27T07:43:39.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:43:39.681 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-05-27T07:43:39.735 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-27T07:43:39.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:43:39.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:43:39.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:43:39.810 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-05-27T07:43:39.913 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-27T07:43:39.924 INFO:tasks.cephadm:Teardown begin 2024-05-27T07:43:39.924 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/cephadm.py", line 2270, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T07:43:39.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:43:39.990 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:43:40.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:43:40.184 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-27T07:43:40.184 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:43:40.246 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:43:40.306 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:43:40.363 INFO:tasks.cephadm:Stopping all daemons... 2024-05-27T07:43:40.363 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-27T07:43:40.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a 2024-05-27T07:43:40.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:40 smithi162 ceph-mon[35676]: pgmap v722: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-05-27T07:43:40.744 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:40 smithi062 ceph-mon[35304]: pgmap v722: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-05-27T07:43:40.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:40 smithi012 ceph-mon[31473]: pgmap v722: 129 pgs: 129 active+clean; 534 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-05-27T07:43:40.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:40 smithi012 systemd[1]: Stopping Ceph mon.a for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:43:41.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:40 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a[31468]: 2024-05-27T07:43:40.958+0000 7fd525800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:41.268 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:40 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a[31468]: 2024-05-27T07:43:40.958+0000 7fd525800700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-27T07:43:41.564 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:41 smithi012 podman[144775]: 2024-05-27 07:43:41.268896326 +0000 UTC m=+0.467895951 container died f4a74345a656417d7e759b6600c0c5dcfb2fff6f335e12494f43eab188220ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:43:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:41 smithi012 podman[144775]: 2024-05-27 07:43:41.709315625 +0000 UTC m=+0.908315247 container cleanup f4a74345a656417d7e759b6600c0c5dcfb2fff6f335e12494f43eab188220ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:43:41.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:41 smithi012 bash[144775]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a 2024-05-27T07:43:42.349 INFO:journalctl@ceph.mon.a.smithi012.stdout:May 27 07:43:42 smithi012 podman[144788]: 2024-05-27 07:43:42.019195202 +0000 UTC m=+0.732504959 container remove f4a74345a656417d7e759b6600c0c5dcfb2fff6f335e12494f43eab188220ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:43:42.707 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.a.service' 2024-05-27T07:43:42.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:42.857 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-27T07:43:42.857 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-27T07:43:42.858 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.b 2024-05-27T07:43:43.244 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:42 smithi062 systemd[1]: Stopping Ceph mon.b for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:43:43.960 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:43 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b[35300]: 2024-05-27T07:43:43.514+0000 7f2bada00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:43.961 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:43 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b[35300]: 2024-05-27T07:43:43.514+0000 7f2bada00700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-27T07:43:43.961 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:43 smithi062 podman[74394]: 2024-05-27 07:43:43.607177239 +0000 UTC m=+0.289034818 container died ccbee1f2bf47cdf85b84ec85569f1ee6169e9ac42c65ac202f777ce3009b000d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:43:44.244 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:44 smithi062 podman[74394]: 2024-05-27 07:43:44.173723494 +0000 UTC m=+0.855581123 container cleanup ccbee1f2bf47cdf85b84ec85569f1ee6169e9ac42c65ac202f777ce3009b000d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-05-27T07:43:44.244 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:44 smithi062 bash[74394]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b 2024-05-27T07:43:44.994 INFO:journalctl@ceph.mon.b.smithi062.stdout:May 27 07:43:44 smithi062 podman[74405]: 2024-05-27 07:43:44.592830675 +0000 UTC m=+0.976500260 container remove ccbee1f2bf47cdf85b84ec85569f1ee6169e9ac42c65ac202f777ce3009b000d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:43:45.224 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.b.service' 2024-05-27T07:43:45.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:45.365 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-27T07:43:45.366 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-27T07:43:45.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.c 2024-05-27T07:43:45.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:45 smithi162 systemd[1]: Stopping Ceph mon.c for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:43:46.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-c[35670]: 2024-05-27T07:43:46.000+0000 7f0a6bc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:46.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-c[35670]: 2024-05-27T07:43:46.000+0000 7f0a6bc00700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-27T07:43:46.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:46 smithi162 podman[77100]: 2024-05-27 07:43:46.310186471 +0000 UTC m=+0.468145307 container died 87b3e6a8271106b1897dace8c986ad13632d4ba2518e7cd5d074f15a6a29f481 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:43:46.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:46 smithi162 podman[77100]: 2024-05-27 07:43:46.877141279 +0000 UTC m=+1.035100106 container cleanup 87b3e6a8271106b1897dace8c986ad13632d4ba2518e7cd5d074f15a6a29f481 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-27T07:43:46.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:46 smithi162 bash[77100]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-c 2024-05-27T07:43:47.232 INFO:journalctl@ceph.mon.c.smithi162.stdout:May 27 07:43:47 smithi162 podman[77112]: 2024-05-27 07:43:47.121041969 +0000 UTC m=+0.792180625 container remove 87b3e6a8271106b1897dace8c986ad13632d4ba2518e7cd5d074f15a6a29f481 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:43:47.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mon.c.service' 2024-05-27T07:43:47.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:47.736 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-27T07:43:47.736 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-27T07:43:47.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T07:43:48.099 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:43:47 smithi012 systemd[1]: Stopping Ceph mgr.x for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:43:48.599 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:43:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:43:48.346+0000 7f9497200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:48.599 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:43:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x[31714]: 2024-05-27T07:43:48.346+0000 7f9497200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-27T07:43:48.599 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:43:48 smithi012 podman[144892]: 2024-05-27 07:43:48.449499247 +0000 UTC m=+0.244931692 container died 0587ea20bc322350ce3c311a3ab3e53a21e46e7aeac9155f19fda338b71452f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:43:49.099 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:43:48 smithi012 podman[144892]: 2024-05-27 07:43:48.964509415 +0000 UTC m=+0.759941843 container cleanup 0587ea20bc322350ce3c311a3ab3e53a21e46e7aeac9155f19fda338b71452f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-27T07:43:49.099 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:43:48 smithi012 bash[144892]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x 2024-05-27T07:43:49.599 INFO:journalctl@ceph.mgr.x.smithi012.stdout:May 27 07:43:49 smithi012 podman[144907]: 2024-05-27 07:43:49.174500675 +0000 UTC m=+0.712632232 container remove 0587ea20bc322350ce3c311a3ab3e53a21e46e7aeac9155f19fda338b71452f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-x, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:43:49.716 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.x.service' 2024-05-27T07:43:49.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:49.826 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-27T07:43:49.826 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-27T07:43:49.826 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.y 2024-05-27T07:43:50.244 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:43:49 smithi062 systemd[1]: Stopping Ceph mgr.y for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:43:50.994 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:43:50 smithi062 podman[74528]: 2024-05-27 07:43:50.620204386 +0000 UTC m=+0.305067155 container died 9873a08d4594b6e194919b5376619a247de8591ecbf8abdcf5106602b012166a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:43:51.494 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:43:51 smithi062 podman[74528]: 2024-05-27 07:43:51.388341415 +0000 UTC m=+1.073204241 container cleanup 9873a08d4594b6e194919b5376619a247de8591ecbf8abdcf5106602b012166a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:43:51.494 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:43:51 smithi062 bash[74528]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y 2024-05-27T07:43:51.994 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:43:51 smithi062 podman[74542]: 2024-05-27 07:43:51.590098338 +0000 UTC m=+0.950807772 container remove 9873a08d4594b6e194919b5376619a247de8591ecbf8abdcf5106602b012166a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-y, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:43:51.995 INFO:journalctl@ceph.mgr.y.smithi062.stdout:May 27 07:43:51 smithi062 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-27T07:43:52.143 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.y.service' 2024-05-27T07:43:52.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:52.269 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-27T07:43:52.269 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-27T07:43:52.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.z 2024-05-27T07:43:52.732 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:43:52 smithi162 systemd[1]: Stopping Ceph mgr.z for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:43:53.232 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:43:52 smithi162 podman[77235]: 2024-05-27 07:43:52.923706682 +0000 UTC m=+0.230131285 container died f3a2800d7af695fb2de69802e1df8cc32f5b1c3bf15ed0d210c0d1319bcb0070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:43:53.667 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:43:53 smithi162 podman[77235]: 2024-05-27 07:43:53.408091873 +0000 UTC m=+0.714516461 container cleanup f3a2800d7af695fb2de69802e1df8cc32f5b1c3bf15ed0d210c0d1319bcb0070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T07:43:53.667 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:43:53 smithi162 bash[77235]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z 2024-05-27T07:43:53.982 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:43:53 smithi162 podman[77249]: 2024-05-27 07:43:53.668356403 +0000 UTC m=+0.737028676 container remove f3a2800d7af695fb2de69802e1df8cc32f5b1c3bf15ed0d210c0d1319bcb0070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T07:43:53.982 INFO:journalctl@ceph.mgr.z.smithi162.stdout:May 27 07:43:53 smithi162 systemd[1]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-05-27T07:43:54.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mgr.z.service' 2024-05-27T07:43:54.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:54.256 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-27T07:43:54.256 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-27T07:43:54.256 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.0 2024-05-27T07:43:54.598 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:43:54 smithi012 systemd[1]: Stopping Ceph osd.0 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:43:55.007 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:43:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0[45876]: 2024-05-27T07:43:54.841+0000 7f082da00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:55.007 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:43:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0[45876]: 2024-05-27T07:43:54.841+0000 7f082da00700 -1 osd.0 159 *** Got signal Terminated *** 2024-05-27T07:43:55.007 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:43:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0[45876]: 2024-05-27T07:43:54.841+0000 7f082da00700 -1 osd.0 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:44:00.849 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:00 smithi012 podman[145013]: 2024-05-27 07:44:00.541246707 +0000 UTC m=+5.849955667 container died 8fe08bb841100c73b69e6bf4db4d845ee205e8aa72a94e2414eea721acfd0781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:44:01.599 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:01 smithi012 podman[145013]: 2024-05-27 07:44:01.177089735 +0000 UTC m=+6.485798688 container cleanup 8fe08bb841100c73b69e6bf4db4d845ee205e8aa72a94e2414eea721acfd0781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:44:01.599 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:01 smithi012 bash[145013]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0 2024-05-27T07:44:01.599 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:01 smithi012 podman[145045]: 2024-05-27 07:44:01.412287654 +0000 UTC m=+0.850842419 container remove 8fe08bb841100c73b69e6bf4db4d845ee205e8aa72a94e2414eea721acfd0781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T07:44:02.599 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:02 smithi012 podman[145131]: 2024-05-27 07:44:02.219072375 +0000 UTC m=+0.049981611 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:44:02.599 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:02 smithi012 podman[145131]: 2024-05-27 07:44:02.363149051 +0000 UTC m=+0.194058283 container create 8ebb4f865eb1feedf5460bba29b74d7af1f161336cac275df195bb8709933197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True) 2024-05-27T07:44:03.099 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:02 smithi012 podman[145131]: 2024-05-27 07:44:02.650425295 +0000 UTC m=+0.481334507 container init 8ebb4f865eb1feedf5460bba29b74d7af1f161336cac275df195bb8709933197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:44:03.099 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:02 smithi012 podman[145131]: 2024-05-27 07:44:02.663551099 +0000 UTC m=+0.494460351 container start 8ebb4f865eb1feedf5460bba29b74d7af1f161336cac275df195bb8709933197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:44:03.099 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:02 smithi012 podman[145131]: 2024-05-27 07:44:02.740764938 +0000 UTC m=+0.571674221 container attach 8ebb4f865eb1feedf5460bba29b74d7af1f161336cac275df195bb8709933197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True) 2024-05-27T07:44:03.505 INFO:journalctl@ceph.osd.0.smithi012.stdout:May 27 07:44:03 smithi012 podman[145157]: 2024-05-27 07:44:03.190998129 +0000 UTC m=+0.034404525 container died 8ebb4f865eb1feedf5460bba29b74d7af1f161336cac275df195bb8709933197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:44:04.053 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.0.service' 2024-05-27T07:44:04.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:04.175 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-27T07:44:04.175 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-27T07:44:04.176 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.1 2024-05-27T07:44:04.744 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:04 smithi062 systemd[1]: Stopping Ceph osd.1 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:44:05.244 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1[39878]: 2024-05-27T07:44:04.791+0000 7f63da400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:44:05.245 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1[39878]: 2024-05-27T07:44:04.791+0000 7f63da400700 -1 osd.1 159 *** Got signal Terminated *** 2024-05-27T07:44:05.245 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1[39878]: 2024-05-27T07:44:04.791+0000 7f63da400700 -1 osd.1 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:44:10.744 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:10 smithi062 podman[74650]: 2024-05-27 07:44:10.440435078 +0000 UTC m=+5.807436722 container died a3e90de9255b319a599bd3a17a9fcbdf393c2acd4058f259f9cb950958698e24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:44:11.136 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:10 smithi062 podman[74650]: 2024-05-27 07:44:10.935514549 +0000 UTC m=+6.302516186 container cleanup a3e90de9255b319a599bd3a17a9fcbdf393c2acd4058f259f9cb950958698e24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-05-27T07:44:11.136 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:10 smithi062 bash[74650]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1 2024-05-27T07:44:11.494 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:11 smithi062 podman[74668]: 2024-05-27 07:44:11.13706696 +0000 UTC m=+0.681963047 container remove a3e90de9255b319a599bd3a17a9fcbdf393c2acd4058f259f9cb950958698e24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:44:12.275 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:12 smithi062 podman[74758]: 2024-05-27 07:44:11.934629332 +0000 UTC m=+0.039593903 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:44:12.275 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:12 smithi062 podman[74758]: 2024-05-27 07:44:12.063251331 +0000 UTC m=+0.168215927 container create 9c41ba0c90f90a439877591ac0ba4403a6afe75bbfe146e2b2a635986581e0e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:44:12.744 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:12 smithi062 podman[74758]: 2024-05-27 07:44:12.384243067 +0000 UTC m=+0.489207663 container init 9c41ba0c90f90a439877591ac0ba4403a6afe75bbfe146e2b2a635986581e0e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:44:12.744 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:12 smithi062 podman[74758]: 2024-05-27 07:44:12.396833533 +0000 UTC m=+0.501798128 container start 9c41ba0c90f90a439877591ac0ba4403a6afe75bbfe146e2b2a635986581e0e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:44:12.745 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:12 smithi062 podman[74758]: 2024-05-27 07:44:12.559356718 +0000 UTC m=+0.664321357 container attach 9c41ba0c90f90a439877591ac0ba4403a6afe75bbfe146e2b2a635986581e0e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-27T07:44:13.242 INFO:journalctl@ceph.osd.1.smithi062.stdout:May 27 07:44:12 smithi062 podman[74784]: 2024-05-27 07:44:12.909459307 +0000 UTC m=+0.032634460 container died 9c41ba0c90f90a439877591ac0ba4403a6afe75bbfe146e2b2a635986581e0e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-1-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:44:13.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.1.service' 2024-05-27T07:44:13.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:13.791 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-27T07:44:13.791 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-27T07:44:13.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.2 2024-05-27T07:44:14.232 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:13 smithi162 systemd[1]: Stopping Ceph osd.2 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:44:14.732 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2[40827]: 2024-05-27T07:44:14.401+0000 7f9b3ea00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:44:14.732 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2[40827]: 2024-05-27T07:44:14.401+0000 7f9b3ea00700 -1 osd.2 159 *** Got signal Terminated *** 2024-05-27T07:44:14.733 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2[40827]: 2024-05-27T07:44:14.401+0000 7f9b3ea00700 -1 osd.2 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:44:20.531 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:20 smithi162 podman[77361]: 2024-05-27 07:44:20.238789883 +0000 UTC m=+5.992307891 container died 63c6536b358d9a6fef517360543be52318f18297265c8342e6b9d75a59137acf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-27T07:44:20.969 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:20 smithi162 podman[77361]: 2024-05-27 07:44:20.660116058 +0000 UTC m=+6.413634060 container cleanup 63c6536b358d9a6fef517360543be52318f18297265c8342e6b9d75a59137acf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T07:44:20.969 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:20 smithi162 bash[77361]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2 2024-05-27T07:44:21.232 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:20 smithi162 podman[77377]: 2024-05-27 07:44:20.970219694 +0000 UTC m=+0.722671075 container remove 63c6536b358d9a6fef517360543be52318f18297265c8342e6b9d75a59137acf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:44:22.225 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:21 smithi162 podman[77467]: 2024-05-27 07:44:21.822591738 +0000 UTC m=+0.033397358 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:44:22.225 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:21 smithi162 podman[77467]: 2024-05-27 07:44:21.954864461 +0000 UTC m=+0.165670138 container create 8eceb202b7aca199211fb8bc3a57c153d157bdf5073e2efe1bb2a568df5b01ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:44:22.482 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:22 smithi162 podman[77467]: 2024-05-27 07:44:22.225231446 +0000 UTC m=+0.436037110 container init 8eceb202b7aca199211fb8bc3a57c153d157bdf5073e2efe1bb2a568df5b01ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:44:22.482 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:22 smithi162 podman[77467]: 2024-05-27 07:44:22.236320353 +0000 UTC m=+0.447125974 container start 8eceb202b7aca199211fb8bc3a57c153d157bdf5073e2efe1bb2a568df5b01ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:44:22.482 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:22 smithi162 podman[77467]: 2024-05-27 07:44:22.357699435 +0000 UTC m=+0.568505080 container attach 8eceb202b7aca199211fb8bc3a57c153d157bdf5073e2efe1bb2a568df5b01ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-27T07:44:22.807 INFO:journalctl@ceph.osd.2.smithi162.stdout:May 27 07:44:22 smithi162 podman[77467]: 2024-05-27 07:44:22.708337638 +0000 UTC m=+0.919143325 container died 8eceb202b7aca199211fb8bc3a57c153d157bdf5073e2efe1bb2a568df5b01ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:44:22.807 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:22.495+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:23.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:22.921+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:23.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.2.service' 2024-05-27T07:44:23.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:23.561 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-27T07:44:23.561 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-27T07:44:23.561 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.3 2024-05-27T07:44:23.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:23 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:23.462+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:23.895 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:23 smithi012 systemd[1]: Stopping Ceph osd.3 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:44:24.151 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:23.895+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:24.533 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3[52789]: 2024-05-27T07:44:24.153+0000 7fa7ea600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:44:24.533 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3[52789]: 2024-05-27T07:44:24.153+0000 7fa7ea600700 -1 osd.3 159 *** Got signal Terminated *** 2024-05-27T07:44:24.533 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3[52789]: 2024-05-27T07:44:24.153+0000 7fa7ea600700 -1 osd.3 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:44:24.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:24.472+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:24.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:24.252+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:25.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:24.897+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:25.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:24.926+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:25.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:25.105+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:25.482 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:25.208+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:25.804 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3[52789]: 2024-05-27T07:44:25.499+0000 7fa7e5e00700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.606078+0000 front 2024-05-27T07:43:59.606270+0000 (oldest deadline 2024-05-27T07:44:25.408714+0000) 2024-05-27T07:44:25.982 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:25.509+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:26.061 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:25.898+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:26.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:25.920+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:26.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:26.062+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:26.482 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:26.225+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:26.482 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:26.074+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:26.848 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3[52789]: 2024-05-27T07:44:26.525+0000 7fa7e5e00700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.606078+0000 front 2024-05-27T07:43:59.606270+0000 (oldest deadline 2024-05-27T07:44:25.408714+0000) 2024-05-27T07:44:26.982 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:26.493+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:27.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:26.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:27.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:26.883+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:27.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:27 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:27.096+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:27.467 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:27.034+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:27.468 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:27.205+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:27.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:27.467+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:27.848 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:27 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3[52789]: 2024-05-27T07:44:27.524+0000 7fa7e5e00700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.606078+0000 front 2024-05-27T07:43:59.606270+0000 (oldest deadline 2024-05-27T07:44:25.408714+0000) 2024-05-27T07:44:28.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:27.880+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:28.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:27 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:27.932+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:28.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:28.132+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:28.482 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:28.157+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:28.482 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:28.014+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:28.849 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3[52789]: 2024-05-27T07:44:28.529+0000 7fa7e5e00700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.606078+0000 front 2024-05-27T07:43:59.606270+0000 (oldest deadline 2024-05-27T07:44:25.408714+0000) 2024-05-27T07:44:28.982 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:28.490+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:29.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:28.925+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:29.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:28 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:28.966+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:29.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:29 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:29.091+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:29.482 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:29.194+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:29.482 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:29.015+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:29.982 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:29.495+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:30.201 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:29 smithi012 podman[145236]: 2024-05-27 07:44:29.901292474 +0000 UTC m=+5.925980629 container died 861e74be575cd27b95400587e0485b33ab8e5e6409d0377da7939cb99d1bca53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:44:30.201 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:29 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:29.928+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:30.202 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:30.073+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:30.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:29.912+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:30.457 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:30.032+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:30.457 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:30.227+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:30.599 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:30 smithi012 podman[145236]: 2024-05-27 07:44:30.40503727 +0000 UTC m=+6.429725435 container cleanup 861e74be575cd27b95400587e0485b33ab8e5e6409d0377da7939cb99d1bca53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:44:30.599 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:30 smithi012 bash[145236]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3 2024-05-27T07:44:30.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:30.458+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:31.094 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:30 smithi012 podman[145263]: 2024-05-27 07:44:30.715169406 +0000 UTC m=+0.793629714 container remove 861e74be575cd27b95400587e0485b33ab8e5e6409d0377da7939cb99d1bca53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:44:31.094 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:30 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:30.923+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:31.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:30.908+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:31.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:30.908+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:31.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:31 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:31.095+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:31.423 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:30.999+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:31.423 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:31.232+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:31.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:31.423+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:31.754 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:31 smithi012 podman[145349]: 2024-05-27 07:44:31.352397941 +0000 UTC m=+0.038928012 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:44:31.754 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:31 smithi012 podman[145349]: 2024-05-27 07:44:31.457648459 +0000 UTC m=+0.144178550 container create 6923c37982920033764b20dc59a36c61f17bb5f974d45e1db11d1ca78e26a4ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:44:32.052 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:31 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:31.947+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:32.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:31 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:31.947+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:32.053 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:32 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:32.053+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:32.053 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:32 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:32.053+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:32.053 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:31 smithi012 podman[145349]: 2024-05-27 07:44:31.754124307 +0000 UTC m=+0.440654438 container init 6923c37982920033764b20dc59a36c61f17bb5f974d45e1db11d1ca78e26a4ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:44:32.053 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:31 smithi012 podman[145349]: 2024-05-27 07:44:31.76891285 +0000 UTC m=+0.455442947 container start 6923c37982920033764b20dc59a36c61f17bb5f974d45e1db11d1ca78e26a4ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:44:32.053 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:31 smithi012 podman[145349]: 2024-05-27 07:44:31.843600025 +0000 UTC m=+0.530130170 container attach 6923c37982920033764b20dc59a36c61f17bb5f974d45e1db11d1ca78e26a4ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:44:32.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:31.887+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:32.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:31.887+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:32.348 INFO:journalctl@ceph.osd.3.smithi012.stdout:May 27 07:44:32 smithi012 podman[145349]: 2024-05-27 07:44:32.245037306 +0000 UTC m=+0.931567434 container died 6923c37982920033764b20dc59a36c61f17bb5f974d45e1db11d1ca78e26a4ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:44:32.415 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:32.014+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:32.416 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:32.014+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:32.416 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:32.245+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:32.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:32.415+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:32.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:32.415+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:33.083 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.3.service' 2024-05-27T07:44:33.204 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:32 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:32.945+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:33.204 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:32 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:32.945+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:33.205 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:33 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:33.010+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:33.205 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:33 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:33.010+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:33.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:33.209 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-27T07:44:33.209 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-27T07:44:33.209 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.4 2024-05-27T07:44:33.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:32.917+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:33.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:32.917+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:33.435 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:33.032+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:33.436 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:33.032+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:33.436 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:33.253+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:33.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:33.435+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:33.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:33.435+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:33.745 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:33 smithi062 systemd[1]: Stopping Ceph osd.4 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:44:33.745 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:33.471+0000 7febe5000700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.218269+0000 front 2024-05-27T07:44:07.218715+0000 (oldest deadline 2024-05-27T07:44:33.020110+0000) 2024-05-27T07:44:34.227 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:33 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:33.910+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:34.228 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:33 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:33.910+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:34.228 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:34 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:34.032+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:34.228 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:34 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:34.032+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:34.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:34.063+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:34.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:34.064+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:34.244 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:33.906+0000 7febee800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:44:34.244 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:33.906+0000 7febee800700 -1 osd.4 159 *** Got signal Terminated *** 2024-05-27T07:44:34.244 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:33.906+0000 7febee800700 -1 osd.4 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:44:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:33.946+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:33.946+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:34.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:34.401+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:34.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:34.401+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:34.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:34.287+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:34.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:34.287+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:34.744 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:34.487+0000 7febe5000700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.218269+0000 front 2024-05-27T07:44:07.218715+0000 (oldest deadline 2024-05-27T07:44:33.020110+0000) 2024-05-27T07:44:34.745 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:34.703+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:35.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:34.924+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:35.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:34.924+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:35.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:34 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:34.960+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:35.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:34 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:34.960+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:35.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:35 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:35.035+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:35.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:35 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:35.035+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:35.391 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:35.281+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:35.391 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:35.281+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:35.391 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:35.073+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:35.392 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:35.073+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:35.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:35.391+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:35.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:35.391+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:35.744 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:35.463+0000 7febe5000700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.218269+0000 front 2024-05-27T07:44:07.218715+0000 (oldest deadline 2024-05-27T07:44:33.020110+0000) 2024-05-27T07:44:35.744 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:35.684+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:36.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:35.884+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:36.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:35.884+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:36.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:36.080+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:36.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:36 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:36.080+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:36.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:35 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:35.970+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:36.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:35 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:35.970+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:36.388 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:36.260+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:36.388 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:36.260+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:36.388 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:36.044+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:36.388 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:36.044+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:36.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:36.389+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:36.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:36.389+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:36.744 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:36 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:36.689+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:36.744 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:36 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:36.451+0000 7febe5000700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.218269+0000 front 2024-05-27T07:44:07.218715+0000 (oldest deadline 2024-05-27T07:44:33.020110+0000) 2024-05-27T07:44:37.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:36 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:36.922+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:37.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:36 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:36.922+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:37.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:37.037+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:37.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:37.037+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:37.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:37.020+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:37.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:37 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:37.020+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:37.377 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:37.297+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:37.377 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:37.297+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:37.378 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:37.094+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:37.378 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:37.094+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:37.696 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:37.467+0000 7febe5000700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.218269+0000 front 2024-05-27T07:44:07.218715+0000 (oldest deadline 2024-05-27T07:44:33.020110+0000) 2024-05-27T07:44:37.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:37.378+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:37.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:37.378+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:37.952 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:37.697+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:38.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:37.952+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:38.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:37 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:37.952+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:38.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:38 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:38.032+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:38.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:38 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:38.032+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:38.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:38 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:38.059+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:38.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:38 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:38.059+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:38.384 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:38.304+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:38.384 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:38.304+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:38.384 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:38.089+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:38.385 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:38.089+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:38.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:38.385+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:38.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:38.385+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:38.917 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:38.673+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:38.917 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4[46600]: 2024-05-27T07:44:38.499+0000 7febe5000700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.218269+0000 front 2024-05-27T07:44:07.218715+0000 (oldest deadline 2024-05-27T07:44:33.020110+0000) 2024-05-27T07:44:39.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:38.917+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:39.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:38 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:38.917+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:39.323 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:39.065+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:39.323 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:39.065+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:39.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:39.107+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:39.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:39.107+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:39.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:39.012+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:39.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:39 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:39.012+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:39.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:39.368+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:39.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:39.368+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:39.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:39.324+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:39.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:39.324+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:39.903 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:39 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:39.664+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:39.903 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:39 smithi062 podman[74876]: 2024-05-27 07:44:39.544684099 +0000 UTC m=+5.845714885 container died 1e8cc33a729ab718e58f7ce5efe6b483775f812ecb0e1d655e899eb61c0f443d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:44:40.245 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:40 smithi062 podman[74876]: 2024-05-27 07:44:40.032599318 +0000 UTC m=+6.333630144 container cleanup 1e8cc33a729ab718e58f7ce5efe6b483775f812ecb0e1d655e899eb61c0f443d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:44:40.245 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:40 smithi062 bash[74876]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4 2024-05-27T07:44:40.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:39 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:39.956+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:40.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:39 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:39.956+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:40.334 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:40.040+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:40.334 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:40.040+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:40.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:40 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:40.036+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:40.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:40 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:40.036+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:40.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:40 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:40.080+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:40.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:40 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:40.080+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:40.696 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:40 smithi062 podman[74894]: 2024-05-27 07:44:40.317511958 +0000 UTC m=+0.756484757 container remove 1e8cc33a729ab718e58f7ce5efe6b483775f812ecb0e1d655e899eb61c0f443d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:44:40.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:40.371+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:40.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:40.371+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:40.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:40.334+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:40.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:40.334+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:40.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:40 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:40.697+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:40.994 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:40 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:40.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:40.994 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:40 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:40.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:41.344 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:41.008+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:41.344 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:41.009+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:41.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:41.095+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:41.348 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:41.095+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:41.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:41.040+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:41.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:41.040+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:41.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:41 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:41.040+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:41.463 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:41 smithi062 podman[74981]: 2024-05-27 07:44:41.08667973 +0000 UTC m=+0.039267178 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:44:41.463 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:41 smithi062 podman[74981]: 2024-05-27 07:44:41.2019335 +0000 UTC m=+0.154520981 container create b6eaf0ed75d5dc5b8efd875545d489c6a3fc54441d9178fce32938781723f422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:44:41.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:41.345+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:41.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:41.345+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:41.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:41.366+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:41.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:41.366+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:41.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:41.366+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:41.744 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:41 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:41.657+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:41.744 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:41 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:41.657+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:41.744 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:41 smithi062 podman[74981]: 2024-05-27 07:44:41.464365796 +0000 UTC m=+0.416953344 container init b6eaf0ed75d5dc5b8efd875545d489c6a3fc54441d9178fce32938781723f422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:44:41.745 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:41 smithi062 podman[74981]: 2024-05-27 07:44:41.475509354 +0000 UTC m=+0.428096798 container start b6eaf0ed75d5dc5b8efd875545d489c6a3fc54441d9178fce32938781723f422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:44:41.745 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:41 smithi062 podman[74981]: 2024-05-27 07:44:41.579488848 +0000 UTC m=+0.532076339 container attach b6eaf0ed75d5dc5b8efd875545d489c6a3fc54441d9178fce32938781723f422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T07:44:42.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:41 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:41.941+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:42.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:41 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:41.941+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:42.245 INFO:journalctl@ceph.osd.4.smithi062.stdout:May 27 07:44:41 smithi062 podman[74981]: 2024-05-27 07:44:41.932564839 +0000 UTC m=+0.885152290 container died b6eaf0ed75d5dc5b8efd875545d489c6a3fc54441d9178fce32938781723f422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:44:42.326 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:42.036+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:42.326 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:42.036+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:42.350 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:42.082+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:42.350 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:42.082+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:42.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:42.005+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:42.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:42.005+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:42.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:42 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:42.005+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:42.710 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.4.service' 2024-05-27T07:44:42.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:42.327+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:42.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:42.327+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:42.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:42.377+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:42.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:42.377+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:42.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:42.377+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:42.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:42.852 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-27T07:44:42.852 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-27T07:44:42.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.5 2024-05-27T07:44:42.931 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:42 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:42.633+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:42.931 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:42 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:42.633+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:43.232 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:42 smithi162 systemd[1]: Stopping Ceph osd.5 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:44:43.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:43.029+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:43.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:43.029+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:43.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:43.029+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:43.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:42 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:42.975+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:43.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:42 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:42.975+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:43.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:43.035+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:43.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:43.035+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:43.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:43.019+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:43.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:43.019+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:43.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:43.019+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:43.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:43.324+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:43.324+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:43.324+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:43.345+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:43.345+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:43.345+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:43.462+0000 7f7677000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:43.462+0000 7f7677000700 -1 osd.5 159 *** Got signal Terminated *** 2024-05-27T07:44:43.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:43.462+0000 7f7677000700 -1 osd.5 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:44:43.979 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:43 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:43.656+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:43.979 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:43 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:43.656+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:44.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:43 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:43.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:44.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:43 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:43.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:44.294 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:43.997+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:44.294 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:43 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:43.997+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:44.294 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:44.019+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:44.294 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:44.019+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:44.294 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:44.019+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:44.316 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:44.002+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:44.316 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:44.002+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:44.316 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:44.002+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:44.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:44.317+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:44.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:44.317+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:44.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:44.317+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:44.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:44.332+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:44.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:44.332+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:44.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:44.332+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:44.969 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:44 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:44.619+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:44.969 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:44 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:44.619+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:45.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:44 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:44.969+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:45.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:44 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:44.969+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:45.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:44 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:44.969+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:45.322 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:45.029+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:45.322 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:45.029+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:45.322 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:45.029+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:45.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:44.961+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:45.350 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:44 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:44.961+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:45.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:45.016+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:45.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:45.016+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:45.351 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:45.016+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:45.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:45.322+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:45.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:45.322+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:45.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:45.322+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:45.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:45.322+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:45.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:45.322+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:45.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:45.322+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:45.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:45.619+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:45.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:45 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:45.619+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:46.336 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:45.989+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:46.336 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:45.989+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:46.336 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:45 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:45.989+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:46.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:45.988+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:46.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:45.988+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:46.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:45 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:45.988+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:46.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:46 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:46.000+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:46.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:46 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:46.000+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:46.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:46 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:46.000+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:46.494 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:46 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:46.000+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:46.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:46 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:46.000+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:46.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:46 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:46.000+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:46.732 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:46.337+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:46.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:46.337+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:46.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:46.337+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:46.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:46.344+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:46.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:46.344+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:46.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:46.344+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:46.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:46 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:46.653+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:46.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:46 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:46.653+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:47.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:46.945+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:47.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:46.945+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:47.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:46 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:46.945+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:47.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:47.018+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:47.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:47.018+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:47.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:47.018+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:47.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:46 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:46.978+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:47.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:46 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:46.978+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:47.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:46 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:46.978+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:47.494 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:47 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:47.046+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:47.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:47 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:47.046+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:47.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:47 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:47.046+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:47.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:47.359+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:47.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:47.359+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:47.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:47.359+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:47.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:47.337+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:47.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:47.337+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:47.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:47.337+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:47.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:47 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:47.607+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:47.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:47 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:47.607+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:48.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:47.911+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:48.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:47.911+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:48.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:47 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:47.911+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:48.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:47.978+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:48.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:47.978+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:48.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:47.978+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:48.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:47.932+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:48.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:47.932+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:48.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:47 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:47.932+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:48.494 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:48 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:48.043+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:48.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:48 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:48.043+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:48.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:48 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:48.043+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:48.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:48.409+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:48.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:48.409+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:48.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:48.409+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:48.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:48.376+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:57.810901+0000 front 2024-05-27T07:43:57.810188+0000 (oldest deadline 2024-05-27T07:44:21.910188+0000) 2024-05-27T07:44:48.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:48.376+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:09.317298+0000 front 2024-05-27T07:44:09.316791+0000 (oldest deadline 2024-05-27T07:44:32.217066+0000) 2024-05-27T07:44:48.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5[47858]: 2024-05-27T07:44:48.376+0000 7f7672800700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:17.223549+0000 front 2024-05-27T07:44:17.223722+0000 (oldest deadline 2024-05-27T07:44:41.323763+0000) 2024-05-27T07:44:48.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:48 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:48.644+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:48.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:48 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:48.644+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:49.232 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:49 smithi162 podman[77584]: 2024-05-27 07:44:49.064427628 +0000 UTC m=+5.747316777 container died c61963426e76d9d2e022b977336ea9ff100a40a15e3259c148d510eb73b87d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T07:44:49.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:48.873+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:49.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:48.873+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:49.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:48 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:48.873+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:49.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:48.974+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:49.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:48.974+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:49.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:48.974+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:49.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:48.955+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:49.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:48.955+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:49.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:48 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:48.955+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:49.494 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:49 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:49.087+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:49.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:49 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:49.087+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:49.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:49 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:49.087+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:49.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:49 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:49.366+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:49.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:49 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:49.366+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:49.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:49 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:49.366+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:49.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:49 smithi162 podman[77584]: 2024-05-27 07:44:49.521141022 +0000 UTC m=+6.204030161 container cleanup c61963426e76d9d2e022b977336ea9ff100a40a15e3259c148d510eb73b87d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:44:49.733 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:49 smithi162 bash[77584]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5 2024-05-27T07:44:49.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:49 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:49.689+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:49.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:49 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:49.689+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:50.232 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:49 smithi162 podman[77604]: 2024-05-27 07:44:49.739199493 +0000 UTC m=+0.663658407 container remove c61963426e76d9d2e022b977336ea9ff100a40a15e3259c148d510eb73b87d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:44:50.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:49 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:49.839+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:50.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:49 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:49.839+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:50.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:49 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:49.839+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:50.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:49.953+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:50.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:49.953+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:50.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:49.953+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:50.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:49.974+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:50.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:49.974+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:50.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:49 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:49.974+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:50.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:50 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:50.059+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:50.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:50 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:50.059+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:50.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:50 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:50.059+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:50.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:50 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:50.059+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:50.572 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:50.344+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:50.572 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:50.344+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:50.573 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:50.344+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:50.861 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:50 smithi162 podman[77691]: 2024-05-27 07:44:50.478655099 +0000 UTC m=+0.033625964 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:44:50.861 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:50 smithi162 podman[77691]: 2024-05-27 07:44:50.598702593 +0000 UTC m=+0.153673485 container create e9fe2b6ce23539db5a93cdd47de5c1eea1ec15b4a799373b71be8f1dedbc69b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:44:50.861 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:50 smithi162 podman[77691]: 2024-05-27 07:44:50.861474996 +0000 UTC m=+0.416445965 container init e9fe2b6ce23539db5a93cdd47de5c1eea1ec15b4a799373b71be8f1dedbc69b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:44:50.861 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:50.820+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:50.861 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:50.820+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:50.862 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:50 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:50.820+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:50.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:50 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:50.688+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:50.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:50 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:50.688+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:51.232 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:50 smithi162 podman[77691]: 2024-05-27 07:44:50.872846751 +0000 UTC m=+0.427817673 container start e9fe2b6ce23539db5a93cdd47de5c1eea1ec15b4a799373b71be8f1dedbc69b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:44:51.233 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:50 smithi162 podman[77691]: 2024-05-27 07:44:50.968229385 +0000 UTC m=+0.523200327 container attach e9fe2b6ce23539db5a93cdd47de5c1eea1ec15b4a799373b71be8f1dedbc69b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:44:51.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:50.988+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:51.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:50.988+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:51.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:50.988+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:51.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:50.971+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:51.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:50.971+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:51.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:50 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:50.971+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:51.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:51 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:51.060+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:51.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:51 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:51.060+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:51.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:51 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:51.060+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:51.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:51 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:51.060+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:51.699 INFO:journalctl@ceph.osd.5.smithi162.stdout:May 27 07:44:51 smithi162 podman[77691]: 2024-05-27 07:44:51.337636575 +0000 UTC m=+0.892607454 container died e9fe2b6ce23539db5a93cdd47de5c1eea1ec15b4a799373b71be8f1dedbc69b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:44:51.700 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:51.383+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:51.700 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:51.383+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:51.700 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:51.383+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:51.982 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:51.827+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:51.983 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:51.827+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:51.983 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:51 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:51.827+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:51.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:51 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:51.715+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:51.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:51 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:51.715+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:52.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.5.service' 2024-05-27T07:44:52.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:52.152 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-27T07:44:52.152 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-27T07:44:52.153 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.6 2024-05-27T07:44:52.235 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:51 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:51.967+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:52.235 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:51 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:51.967+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:52.235 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:51 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:51.967+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:52.236 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:52.010+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:52.236 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:52.010+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:52.236 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:52.010+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:52.359 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:52 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:52.054+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:52.359 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:52 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:52.054+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:52.359 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:52 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:52.054+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:52.359 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:52 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:52.054+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:52.598 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:52 smithi012 systemd[1]: Stopping Ceph osd.6 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:44:52.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:52.427+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:52.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:52.427+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:52.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:52.427+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:52.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:52 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:52.679+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:52.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:52 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:52.679+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:52.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:52 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:52.679+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:53.052 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:52.779+0000 7fa964400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:44:53.052 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:52.780+0000 7fa964400700 -1 osd.6 159 *** Got signal Terminated *** 2024-05-27T07:44:53.052 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:52.780+0000 7fa964400700 -1 osd.6 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:44:53.052 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:52.969+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:53.052 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:52.969+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:53.052 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:52 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:52.969+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:53.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:52.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:53.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:52.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:53.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:52.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:53.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:52 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:52.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:44:53.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:53.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:53.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:53.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:53.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:53.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:53.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:53.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:44:53.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:53 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:53.078+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:53.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:53 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:53.078+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:53.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:53 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:53.078+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:53.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:53 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:53.078+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:53.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:53 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:53.401+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:53.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:53 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:53.401+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:53.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:53 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:53.401+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:53.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:53 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:53.693+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:53.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:53 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:53.693+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:53.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:53 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:53.693+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:54.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:53 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:53.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:54.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:53 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:53.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:54.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:53 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:53.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:54.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:53 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:53.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:44:54.251 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:53.950+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:54.251 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:53.950+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:54.251 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:53.950+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:54.251 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:53 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:53.950+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.948767+0000 front 2024-05-27T07:44:28.948840+0000 (oldest deadline 2024-05-27T07:44:53.049065+0000) 2024-05-27T07:44:54.251 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:54.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:54.252 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:54.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:54.252 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:54.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:54.252 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:54.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:44:54.494 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.030+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:54.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.030+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:54.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.030+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:54.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.030+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:54.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:54 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:54.433+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:54.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:54 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:54.433+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:54.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:54 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:54.433+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:54.983 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:54.683+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:54.984 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:54.683+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:54.984 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:54.683+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:55.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:54 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:54.821+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:55.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:54 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:54.821+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:55.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:54 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:54.821+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:55.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:54 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:54.821+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:44:55.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.984+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:55.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.984+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:55.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.984+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:55.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:54 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:54.984+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:55.285 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:55.000+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:55.285 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:55.000+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:55.285 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:55.000+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:55.285 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:55.000+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:44:55.285 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:54.993+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:55.285 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:54.993+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:55.285 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:54.993+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:55.286 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:54 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:54.993+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.948767+0000 front 2024-05-27T07:44:28.948840+0000 (oldest deadline 2024-05-27T07:44:53.049065+0000) 2024-05-27T07:44:55.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:55.391+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:55.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:55.391+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:55.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:55.391+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:55.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:55 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:55.730+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:55.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:55 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:55.730+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:55.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:55 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:55.730+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:56.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:55.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:56.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:55.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:56.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:55.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:56.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:55 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:55.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:44:56.284 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.013+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:56.284 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.013+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:56.284 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.013+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:56.284 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.013+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:56.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:55.976+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:56.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:55.976+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:56.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:55.976+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:56.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:55 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:55.976+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.948767+0000 front 2024-05-27T07:44:28.948840+0000 (oldest deadline 2024-05-27T07:44:53.049065+0000) 2024-05-27T07:44:56.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:56 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:56.028+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:56.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:56 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:56.028+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:56.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:56 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:56.028+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:56.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:56 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:56.028+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:44:56.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:56.426+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:56.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:56.426+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:56.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:56.426+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:56.980 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:56.725+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:56.980 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:56.725+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:56.980 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:56.725+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:57.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:56.838+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:57.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:56.838+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:57.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:56.838+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:57.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:56 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:56.838+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:44:57.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:57.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:57.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:57.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:56 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:56.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:57.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:57.002+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.538611+0000 front 2024-05-27T07:43:58.538382+0000 (oldest deadline 2024-05-27T07:44:22.539747+0000) 2024-05-27T07:44:57.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:57.002+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.442095+0000 front 2024-05-27T07:44:08.442523+0000 (oldest deadline 2024-05-27T07:44:31.342413+0000) 2024-05-27T07:44:57.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:57.002+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:13.644862+0000 front 2024-05-27T07:44:13.644796+0000 (oldest deadline 2024-05-27T07:44:45.446574+0000) 2024-05-27T07:44:57.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6[59432]: 2024-05-27T07:44:57.002+0000 7fa95fc00700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.948767+0000 front 2024-05-27T07:44:28.948840+0000 (oldest deadline 2024-05-27T07:44:53.049065+0000) 2024-05-27T07:44:57.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:57.014+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:57.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:57.014+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:57.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:57.014+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:57.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:57 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:57.014+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:44:57.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:57 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:57.390+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:57.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:57 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:57.390+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:57.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:57 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:57.390+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:58.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:57 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:57.817+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:58.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:57 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:57.817+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:58.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:57 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:57.817+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:58.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:57 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:57.817+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:44:58.245 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:57 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:57.764+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:58.245 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:57 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:57.764+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:58.245 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:57 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:57.764+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:58.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.010+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:58.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.010+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:58.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.010+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:58.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.010+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:58.348 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:58 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:58.063+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:58.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:58 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:58.063+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:58.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:58 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:58.063+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:58.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:58 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:58.063+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:44:58.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:58 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:58.403+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:58.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:58 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:58.403+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:58.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:58 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:58.403+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:58.745 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:58 smithi012 podman[145452]: 2024-05-27 07:44:58.407284493 +0000 UTC m=+5.765697409 container died cbca1932ba4f408484fa42cd3ff35d04d35bfe816821fa5c0dbdf00ec54ad6fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-27T07:44:59.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:58 smithi012 podman[145452]: 2024-05-27 07:44:58.964575116 +0000 UTC m=+6.322988056 container cleanup cbca1932ba4f408484fa42cd3ff35d04d35bfe816821fa5c0dbdf00ec54ad6fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:44:59.053 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:58 smithi012 bash[145452]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6 2024-05-27T07:44:59.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:58 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:58.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:44:59.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:58 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:58.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:44:59.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:58 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:58.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:44:59.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:58 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:58.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:44:59.244 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:58.759+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:59.244 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:58.759+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:59.244 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:58.759+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:59.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:59.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:59.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:59.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:58.979+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:44:59.309 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:59 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:59.053+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:44:59.309 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:59 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:59.053+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:44:59.309 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:59 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:59.053+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:44:59.309 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:44:59 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:44:59.053+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:44:59.598 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:44:59 smithi012 podman[145485]: 2024-05-27 07:44:59.308404775 +0000 UTC m=+0.882708737 container remove cbca1932ba4f408484fa42cd3ff35d04d35bfe816821fa5c0dbdf00ec54ad6fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T07:44:59.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:59.366+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:44:59.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:59.366+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:44:59.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:44:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:44:59.366+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:44:59.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:59.714+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:44:59.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:59.714+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:44:59.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:44:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:44:59.714+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:44:59.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:59.931+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:44:59.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:59.931+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:44:59.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:59.931+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:44:59.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:44:59 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:44:59.931+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:00.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:59.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:00.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:59.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:00.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:59.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:00.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:44:59 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:44:59.866+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:00.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:45:00 smithi012 podman[145572]: 2024-05-27 07:45:00.124665246 +0000 UTC m=+0.036068100 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:00.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:45:00 smithi012 podman[145572]: 2024-05-27 07:45:00.22568416 +0000 UTC m=+0.137087045 container create 75d1f036ba2e9a979f0c195b58ea72a08bb4a103569f936bbcba4f6e3dda7387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:45:00.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:00.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:00.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:00.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:00.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:00.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:00.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:00 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:00.051+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:00.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:00.363+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:00.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:00.363+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:00.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:00.363+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:00.734 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:00.363+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:00.849 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:45:00 smithi012 podman[145572]: 2024-05-27 07:45:00.49605942 +0000 UTC m=+0.407462288 container init 75d1f036ba2e9a979f0c195b58ea72a08bb4a103569f936bbcba4f6e3dda7387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:45:00.849 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:45:00 smithi012 podman[145572]: 2024-05-27 07:45:00.507930772 +0000 UTC m=+0.419333663 container start 75d1f036ba2e9a979f0c195b58ea72a08bb4a103569f936bbcba4f6e3dda7387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-27T07:45:00.849 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:45:00 smithi012 podman[145572]: 2024-05-27 07:45:00.628265114 +0000 UTC m=+0.539668046 container attach 75d1f036ba2e9a979f0c195b58ea72a08bb4a103569f936bbcba4f6e3dda7387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:00.686+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:00.686+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:00.686+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:00.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:00.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:00.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:00.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:00.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:00 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:00.930+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:01.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:00.871+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:01.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:00.871+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:01.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:00.871+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:01.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:00 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:00.871+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:01.349 INFO:journalctl@ceph.osd.6.smithi012.stdout:May 27 07:45:00 smithi012 podman[145572]: 2024-05-27 07:45:00.983828375 +0000 UTC m=+0.895231222 container died 75d1f036ba2e9a979f0c195b58ea72a08bb4a103569f936bbcba4f6e3dda7387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:45:01.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:01 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:01.057+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:01.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:01 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:01.057+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:01.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:01 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:01.057+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:01.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:01 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:01.057+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:01.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:01 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:01.057+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:01.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:01.365+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:01.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:01.365+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:01.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:01.365+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:01.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:01.365+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:01.994 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:01.645+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:01.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:01.645+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:01.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:01.645+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:01.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:01.892+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:01.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:01.892+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:01.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:01.892+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:01.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:01.892+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:01.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:01 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:01.892+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:02.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.6.service' 2024-05-27T07:45:02.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:02.201 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-27T07:45:02.201 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-27T07:45:02.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.7 2024-05-27T07:45:02.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:01.850+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:02.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:01.850+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:02.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:01.850+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:02.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:01 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:01.850+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:02.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:02 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:02.103+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:02.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:02 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:02.103+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:02.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:02 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:02.103+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:02.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:02 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:02.103+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:02.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:02 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:02.103+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:02.645 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 systemd[1]: Stopping Ceph osd.7 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:45:02.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:02.388+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:02.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:02.388+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:02.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:02.388+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:02.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:02.388+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:02.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:02.862+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:02.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:02.862+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:02.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:02.862+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:02.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:02.862+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:02.862+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:02.645+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:02.645+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:02.645+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:02.645+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.032610+0000 front 2024-05-27T07:44:37.032771+0000 (oldest deadline 2024-05-27T07:45:02.332787+0000) 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:02.819+0000 7f18a9c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:02.819+0000 7f18a9c00700 -1 osd.7 159 *** Got signal Terminated *** 2024-05-27T07:45:02.996 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:02 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:02.819+0000 7f18a9c00700 -1 osd.7 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:45:03.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:02.807+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:03.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:02.807+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:03.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:02.807+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:03.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:02 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:02.807+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:03.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:03.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:03.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:03.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:03.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:03.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:03.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:03.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:03.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:03 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:03.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:03.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:03.343+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:03.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:03.343+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:03.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:03.343+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:03.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:03.343+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:03.908 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:03.629+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:03.908 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:03.629+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:03.908 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:03.630+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:03.908 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:03.630+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.032610+0000 front 2024-05-27T07:44:37.032771+0000 (oldest deadline 2024-05-27T07:45:02.332787+0000) 2024-05-27T07:45:04.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:03.815+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:04.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:03.815+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:04.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:03.815+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:04.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:03 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:03.815+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:04.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:03.907+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:04.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:03.907+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:04.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:03.907+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:04.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:03.907+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:04.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:03.907+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:04.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:04 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:04.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:04.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:04 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:04.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:04.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:04 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:04.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:04.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:04 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:04.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:04.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:04 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:04.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:04.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:04.297+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:04.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:04.297+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:04.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:04.297+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:04.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:04.297+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:04.873 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:04.586+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:04.873 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:04.586+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:04.873 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:04.586+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:04.873 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:04.586+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.032610+0000 front 2024-05-27T07:44:37.032771+0000 (oldest deadline 2024-05-27T07:45:02.332787+0000) 2024-05-27T07:45:05.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:04.786+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:05.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:04.786+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:05.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:04.786+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:05.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:04.786+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:05.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:04 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:04.786+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:05.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:04.872+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:05.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:04.872+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:05.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:04.872+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:05.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:04.872+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:05.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:04 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:04.872+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:05.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:05.074+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:05.869 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:05.074+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:05.869 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:05.074+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:05.869 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:05.074+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:05.869 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:05 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:05.074+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:05.870 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:05.563+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:05.870 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:05.563+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:05.870 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:05.563+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:05.870 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:05.563+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.032610+0000 front 2024-05-27T07:44:37.032771+0000 (oldest deadline 2024-05-27T07:45:02.332787+0000) 2024-05-27T07:45:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:05.326+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:05.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:05.326+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:05.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:05.326+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:05.871 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:05.326+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:06.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:05.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:06.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:05.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:06.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:05.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:06.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:05.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:06.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:05 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:05.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:06.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:05.842+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:06.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:05.842+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:06.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:05.842+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:06.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:05.842+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:06.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:05 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:05.842+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:06.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:06.069+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:06.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:06.069+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:06.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:06.069+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:06.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:06.069+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:06.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:06 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:06.069+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:06.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:06.368+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:06.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:06.368+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:06.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:06.368+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:06.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:06.368+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:06.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:06.600+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:06.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:06.600+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:06.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:06.600+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:06.995 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:06.601+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.032610+0000 front 2024-05-27T07:44:37.032771+0000 (oldest deadline 2024-05-27T07:45:02.332787+0000) 2024-05-27T07:45:06.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:06.805+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:06.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:06.805+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:06.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:06.805+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:06.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:06.805+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:06.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:06 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:06.805+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:07.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:06.843+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:07.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:06.843+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:07.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:06.843+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:07.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:06.843+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:07.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:06 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:06.843+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:07.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:07.029+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:07.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:07.029+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:07.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:07.029+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:07.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:07.029+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:07.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:07 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:07.029+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:07.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:07.378+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:07.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:07.378+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:07.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:07.378+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:07.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:07.378+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:07.745 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:07.576+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.916708+0000 front 2024-05-27T07:44:07.916796+0000 (oldest deadline 2024-05-27T07:44:33.816940+0000) 2024-05-27T07:45:07.745 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:07.576+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.520682+0000 front 2024-05-27T07:44:18.520770+0000 (oldest deadline 2024-05-27T07:44:40.820895+0000) 2024-05-27T07:45:07.745 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:07.576+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:27.825815+0000 front 2024-05-27T07:44:27.826133+0000 (oldest deadline 2024-05-27T07:44:51.925709+0000) 2024-05-27T07:45:07.745 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7[53553]: 2024-05-27T07:45:07.576+0000 7f18a0400700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.032610+0000 front 2024-05-27T07:44:37.032771+0000 (oldest deadline 2024-05-27T07:45:02.332787+0000) 2024-05-27T07:45:08.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:07.874+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:10.041 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:07.874+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:10.041 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:07.874+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:10.041 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:07.874+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:10.041 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:07 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:07.874+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:10.043 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:07.837+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:10.043 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:07.837+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:10.043 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:07.837+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:10.043 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:07.837+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:10.043 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:07 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:07.837+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:10.043 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:08.428+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:10.043 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:08.428+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:08.428+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:08.428+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:08.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:08.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:08.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:08.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:08 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:08.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:09.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:09.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:10.044 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:09.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:09.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:09 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:09.076+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:08 smithi062 podman[75109]: 2024-05-27 07:45:08.554692334 +0000 UTC m=+5.887228735 container died a45fa7621ee4da7a0a11dfd94d715c3be81fbbda5c69864806f954c9f226e95d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:08.835+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:08.835+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:08.835+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:08.835+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:10.045 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:08 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:08.835+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:08 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:08.794+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:08 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:08.794+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:08 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:08.794+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:08 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:08.794+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:08 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:08.794+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:09.433+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:09.433+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:09.433+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:10.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:09.433+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:08 smithi062 podman[75109]: 2024-05-27 07:45:08.970593106 +0000 UTC m=+6.303129504 container cleanup a45fa7621ee4da7a0a11dfd94d715c3be81fbbda5c69864806f954c9f226e95d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:08 smithi062 bash[75109]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:09 smithi062 podman[75127]: 2024-05-27 07:45:09.238478221 +0000 UTC m=+0.665511518 container remove a45fa7621ee4da7a0a11dfd94d715c3be81fbbda5c69864806f954c9f226e95d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:09 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:09.758+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:09 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:09.758+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:09 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:09.758+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:09 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:09.758+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:10.047 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:09 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:09.758+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:10.048 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:09 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:09.758+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:10.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:09.856+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:10.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:09.856+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:10.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:09.856+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:10.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:09.856+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:10.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:09 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:09.856+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:10.276 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:10 smithi062 podman[75212]: 2024-05-27 07:45:09.92776881 +0000 UTC m=+0.040548167 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:10.276 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:10 smithi062 podman[75212]: 2024-05-27 07:45:10.049169314 +0000 UTC m=+0.161948727 container create e383b1fd53f9dd385e5c1e255214d4092b1806d46d50cff45cd98b2a469f2197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:45:10.276 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:10 smithi062 podman[75212]: 2024-05-27 07:45:10.276704154 +0000 UTC m=+0.389483603 container init e383b1fd53f9dd385e5c1e255214d4092b1806d46d50cff45cd98b2a469f2197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-27T07:45:10.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:10 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:10.036+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:10.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:10 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:10.036+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:10.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:10 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:10.036+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:10.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:10 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:10.036+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:10.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:10 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:10.036+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:10.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:10.408+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:10.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:10.408+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:10.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:10.408+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:10.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:10.408+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:10.745 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:10 smithi062 podman[75212]: 2024-05-27 07:45:10.289528429 +0000 UTC m=+0.402307841 container start e383b1fd53f9dd385e5c1e255214d4092b1806d46d50cff45cd98b2a469f2197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:45:10.745 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:10 smithi062 podman[75212]: 2024-05-27 07:45:10.383744065 +0000 UTC m=+0.496523439 container attach e383b1fd53f9dd385e5c1e255214d4092b1806d46d50cff45cd98b2a469f2197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:45:11.127 INFO:journalctl@ceph.osd.7.smithi062.stdout:May 27 07:45:10 smithi062 podman[75238]: 2024-05-27 07:45:10.817910372 +0000 UTC m=+0.033302713 container died e383b1fd53f9dd385e5c1e255214d4092b1806d46d50cff45cd98b2a469f2197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:45:11.128 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:10 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:10.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:11.128 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:10 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:10.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:11.128 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:10 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:10.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:11.128 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:10 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:10.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:11.128 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:10 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:10.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:11.128 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:10 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:10.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:11.232 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:10.810+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:11.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:10.810+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:11.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:10.810+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:11.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:10.810+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:11.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:10 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:10.810+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:11.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:11.054+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:11.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:11.054+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:11.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:11.054+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:11.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:11.054+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:11.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:11.054+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:11.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:11 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:11.054+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:11.538 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.7.service' 2024-05-27T07:45:11.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:11.661 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-27T07:45:11.661 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-27T07:45:11.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.8 2024-05-27T07:45:11.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:11.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:11.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:11.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:11.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:11.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:11.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:11.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:12.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:11 smithi162 systemd[1]: Stopping Ceph osd.8 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:45:12.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:11.795+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:12.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:11.795+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:12.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:11.795+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:12.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:11.795+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:12.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:11.795+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:12.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:11 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:11.795+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.046636+0000 front 2024-05-27T07:44:47.046697+0000 (oldest deadline 2024-05-27T07:45:11.146822+0000) 2024-05-27T07:45:12.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:11 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:11.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:12.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:11 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:11.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:12.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:11 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:11.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:12.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:11 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:11.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:12.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:11 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:11.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:12.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:11 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:11.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:12.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:12.011+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:12.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:12.011+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:12.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:12.011+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:12.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:12.011+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:12.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:12.011+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:12.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:12 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:12.011+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:12.732 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.245+0000 7fe98a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:45:12.733 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.245+0000 7fe98a000700 -1 osd.8 159 *** Got signal Terminated *** 2024-05-27T07:45:12.733 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.245+0000 7fe98a000700 -1 osd.8 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:45:12.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:12.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:12.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:12.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:12.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:12.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:12.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:12.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:12.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:12.430+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:12.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:12.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:12.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:12.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:12.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:12.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:12.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:12.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:12.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:12.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:12.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:12 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:12.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:13.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.805+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:13.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.805+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:13.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.805+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:13.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.805+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:13.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.805+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:13.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:12 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:12.805+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.046636+0000 front 2024-05-27T07:44:47.046697+0000 (oldest deadline 2024-05-27T07:45:11.146822+0000) 2024-05-27T07:45:13.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:13.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:13.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:13.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:13.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:13.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:13.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:13.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:13.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:13.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:13.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:13 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:13.046+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:13.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:13.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:13.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:13.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:13.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:13.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:13.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:13.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:13.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:13.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:14.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:13.794+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:14.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:13.794+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:14.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:13.794+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:14.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:13.794+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:14.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:13.794+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:14.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:13 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:13.794+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.046636+0000 front 2024-05-27T07:44:47.046697+0000 (oldest deadline 2024-05-27T07:45:11.146822+0000) 2024-05-27T07:45:14.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:13.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:14.244 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:13.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:14.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:13.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:14.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:13.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:14.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:13.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:14.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:13 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:13.759+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:14.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:14 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:14.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:14.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:14 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:14.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:14.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:14 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:14.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:14.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:14 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:14.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:14.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:14 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:14.080+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:14.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:14 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:14.080+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:14.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:14.437+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:14.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:14.437+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:14.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:14.437+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:14.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:14.437+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:14.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:14.437+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:15.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:14.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:15.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:14.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:15.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:14.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:15.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:14.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:15.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:14.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:15.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:14 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:14.844+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.046636+0000 front 2024-05-27T07:44:47.046697+0000 (oldest deadline 2024-05-27T07:45:11.146822+0000) 2024-05-27T07:45:15.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:14.751+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:15.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:14.751+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:15.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:14.751+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:15.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:14.751+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:15.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:14.751+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:15.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:14 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:14.751+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:15.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:15.091+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:15.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:15.091+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:15.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:15.091+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:15.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:15.091+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:15.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:15.091+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:15.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:15 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:15.091+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:15.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:15.480+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:15.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:15.480+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:15.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:15.480+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:15.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:15.480+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:15.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:15.480+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:15.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:15.742+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:15.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:15.742+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:15.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:15.742+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:15.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:15.742+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:15.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:15.742+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:15.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:15 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:15.742+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:16.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:15.833+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:16.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:15.833+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:16.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:15.833+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:16.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:15.833+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:16.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:15.833+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:16.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:15 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:15.833+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.046636+0000 front 2024-05-27T07:44:47.046697+0000 (oldest deadline 2024-05-27T07:45:11.146822+0000) 2024-05-27T07:45:16.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:16.045+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:16.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:16.045+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:16.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:16.045+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:16.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:16.045+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:16.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:16.045+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:16.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:16 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:16.045+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:16.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:16.460+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:16.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:16.460+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:16.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:16.460+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:16.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:16.460+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:16.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:16.460+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:16.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:16.735+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:16.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:16.735+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:16.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:16.735+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:16.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:16.735+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:16.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:16.735+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:16.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:16 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:16.735+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:17.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:16.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.427592+0000 front 2024-05-27T07:43:59.428152+0000 (oldest deadline 2024-05-27T07:44:25.327669+0000) 2024-05-27T07:45:17.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:16.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.131775+0000 front 2024-05-27T07:44:08.131745+0000 (oldest deadline 2024-05-27T07:44:31.031755+0000) 2024-05-27T07:45:17.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:16.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.636190+0000 front 2024-05-27T07:44:18.636063+0000 (oldest deadline 2024-05-27T07:44:42.736199+0000) 2024-05-27T07:45:17.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:16.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.940969+0000 front 2024-05-27T07:44:28.941060+0000 (oldest deadline 2024-05-27T07:44:51.841239+0000) 2024-05-27T07:45:17.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:16.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:38.845283+0000 front 2024-05-27T07:44:38.844707+0000 (oldest deadline 2024-05-27T07:45:04.144830+0000) 2024-05-27T07:45:17.233 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:16 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8[54521]: 2024-05-27T07:45:16.825+0000 7fe985800700 -1 osd.8 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.046636+0000 front 2024-05-27T07:44:47.046697+0000 (oldest deadline 2024-05-27T07:45:11.146822+0000) 2024-05-27T07:45:17.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:17.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:17.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:17.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:17.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:17.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:17.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:17.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:17.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:17.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:17.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:17 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:17.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:17.783 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:17.500+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:17.783 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:17.500+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:17.783 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:17.500+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:17.784 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:17.500+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:17.784 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:17 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:17.500+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:17.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:17.710+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:17.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:17.710+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:17.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:17.710+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:17.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:17.710+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:17.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:17.710+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:17.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:17 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:17.710+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:18.078 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:17 smithi162 podman[77808]: 2024-05-27 07:45:17.784444077 +0000 UTC m=+5.677513767 container died bdc96db15df114ce4998718a35d21ed5e3c452b443b81b136ebb5f5efed518e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:45:18.483 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:18 smithi162 podman[77808]: 2024-05-27 07:45:18.172357943 +0000 UTC m=+6.065427596 container cleanup bdc96db15df114ce4998718a35d21ed5e3c452b443b81b136ebb5f5efed518e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:45:18.483 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:18 smithi162 bash[77808]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8 2024-05-27T07:45:18.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:18.113+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:18.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:18.113+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:18.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:18.113+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:18.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:18.113+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:18.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:18.113+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:18.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:18 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:18.113+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:18.982 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:18 smithi162 podman[77824]: 2024-05-27 07:45:18.490272936 +0000 UTC m=+0.686310378 container remove bdc96db15df114ce4998718a35d21ed5e3c452b443b81b136ebb5f5efed518e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True) 2024-05-27T07:45:18.982 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:18.501+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:18.982 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:18.501+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:18.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:18.501+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:18.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:18.501+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:18.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:18 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:18.501+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:19.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:18.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:19.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:18.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:19.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:18.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:19.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:18.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:19.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:18.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:19.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:18 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:18.755+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:19.346 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:19 smithi162 podman[77910]: 2024-05-27 07:45:18.983106288 +0000 UTC m=+0.033766736 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:19.346 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:19 smithi162 podman[77910]: 2024-05-27 07:45:19.083050549 +0000 UTC m=+0.133711034 container create 1066ac7c581c1caa3550e48ac15157e4d3bb5bce9022179e76ddf50a1c4ac56f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:45:19.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:19.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:19.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:19.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:19.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:19.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:19.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:19.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:19.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:19.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:19.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:19 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:19.079+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:19.732 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:19 smithi162 podman[77910]: 2024-05-27 07:45:19.345643414 +0000 UTC m=+0.396303882 container init 1066ac7c581c1caa3550e48ac15157e4d3bb5bce9022179e76ddf50a1c4ac56f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:19.733 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:19 smithi162 podman[77910]: 2024-05-27 07:45:19.359262432 +0000 UTC m=+0.409922905 container start 1066ac7c581c1caa3550e48ac15157e4d3bb5bce9022179e76ddf50a1c4ac56f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:45:19.733 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:19 smithi162 podman[77910]: 2024-05-27 07:45:19.427297003 +0000 UTC m=+0.477957488 container attach 1066ac7c581c1caa3550e48ac15157e4d3bb5bce9022179e76ddf50a1c4ac56f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:45:19.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:19.539+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:19.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:19.539+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:19.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:19.539+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:19.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:19.539+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:19.734 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:19 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:19.539+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:20.159 INFO:journalctl@ceph.osd.8.smithi162.stdout:May 27 07:45:19 smithi162 podman[77936]: 2024-05-27 07:45:19.857195376 +0000 UTC m=+0.030843774 container died 1066ac7c581c1caa3550e48ac15157e4d3bb5bce9022179e76ddf50a1c4ac56f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:45:20.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:19.801+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:20.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:19.801+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:20.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:19.801+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:20.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:19.801+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:20.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:19.801+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:20.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:19 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:19.801+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:20.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.8.service' 2024-05-27T07:45:20.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:20.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:20.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:20.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:20.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:20.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:20.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:20.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:20.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:20.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:20.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:20 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:20.109+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:20.645 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:20.646 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-27T07:45:20.646 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-27T07:45:20.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.9 2024-05-27T07:45:20.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:20.503+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:20.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:20.503+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:20.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:20.503+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:20.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:20.503+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:20.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:20 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:20.503+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:21.098 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:20 smithi012 systemd[1]: Stopping Ceph osd.9 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:45:21.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:20.817+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:21.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:20.817+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:21.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:20.817+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:21.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:20.817+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:21.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:20.817+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:21.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:20.817+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:21.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:20 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:20.817+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:21.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.154+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.154+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.154+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.154+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.154+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.154+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.207+0000 7fded9600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.207+0000 7fded9600700 -1 osd.9 159 *** Got signal Terminated *** 2024-05-27T07:45:21.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:21 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:21.207+0000 7fded9600700 -1 osd.9 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:45:21.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:21.481+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:21.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:21.481+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:21.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:21.481+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:21.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:21.481+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:21.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:21 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:21.481+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:22.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:21.826+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:22.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:21.826+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:22.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:21.826+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:22.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:21.826+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:22.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:21.826+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:22.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:21.826+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:22.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:21 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:21.826+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:22.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:22.124+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:22.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:22.124+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:22.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:22.124+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:22.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:22.124+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:22.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:22.124+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:22.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:22 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:22.124+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:22.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:22.476+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:22.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:22.476+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:22.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:22.476+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:22.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:22.476+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:22.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:22.476+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:22.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:22 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:22.476+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:23.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:22.797+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:23.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:22.797+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:23.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:22.797+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:23.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:22.797+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:23.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:22.797+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:23.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:22.797+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:23.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:22 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:22.797+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:23.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:23.086+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:23.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:23.086+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:23.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:23.086+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:23.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:23.086+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:23.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:23.086+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:23.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:23 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:23.086+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:23.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:23 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:23.443+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:23.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:23 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:23.443+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:23.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:23 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:23.443+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:23.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:23 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:23.443+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:23.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:23 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:23.443+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:23.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:23 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:23.443+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:24.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:23 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:23.800+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:24.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:23 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:23.800+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:24.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:23 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:23.800+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:24.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:23 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:23.800+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:24.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:23 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:23.800+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:24.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:23 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:23.800+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:24.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:23 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:23.800+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:24.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:24.128+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:24.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:24.128+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:24.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:24.128+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:24.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:24.128+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:24.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:24.128+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:24.600 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:24 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:24.128+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:24.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:24.483+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:24.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:24.483+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:24.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:24.483+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:24.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:24.483+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:24.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:24.483+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:24.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:24 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:24.483+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:25.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:24.780+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:25.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:24.780+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:25.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:24.781+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:25.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:24.781+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:25.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:24.781+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:25.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:24.781+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:25.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:24 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:24.781+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:25.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:25.129+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:25.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:25.129+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:25.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:25.129+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:25.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:25.129+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:25.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:25.130+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:25.599 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:25 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:25.130+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:25.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:25.440+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:25.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:25.440+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:25.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:25.440+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:25.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:25.440+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:25.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:25.440+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:25.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:25 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:25.440+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:26.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:25.760+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:26.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:25.760+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:26.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:25.760+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:26.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:25.760+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:26.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:25.760+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:26.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:25.760+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:26.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:25 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:25.760+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:26.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:26.164+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.259586+0000 front 2024-05-27T07:43:59.259904+0000 (oldest deadline 2024-05-27T07:44:24.559463+0000) 2024-05-27T07:45:26.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:26.164+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:08.662132+0000 front 2024-05-27T07:44:08.662200+0000 (oldest deadline 2024-05-27T07:44:31.562432+0000) 2024-05-27T07:45:26.349 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:26.164+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:15.368420+0000 front 2024-05-27T07:44:15.368356+0000 (oldest deadline 2024-05-27T07:44:40.668580+0000) 2024-05-27T07:45:26.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:26.164+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:26.570822+0000 front 2024-05-27T07:44:26.571158+0000 (oldest deadline 2024-05-27T07:44:52.471118+0000) 2024-05-27T07:45:26.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:26.164+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.573738+0000 front 2024-05-27T07:44:36.573849+0000 (oldest deadline 2024-05-27T07:45:00.673260+0000) 2024-05-27T07:45:26.350 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:26 smithi012 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9[66288]: 2024-05-27T07:45:26.164+0000 7fdecfe00700 -1 osd.9 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:48.177685+0000 front 2024-05-27T07:44:48.177490+0000 (oldest deadline 2024-05-27T07:45:10.477601+0000) 2024-05-27T07:45:26.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:26.461+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:26.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:26.461+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:26.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:26.461+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:26.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:26.461+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:26.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:26.461+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:26.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:26 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:26.461+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:26.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:26.737+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:26.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:26.737+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:26.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:26.737+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:26.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:26.737+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:26.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:26.737+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:26.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:26.737+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:26.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:26 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:26.737+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:27.269 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:26 smithi012 podman[145677]: 2024-05-27 07:45:26.994442809 +0000 UTC m=+5.923921333 container died 453f2bb9eec6ecde50fbe4078c220d5e52f0c80726cdb65d63f898079e18f224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:45:27.535 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:27 smithi012 podman[145677]: 2024-05-27 07:45:27.506897477 +0000 UTC m=+6.436376041 container cleanup 453f2bb9eec6ecde50fbe4078c220d5e52f0c80726cdb65d63f898079e18f224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:27.535 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:27 smithi012 bash[145677]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9 2024-05-27T07:45:27.849 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:27 smithi012 podman[145705]: 2024-05-27 07:45:27.6748867 +0000 UTC m=+0.661346254 container remove 453f2bb9eec6ecde50fbe4078c220d5e52f0c80726cdb65d63f898079e18f224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:45:27.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:27.497+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:27.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:27.497+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:27.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:27.497+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:27.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:27.497+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:27.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:27.497+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:27.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:27 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:27.497+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:27.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:27.720+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:27.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:27.720+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:27.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:27.720+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:27.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:27.720+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:27.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:27.720+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:27.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:27.720+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:27.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:27 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:27.720+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:28.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:28.469+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:28.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:28.469+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:28.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:28.469+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:28.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:28.469+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:28.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:28.469+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:28.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:28 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:28.469+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:28.733 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:28 smithi012 podman[145792]: 2024-05-27 07:45:28.376704073 +0000 UTC m=+0.033863138 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:28.734 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:28 smithi012 podman[145792]: 2024-05-27 07:45:28.475944227 +0000 UTC m=+0.133103334 container create 9c45d0cb2a49d4c5bb66f1c72da37edd92057aeb35a1b8535ba11895d38526e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:45:28.734 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:28 smithi012 podman[145792]: 2024-05-27 07:45:28.721294967 +0000 UTC m=+0.378454050 container init 9c45d0cb2a49d4c5bb66f1c72da37edd92057aeb35a1b8535ba11895d38526e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-27T07:45:28.734 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:28 smithi012 podman[145792]: 2024-05-27 07:45:28.733302125 +0000 UTC m=+0.390461234 container start 9c45d0cb2a49d4c5bb66f1c72da37edd92057aeb35a1b8535ba11895d38526e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:45:28.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:28.717+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:28.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:28.717+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:28.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:28.717+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:28.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:28.717+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:28.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:28.717+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:28.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:28.717+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:28.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:28 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:28.717+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:29.099 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:28 smithi012 podman[145792]: 2024-05-27 07:45:28.836598679 +0000 UTC m=+0.493757770 container attach 9c45d0cb2a49d4c5bb66f1c72da37edd92057aeb35a1b8535ba11895d38526e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:45:29.542 INFO:journalctl@ceph.osd.9.smithi012.stdout:May 27 07:45:29 smithi012 podman[145792]: 2024-05-27 07:45:29.208019377 +0000 UTC m=+0.865178517 container died 9c45d0cb2a49d4c5bb66f1c72da37edd92057aeb35a1b8535ba11895d38526e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:45:29.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:29.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:29.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:29.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:29.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:29.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:29.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:29.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:29.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:29.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:29.734 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:29 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:29.455+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:29.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:29.684+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:29.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:29.684+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:29.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:29.684+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:29.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:29.684+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:29.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:29.684+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:29.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:29.684+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:29.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:29 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:29.684+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:30.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.9.service' 2024-05-27T07:45:30.128 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:30.128 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-27T07:45:30.129 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-27T07:45:30.129 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.10 2024-05-27T07:45:30.494 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 systemd[1]: Stopping Ceph osd.10 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:45:30.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:30.491+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:30.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:30.491+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:30.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:30.491+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:30.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:30.491+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:30.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:30.491+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:30.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:30.491+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:30.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:30 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:30.491+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:30.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.691+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:30.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.691+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:30.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.691+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:30.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.691+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:30.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.691+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:30.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.691+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:30.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.691+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:30.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.715+0000 7fd380200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:45:30.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.715+0000 7fd380200700 -1 osd.10 159 *** Got signal Terminated *** 2024-05-27T07:45:30.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:30 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:30.715+0000 7fd380200700 -1 osd.10 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:45:31.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:31.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:31.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:31.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:31.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:31.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:31.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:31.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:31.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:31.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:31.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:31.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:31.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:31 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:31.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:31.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:31.736+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:31.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:31.736+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:31.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:31.736+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:31.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:31.736+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:31.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:31.736+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:31.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:31.736+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:31.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:31 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:31.736+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:32.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:32.496+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:32.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:32.496+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:32.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:32.496+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:32.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:32.496+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:32.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:32.496+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:32.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:32.496+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:32.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:32 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:32.496+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:33.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:32.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:33.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:32.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:33.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:32.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:33.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:32.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:33.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:32.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:33.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:32.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:33.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:32 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:32.775+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:33.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:33.499+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:33.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:33.499+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:33.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:33.499+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:33.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:33.499+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:33.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:33.499+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:33.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:33.499+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:33.733 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:33 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:33.499+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:33.766+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:33.766+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:33.766+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:33.766+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:33.766+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:33.766+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:34.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:33 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:33.766+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:34.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:34.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:34.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:34.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:34.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:34.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:34.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:34.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:34.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:34.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:34.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:34.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:34.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:34 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:34.504+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:35.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:34.757+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:35.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:34.757+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:35.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:34.757+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:35.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:34.757+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:35.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:34.757+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:35.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:34.757+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:35.246 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:34 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:34.757+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:35.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:35.550+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:35.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:35.550+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:35.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:35.550+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:35.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:35.550+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:35.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:35.550+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:35.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:35.550+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:35.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:35 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:35.550+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:35.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:35.712+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:58.883303+0000 front 2024-05-27T07:43:58.883402+0000 (oldest deadline 2024-05-27T07:44:24.783513+0000) 2024-05-27T07:45:35.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:35.712+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:04.784761+0000 front 2024-05-27T07:44:04.785590+0000 (oldest deadline 2024-05-27T07:44:30.684940+0000) 2024-05-27T07:45:35.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:35.712+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.288681+0000 front 2024-05-27T07:44:18.288801+0000 (oldest deadline 2024-05-27T07:44:44.188922+0000) 2024-05-27T07:45:35.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:35.712+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-05-27T07:44:28.291169+0000 front 2024-05-27T07:44:28.291554+0000 (oldest deadline 2024-05-27T07:44:49.991412+0000) 2024-05-27T07:45:35.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:35.712+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:37.594715+0000 front 2024-05-27T07:44:37.594893+0000 (oldest deadline 2024-05-27T07:45:00.494969+0000) 2024-05-27T07:45:35.995 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:35.712+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:45.599217+0000 front 2024-05-27T07:44:45.599553+0000 (oldest deadline 2024-05-27T07:45:09.699047+0000) 2024-05-27T07:45:35.996 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:35 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10[60684]: 2024-05-27T07:45:35.712+0000 7fd376a00700 -1 osd.10 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.701960+0000 front 2024-05-27T07:44:56.702007+0000 (oldest deadline 2024-05-27T07:45:20.201750+0000) 2024-05-27T07:45:36.462 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:36 smithi062 podman[75329]: 2024-05-27 07:45:36.186841396 +0000 UTC m=+5.608895633 container died 3cad76387a2ab59cd104a0083d792c384058e7ba607c2845f0e90fa5646afca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:45:36.714 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:36 smithi062 podman[75329]: 2024-05-27 07:45:36.708565144 +0000 UTC m=+6.130619415 container cleanup 3cad76387a2ab59cd104a0083d792c384058e7ba607c2845f0e90fa5646afca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:45:36.714 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:36 smithi062 bash[75329]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10 2024-05-27T07:45:36.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:36.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:36.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:36.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:36.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:36.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:36.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:36.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:36.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:36.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:36.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:36.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:36.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:36 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:36.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:37.245 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:36 smithi062 podman[75348]: 2024-05-27 07:45:36.90969687 +0000 UTC m=+0.708012729 container remove 3cad76387a2ab59cd104a0083d792c384058e7ba607c2845f0e90fa5646afca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:45:37.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:37.600+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:37.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:37.600+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:37.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:37.600+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:37.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:37.600+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:37.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:37.600+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:37.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:37.600+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:37.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:37 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:37.600+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:38.114 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:37 smithi062 podman[75437]: 2024-05-27 07:45:37.659349169 +0000 UTC m=+0.041422389 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:38.114 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:37 smithi062 podman[75437]: 2024-05-27 07:45:37.752349036 +0000 UTC m=+0.134422262 container create 00cc5756962636f0cd4ca6db1f653062b735c13070517af022dca7bd12b22d1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:45:38.114 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:38 smithi062 podman[75437]: 2024-05-27 07:45:38.114423711 +0000 UTC m=+0.496497000 container init 00cc5756962636f0cd4ca6db1f653062b735c13070517af022dca7bd12b22d1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:45:38.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:38 smithi062 podman[75437]: 2024-05-27 07:45:38.12627628 +0000 UTC m=+0.508349502 container start 00cc5756962636f0cd4ca6db1f653062b735c13070517af022dca7bd12b22d1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:45:38.495 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:38 smithi062 podman[75437]: 2024-05-27 07:45:38.204680722 +0000 UTC m=+0.586753998 container attach 00cc5756962636f0cd4ca6db1f653062b735c13070517af022dca7bd12b22d1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T07:45:38.912 INFO:journalctl@ceph.osd.10.smithi062.stdout:May 27 07:45:38 smithi062 podman[75437]: 2024-05-27 07:45:38.599232301 +0000 UTC m=+0.981305491 container died 00cc5756962636f0cd4ca6db1f653062b735c13070517af022dca7bd12b22d1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:45:38.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:38.643+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:38.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:38.643+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:38.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:38.643+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:38.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:38.643+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:38.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:38.643+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:38.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:38.643+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:38.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:38 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:38.643+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:39.276 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.10.service' 2024-05-27T07:45:39.492 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:39.493 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-27T07:45:39.493 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-27T07:45:39.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.11 2024-05-27T07:45:39.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:39.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:39.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:39.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:39.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:39.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:39.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:39.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:39.606+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-05-27T07:45:15.969177+0000 front 2024-05-27T07:45:15.969309+0000 (oldest deadline 2024-05-27T07:45:38.869059+0000) 2024-05-27T07:45:39.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:39 smithi162 systemd[1]: Stopping Ceph osd.11 for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:45:40.483 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.125+0000 7f9915a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:45:40.483 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.125+0000 7f9915a00700 -1 osd.11 159 *** Got signal Terminated *** 2024-05-27T07:45:40.483 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.125+0000 7f9915a00700 -1 osd.11 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:45:40.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:40.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:40.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:40.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:40.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:40.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:40.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:40.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:40 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:40.576+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-05-27T07:45:15.969177+0000 front 2024-05-27T07:45:15.969309+0000 (oldest deadline 2024-05-27T07:45:38.869059+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:41.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:41 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:41.570+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-05-27T07:45:15.969177+0000 front 2024-05-27T07:45:15.969309+0000 (oldest deadline 2024-05-27T07:45:38.869059+0000) 2024-05-27T07:45:42.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:42.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:42.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:42.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:42.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:42.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:42.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:42.984 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:42 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:42.578+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-05-27T07:45:15.969177+0000 front 2024-05-27T07:45:15.969309+0000 (oldest deadline 2024-05-27T07:45:38.869059+0000) 2024-05-27T07:45:43.985 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:43.985 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:43.986 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:43.986 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:43.986 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:43.986 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:43.986 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:43.986 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:43 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:43.584+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-05-27T07:45:15.969177+0000 front 2024-05-27T07:45:15.969309+0000 (oldest deadline 2024-05-27T07:45:38.869059+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-05-27T07:43:59.441719+0000 front 2024-05-27T07:43:59.441620+0000 (oldest deadline 2024-05-27T07:44:24.141669+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-05-27T07:44:07.644061+0000 front 2024-05-27T07:44:07.644162+0000 (oldest deadline 2024-05-27T07:44:33.544348+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-05-27T07:44:18.048424+0000 front 2024-05-27T07:44:18.049420+0000 (oldest deadline 2024-05-27T07:44:42.748419+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-05-27T07:44:36.653976+0000 front 2024-05-27T07:44:36.654294+0000 (oldest deadline 2024-05-27T07:45:00.154196+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-05-27T07:44:47.658978+0000 front 2024-05-27T07:44:47.659312+0000 (oldest deadline 2024-05-27T07:45:12.260006+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-05-27T07:44:56.862981+0000 front 2024-05-27T07:44:56.863016+0000 (oldest deadline 2024-05-27T07:45:22.162462+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-05-27T07:45:05.064635+0000 front 2024-05-27T07:45:05.064593+0000 (oldest deadline 2024-05-27T07:45:29.763874+0000) 2024-05-27T07:45:44.983 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:44 smithi162 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11[61455]: 2024-05-27T07:45:44.591+0000 7f990c200700 -1 osd.11 159 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-05-27T07:45:15.969177+0000 front 2024-05-27T07:45:15.969309+0000 (oldest deadline 2024-05-27T07:45:38.869059+0000) 2024-05-27T07:45:45.934 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:45 smithi162 podman[78025]: 2024-05-27 07:45:45.657269779 +0000 UTC m=+5.691633672 container died c3ad7d65d3120548e6c958833af8d026d716b979e6b267e41f6581342c51fc4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:45:46.232 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:46 smithi162 podman[78025]: 2024-05-27 07:45:46.038802075 +0000 UTC m=+6.073166030 container cleanup c3ad7d65d3120548e6c958833af8d026d716b979e6b267e41f6581342c51fc4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:45:46.233 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:46 smithi162 bash[78025]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11 2024-05-27T07:45:46.732 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:46 smithi162 podman[78043]: 2024-05-27 07:45:46.322883273 +0000 UTC m=+0.657190064 container remove c3ad7d65d3120548e6c958833af8d026d716b979e6b267e41f6581342c51fc4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:45:47.118 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:46 smithi162 podman[78130]: 2024-05-27 07:45:46.865648112 +0000 UTC m=+0.117261597 container create b2d5ea2f2539488fa1013c23db7c44cd0c6da89f2c982857a088c2773fb5fe7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:45:47.118 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:46 smithi162 podman[78130]: 2024-05-27 07:45:46.780785213 +0000 UTC m=+0.032398673 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:45:47.118 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:47 smithi162 podman[78130]: 2024-05-27 07:45:47.118737747 +0000 UTC m=+0.370351223 container init b2d5ea2f2539488fa1013c23db7c44cd0c6da89f2c982857a088c2773fb5fe7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-27T07:45:47.482 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:47 smithi162 podman[78130]: 2024-05-27 07:45:47.130669198 +0000 UTC m=+0.382282721 container start b2d5ea2f2539488fa1013c23db7c44cd0c6da89f2c982857a088c2773fb5fe7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:45:47.483 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:47 smithi162 podman[78130]: 2024-05-27 07:45:47.193035247 +0000 UTC m=+0.444648793 container attach b2d5ea2f2539488fa1013c23db7c44cd0c6da89f2c982857a088c2773fb5fe7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:45:47.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:May 27 07:45:47 smithi162 podman[78130]: 2024-05-27 07:45:47.603204333 +0000 UTC m=+0.854817836 container died b2d5ea2f2539488fa1013c23db7c44cd0c6da89f2c982857a088c2773fb5fe7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:45:48.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@osd.11.service' 2024-05-27T07:45:48.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:48.328 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-27T07:45:48.328 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-27T07:45:48.328 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a 2024-05-27T07:45:48.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.a.service' 2024-05-27T07:45:48.553 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:48.554 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-27T07:45:48.554 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-27T07:45:48.554 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d 2024-05-27T07:45:48.699 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.d.service' 2024-05-27T07:45:48.807 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:48.807 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-27T07:45:48.807 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-27T07:45:48.807 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g 2024-05-27T07:45:48.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.g.service' 2024-05-27T07:45:49.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:49.058 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-27T07:45:49.058 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-27T07:45:49.058 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j 2024-05-27T07:45:49.210 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.j.service' 2024-05-27T07:45:49.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:49.318 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-27T07:45:49.318 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-27T07:45:49.318 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.b 2024-05-27T07:45:49.744 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:45:49 smithi062 systemd[1]: Stopping Ceph mds.b for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:45:50.246 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:45:49 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b[66152]: 2024-05-27T07:45:49.891+0000 7f6e11000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:45:50.246 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:45:49 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b[66152]: 2024-05-27T07:45:49.891+0000 7f6e11000700 -1 mds.b *** got signal Terminated *** 2024-05-27T07:45:57.363 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:45:57 smithi062 podman[75541]: 2024-05-27 07:45:57.096712582 +0000 UTC m=+7.328738233 container died 7e9040046a30aa48fda6d0795ba65937c9ac8ad47d9f0645d3e606a80f43e42b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:45:57.678 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:45:57 smithi062 podman[75541]: 2024-05-27 07:45:57.476124736 +0000 UTC m=+7.708150363 container cleanup 7e9040046a30aa48fda6d0795ba65937c9ac8ad47d9f0645d3e606a80f43e42b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:45:57.679 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:45:57 smithi062 bash[75541]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b 2024-05-27T07:45:57.994 INFO:journalctl@ceph.mds.b.smithi062.stdout:May 27 07:45:57 smithi062 podman[75554]: 2024-05-27 07:45:57.677534803 +0000 UTC m=+0.565749598 container remove 7e9040046a30aa48fda6d0795ba65937c9ac8ad47d9f0645d3e606a80f43e42b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:45:58.225 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.b.service' 2024-05-27T07:45:58.339 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:45:58.339 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-27T07:45:58.340 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-27T07:45:58.340 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.e 2024-05-27T07:45:58.744 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:45:58 smithi062 systemd[1]: Stopping Ceph mds.e for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:45:59.245 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:45:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e[65541]: 2024-05-27T07:45:58.969+0000 7fe678200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:45:59.245 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:45:58 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e[65541]: 2024-05-27T07:45:58.969+0000 7fe678200700 -1 mds.e *** got signal Terminated *** 2024-05-27T07:46:01.626 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:46:01 smithi062 podman[75662]: 2024-05-27 07:46:01.359235553 +0000 UTC m=+2.523991652 container died c6bd01fa6bb2f14932df7b4e355ec8be844d3ad2ca4d49d54a7c816b524cc1a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:46:01.930 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:46:01 smithi062 podman[75662]: 2024-05-27 07:46:01.738033647 +0000 UTC m=+2.902789765 container cleanup c6bd01fa6bb2f14932df7b4e355ec8be844d3ad2ca4d49d54a7c816b524cc1a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:46:01.930 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:46:01 smithi062 bash[75662]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e 2024-05-27T07:46:02.244 INFO:journalctl@ceph.mds.e.smithi062.stdout:May 27 07:46:01 smithi062 podman[75677]: 2024-05-27 07:46:01.930992573 +0000 UTC m=+0.550689498 container remove c6bd01fa6bb2f14932df7b4e355ec8be844d3ad2ca4d49d54a7c816b524cc1a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-27T07:46:02.462 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.e.service' 2024-05-27T07:46:02.578 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:46:02.579 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-27T07:46:02.579 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-27T07:46:02.579 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.h 2024-05-27T07:46:02.995 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:46:02 smithi062 systemd[1]: Stopping Ceph mds.h for 55c251ba-1bf7-11ef-bc9b-c7b262605968... 2024-05-27T07:46:03.495 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:46:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h[65840]: 2024-05-27T07:46:03.196+0000 7f8e3c000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:46:03.495 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:46:03 smithi062 ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h[65840]: 2024-05-27T07:46:03.196+0000 7f8e3c000700 -1 mds.h *** got signal Terminated *** 2024-05-27T07:46:08.639 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:46:08 smithi062 podman[75785]: 2024-05-27 07:46:08.3815321 +0000 UTC m=+5.305115431 container died 6a0fbf2d97ca3acdb56b1a6d99ad025ead46245de5c0da57c74d79ec05d01fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T07:46:08.925 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:46:08 smithi062 podman[75785]: 2024-05-27 07:46:08.750042708 +0000 UTC m=+5.673626046 container cleanup 6a0fbf2d97ca3acdb56b1a6d99ad025ead46245de5c0da57c74d79ec05d01fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-27T07:46:08.925 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:46:08 smithi062 bash[75785]: ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h 2024-05-27T07:46:09.244 INFO:journalctl@ceph.mds.h.smithi062.stdout:May 27 07:46:08 smithi062 podman[75797]: 2024-05-27 07:46:08.925808484 +0000 UTC m=+0.522715911 container remove 6a0fbf2d97ca3acdb56b1a6d99ad025ead46245de5c0da57c74d79ec05d01fbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968-mds-h, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-27T07:46:09.454 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.h.service' 2024-05-27T07:46:09.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:46:09.561 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-27T07:46:09.561 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-27T07:46:09.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k 2024-05-27T07:46:09.710 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.k.service' 2024-05-27T07:46:09.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:46:09.820 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-27T07:46:09.820 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-27T07:46:09.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c 2024-05-27T07:46:09.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.c.service' 2024-05-27T07:46:10.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:46:10.032 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-27T07:46:10.032 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-27T07:46:10.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f 2024-05-27T07:46:10.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.f.service' 2024-05-27T07:46:10.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:46:10.263 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-27T07:46:10.263 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-27T07:46:10.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i 2024-05-27T07:46:10.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.i.service' 2024-05-27T07:46:10.507 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:46:10.507 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-27T07:46:10.507 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-27T07:46:10.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l 2024-05-27T07:46:10.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-55c251ba-1bf7-11ef-bc9b-c7b262605968@mds.l.service' 2024-05-27T07:46:10.684 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:46:10.684 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-27T07:46:10.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:46:11.018 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:46:16.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:46:16.546 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:46:21.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:46:22.072 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:46:27.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:46:27.104 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:46:27.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:46:27.292 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-27T07:46:27.294 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi012/crash 2024-05-27T07:46:27.295 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:46:27.395 INFO:teuthology.orchestra.run.smithi012.stderr:tar: /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:46:27.395 INFO:teuthology.orchestra.run.smithi012.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:46:27.400 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi062/crash 2024-05-27T07:46:27.401 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:46:27.500 INFO:teuthology.orchestra.run.smithi062.stderr:tar: /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:46:27.500 INFO:teuthology.orchestra.run.smithi062.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:46:27.505 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi162/crash 2024-05-27T07:46:27.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:46:27.598 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:46:27.599 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:46:27.602 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-27T07:46:27.603 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-27T07:46:27.721 INFO:teuthology.orchestra.run.smithi012.stdout:2024-05-27T07:33:10.503526+0000 mon.a (mon.0) 1630 : cluster [WRN] Health check failed: 9 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:46:27.722 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-27T07:46:27.722 INFO:tasks.cephadm:Compressing logs... 2024-05-27T07:46:27.723 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:46:27.727 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:46:27.734 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:46:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-05-27T07:46:27.786 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.0.log 2024-05-27T07:46:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:46:27.788 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:46:27.789 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:46:27.791 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-05-27T07:46:27.792 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-05-27T07:46:27.794 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-05-27T07:46:27.796 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-05-27T07:46:27.797 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.3.log: 87.6% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:46:27.797 INFO:teuthology.orchestra.run.smithi012.stderr: 90.3% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:46:27.798 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-05-27T07:46:27.799 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-05-27T07:46:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.x.log: 94.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:46:27.801 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-05-27T07:46:27.804 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.a.log: /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:46:27.805 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-05-27T07:46:27.807 INFO:teuthology.orchestra.run.smithi062.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose -- 2024-05-27T07:46:27.807 INFO:teuthology.orchestra.run.smithi062.stderr: /var/log/ceph/cephadm.log 2024-05-27T07:46:27.808 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:46:27.809 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:46:27.811 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-05-27T07:46:27.811 INFO:teuthology.orchestra.run.smithi062.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:46:27.811 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log: 84.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:46:27.812 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-05-27T07:46:27.813 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-05-27T07:46:27.815 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-05-27T07:46:27.815 INFO:teuthology.orchestra.run.smithi062.stderr: 90.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:46:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:46:27.816 INFO:teuthology.orchestra.run.smithi062.stderr: 87.6% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:46:27.816 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-05-27T07:46:27.817 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-05-27T07:46:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-05-27T07:46:27.818 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:46:27.819 INFO:teuthology.orchestra.run.smithi162.stderr:: No such file or directory 2024-05-27T07:46:27.819 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:46:27.819 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-05-27T07:46:27.820 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-05-27T07:46:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-05-27T07:46:27.821 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.h.log: /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:46:27.821 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-05-27T07:46:27.822 INFO:teuthology.orchestra.run.smithi162.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:46:27.822 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:46:27.823 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-05-27T07:46:27.823 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-27T07:46:27.824 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-05-27T07:46:27.825 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-05-27T07:46:27.825 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-05-27T07:46:27.826 INFO:teuthology.orchestra.run.smithi162.stderr: 87.6% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:46:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-05-27T07:46:27.828 INFO:teuthology.orchestra.run.smithi062.stderr: 84.2%gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:46:27.828 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:46:27.829 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-05-27T07:46:27.829 INFO:teuthology.orchestra.run.smithi162.stderr: 90.5% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:46:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:46:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-05-27T07:46:27.834 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.i.log: /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-05-27T07:46:27.834 INFO:teuthology.orchestra.run.smithi162.stderr: 84.1% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:46:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-27T07:46:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log: /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-05-27T07:46:27.838 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-05-27T07:46:27.838 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:46:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.k.log: 92.5% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-05-27T07:46:27.845 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.8.log: /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-05-27T07:46:27.846 INFO:teuthology.orchestra.run.smithi162.stderr: 92.6% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-05-27T07:46:27.854 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:46:27.866 INFO:teuthology.orchestra.run.smithi062.stderr: 90.9% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:46:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.l.log: 90.9% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:46:28.417 INFO:teuthology.orchestra.run.smithi162.stderr: 93.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-05-27T07:46:28.620 INFO:teuthology.orchestra.run.smithi162.stderr: 91.8% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-05-27T07:46:28.730 INFO:teuthology.orchestra.run.smithi012.stderr: 90.2% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-05-27T07:46:28.801 INFO:teuthology.orchestra.run.smithi012.stderr: 94.0% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-05-27T07:46:28.868 INFO:teuthology.orchestra.run.smithi062.stderr: 91.7% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-05-27T07:46:29.322 INFO:teuthology.orchestra.run.smithi162.stderr: 94.9% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-05-27T07:46:29.361 INFO:teuthology.orchestra.run.smithi062.stderr: 94.7% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-05-27T07:46:29.619 INFO:teuthology.orchestra.run.smithi012.stderr: 94.8% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-05-27T07:46:30.606 INFO:teuthology.orchestra.run.smithi162.stderr: 95.3% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-05-27T07:46:30.638 INFO:teuthology.orchestra.run.smithi012.stderr: 90.2% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-05-27T07:46:31.125 INFO:teuthology.orchestra.run.smithi162.stderr: 95.3% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-05-27T07:46:31.400 INFO:teuthology.orchestra.run.smithi012.stderr: 95.3% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-05-27T07:46:31.968 INFO:teuthology.orchestra.run.smithi012.stderr: 95.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-05-27T07:46:36.186 INFO:teuthology.orchestra.run.smithi062.stderr: 94.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-05-27T07:46:42.931 INFO:teuthology.orchestra.run.smithi162.stderr: 91.9% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-05-27T07:46:44.041 INFO:teuthology.orchestra.run.smithi162.stderr: 91.6% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-05-27T07:46:47.160 INFO:teuthology.orchestra.run.smithi012.stderr: 91.8% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-05-27T07:46:48.089 INFO:teuthology.orchestra.run.smithi062.stderr: 91.9% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-05-27T07:46:48.684 INFO:teuthology.orchestra.run.smithi062.stderr: 91.8% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-05-27T07:46:50.107 INFO:teuthology.orchestra.run.smithi062.stderr: 93.9% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-05-27T07:46:51.262 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-05-27T07:46:53.204 INFO:teuthology.orchestra.run.smithi062.stderr: 92.2% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-05-27T07:46:59.595 INFO:teuthology.orchestra.run.smithi162.stderr: 91.8% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-05-27T07:47:00.470 INFO:teuthology.orchestra.run.smithi162.stderr: 91.7% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-05-27T07:47:00.473 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-05-27T07:47:00.473 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m32.686s 2024-05-27T07:47:00.474 INFO:teuthology.orchestra.run.smithi162.stderr:user 1m36.777s 2024-05-27T07:47:00.474 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m7.395s 2024-05-27T07:47:01.128 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-05-27T07:47:04.444 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-05-27T07:47:04.527 INFO:teuthology.orchestra.run.smithi062.stderr: 93.5% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-05-27T07:47:05.551 INFO:teuthology.orchestra.run.smithi062.stderr: 91.4% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-05-27T07:47:05.556 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-05-27T07:47:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m37.776s 2024-05-27T07:47:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:user 2m40.558s 2024-05-27T07:47:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m11.118s 2024-05-27T07:47:13.750 INFO:teuthology.orchestra.run.smithi012.stderr: 91.7% -- replaced with /var/log/ceph/55c251ba-1bf7-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-05-27T07:47:13.755 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-05-27T07:47:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m45.995s 2024-05-27T07:47:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:user 2m37.663s 2024-05-27T07:47:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m11.755s 2024-05-27T07:47:13.759 INFO:tasks.cephadm:Archiving logs... 2024-05-27T07:47:13.759 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi012/log 2024-05-27T07:47:13.854 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:47:36.884 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi062/log 2024-05-27T07:47:36.889 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:47:56.729 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi162/log 2024-05-27T07:47:56.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:48:12.372 INFO:tasks.cephadm:Removing cluster... 2024-05-27T07:48:12.372 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 --force 2024-05-27T07:48:12.714 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:48:13.782 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 --force 2024-05-27T07:48:14.114 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:48:15.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 --force 2024-05-27T07:48:15.557 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 55c251ba-1bf7-11ef-bc9b-c7b262605968 2024-05-27T07:48:16.452 INFO:tasks.cephadm:Removing cephadm ... 2024-05-27T07:48:16.452 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T07:48:16.515 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T07:48:16.579 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T07:48:16.637 INFO:tasks.cephadm:Teardown complete 2024-05-27T07:48:16.637 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-27T07:48:16.650 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/install/__init__.py", line 632, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T07:48:16.651 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-27T07:48:16.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:48:16.659 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:48:16.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:48:16.833 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-27T07:48:16.844 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-27T07:48:16.844 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:48:16.852 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:48:16.860 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:48:16.903 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:48:16.907 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:48:16.912 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:48:16.914 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:48:16.914 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-05-27T07:48:16.914 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 55 +941us[ +941us] +/- 73ms 2024-05-27T07:48:16.914 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -376us[ -356us] +/- 66ms 2024-05-27T07:48:16.915 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -642us[ -642us] +/- 90ms 2024-05-27T07:48:16.915 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:48:16.917 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:48:16.917 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-05-27T07:48:16.917 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 55 +943us[ +943us] +/- 73ms 2024-05-27T07:48:16.917 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -373us[ -371us] +/- 66ms 2024-05-27T07:48:16.917 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -641us[ -641us] +/- 90ms 2024-05-27T07:48:16.917 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:48:16.922 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:48:16.922 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-05-27T07:48:16.922 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 56 +943us[ +943us] +/- 73ms 2024-05-27T07:48:16.922 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 8 377 183 -286us[ -287us] +/- 65ms 2024-05-27T07:48:16.922 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -655us[ -655us] +/- 90ms 2024-05-27T07:48:16.922 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:48:16.924 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-27T07:48:16.935 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-27T07:48:16.935 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-27T07:48:16.972 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T07:48:17.174 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T07:48:17.383 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T07:48:17.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:48:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:48:17.686 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:48:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:48:17.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:48:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:48:17.877 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-27T07:48:17.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-27T07:48:17.947 INFO:teuthology.task.internal:Duration was 3180.136094 seconds 2024-05-27T07:48:17.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-27T07:48:17.988 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-27T07:48:17.989 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:48:17.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:48:18.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:48:18.138 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:48:18.141 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:48:18.148 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:48:18.595 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-27T07:48:18.595 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-05-27T07:48:18.595 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:48:18.673 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-05-27T07:48:18.673 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:48:18.751 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-05-27T07:48:18.751 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:48:18.823 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-27T07:48:18.823 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:48:18.831 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:48:18.838 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:48:18.927 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-27T07:48:18.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:48:18.934 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:48:18.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:48:20.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-27T07:48:20.425 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-27T07:48:20.425 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:48:20.507 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:48:20.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:48:20.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-27T07:48:20.711 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:48:20.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:48:20.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:48:20.816 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-05-27T07:48:20.823 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-05-27T07:48:20.825 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-05-27T07:48:21.003 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:48:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:48:21.061 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:48:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:48:21.120 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:48:21.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:48:21.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-27T07:48:21.186 INFO:teuthology.task.internal:Transferring archived files... 2024-05-27T07:48:21.188 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi012 2024-05-27T07:48:21.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T07:48:21.329 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi062 2024-05-27T07:48:21.329 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T07:48:21.434 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727636/remote/smithi162 2024-05-27T07:48:21.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T07:48:21.535 INFO:teuthology.task.internal:Removing archive directory... 2024-05-27T07:48:21.535 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T07:48:21.545 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T07:48:21.553 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T07:48:21.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-27T07:48:21.624 INFO:teuthology.task.internal:Not uploading archives. 2024-05-27T07:48:21.624 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-27T07:48:21.650 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-27T07:48:21.650 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:48:21.658 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:48:21.666 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:48:21.713 INFO:teuthology.orchestra.run.smithi012.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:48 /home/ubuntu/cephtest 2024-05-27T07:48:21.718 INFO:teuthology.orchestra.run.smithi062.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:48 /home/ubuntu/cephtest 2024-05-27T07:48:21.722 INFO:teuthology.orchestra.run.smithi162.stdout: 393251 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:48 /home/ubuntu/cephtest 2024-05-27T07:48:21.726 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-27T07:48:21.739 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-27T07:48:21.775 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T07:48:21.779 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} duration: 3180.1360936164856 failure_reason: 'Command failed on smithi012 with status 22: ''sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 55c251ba-1bf7-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump''' flavor: default mon.a-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 mon.b-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 mon.c-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 owner: scheduled_vshankar@teuthology status: fail success: false 2024-05-27T07:48:21.780 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T07:48:21.865 INFO:teuthology.run:FAIL